2024-02-02

Title: Using Multi-Temporal Sentinel-1 and Sentinel-2 data for water bodies mapping

• Authors: Luigi Russo, Francesco Mauro, Babak Memar, Alessandro Sebastianelli, Paolo Gamba, Silvia Liberata Ullo
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.00023
• Pdf URL: https://arxiv.org/pdf/2402.00023
• Copy Paste: [[2402.00023]] Using Multi-Temporal Sentinel-1 and Sentinel-2 data for water bodies mapping(https://arxiv.org/abs/2402.00023)
• Keywords: segmentation
• Abstract: Climate change is intensifying extreme weather events, causing both water scarcity and severe rainfall unpredictability, and posing threats to sustainable development, biodiversity, and access to water and sanitation. This paper aims to provide valuable insights for comprehensive water resource monitoring under diverse meteorological conditions. An extension of the SEN2DWATER dataset is proposed to enhance its capabilities for water basin segmentation. Through the integration of temporally and spatially aligned radar information from Sentinel-1 data with the existing multispectral Sentinel-2 data, a novel multisource and multitemporal dataset is generated. Benchmarking the enhanced dataset involves the application of indices such as the Soil Water Index (SWI) and Normalized Difference Water Index (NDWI), along with an unsupervised Machine Learning (ML) classifier (k-means clustering). Promising results are obtained and potential future developments and applications arising from this research are also explored.

Title: LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition

• Authors: Youbing Hu, Yun Cheng, Anqi Lu, Zhiqiang Cao, Dawei Wei, Jie Liu, Zhijun Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00033
• Pdf URL: https://arxiv.org/pdf/2402.00033
• Copy Paste: [[2402.00033]] LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition(https://arxiv.org/abs/2402.00033)
• Keywords: transformer
• Abstract: The Vision Transformer (ViT) excels in accuracy when handling high-resolution images, yet it confronts the challenge of significant spatial redundancy, leading to increased computational and memory requirements. To address this, we present the Localization and Focus Vision Transformer (LF-ViT). This model operates by strategically curtailing computational demands without impinging on performance. In the Localization phase, a reduced-resolution image is processed; if a definitive prediction remains elusive, our pioneering Neighborhood Global Class Attention (NGCA) mechanism is triggered, effectively identifying and spotlighting class-discriminative regions based on initial findings. Subsequently, in the Focus phase, this designated region is used from the original image to enhance recognition. Uniquely, LF-ViT employs consistent parameters across both phases, ensuring seamless end-to-end optimization. Our empirical tests affirm LF-ViT's prowess: it remarkably decreases Deit-S's FLOPs by 63\% and concurrently amplifies throughput twofold. Code of this project is at https://github.com/edgeai1/LF-ViT.git.

Title: Robustness Assessment of a Runway Object Classifier for Safe Aircraft Taxiing

• Authors: Yizhak Elboher, Raya Elsaleh, Omri Isac, Mélanie Ducoffe, Audrey Galametz, Guillaume Povéda, Ryma Boumazouza, Noémie Cohen, Guy Katz
• Subjects: cs.CV, cs.LG, cs.LO
• Abstract URL: https://arxiv.org/abs/2402.00035
• Pdf URL: https://arxiv.org/pdf/2402.00035
• Copy Paste: [[2402.00035]] Robustness Assessment of a Runway Object Classifier for Safe Aircraft Taxiing(https://arxiv.org/abs/2402.00035)
• Keywords: robust
• Abstract: As deep neural networks (DNNs) are becoming the prominent solution for many computational problems, the aviation industry seeks to explore their potential in alleviating pilot workload and in improving operational safety. However, the use of DNNs in this type of safety-critical applications requires a thorough certification process. This need can be addressed through formal verification, which provides rigorous assurances -- e.g.,~by proving the absence of certain mispredictions. In this case-study paper, we demonstrate this process using an image-classifier DNN currently under development at Airbus and intended for use during the aircraft taxiing phase. We use formal methods to assess this DNN's robustness to three common image perturbation types: noise, brightness and contrast, and some of their combinations. This process entails multiple invocations of the underlying verifier, which might be computationally expensive; and we therefore propose a method that leverages the monotonicity of these robustness properties, as well as the results of past verification queries, in order to reduce the overall number of verification queries required by nearly 60%. Our results provide an indication of the level of robustness achieved by the DNN classifier under study, and indicate that it is considerably more vulnerable to noise than to brightness or contrast perturbations.

Title: TrackGPT -- A generative pre-trained transformer for cross-domain entity trajectory forecasting

• Authors: Nicholas Stroh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00066
• Pdf URL: https://arxiv.org/pdf/2402.00066
• Copy Paste: [[2402.00066]] TrackGPT -- A generative pre-trained transformer for cross-domain entity trajectory forecasting(https://arxiv.org/abs/2402.00066)
• Keywords: defense, transformer, generative, large language model
• Abstract: The forecasting of entity trajectories at future points in time is a critical capability gap in applications across both Commercial and Defense sectors. Transformers, and specifically Generative Pre-trained Transformer (GPT) networks have recently revolutionized several fields of Artificial Intelligence, most notably Natural Language Processing (NLP) with the advent of Large Language Models (LLM) like OpenAI's ChatGPT. In this research paper, we introduce TrackGPT, a GPT-based model for entity trajectory forecasting that has shown utility across both maritime and air domains, and we expect to perform well in others. TrackGPT stands as a pioneering GPT model capable of producing accurate predictions across diverse entity time series datasets, demonstrating proficiency in generating both long-term forecasts with sustained accuracy and short-term forecasts with high precision. We present benchmarks against state-of-the-art deep learning techniques, showing that TrackGPT's forecasting capability excels in terms of accuracy, reliability, and modularity. Importantly, TrackGPT achieves these results while remaining domain-agnostic and requiring minimal data features (only location and time) compared to models achieving similar performance. In conclusion, our findings underscore the immense potential of applying GPT architectures to the task of entity trajectory forecasting, exemplified by the innovative TrackGPT model.

Title: GPT4Battery: An LLM-driven Framework for Adaptive State of Health Estimation of Raw Li-ion Batteries

• Authors: Yuyuan Feng, Guosheng Hu, Zhihong Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00068
• Pdf URL: https://arxiv.org/pdf/2402.00068
• Copy Paste: [[2402.00068]] GPT4Battery: An LLM-driven Framework for Adaptive State of Health Estimation of Raw Li-ion Batteries(https://arxiv.org/abs/2402.00068)
• Keywords: large language model
• Abstract: State of health (SOH) is a crucial indicator for assessing the degradation level of batteries that cannot be measured directly but requires estimation. Accurate SOH estimation enhances detection, control, and feedback for Li-ion batteries, allowing for safe and efficient energy management and guiding the development of new-generation batteries. Despite the significant progress in data-driven SOH estimation, the time and resource-consuming degradation experiments for generating lifelong training data pose a challenge in establishing one large model capable of handling diverse types of Li-ion batteries, e.g., cross-chemistry, cross-manufacturer, and cross-capacity. Hence, this paper utilizes the strong generalization capability of large language model (LLM) to proposes a novel framework for adaptable SOH estimation across diverse batteries. To match the real scenario where unlabeled data sequentially arrives in use with distribution shifts, the proposed model is modified by a test-time training technique to ensure estimation accuracy even at the battery's end of life. The validation results demonstrate that the proposed framework achieves state-of-the-art accuracy on four widely recognized datasets collected from 62 batteries. Furthermore, we analyze the theoretical challenges of cross-battery estimation and provide a quantitative explanation of the effectiveness of our method.

Title: Unraveling the Impact of Initial Choices and In-Loop Interventions on Learning Dynamics in Autonomous Scanning Probe Microscopy

• Authors: Boris N. Slautin, Yongtao Liu, Hiroshi Funakubo, Sergei V. Kalinin
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2402.00071
• Pdf URL: https://arxiv.org/pdf/2402.00071
• Copy Paste: [[2402.00071]] Unraveling the Impact of Initial Choices and In-Loop Interventions on Learning Dynamics in Autonomous Scanning Probe Microscopy(https://arxiv.org/abs/2402.00071)
• Keywords: robust
• Abstract: The current focus in Autonomous Experimentation (AE) is on developing robust workflows to conduct the AE effectively. This entails the need for well-defined approaches to guide the AE process, including strategies for hyperparameter tuning and high-level human interventions within the workflow loop. This paper presents a comprehensive analysis of the influence of initial experimental conditions and in-loop interventions on the learning dynamics of Deep Kernel Learning (DKL) within the realm of AE in Scanning Probe Microscopy. We explore the concept of 'seed effect', where the initial experiment setup has a substantial impact on the subsequent learning trajectory. Additionally, we introduce an approach of the seed point interventions in AE allowing the operator to influence the exploration process. Using a dataset from Piezoresponse Force Microscopy (PFM) on PbTiO3 thin films, we illustrate the impact of the 'seed effect' and in-loop seed interventions on the effectiveness of DKL in predicting material properties. The study highlights the importance of initial choices and adaptive interventions in optimizing learning rates and enhancing the efficiency of automated material characterization. This work offers valuable insights into designing more robust and effective AE workflows in microscopy with potential applications across various characterization techniques. The analysis code that supports the funding is publicly available at https://github.com/Slautin/2024_Seed_effect_DKL_BO.

Title: D-Nikud: Enhancing Hebrew Diacritization with LSTM and Pretrained Models

• Authors: Adi Rosenthal, Nadav Shaked
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00075
• Pdf URL: https://arxiv.org/pdf/2402.00075
• Copy Paste: [[2402.00075]] D-Nikud: Enhancing Hebrew Diacritization with LSTM and Pretrained Models(https://arxiv.org/abs/2402.00075)
• Keywords: transformer
• Abstract: D-Nikud, a novel approach to Hebrew diacritization that integrates the strengths of LSTM networks and BERT-based (transformer) pre-trained model. Inspired by the methodologies employed in Nakdimon, we integrate it with the TavBERT pre-trained model, our system incorporates advanced architectural choices and diverse training data. Our experiments showcase state-of-the-art results on several benchmark datasets, with a particular emphasis on modern texts and more specified diacritization like gender.

Title: Common Sense Reasoning for Deep Fake Detection

• Authors: Yue Zhang, Ben Colman, Ali Shahriyari, Gaurav Bharaj
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00126
• Pdf URL: https://arxiv.org/pdf/2402.00126
• Copy Paste: [[2402.00126]] Common Sense Reasoning for Deep Fake Detection(https://arxiv.org/abs/2402.00126)
• Keywords: extraction, interpretability, transformer
• Abstract: State-of-the-art approaches rely on image-based features extracted via neural networks for the deepfake detection binary classification. While these approaches trained in the supervised sense extract likely fake features, they may fall short in representing unnatural `non-physical' semantic facial attributes -- blurry hairlines, double eyebrows, rigid eye pupils, or unnatural skin shading. However, such facial attributes are generally easily perceived by humans via common sense reasoning. Furthermore, image-based feature extraction methods that provide visual explanation via saliency maps can be hard to be interpreted by humans. To address these challenges, we propose the use of common sense reasoning to model deepfake detection, and extend it to the Deepfake Detection VQA (DD-VQA) task with the aim to model human intuition in explaining the reason behind labeling an image as either real or fake. To this end, we introduce a new dataset that provides answers to the questions related to the authenticity of an image, along with its corresponding explanations. We also propose a Vision and Language Transformer-based framework for the DD-VQA task, incorporating text and image aware feature alignment formulations. Finally, we evaluate our method on both the performance of deepfake detection and the quality of the generated explanations. We hope that this task inspires researchers to explore new avenues for enhancing language-based interpretability and cross-modality applications in the realm of deepfake detection.

Title: Real-time Traffic Object Detection for Autonomous Driving

• Authors: Abdul Hannan Khan, Syed Tahseen Raza Rizvi, Andreas Dengel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00128
• Pdf URL: https://arxiv.org/pdf/2402.00128
• Copy Paste: [[2402.00128]] Real-time Traffic Object Detection for Autonomous Driving(https://arxiv.org/abs/2402.00128)
• Keywords: robust
• Abstract: With recent advances in computer vision, it appears that autonomous driving will be part of modern society sooner rather than later. However, there are still a significant number of concerns to address. Although modern computer vision techniques demonstrate superior performance, they tend to prioritize accuracy over efficiency, which is a crucial aspect of real-time applications. Large object detection models typically require higher computational power, which is achieved by using more sophisticated onboard hardware. For autonomous driving, these requirements translate to increased fuel costs and, ultimately, a reduction in mileage. Further, despite their computational demands, the existing object detectors are far from being real-time. In this research, we assess the robustness of our previously proposed, highly efficient pedestrian detector LSFM on well-established autonomous driving benchmarks, including diverse weather conditions and nighttime scenes. Moreover, we extend our LSFM model for general object detection to achieve real-time object detection in traffic scenes. We evaluate its performance, low latency, and generalizability on traffic object detection datasets. Furthermore, we discuss the inadequacy of the current key performance indicator employed by object detection systems in the context of autonomous driving and propose a more suitable alternative that incorporates real-time requirements.

Title: CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibration

• Authors: Daniele Cattaneo, Abhinav Valada
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.00129
• Pdf URL: https://arxiv.org/pdf/2402.00129
• Copy Paste: [[2402.00129]] CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibration(https://arxiv.org/abs/2402.00129)
• Keywords: robust
• Abstract: LiDARs are widely used for mapping and localization in dynamic environments. However, their high cost limits their widespread adoption. On the other hand, monocular localization in LiDAR maps using inexpensive cameras is a cost-effective alternative for large-scale deployment. Nevertheless, most existing approaches struggle to generalize to new sensor setups and environments, requiring retraining or fine-tuning. In this paper, we present CMRNext, a novel approach for camera-LIDAR matching that is independent of sensor-specific parameters, generalizable, and can be used in the wild for monocular localization in LiDAR maps and camera-LiDAR extrinsic calibration. CMRNext exploits recent advances in deep neural networks for matching cross-modal data and standard geometric techniques for robust pose estimation. We reformulate the point-pixel matching problem as an optical flow estimation problem and solve the Perspective-n-Point problem based on the resulting correspondences to find the relative pose between the camera and the LiDAR point cloud. We extensively evaluate CMRNext on six different robotic platforms, including three publicly available datasets and three in-house robots. Our experimental evaluations demonstrate that CMRNext outperforms existing approaches on both tasks and effectively generalizes to previously unseen environments and sensor setups in a zero-shot manner. We make the code and pre-trained models publicly available at this http URL .

Title: Multimodal Neurodegenerative Disease Subtyping Explained by ChatGPT

• Authors: Diego Machado Reyes, Hanqing Chao, Juergen Hahn, Li Shen, Pingkun Yan
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00137
• Pdf URL: https://arxiv.org/pdf/2402.00137
• Copy Paste: [[2402.00137]] Multimodal Neurodegenerative Disease Subtyping Explained by ChatGPT(https://arxiv.org/abs/2402.00137)
• Keywords: explainability, generative, large language model
• Abstract: Alzheimer's disease (AD) is the most prevalent neurodegenerative disease; yet its currently available treatments are limited to stopping disease progression. Moreover, effectiveness of these treatments is not guaranteed due to the heterogenetiy of the disease. Therefore, it is essential to be able to identify the disease subtypes at a very early stage. Current data driven approaches are able to classify the subtypes at later stages of AD or related disorders, but struggle when predicting at the asymptomatic or prodromal stage. Moreover, most existing models either lack explainability behind the classification or only use a single modality for the assessment, limiting scope of its analysis. Thus, we propose a multimodal framework that uses early-stage indicators such as imaging, genetics and clinical assessments to classify AD patients into subtypes at early stages. Similarly, we build prompts and use large language models, such as ChatGPT, to interpret the findings of our model. In our framework, we propose a tri-modal co-attention mechanism (Tri-COAT) to explicitly learn the cross-modal feature associations. Our proposed model outperforms baseline models and provides insight into key cross-modal feature associations supported by known biological mechanisms.

Title: Fully Data-Driven Model for Increasing Sampling Rate Frequency of Seismic Data using Super-Resolution Generative Adversarial Networks

• Authors: Navid Gholizadeh, Javad Katebi
• Subjects: cs.LG, eess.SP, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2402.00153
• Pdf URL: https://arxiv.org/pdf/2402.00153
• Copy Paste: [[2402.00153]] Fully Data-Driven Model for Increasing Sampling Rate Frequency of Seismic Data using Super-Resolution Generative Adversarial Networks(https://arxiv.org/abs/2402.00153)
• Keywords: generative
• Abstract: High-quality data is one of the key requirements for any engineering application. In earthquake engineering practice, accurate data is pivotal in predicting the response of structure or damage detection process in an Structural Health Monitoring (SHM) application with less uncertainty. However, obtaining high-resolution data is fraught with challenges, such as significant costs, extensive data channels, and substantial storage requirements. To address these challenges, this study employs super-resolution generative adversarial networks (SRGANs) to improve the resolution of time-history data such as the data obtained by a sensor network in an SHM application, marking the first application of SRGANs in earthquake engineering domain. The time-series data are transformed into RGB values, converting raw data into images. SRGANs are then utilized to upscale these low-resolution images, thereby enhancing the overall sensor resolution. This methodology not only offers potential reductions in data storage requirements but also simplifies the sensor network, which could result in lower installation and maintenance costs. The proposed SRGAN method is rigorously evaluated using real seismic data, and its performance is compared with traditional enhancement techniques. The findings of this study pave the way for cost-effective and efficient improvements in the resolution of sensors used in SHM systems, with promising implications for the safety and sustainability of infrastructures worldwide.

Title: Large Language Models for Mathematical Reasoning: Progresses and Challenges

• Authors: Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, Wenpeng Yin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00157
• Pdf URL: https://arxiv.org/pdf/2402.00157
• Copy Paste: [[2402.00157]] Large Language Models for Mathematical Reasoning: Progresses and Challenges(https://arxiv.org/abs/2402.00157)
• Keywords: large language model
• Abstract: Mathematical reasoning serves as a cornerstone for assessing the fundamental cognitive capabilities of human intelligence. In recent times, there has been a notable surge in the development of Large Language Models (LLMs) geared towards the automated resolution of mathematical problems. However, the landscape of mathematical problem types is vast and varied, with LLM-oriented techniques undergoing evaluation across diverse datasets and settings. This diversity makes it challenging to discern the true advancements and obstacles within this burgeoning field. This survey endeavors to address four pivotal dimensions: i) a comprehensive exploration of the various mathematical problems and their corresponding datasets that have been investigated; ii) an examination of the spectrum of LLM-oriented techniques that have been proposed for mathematical problem-solving; iii) an overview of factors and concerns affecting LLMs in solving math; and iv) an elucidation of the persisting challenges within this domain. To the best of our knowledge, this survey stands as one of the first extensive examinations of the landscape of LLMs in the realm of mathematics, providing a holistic perspective on the current state, accomplishments, and future challenges in this rapidly evolving field.

Title: Multimodal Clinical Pseudo-notes for Emergency Department Prediction Tasks using Multiple Embedding Model for EHR (MEME)

• Authors: Simon A. Lee, Sujay Jain, Alex Chen, Arabdha Biswas, Jennifer Fang, Akos Rudas, Jeffrey N. Chiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00160
• Pdf URL: https://arxiv.org/pdf/2402.00160
• Copy Paste: [[2402.00160]] Multimodal Clinical Pseudo-notes for Emergency Department Prediction Tasks using Multiple Embedding Model for EHR (MEME)(https://arxiv.org/abs/2402.00160)
• Keywords: large language model
• Abstract: In this work, we introduce Multiple Embedding Model for EHR (MEME), an approach that views Electronic Health Records (EHR) as multimodal data. This approach incorporates "pseudo-notes", textual representations of tabular EHR concepts such as diagnoses and medications, and allows us to effectively employ Large Language Models (LLMs) for EHR representation. This framework also adopts a multimodal approach, embedding each EHR modality separately. We demonstrate the effectiveness of MEME by applying it to several tasks within the Emergency Department across multiple hospital systems. Our findings show that MEME surpasses the performance of both single modality embedding methods and traditional machine learning approaches. However, we also observe notable limitations in generalizability across hospital institutions for all tested models.

Title: De-identification is not always enough

• Authors: Atiquer Rahman Sarkar, Yao-Shun Chuang, Noman Mohammed, Xiaoqian Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00179
• Pdf URL: https://arxiv.org/pdf/2402.00179
• Copy Paste: [[2402.00179]] De-identification is not always enough(https://arxiv.org/abs/2402.00179)
• Keywords: privacy, protect, attack, membership infer, generative, large language model
• Abstract: For sharing privacy-sensitive data, de-identification is commonly regarded as adequate for safeguarding privacy. Synthetic data is also being considered as a privacy-preserving alternative. Recent successes with numerical and tabular data generative models and the breakthroughs in large generative language models raise the question of whether synthetically generated clinical notes could be a viable alternative to real notes for research purposes. In this work, we demonstrated that (i) de-identification of real clinical notes does not protect records against a membership inference attack, (ii) proposed a novel approach to generate synthetic clinical notes using the current state-of-the-art large language models, (iii) evaluated the performance of the synthetically generated notes in a clinical domain task, and (iv) proposed a way to mount a membership inference attack where the target model is trained with synthetic data. We observed that when synthetically generated notes closely match the performance of real data, they also exhibit similar privacy concerns to the real data. Whether other approaches to synthetically generated clinical notes could offer better trade-offs and become a better alternative to sensitive real notes warrants further investigation.

Title: Dataset Condensation Driven Machine Unlearning

• Authors: Junaid Iqbal Khan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00195
• Pdf URL: https://arxiv.org/pdf/2402.00195
• Copy Paste: [[2402.00195]] Dataset Condensation Driven Machine Unlearning(https://arxiv.org/abs/2402.00195)
• Keywords: privacy, attack, membership infer
• Abstract: The current trend in data regulation requirements and privacy-preserving machine learning has emphasized the importance of machine unlearning. The naive approach to unlearning training data by retraining over the complement of the forget samples is susceptible to computational challenges. These challenges have been effectively addressed through a collection of techniques falling under the umbrella of machine unlearning. However, there still exists a lack of sufficiency in handling persistent computational challenges in harmony with the utility and privacy of unlearned model. We attribute this to the lack of work on improving the computational complexity of approximate unlearning from the perspective of the training dataset. In this paper, we aim to fill this gap by introducing dataset condensation as an essential component of machine unlearning in the context of image classification. To achieve this goal, we propose new dataset condensation techniques and an innovative unlearning scheme that strikes a balance between machine unlearning privacy, utility, and efficiency. Furthermore, we present a novel and effective approach to instrumenting machine unlearning and propose its application in defending against membership inference and model inversion attacks. Additionally, we explore a new application of our approach, which involves removing data from `condensed model', which can be employed to quickly train any arbitrary model without being influenced by unlearning samples.

Title: An Experiment on Feature Selection using Logistic Regression

• Authors: Raisa Islam, Subhasish Mazumdar, Rakibul Islam
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00201
• Pdf URL: https://arxiv.org/pdf/2402.00201
• Copy Paste: [[2402.00201]] An Experiment on Feature Selection using Logistic Regression(https://arxiv.org/abs/2402.00201)
• Keywords: explainability
• Abstract: In supervised machine learning, feature selection plays a very important role by potentially enhancing explainability and performance as measured by computing time and accuracy-related metrics. In this paper, we investigate a method for feature selection based on the well-known L1 and L2 regularization strategies associated with logistic regression (LR). It is well known that the learned coefficients, which serve as weights, can be used to rank the features. Our approach is to synthesize the findings of L1 and L2 regularization. For our experiment, we chose the CIC-IDS2018 dataset owing partly to its size and also to the existence of two problematic classes that are hard to separate. We report first with the exclusion of one of them and then with its inclusion. We ranked features first with L1 and then with L2, and then compared logistic regression with L1 (LR+L1) against that with L2 (LR+L2) by varying the sizes of the feature sets for each of the two rankings. We found no significant difference in accuracy between the two methods once the feature set is selected. We chose a synthesis, i.e., only those features that were present in both the sets obtained from L1 and that from L2, and experimented with it on more complex models like Decision Tree and Random Forest and observed that the accuracy was very close in spite of the small size of the feature set. Additionally, we also report on the standard metrics: accuracy, precision, recall, and f1-score.

Title: Decentralised, Collaborative, and Privacy-preserving Machine Learning for Multi-Hospital Data

• Authors: Congyu Fang, Adam Dziedzic, Lin Zhang, Laura Oliva, Amol Verma, Fahad Razak, Nicolas Papernot, Bo Wang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.00205
• Pdf URL: https://arxiv.org/pdf/2402.00205
• Copy Paste: [[2402.00205]] Decentralised, Collaborative, and Privacy-preserving Machine Learning for Multi-Hospital Data(https://arxiv.org/abs/2402.00205)
• Keywords: privacy
• Abstract: Machine Learning (ML) has demonstrated its great potential on medical data analysis. Large datasets collected from diverse sources and settings are essential for ML models in healthcare to achieve better accuracy and generalizability. Sharing data across different healthcare institutions is challenging because of complex and varying privacy and regulatory requirements. Hence, it is hard but crucial to allow multiple parties to collaboratively train an ML model leveraging the private datasets available at each party without the need for direct sharing of those datasets or compromising the privacy of the datasets through collaboration. In this paper, we address this challenge by proposing Decentralized, Collaborative, and Privacy-preserving ML for Multi-Hospital Data (DeCaPH). It offers the following key benefits: (1) it allows different parties to collaboratively train an ML model without transferring their private datasets; (2) it safeguards patient privacy by limiting the potential privacy leakage arising from any contents shared across the parties during the training process; and (3) it facilitates the ML model training without relying on a centralized server. We demonstrate the generalizability and power of DeCaPH on three distinct tasks using real-world distributed medical datasets: patient mortality prediction using electronic health records, cell-type classification using single-cell human genomes, and pathology identification using chest radiology images. We demonstrate that the ML models trained with DeCaPH framework have an improved utility-privacy trade-off, showing it enables the models to have good performance while preserving the privacy of the training data points. In addition, the ML models trained with DeCaPH framework in general outperform those trained solely with the private datasets from individual parties, showing that DeCaPH enhances the model generalizability.

Title: MP-SL: Multihop Parallel Split Learning

• Authors: Joana Tirana, Spyros Lalis, Dimitris Chatzopoulos
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.00208
• Pdf URL: https://arxiv.org/pdf/2402.00208
• Copy Paste: [[2402.00208]] MP-SL: Multihop Parallel Split Learning(https://arxiv.org/abs/2402.00208)
• Keywords: federate
• Abstract: Federated Learning (FL) stands out as a widely adopted protocol facilitating the training of Machine Learning (ML) models while maintaining decentralized data. However, challenges arise when dealing with a heterogeneous set of participating devices, causing delays in the training process, particularly among devices with limited resources. Moreover, the task of training ML models with a vast number of parameters demands computing and memory resources beyond the capabilities of small devices, such as mobile and Internet of Things (IoT) devices. To address these issues, techniques like Parallel Split Learning (SL) have been introduced, allowing multiple resource-constrained devices to actively participate in collaborative training processes with assistance from resourceful compute nodes. Nonetheless, a drawback of Parallel SL is the substantial memory allocation required at the compute nodes, for instance training VGG-19 with 100 participants needs 80 GB. In this paper, we introduce Multihop Parallel SL (MP-SL), a modular and extensible ML as a Service (MLaaS) framework designed to facilitate the involvement of resource-constrained devices in collaborative and distributed ML model training. Notably, to alleviate memory demands per compute node, MP-SL supports multihop Parallel SL-based training. This involves splitting the model into multiple parts and utilizing multiple compute nodes in a pipelined manner. Extensive experimentation validates MP-SL's capability to handle system heterogeneity, demonstrating that the multihop configuration proves more efficient than horizontally scaled one-hop Parallel SL setups, especially in scenarios involving more cost-effective compute nodes.

Title: A Circuit Approach to Constructing Blockchains on Blockchains

• Authors: Ertem Nusret Tas, David Tse, Yifei Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.00220
• Pdf URL: https://arxiv.org/pdf/2402.00220
• Copy Paste: [[2402.00220]] A Circuit Approach to Constructing Blockchains on Blockchains(https://arxiv.org/abs/2402.00220)
• Keywords: secure
• Abstract: Since the creation of Bitcoin 15 years ago, there has been an explosion in the number of permissionless blockchains. Each of these blockchains provides an open ledger that anyone can read from and write to. In this multi-chain world, an important question emerges: how can we build a more secure overlay blockchain by reading from and writing to a given set of blockchains? Drawing an analogy with switching circuits, we approach the problem by defining two basic compositional operations between blockchains, serial and triangular compositions, and use these operations as building blocks to construct general overlay blockchains. Under the partially synchronous setting, we have the following results: 1) the serial composition, between two blockchains, yields an overlay blockchain that is safe if at least one of the two underlay blockchains is safe and that is live if both underlay blockchains are live; 2) the triangular composition between three blockchains, akin to parallel composition of switching circuits, yields an overlay blockchain that is safe if all underlay blockchains are safe and that is live if at least half of them are live; 3) repeated composition of these two basic operations can yield all possible tradeoffs of safety and liveness for an overlay blockchain built on arbitrary number of underlay chains. The results are also extended to the synchronous setting.

Title: Exploring the limits of decoder-only models trained on public speech recognition corpora

• Authors: Ankit Gupta, George Saon, Brian Kingsbury
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2402.00235
• Pdf URL: https://arxiv.org/pdf/2402.00235
• Copy Paste: [[2402.00235]] Exploring the limits of decoder-only models trained on public speech recognition corpora(https://arxiv.org/abs/2402.00235)
• Keywords: transformer, large language model
• Abstract: The emergence of industrial-scale speech recognition (ASR) models such as Whisper and USM, trained on 1M hours of weakly labelled and 12M hours of audio only proprietary data respectively, has led to a stronger need for large scale public ASR corpora and competitive open source pipelines. Unlike the said models, large language models are typically based on Transformer decoders, and it remains unclear if decoder-only models trained on public data alone can deliver competitive performance. In this work, we investigate factors such as choice of training datasets and modeling components necessary for obtaining the best performance using public English ASR corpora alone. Our Decoder-Only Transformer for ASR (DOTA) model comprehensively outperforms the encoder-decoder open source replication of Whisper (OWSM) on nearly all English ASR benchmarks and outperforms Whisper large-v3 on 7 out of 15 test sets. We release our codebase and model checkpoints under permissive license.

Title: Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary

• Authors: Takashi Morita
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.00236
• Pdf URL: https://arxiv.org/pdf/2402.00236
• Copy Paste: [[2402.00236]] Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary(https://arxiv.org/abs/2402.00236)
• Keywords: transformer
• Abstract: This study discusses the effects of positional encoding on recurrent neural networks (RNNs) utilizing synthetic benchmarks. Positional encoding "time-stamps" data points in time series and complements the capabilities of Transformer neural networks, which lack an inherent mechanism for representing the data order. By contrast, RNNs can encode the temporal information of data points on their own, rendering their use of positional encoding seemingly "redundant". Nonetheless, empirical investigations reveal the effectiveness of positional encoding even when coupled with RNNs, specifically for handling a large vocabulary that yields diverse observations. These findings pave the way for a new line of research on RNNs, concerning the combination of input-driven and autonomous time representation. Additionally, biological implications of the computational/simulational results are discussed, in the light of the affinity between the sinusoidal implementation of positional encoding and neural oscillations in biological brains.

Title: CNN-FL for Biotechnology Industry Empowered by Internet-of-BioNano Things and Digital Twins

• Authors: Mohammad (Behdad)Jamshidi, Dinh Thai Hoang, Diep N. Nguyen
• Subjects: cs.LG, eess.IV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2402.00238
• Pdf URL: https://arxiv.org/pdf/2402.00238
• Copy Paste: [[2402.00238]] CNN-FL for Biotechnology Industry Empowered by Internet-of-BioNano Things and Digital Twins(https://arxiv.org/abs/2402.00238)
• Keywords: security, robust, federate
• Abstract: Digital twins (DTs) are revolutionizing the biotechnology industry by enabling sophisticated digital representations of biological assets, microorganisms, drug development processes, and digital health applications. However, digital twinning at micro and nano scales, particularly in modeling complex entities like bacteria, presents significant challenges in terms of requiring advanced Internet of Things (IoT) infrastructure and computing approaches to achieve enhanced accuracy and scalability. In this work, we propose a novel framework that integrates the Internet of Bio-Nano Things (IoBNT) with advanced machine learning techniques, specifically convolutional neural networks (CNN) and federated learning (FL), to effectively tackle the identified challenges. Within our framework, IoBNT devices are deployed to gather image-based biological data across various physical environments, leveraging the strong capabilities of CNNs for robust machine vision and pattern recognition. Subsequently, FL is utilized to aggregate insights from these disparate data sources, creating a refined global model that continually enhances accuracy and predictive reliability, which is crucial for the effective deployment of DTs in biotechnology. The primary contribution is the development of a novel framework that synergistically combines CNN and FL, augmented by the capabilities of the IoBNT. This novel approach is specifically tailored to enhancing DTs in the biotechnology industry. The results showcase enhancements in the reliability and safety of microorganism DTs, while preserving their accuracy. Furthermore, the proposed framework excels in energy efficiency and security, offering a user-friendly and adaptable solution. This broadens its applicability across diverse sectors, including biotechnology and pharmaceutical industries, as well as clinical and hospital settings.

Title: Spectral Norm of Convolutional Layers with Circular and Zero Paddings

• Authors: Blaise Delattre, Quentin Barthélemy, Alexandre Allauzen
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00240
• Pdf URL: https://arxiv.org/pdf/2402.00240
• Copy Paste: [[2402.00240]] Spectral Norm of Convolutional Layers with Circular and Zero Paddings(https://arxiv.org/abs/2402.00240)
• Keywords: robust
• Abstract: This paper leverages the use of \emph{Gram iteration} an efficient, deterministic, and differentiable method for computing spectral norm with an upper bound guarantee. Designed for circular convolutional layers, we generalize the use of the Gram iteration to zero padding convolutional layers and prove its quadratic convergence. We also provide theorems for bridging the gap between circular and zero padding convolution's spectral norm. We design a \emph{spectral rescaling} that can be used as a competitive $1$-Lipschitz layer that enhances network robustness. Demonstrated through experiments, our method outperforms state-of-the-art techniques in precision, computational cost, and scalability. The code of experiments is available at https://github.com/blaisedelattre/lip4conv.

Title: LRDif: Diffusion Models for Under-Display Camera Emotion Recognition

• Authors: Zhifeng Wang, Kaihao Zhang, Ramesh Sankaranarayana
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00250
• Pdf URL: https://arxiv.org/pdf/2402.00250
• Copy Paste: [[2402.00250]] LRDif: Diffusion Models for Under-Display Camera Emotion Recognition(https://arxiv.org/abs/2402.00250)
• Keywords: robust, extraction, diffusion, transformer
• Abstract: This study introduces LRDif, a novel diffusion-based framework designed specifically for facial expression recognition (FER) within the context of under-display cameras (UDC). To address the inherent challenges posed by UDC's image degradation, such as reduced sharpness and increased noise, LRDif employs a two-stage training strategy that integrates a condensed preliminary extraction network (FPEN) and an agile transformer network (UDCformer) to effectively identify emotion labels from UDC images. By harnessing the robust distribution mapping capabilities of Diffusion Models (DMs) and the spatial dependency modeling strength of transformers, LRDif effectively overcomes the obstacles of noise and distortion inherent in UDC environments. Comprehensive experiments on standard FER datasets including RAF-DB, KDEF, and FERPlus, LRDif demonstrate state-of-the-art performance, underscoring its potential in advancing FER applications. This work not only addresses a significant gap in the literature by tackling the UDC challenge in FER but also sets a new benchmark for future research in the field.

Title: Efficient Non-Parametric Uncertainty Quantification for Black-Box Large Language Models and Decision Planning

• Authors: Yao-Hung Hubert Tsai, Walter Talbott, Jian Zhang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00251
• Pdf URL: https://arxiv.org/pdf/2402.00251
• Copy Paste: [[2402.00251]] Efficient Non-Parametric Uncertainty Quantification for Black-Box Large Language Models and Decision Planning(https://arxiv.org/abs/2402.00251)
• Keywords: large language model
• Abstract: Step-by-step decision planning with large language models (LLMs) is gaining attention in AI agent development. This paper focuses on decision planning with uncertainty estimation to address the hallucination problem in language models. Existing approaches are either white-box or computationally demanding, limiting use of black-box proprietary LLMs within budgets. The paper's first contribution is a non-parametric uncertainty quantification method for LLMs, efficiently estimating point-wise dependencies between input-decision on the fly with a single inference, without access to token logits. This estimator informs the statistical interpretation of decision trustworthiness. The second contribution outlines a systematic design for a decision-making agent, generating actions like ``turn on the bathroom light'' based on user prompts such as ``take a bath''. Users will be asked to provide preferences when more than one action has high estimated point-wise dependencies. In conclusion, our uncertainty estimation and decision-making agent design offer a cost-efficient approach for AI agent development.

Title: Does \textsc{DetectGPT} Fully Utilize Perturbation? Selective Perturbation on Model-Based Contrastive Learning Detector would be Better

• Authors: Shengchao Liu, Xiaoming Liu, Yichen Wang, Zehua Cheng, Chengzhengxu Li, Zhaohan Zhang, Yu Lan, Chao Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00263
• Pdf URL: https://arxiv.org/pdf/2402.00263
• Copy Paste: [[2402.00263]] Does \textsc{DetectGPT} Fully Utilize Perturbation? Selective Perturbation on Model-Based Contrastive Learning Detector would be Better(https://arxiv.org/abs/2402.00263)
• Keywords: robust, large language model
• Abstract: The burgeoning capabilities of large language models (LLMs) have raised growing concerns about abuse. DetectGPT, a zero-shot metric-based unsupervised machine-generated text detector, first introduces perturbation and shows great performance improvement. However, DetectGPT's random perturbation strategy might introduce noise, limiting the distinguishability and further performance improvements. Moreover, its logit regression module relies on setting the threshold, which harms the generalizability and applicability of individual or small-batch inputs. Hence, we propose a novel detector, \modelname{}, which uses selective strategy perturbation to relieve the important information loss caused by random masking, and multi-pair contrastive learning to capture the implicit pattern information during perturbation, facilitating few-shot performance. The experiments show that \modelname{} outperforms the SOTA method by 1.20\% in accuracy on average on four public datasets. We further analyze the effectiveness, robustness, and generalization of our perturbation method.

Title: Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues

• Authors: Soufiane Belharbi, Marco Pedersoli, Alessandro Lameiras Koerich, Simon Bacon, Eric Granger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00281
• Pdf URL: https://arxiv.org/pdf/2402.00281
• Copy Paste: [[2402.00281]] Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues(https://arxiv.org/abs/2402.00281)
• Keywords: interpretability, transformer
• Abstract: While state-of-the-art facial expression recognition (FER) classifiers achieve a high level of accuracy, they lack interpretability, an important aspect for end-users. To recognize basic facial expressions, experts resort to a codebook associating a set of spatial action units to a facial expression. In this paper, we follow the same expert footsteps, and propose a learning strategy that allows us to explicitly incorporate spatial action units (aus) cues into the classifier's training to build a deep interpretable model. In particular, using this aus codebook, input image expression label, and facial landmarks, a single action units heatmap is built to indicate the most discriminative regions of interest in the image w.r.t the facial expression. We leverage this valuable spatial cue to train a deep interpretable classifier for FER. This is achieved by constraining the spatial layer features of a classifier to be correlated with \aus map. Using a composite loss, the classifier is trained to correctly classify an image while yielding interpretable visual layer-wise attention correlated with aus maps, simulating the experts' decision process. This is achieved using only the image class expression as supervision and without any extra manual annotations. Moreover, our method is generic. It can be applied to any CNN- or transformer-based deep classifier without the need for architectural change or adding significant training time. Our extensive evaluation on two public benchmarks RAFDB, and AFFECTNET datasets shows that our proposed strategy can improve layer-wise interpretability without degrading classification performance. In addition, we explore a common type of interpretable classifiers that rely on Class-Activation Mapping methods (CAMs), and we show that our training technique improves the CAM interpretability.

Title: Multimodal Embodied Interactive Agent for Cafe Scene

• Authors: Yang Liu, Xinshuai Song, Kaixuan Jiang, Weixing Chen, Jingzhou Luo, Guanbin Li, Liang Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00290
• Pdf URL: https://arxiv.org/pdf/2402.00290
• Copy Paste: [[2402.00290]] Multimodal Embodied Interactive Agent for Cafe Scene(https://arxiv.org/abs/2402.00290)
• Keywords: large language model
• Abstract: With the surge in the development of large language models, embodied intelligence has attracted increasing attention. Nevertheless, prior works on embodied intelligence typically encode scene or historical memory in an unimodal manner, either visual or linguistic, which complicates the alignment of the model's action planning with embodied control. To overcome this limitation, we introduce the Multimodal Embodied Interactive Agent (MEIA), capable of translating high-level tasks expressed in natural language into a sequence of executable actions. Specifically, we propose a novel Multimodal Environment Memory (MEM) module, facilitating the integration of embodied control with large models through the visual-language memory of scenes. This capability enables MEIA to generate executable action plans based on diverse requirements and the robot's capabilities. We conduct experiments in a dynamic virtual cafe environment, utilizing multiple large models through zero-shot learning, and carefully design scenarios for various situations. The experimental results showcase the promising performance of our MEIA in various embodied interactive tasks.

Title: FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation

• Authors: Takuma Yagi, Misaki Ohashi, Yifei Huang, Ryosuke Furuta, Shungo Adachi, Toutai Mitsuyama, Yoichi Sato
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00293
• Pdf URL: https://arxiv.org/pdf/2402.00293
• Copy Paste: [[2402.00293]] FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation(https://arxiv.org/abs/2402.00293)
• Keywords: segmentation
• Abstract: In the development of science, accurate and reproducible documentation of the experimental process is crucial. Automatic recognition of the actions in experiments from videos would help experimenters by complementing the recording of experiments. Towards this goal, we propose FineBio, a new fine-grained video dataset of people performing biological experiments. The dataset consists of multi-view videos of 32 participants performing mock biological experiments with a total duration of 14.5 hours. One experiment forms a hierarchical structure, where a protocol consists of several steps, each further decomposed into a set of atomic operations. The uniqueness of biological experiments is that while they require strict adherence to steps described in each protocol, there is freedom in the order of atomic operations. We provide hierarchical annotation on protocols, steps, atomic operations, object locations, and their manipulation states, providing new challenges for structured activity understanding and hand-object interaction recognition. To find out challenges on activity understanding in biological experiments, we introduce baseline models and results on four different tasks, including (i) step segmentation, (ii) atomic operation detection (iii) object detection, and (iv) manipulated/affected object detection. Dataset and code are available from https://github.com/aistairc/FineBio.

Title: Comparative Evaluation of Traditional and Deep Learning-Based Segmentation Methods for Spoil Pile Delineation Using UAV Images

• Authors: Sureka Thiruchittampalam, Bikram P. Banerjee, Nancy F. Glenn, Simit Raval
• Subjects: cs.CV, stat.AP
• Abstract URL: https://arxiv.org/abs/2402.00295
• Pdf URL: https://arxiv.org/pdf/2402.00295
• Copy Paste: [[2402.00295]] Comparative Evaluation of Traditional and Deep Learning-Based Segmentation Methods for Spoil Pile Delineation Using UAV Images(https://arxiv.org/abs/2402.00295)
• Keywords: extraction, segmentation
• Abstract: The stability of mine dumps is contingent upon the precise arrangement of spoil piles, taking into account their geological and geotechnical attributes. Yet, on-site characterisation of individual piles poses a formidable challenge. The utilisation of image-based techniques for spoil pile characterisation, employing remotely acquired data through unmanned aerial systems, is a promising complementary solution. Image processing, such as object-based classification and feature extraction, are dependent upon effective segmentation. This study refines and juxtaposes various segmentation approaches, specifically colour-based and morphology-based techniques. The objective is to enhance and evaluate avenues for object-based analysis for spoil characterisation within the context of mining environments. Furthermore, a comparative analysis is conducted between conventional segmentation approaches and those rooted in deep learning methodologies. Among the diverse segmentation approaches evaluated, the morphology-based deep learning segmentation approach, Segment Anything Model (SAM), exhibited superior performance in comparison to other approaches. This outcome underscores the efficacy of incorporating advanced morphological and deep learning techniques for accurate and efficient spoil pile characterisation. The findings of this study contribute valuable insights to the optimisation of segmentation strategies, thereby advancing the application of image-based techniques for the characterisation of spoil piles in mining environments.

Title: Self-supervised learning of video representations from a child's perspective

• Authors: A. Emin Orhan, Wentao Wang, Alex N. Wang, Mengye Ren, Brenden M. Lake
• Subjects: cs.CV, cs.LG, cs.NE, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2402.00300
• Pdf URL: https://arxiv.org/pdf/2402.00300
• Copy Paste: [[2402.00300]] Self-supervised learning of video representations from a child's perspective(https://arxiv.org/abs/2402.00300)
• Keywords: robust
• Abstract: Children learn powerful internal models of the world around them from a few years of egocentric visual experience. Can such internal models be learned from a child's visual experience with highly generic learning algorithms or do they require strong inductive biases? Recent advances in collecting large-scale, longitudinal, developmentally realistic video datasets and generic self-supervised learning (SSL) algorithms are allowing us to begin to tackle this nature vs. nurture question. However, existing work typically focuses on image-based SSL algorithms and visual capabilities that can be learned from static images (e.g. object recognition), thus ignoring temporal aspects of the world. To close this gap, here we train self-supervised video models on longitudinal, egocentric headcam recordings collected from a child over a two year period in their early development (6-31 months). The resulting models are highly effective at facilitating the learning of action concepts from a small number of labeled examples; they have favorable data size scaling properties; and they display emergent video interpolation capabilities. Video models also learn more robust object representations than image-based models trained with the exact same data. These results suggest that important temporal aspects of a child's internal model of the world may be learnable from their visual experience using highly generic learning algorithms and without strong inductive biases.

Title: Invariance-powered Trustworthy Defense via Remove Then Restore

• Authors: Xiaowei Fu, Yuhang Zhou, Lina Ma, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00304
• Pdf URL: https://arxiv.org/pdf/2402.00304
• Copy Paste: [[2402.00304]] Invariance-powered Trustworthy Defense via Remove Then Restore(https://arxiv.org/abs/2402.00304)
• Keywords: defense, attack, robust
• Abstract: Adversarial attacks pose a challenge to the deployment of deep neural networks (DNNs), while previous defense models overlook the generalization to various attacks. Inspired by targeted therapies for cancer, we view adversarial samples as local lesions of natural benign samples, because a key finding is that salient attack in an adversarial sample dominates the attacking process, while trivial attack unexpectedly provides trustworthy evidence for obtaining generalizable robustness. Based on this finding, a Pixel Surgery and Semantic Regeneration (PSSR) model following the targeted therapy mechanism is developed, which has three merits: 1) To remove the salient attack, a score-based Pixel Surgery module is proposed, which retains the trivial attack as a kind of invariance information. 2) To restore the discriminative content, a Semantic Regeneration module based on a conditional alignment extrapolator is proposed, which achieves pixel and semantic consistency. 3) To further harmonize robustness and accuracy, an intractable problem, a self-augmentation regularizer with adversarial R-drop is designed. Experiments on numerous benchmarks show the superiority of PSSR.

Title: Online Distribution Learning with Local Private Constraints

• Authors: Jin Sima, Changlong Wu, Olgica Milenkovic, Wojciech Szpankowski
• Subjects: cs.LG, cs.CR, cs.DS, cs.IT
• Abstract URL: https://arxiv.org/abs/2402.00315
• Pdf URL: https://arxiv.org/pdf/2402.00315
• Copy Paste: [[2402.00315]] Online Distribution Learning with Local Private Constraints(https://arxiv.org/abs/2402.00315)
• Keywords: privacy
• Abstract: We study the problem of online conditional distribution estimation with \emph{unbounded} label sets under local differential privacy. Let $\mathcal{F}$ be a distribution-valued function class with unbounded label set. We aim at estimating an \emph{unknown} function $f\in \mathcal{F}$ in an online fashion so that at time $t$ when the context $\boldsymbol{x}_t$ is provided we can generate an estimate of $f(\boldsymbol{x}_t)$ under KL-divergence knowing only a privatized version of the true labels sampling from $f(\boldsymbol{x}_t)$. The ultimate objective is to minimize the cumulative KL-risk of a finite horizon $T$. We show that under $(\epsilon,0)$-local differential privacy of the privatized labels, the KL-risk grows as $\tilde{\Theta}(\frac{1}{\epsilon}\sqrt{KT})$ upto poly-logarithmic factors where $K=|\mathcal{F}|$. This is in stark contrast to the $\tilde{\Theta}(\sqrt{T\log K})$ bound demonstrated by Wu et al. (2023a) for bounded label sets. As a byproduct, our results recover a nearly tight upper bound for the hypothesis selection problem of gopi et al. (2020) established only for the batch setting.

Title: Analog-digital Scheduling for Federated Learning: A Communication-Efficient Approach

• Authors: Muhammad Faraz Ul Abrar, Nicolò Michelusi
• Subjects: cs.LG, cs.IT, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.00318
• Pdf URL: https://arxiv.org/pdf/2402.00318
• Copy Paste: [[2402.00318]] Analog-digital Scheduling for Federated Learning: A Communication-Efficient Approach(https://arxiv.org/abs/2402.00318)
• Keywords: federate
• Abstract: Over-the-air (OTA) computation has recently emerged as a communication-efficient Federated Learning (FL) paradigm to train machine learning models over wireless networks. However, its performance is limited by the device with the worst SNR, resulting in fast yet noisy updates. On the other hand, allocating orthogonal resource blocks (RB) to individual devices via digital channels mitigates the noise problem, at the cost of increased communication latency. In this paper, we address this discrepancy and present ADFL, a novel Analog-Digital FL scheme: in each round, the parameter server (PS) schedules each device to either upload its gradient via the analog OTA scheme or transmit its quantized gradient over an orthogonal RB using the ``digital" scheme. Focusing on a single FL round, we cast the optimal scheduling problem as the minimization of the mean squared error (MSE) on the estimated global gradient at the PS, subject to a delay constraint, yielding the optimal device scheduling configuration and quantization bits for the digital devices. Our simulation results show that ADFL, by scheduling most of the devices in the OTA scheme while also occasionally employing the digital scheme for a few devices, consistently outperforms OTA-only and digital-only schemes, in both i.i.d. and non-i.i.d. settings.

Title: Comparing Spectral Bias and Robustness For Two-Layer Neural Networks: SGD vs Adaptive Random Fourier Features

• Authors: Aku Kammonen, Lisi Liang, Anamika Pandey, Raúl Tempone
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.00332
• Pdf URL: https://arxiv.org/pdf/2402.00332
• Copy Paste: [[2402.00332]] Comparing Spectral Bias and Robustness For Two-Layer Neural Networks: SGD vs Adaptive Random Fourier Features(https://arxiv.org/abs/2402.00332)
• Keywords: attack, robust
• Abstract: We present experimental results highlighting two key differences resulting from the choice of training algorithm for two-layer neural networks. The spectral bias of neural networks is well known, while the spectral bias dependence on the choice of training algorithm is less studied. Our experiments demonstrate that an adaptive random Fourier features algorithm (ARFF) can yield a spectral bias closer to zero compared to the stochastic gradient descent optimizer (SGD). Additionally, we train two identically structured classifiers, employing SGD and ARFF, to the same accuracy levels and empirically assess their robustness against adversarial noise attacks.

Title: Survey of Privacy Threats and Countermeasures in Federated Learning

• Authors: Masahiro Hayashitani, Junki Mori, Isamu Teranishi
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.00342
• Pdf URL: https://arxiv.org/pdf/2402.00342
• Copy Paste: [[2402.00342]] Survey of Privacy Threats and Countermeasures in Federated Learning(https://arxiv.org/abs/2402.00342)
• Keywords: privacy, federate
• Abstract: Federated learning is widely considered to be as a privacy-aware learning method because no training data is exchanged directly between clients. Nevertheless, there are threats to privacy in federated learning, and privacy countermeasures have been studied. However, we note that common and unique privacy threats among typical types of federated learning have not been categorized and described in a comprehensive and specific way. In this paper, we describe privacy threats and countermeasures for the typical types of federated learning; horizontal federated learning, vertical federated learning, and transfer federated learning.

Title: IndiVec: An Exploration of Leveraging Large Language Models for Media Bias Detection with Fine-Grained Bias Indicators

• Authors: Luyang Lin, Lingzhi Wang, Xiaoyan Zhao, Jing Li, Kam-Fai Wong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00345
• Pdf URL: https://arxiv.org/pdf/2402.00345
• Copy Paste: [[2402.00345]] IndiVec: An Exploration of Leveraging Large Language Models for Media Bias Detection with Fine-Grained Bias Indicators(https://arxiv.org/abs/2402.00345)
• Keywords: robust, explainability, large language model
• Abstract: This study focuses on media bias detection, crucial in today's era of influential social media platforms shaping individual attitudes and opinions. In contrast to prior work that primarily relies on training specific models tailored to particular datasets, resulting in limited adaptability and subpar performance on out-of-domain data, we introduce a general bias detection framework, IndiVec, built upon large language models. IndiVec begins by constructing a fine-grained media bias database, leveraging the robust instruction-following capabilities of large language models and vector database techniques. When confronted with new input for bias detection, our framework automatically selects the most relevant indicator from the vector database and employs majority voting to determine the input's bias label. IndiVec excels compared to previous methods due to its adaptability (demonstrating consistent performance across diverse datasets from various sources) and explainability (providing explicit top-k indicators to interpret bias predictions). Experimental results on four political bias datasets highlight IndiVec's significant superiority over baselines. Furthermore, additional experiments and analysis provide profound insights into the framework's effectiveness.

Title: ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update

• Authors: Liyuan Mao, Haoran Xu, Weinan Zhang, Xianyuan Zhan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00348
• Pdf URL: https://arxiv.org/pdf/2402.00348
• Copy Paste: [[2402.00348]] ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update(https://arxiv.org/abs/2402.00348)
• Keywords: robust
• Abstract: In this study, we investigate the DIstribution Correction Estimation (DICE) methods, an important line of work in offline reinforcement learning (RL) and imitation learning (IL). DICE-based methods impose state-action-level behavior constraint, which is an ideal choice for offline learning. However, they typically perform much worse than current state-of-the-art (SOTA) methods that solely use action-level behavior constraint. After revisiting DICE-based methods, we find there exist two gradient terms when learning the value function using true-gradient update: forward gradient (taken on the current state) and backward gradient (taken on the next state). Using forward gradient bears a large similarity to many offline RL methods, and thus can be regarded as applying action-level constraint. However, directly adding the backward gradient may degenerate or cancel out its effect if these two gradients have conflicting directions. To resolve this issue, we propose a simple yet effective modification that projects the backward gradient onto the normal plane of the forward gradient, resulting in an orthogonal-gradient update, a new learning rule for DICE-based methods. We conduct thorough theoretical analyses and find that the projected backward gradient brings state-level behavior regularization, which reveals the mystery of DICE-based methods: the value learning objective does try to impose state-action-level constraint, but needs to be used in a corrected way. Through toy examples and extensive experiments on complex offline RL and IL tasks, we demonstrate that DICE-based methods using orthogonal-gradient updates (O-DICE) achieve SOTA performance and great robustness.

Title: Machine Unlearning for Image-to-Image Generative Models

• Authors: Guihong Li, Hsiang Hsu, Chun-Fu (Richard)Chen, Radu Marculescu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00351
• Pdf URL: https://arxiv.org/pdf/2402.00351
• Copy Paste: [[2402.00351]] Machine Unlearning for Image-to-Image Generative Models(https://arxiv.org/abs/2402.00351)
• Keywords: generative
• Abstract: Machine unlearning has emerged as a new paradigm to deliberately forget data samples from a given model in order to adhere to stringent regulations. However, existing machine unlearning methods have been primarily focused on classification models, leaving the landscape of unlearning for generative models relatively unexplored. This paper serves as a bridge, addressing the gap by providing a unifying framework of machine unlearning for image-to-image generative models. Within this framework, we propose a computationally-efficient algorithm, underpinned by rigorous theoretical analysis, that demonstrates negligible performance degradation on the retain samples, while effectively removing the information from the forget samples. Empirical studies on two large-scale datasets, ImageNet-1K and Places-365, further show that our algorithm does not rely on the availability of the retain samples, which further complies with data retention policy. To our best knowledge, this work is the first that represents systemic, theoretical, empirical explorations of machine unlearning specifically tailored for image-to-image generative models. Our code is available at https://github.com/jpmorganchase/l2l-generator-unlearning.

Title: High-Quality Medical Image Generation from Free-hand Sketch

• Authors: Quan Huu Cap, Atsushi Fukuda
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00353
• Pdf URL: https://arxiv.org/pdf/2402.00353
• Copy Paste: [[2402.00353]] High-Quality Medical Image Generation from Free-hand Sketch(https://arxiv.org/abs/2402.00353)
• Keywords: robust, segmentation
• Abstract: Generating medical images from human-drawn free-hand sketches holds promise for various important medical imaging applications. Due to the extreme difficulty in collecting free-hand sketch data in the medical domain, most deep learning-based methods have been proposed to generate medical images from the synthesized sketches (e.g., edge maps or contours of segmentation masks from real images). However, these models often fail to generalize on the free-hand sketches, leading to unsatisfactory results. In this paper, we propose a practical free-hand sketch-to-image generation model called Sketch2MedI that learns to represent sketches in StyleGAN's latent space and generate medical images from it. Thanks to the ability to encode sketches into this meaningful representation space, Sketch2MedI only requires synthesized sketches for training, enabling a cost-effective learning process. Our Sketch2MedI demonstrates a robust generalization to free-hand sketches, resulting in high-quality and realistic medical image generations. Comparative evaluations of Sketch2MedI against the pix2pix, CycleGAN, UNIT, and U-GAT-IT models show superior performance in generating pharyngeal images, both quantitative and qualitative across various metrics.

Title: Adaptive Primal-Dual Method for Safe Reinforcement Learning

• Authors: Weiqin Chen, James Onyejizu, Long Vu, Lan Hoang, Dharmashankar Subramanian, Koushik Kar, Sandipan Mishra, Santiago Paternain
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2402.00355
• Pdf URL: https://arxiv.org/pdf/2402.00355
• Copy Paste: [[2402.00355]] Adaptive Primal-Dual Method for Safe Reinforcement Learning(https://arxiv.org/abs/2402.00355)
• Keywords: robust
• Abstract: Primal-dual methods have a natural application in Safe Reinforcement Learning (SRL), posed as a constrained policy optimization problem. In practice however, applying primal-dual methods to SRL is challenging, due to the inter-dependency of the learning rate (LR) and Lagrangian multipliers (dual variables) each time an embedded unconstrained RL problem is solved. In this paper, we propose, analyze and evaluate adaptive primal-dual (APD) methods for SRL, where two adaptive LRs are adjusted to the Lagrangian multipliers so as to optimize the policy in each iteration. We theoretically establish the convergence, optimality and feasibility of the APD algorithm. Finally, we conduct numerical evaluation of the practical APD algorithm with four well-known environments in Bullet-Safey-Gym employing two state-of-the-art SRL algorithms: PPO-Lagrangian and DDPG-Lagrangian. All experiments show that the practical APD algorithm outperforms (or achieves comparable performance) and attains more stable training than the constant LR cases. Additionally, we substantiate the robustness of selecting the two adaptive LRs by empirical evidence.

Title: Securing Cloud-Based Internet of Things: Challenges and Mitigations

• Authors: Nivedita Singh, Rajkumar Buyya, Hyoungshich Kim
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.00356
• Pdf URL: https://arxiv.org/pdf/2402.00356
• Copy Paste: [[2402.00356]] Securing Cloud-Based Internet of Things: Challenges and Mitigations(https://arxiv.org/abs/2402.00356)
• Keywords: security, privacy, attack
• Abstract: The Internet of Things (IoT) has seen remarkable advancements in recent years, leading to a paradigm shift in the digital landscape. However, these technological strides have also brought new challenges, particularly in terms of cybersecurity. IoT devices are inherently connected to the internet, which makes them more vulnerable to attack. In addition, IoT services often handle sensitive user data, which could be misused by malicious actors or unauthorized service providers. As more mainstream service providers emerge without uniform regulations, these security risks are expected to escalate exponentially. The task of maintaining the security of IoT devices while they interact with cloud services is also challenging. Newer IoT services, especially those developed and deployed via Platform-as-a-Service (PaaS) and Infrastructure-as-a-Service (IaaS) models, pose additional security threats. Although IoT devices are becoming more affordable and ubiquitous, their growing complexity could expose users to heightened security and privacy risks. This paper highlights these pressing security concerns associated with the widespread adoption of IoT devices and services. We propose potential solutions to bridge the existing security gaps and expect future challenges. Our approach entails a comprehensive exploration of the key security challenges that IoT services are currently facing. We also suggest proactive strategies to mitigate these risks, strengthening the overall security of IoT devices and services.

Title: Safety of Multimodal Large Language Models on Images and Text

• Authors: Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, Yu Qiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00357
• Pdf URL: https://arxiv.org/pdf/2402.00357
• Copy Paste: [[2402.00357]] Safety of Multimodal Large Language Models on Images and Text(https://arxiv.org/abs/2402.00357)
• Keywords: defense, attack, large language model
• Abstract: Attracted by the impressive power of Multimodal Large Language Models (MLLMs), the public is increasingly utilizing them to improve the efficiency of daily work. Nonetheless, the vulnerabilities of MLLMs to unsafe instructions bring huge safety risks when these models are deployed in real-world scenarios. In this paper, we systematically survey current efforts on the evaluation, attack, and defense of MLLMs' safety on images and text. We begin with introducing the overview of MLLMs on images and text and understanding of safety, which helps researchers know the detailed scope of our survey. Then, we review the evaluation datasets and metrics for measuring the safety of MLLMs. Next, we comprehensively present attack and defense techniques related to MLLMs' safety. Finally, we analyze several unsolved issues and discuss promising research directions.

Title: Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration

• Authors: Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, Yulia Tsvetkov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00367
• Pdf URL: https://arxiv.org/pdf/2402.00367
• Copy Paste: [[2402.00367]] Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration(https://arxiv.org/abs/2402.00367)
• Keywords: large language model
• Abstract: Despite efforts to expand the knowledge of large language models (LLMs), knowledge gaps -- missing or outdated information in LLMs -- might always persist given the evolving nature of knowledge. In this work, we study approaches to identify LLM knowledge gaps and abstain from answering questions when knowledge gaps are present. We first adapt existing approaches to model calibration or adaptation through fine-tuning/prompting and analyze their ability to abstain from generating low-confidence outputs. Motivated by their failures in self-reflection and over-reliance on held-out sets, we propose two novel approaches that are based on model collaboration, i.e., LLMs probing other LLMs for knowledge gaps, either cooperatively or competitively. Extensive experiments with three LLMs on four QA tasks featuring diverse knowledge domains demonstrate that both cooperative and competitive approaches to unveiling LLM knowledge gaps achieve up to 19.3% improvements on abstain accuracy against the strongest baseline. Further analysis reveals that our proposed mechanisms could help identify failure cases in retrieval augmentation and pinpoint knowledge gaps in multi-hop reasoning.

Title: What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detection

• Authors: Shangbin Feng, Herun Wan, Ningnan Wang, Zhaoxuan Tan, Minnan Luo, Yulia Tsvetkov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00371
• Pdf URL: https://arxiv.org/pdf/2402.00371
• Copy Paste: [[2402.00371]] What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detection(https://arxiv.org/abs/2402.00371)
• Keywords: large language model
• Abstract: Social media bot detection has always been an arms race between advancements in machine learning bot detectors and adversarial bot strategies to evade detection. In this work, we bring the arms race to the next level by investigating the opportunities and risks of state-of-the-art large language models (LLMs) in social bot detection. To investigate the opportunities, we design novel LLM-based bot detectors by proposing a mixture-of-heterogeneous-experts framework to divide and conquer diverse user information modalities. To illuminate the risks, we explore the possibility of LLM-guided manipulation of user textual and structured information to evade detection. Extensive experiments with three LLMs on two datasets demonstrate that instruction tuning on merely 1,000 annotated examples produces specialized LLMs that outperform state-of-the-art baselines by up to 9.1% on both datasets, while LLM-guided manipulation strategies could significantly bring down the performance of existing bot detectors by up to 29.6% and harm the calibration and reliability of bot detection systems.

Title: Efficient Exploration for LLMs

• Authors: Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, Benjamin Van Roy
• Subjects: cs.LG, cs.AI, cs.CL, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.00396
• Pdf URL: https://arxiv.org/pdf/2402.00396
• Copy Paste: [[2402.00396]] Efficient Exploration for LLMs(https://arxiv.org/abs/2402.00396)
• Keywords: large language model
• Abstract: We present evidence of substantial benefit from efficient exploration in gathering human feedback to improve large language models. In our experiments, an agent sequentially generates queries while fitting a reward model to the feedback received. Our best-performing agent generates queries using double Thompson sampling, with uncertainty represented by an epistemic neural network. Our results demonstrate that efficient exploration enables high levels of performance with far fewer queries. Further, both uncertainty estimation and the choice of exploration scheme play critical roles.

Title: Multi-scale Traffic Pattern Bank for Cross-city Few-shot Traffic Forecasting

• Authors: Zhanyu Liu, Guanjie Zheng, Yanwei Yu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00397
• Pdf URL: https://arxiv.org/pdf/2402.00397
• Copy Paste: [[2402.00397]] Multi-scale Traffic Pattern Bank for Cross-city Few-shot Traffic Forecasting(https://arxiv.org/abs/2402.00397)
• Keywords: robust
• Abstract: Traffic forecasting is crucial for intelligent transportation systems (ITS), aiding in efficient resource allocation and effective traffic control. However, its effectiveness often relies heavily on abundant traffic data, while many cities lack sufficient data due to limited device support, posing a significant challenge for traffic forecasting. Recognizing this challenge, we have made a noteworthy observation: traffic patterns exhibit similarities across diverse cities. Building on this key insight, we propose a solution for the cross-city few-shot traffic forecasting problem called Multi-scale Traffic Pattern Bank (MTPB). Primarily, MTPB initiates its learning process by leveraging data-rich source cities, effectively acquiring comprehensive traffic knowledge through a spatial-temporal-aware pre-training process. Subsequently, the framework employs advanced clustering techniques to systematically generate a multi-scale traffic pattern bank derived from the learned knowledge. Next, the traffic data of the data-scarce target city could query the traffic pattern bank, facilitating the aggregation of meta-knowledge. This meta-knowledge, in turn, assumes a pivotal role as a robust guide in subsequent processes involving graph reconstruction and forecasting. Empirical assessments conducted on real-world traffic datasets affirm the superior performance of MTPB, surpassing existing methods across various categories and exhibiting numerous attributes conducive to the advancement of cross-city few-shot forecasting methodologies. The code is available in https://github.com/zhyliu00/MTPB.

Title: Investigating Bias Representations in Llama 2 Chat via Activation Steering

• Authors: Dawn Lu, Nina Rimsky
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00402
• Pdf URL: https://arxiv.org/pdf/2402.00402
• Copy Paste: [[2402.00402]] Investigating Bias Representations in Llama 2 Chat via Activation Steering(https://arxiv.org/abs/2402.00402)
• Keywords: large language model
• Abstract: We address the challenge of societal bias in Large Language Models (LLMs), focusing on the Llama 2 7B Chat model. As LLMs are increasingly integrated into decision-making processes with substantial societal impact, it becomes imperative to ensure these models do not reinforce existing biases. Our approach employs activation steering to probe for and mitigate biases related to gender, race, and religion. This method manipulates model activations to direct responses towards or away from biased outputs, utilizing steering vectors derived from the StereoSet dataset and custom GPT4 generated gender bias prompts. Our findings reveal inherent gender bias in Llama 2 7B Chat, persisting even after Reinforcement Learning from Human Feedback (RLHF). We also observe a predictable negative correlation between bias and the model's tendency to refuse responses. Significantly, our study uncovers that RLHF tends to increase the similarity in the model's representation of different forms of societal biases, which raises questions about the model's nuanced understanding of different forms of bias. This work also provides valuable insights into effective red-teaming strategies for LLMs using activation steering, particularly emphasizing the importance of integrating a refusal vector.

Title: Hidding the Ghostwriters: An Adversarial Evaluation of AI-Generated Student Essay Detection

• Authors: Xinlin Peng, Ying Zhou, Ben He, Le Sun, Yingfei Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00412
• Pdf URL: https://arxiv.org/pdf/2402.00412
• Copy Paste: [[2402.00412]] Hidding the Ghostwriters: An Adversarial Evaluation of AI-Generated Student Essay Detection(https://arxiv.org/abs/2402.00412)
• Keywords: attack, robust, large language model
• Abstract: Large language models (LLMs) have exhibited remarkable capabilities in text generation tasks. However, the utilization of these models carries inherent risks, including but not limited to plagiarism, the dissemination of fake news, and issues in educational exercises. Although several detectors have been proposed to address these concerns, their effectiveness against adversarial perturbations, specifically in the context of student essay writing, remains largely unexplored. This paper aims to bridge this gap by constructing AIG-ASAP, an AI-generated student essay dataset, employing a range of text perturbation methods that are expected to generate high-quality essays while evading detection. Through empirical experiments, we assess the performance of current AIGC detectors on the AIG-ASAP dataset. The results reveal that the existing detectors can be easily circumvented using straightforward automatic adversarial attacks. Specifically, we explore word substitution and sentence substitution perturbation methods that effectively evade detection while maintaining the quality of the generated essays. This highlights the urgent need for more accurate and robust methods to detect AI-generated student essays in the education domain.

Title: Prompt-Time Symbolic Knowledge Capture with Large Language Models

• Authors: Tolga Çöplü, Arto Bendiken, Andrii Skomorokhov, Eduard Bateiko, Stephen Cobb, Joshua J. Bouw (Haltia, Inc.)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00414
• Pdf URL: https://arxiv.org/pdf/2402.00414
• Copy Paste: [[2402.00414]] Prompt-Time Symbolic Knowledge Capture with Large Language Models(https://arxiv.org/abs/2402.00414)
• Keywords: large language model
• Abstract: Augmenting large language models (LLMs) with user-specific knowledge is crucial for real-world applications, such as personal AI assistants. However, LLMs inherently lack mechanisms for prompt-driven knowledge capture. This paper investigates utilizing the existing LLM capabilities to enable prompt-driven knowledge capture, with a particular emphasis on knowledge graphs. We address this challenge by focusing on prompt-to-triple (P2T) generation. We explore three methods: zero-shot prompting, few-shot prompting, and fine-tuning, and then assess their performance via a specialized synthetic dataset. Our code and datasets are publicly available at https://github.com/HaltiaAI/paper-PTSKC.

Title: Short: Benchmarking transferable adversarial attacks

• Authors: Zhibo Jin, Jiayu Zhang, Zhiyu Zhu, Huaming Chen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00418
• Pdf URL: https://arxiv.org/pdf/2402.00418
• Copy Paste: [[2402.00418]] Short: Benchmarking transferable adversarial attacks(https://arxiv.org/abs/2402.00418)
• Keywords: attack, robust, generative
• Abstract: The robustness of deep learning models against adversarial attacks remains a pivotal concern. This study presents, for the first time, an exhaustive review of the transferability aspect of adversarial attacks. It systematically categorizes and critically evaluates various methodologies developed to augment the transferability of adversarial attacks. This study encompasses a spectrum of techniques, including Generative Structure, Semantic Similarity, Gradient Editing, Target Modification, and Ensemble Approach. Concurrently, this paper introduces a benchmark framework \textit{TAA-Bench}, integrating ten leading methodologies for adversarial attack transferability, thereby providing a standardized and systematic platform for comparative analysis across diverse model architectures. Through comprehensive scrutiny, we delineate the efficacy and constraints of each method, shedding light on their underlying operational principles and practical utility. This review endeavors to be a quintessential resource for both scholars and practitioners in the field, charting the complex terrain of adversarial transferability and setting a foundation for future explorations in this vital sector. The associated codebase is accessible at: https://github.com/KxPlaug/TAA-Bench

Title: From PARIS to LE-PARIS: Toward Patent Response Automation with Recommender Systems and Collaborative Large Language Models

• Authors: Jung-Mei Chu, Hao-Cheng Lo, Jieh Hsiang, Chun-Chieh Cho
• Subjects: cs.CL, cs.HC, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00421
• Pdf URL: https://arxiv.org/pdf/2402.00421
• Copy Paste: [[2402.00421]] From PARIS to LE-PARIS: Toward Patent Response Automation with Recommender Systems and Collaborative Large Language Models(https://arxiv.org/abs/2402.00421)
• Keywords: large language model
• Abstract: In patent prosecution, timely and effective responses to Office Actions (OAs) are crucial for acquiring patents, yet past automation and AI research have scarcely addressed this aspect. To address this gap, our study introduces the Patent Office Action Response Intelligence System (PARIS) and its advanced version, the Large Language Model Enhanced PARIS (LE-PARIS). These systems are designed to expedite the efficiency of patent attorneys in collaboratively handling OA responses. The systems' key features include the construction of an OA Topics Database, development of Response Templates, and implementation of Recommender Systems and LLM-based Response Generation. Our validation involves a multi-paradigmatic analysis using the USPTO Office Action database and longitudinal data of attorney interactions with our systems over six years. Through five studies, we examine the constructiveness of OA topics (studies 1 and 2) using topic modeling and the proposed Delphi process, the efficacy of our proposed hybrid recommender system tailored for OA (both LLM-based and non-LLM-based) (study 3), the quality of response generation (study 4), and the practical value of the systems in real-world scenarios via user studies (study 5). Results demonstrate that both PARIS and LE-PARIS significantly meet key metrics and positively impact attorney performance.

Title: Merging Multi-Task Models via Weight-Ensembling Mixture of Experts

• Authors: Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, Dacheng Tao
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00433
• Pdf URL: https://arxiv.org/pdf/2402.00433
• Copy Paste: [[2402.00433]] Merging Multi-Task Models via Weight-Ensembling Mixture of Experts(https://arxiv.org/abs/2402.00433)
• Keywords: robust, transformer
• Abstract: Merging various task-specific Transformer-based models trained on different tasks into a single unified model can execute all the tasks concurrently. Previous methods, exemplified by task arithmetic, have been proven to be both effective and scalable. Existing methods have primarily focused on seeking a static optimal solution within the original model parameter space. A notable challenge is mitigating the interference between parameters of different models, which can substantially deteriorate performance. In this paper, we propose to merge most of the parameters while upscaling the MLP of the Transformer layers to a weight-ensembling mixture of experts (MoE) module, which can dynamically integrate shared and task-specific knowledge based on the input, thereby providing a more flexible solution that can adapt to the specific needs of each instance. Our key insight is that by identifying and separating shared knowledge and task-specific knowledge, and then dynamically integrating them, we can mitigate the parameter interference problem to a great extent. We conduct the conventional multi-task model merging experiments and evaluate the generalization and robustness of our method. The results demonstrate the effectiveness of our method and provide a comprehensive understanding of our method. The code is available at https://anonymous.4open.science/r/weight-ensembling_MoE-67C9/

Title: Dual-Student Knowledge Distillation Networks for Unsupervised Anomaly Detection

• Authors: Liyi Yao, Shaobing Gao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00448
• Pdf URL: https://arxiv.org/pdf/2402.00448
• Copy Paste: [[2402.00448]] Dual-Student Knowledge Distillation Networks for Unsupervised Anomaly Detection(https://arxiv.org/abs/2402.00448)
• Keywords: segmentation
• Abstract: Due to the data imbalance and the diversity of defects, student-teacher networks (S-T) are favored in unsupervised anomaly detection, which explores the discrepancy in feature representation derived from the knowledge distillation process to recognize anomalies. However, vanilla S-T network is not stable. Employing identical structures to construct the S-T network may weaken the representative discrepancy on anomalies. But using different structures can increase the likelihood of divergent performance on normal data. To address this problem, we propose a novel dual-student knowledge distillation (DSKD) architecture. Different from other S-T networks, we use two student networks a single pre-trained teacher network, where the students have the same scale but inverted structures. This framework can enhance the distillation effect to improve the consistency in recognition of normal data, and simultaneously introduce diversity for anomaly representation. To explore high-dimensional semantic information to capture anomaly clues, we employ two strategies. First, a pyramid matching mode is used to perform knowledge distillation on multi-scale feature maps in the intermediate layers of networks. Second, an interaction is facilitated between the two student networks through a deep feature embedding module, which is inspired by real-world group discussions. In terms of classification, we obtain pixel-wise anomaly segmentation maps by measuring the discrepancy between the output feature maps of the teacher and student networks, from which an anomaly score is computed for sample-wise determination. We evaluate DSKD on three benchmark datasets and probe the effects of internal modules through ablation experiments. The results demonstrate that DSKD can achieve exceptional performance on small models like ResNet18 and effectively improve vanilla S-T networks.

Title: SA-MDKIF: A Scalable and Adaptable Medical Domain Knowledge Injection Framework for Large Language Models

• Authors: Tianhan Xu, Zhe Hu, Ling Chen, Bin Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00474
• Pdf URL: https://arxiv.org/pdf/2402.00474
• Copy Paste: [[2402.00474]] SA-MDKIF: A Scalable and Adaptable Medical Domain Knowledge Injection Framework for Large Language Models(https://arxiv.org/abs/2402.00474)
• Keywords: large language model
• Abstract: Recent advances in large language models (LLMs) have demonstrated exceptional performance in various natural language processing (NLP) tasks. However, their effective application in the medical domain is hampered by a lack of medical domain knowledge. In this study, we present SA-MDKIF, a scalable and adaptable framework that aims to inject medical knowledge into general-purpose LLMs through instruction tuning, thereby enabling adaptability for various downstream tasks. SA-MDKIF consists of two stages: skill training and skill adaptation. In the first stage, we define 12 basic medical skills and use AdaLoRA to train these skills based on uniformly formatted instructional datasets that we have constructed. In the next stage, we train the skill router using task-specific downstream data and use this router to integrate the acquired skills with LLMs during inference. Experimental results on 9 different medical tasks show that SA-MDKIF improves performance by 10-20% compared to the original LLMs. Notably, this improvement is particularly pronounced for unseen medical tasks, showing an improvement of up to 30%.

Title: EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models

• Authors: Xuchen Pan, Yanxi Chen, Yaliang Li, Bolin Ding, Jingren Zhou
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00518
• Pdf URL: https://arxiv.org/pdf/2402.00518
• Copy Paste: [[2402.00518]] EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models(https://arxiv.org/abs/2402.00518)
• Keywords: large language model
• Abstract: This work introduces EE-Tuning, a lightweight and economical solution to training/tuning early-exit large language models (LLMs). In contrast to the common approach of full-parameter pre-training, EE-Tuning augments any pre-trained (and possibly fine-tuned) standard LLM with additional early-exit layers that are tuned in a parameter-efficient manner, which requires significantly less computational resources and training data. Our implementation of EE-Tuning achieves outstanding training efficiency via extensive performance optimizations, as well as scalability due to its full compatibility with 3D parallelism. Results of systematic experiments validate the efficacy of EE-Tuning, confirming that effective early-exit LLM inference can be achieved with a limited training budget. In hope of making early-exit LLMs accessible to the community, we release the source code of our implementation of EE-Tuning at https://github.com/pan-x-c/EE-LLM.

Title: Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling

• Authors: Mingze Wang, Weinan E
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.00522
• Pdf URL: https://arxiv.org/pdf/2402.00522
• Copy Paste: [[2402.00522]] Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling(https://arxiv.org/abs/2402.00522)
• Keywords: transformer
• Abstract: We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through which different components of Transformer, such as the dot-product self-attention, positional encoding and feed-forward layer, affect its expressive power, and we study their combined effects through establishing explicit approximation rates. Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads, and these insights also provide natural suggestions for alternative architectures.

Title: A Manifold Representation of the Key in Vision Transformers

• Authors: Li Meng, Morten Goodwin, Anis Yazidi, Paal Engelstad
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00534
• Pdf URL: https://arxiv.org/pdf/2402.00534
• Copy Paste: [[2402.00534]] A Manifold Representation of the Key in Vision Transformers(https://arxiv.org/abs/2402.00534)
• Keywords: transformer, segmentation
• Abstract: Vision Transformers implement multi-head self-attention (MSA) via stacking multiple attention blocks. The query, key, and value are often intertwined and generated within those blocks via a single, shared linear transformation. This paper explores the concept of disentangling the key from the query and value, and adopting a manifold representation for the key. Our experiments reveal that decoupling and endowing the key with a manifold structure can enhance the model performance. Specifically, ViT-B exhibits a 0.87% increase in top-1 accuracy, while Swin-T sees a boost of 0.52% in top-1 accuracy on the ImageNet-1K dataset, with eight charts in the manifold key. Our approach also yields positive results in object detection and instance segmentation tasks on the COCO dataset. Through detailed ablation studies, we establish that these performance gains are not merely due to the simplicity of adding more parameters and computations. Future research may investigate strategies for cutting the budget of such representations and aim for further performance improvements based on our findings.

Title: Masked Conditional Diffusion Model for Enhancing Deepfake Detection

• Authors: Tiewen Chen, Shanmin Yang, Shu Hu, Zhenghan Fang, Ying Fu, Xi Wu, Xin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00541
• Pdf URL: https://arxiv.org/pdf/2402.00541
• Copy Paste: [[2402.00541]] Masked Conditional Diffusion Model for Enhancing Deepfake Detection(https://arxiv.org/abs/2402.00541)
• Keywords: robust, diffusion
• Abstract: Recent studies on deepfake detection have achieved promising results when training and testing faces are from the same dataset. However, their results severely degrade when confronted with forged samples that the model has not yet seen during training. In this paper, deepfake data to help detect deepfakes. this paper present we put a new insight into diffusion model-based data augmentation, and propose a Masked Conditional Diffusion Model (MCDM) for enhancing deepfake detection. It generates a variety of forged faces from a masked pristine one, encouraging the deepfake detection model to learn generic and robust representations without overfitting to special artifacts. Extensive experiments demonstrate that forgery images generated with our method are of high quality and helpful to improve the performance of deepfake detection models.

Title: Secure Supervised Learning-Based Smart Home Authentication Framework

• Authors: K. Swapna Sudha, N. Jeyanthi, Celestine Iwendi
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00568
• Pdf URL: https://arxiv.org/pdf/2402.00568
• Copy Paste: [[2402.00568]] Secure Supervised Learning-Based Smart Home Authentication Framework(https://arxiv.org/abs/2402.00568)
• Keywords: secure, security, privacy, attack
• Abstract: The Smart home possesses the capability of facilitating home services to their users with the systematic advance in The Internet of Things (IoT) and information and communication technologies (ICT) in recent decades. The home service offered by the smart devices helps the users in utilize maximized level of comfort for the objective of improving life quality. As the user and smart devices communicate through an insecure channel, the smart home environment is prone to security and privacy problems. A secure authentication protocol needs to be established between the smart devices and the user, such that a situation for device authentication can be made feasible in smart home environments. Most of the existing smart home authentication protocols were identified to fail in facilitating a secure mutual authentication and increases the possibility of lunching the attacks of session key disclosure, impersonation and stolen smart device. In this paper, Secure Supervised Learning-based Smart Home Authentication Framework (SSL-SHAF) is proposed as are liable mutual authentication that can be contextually imposed for better security. The formal analysis of the proposed SSL-SHAF confirmed better resistance against session key disclosure, impersonation and stolen smart device attacks. The results of SSL-SHAF confirmed minimized computational costs and security compared to the baseline protocols considered for investigation.

Title: Diffusion-based Light Field Synthesis

• Authors: Ruisheng Gao, Yutong Liu, Zeyu Xiao, Zhiwei Xiong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00575
• Pdf URL: https://arxiv.org/pdf/2402.00575
• Copy Paste: [[2402.00575]] Diffusion-based Light Field Synthesis(https://arxiv.org/abs/2402.00575)
• Keywords: robust, diffusion, generative
• Abstract: Light fields (LFs), conducive to comprehensive scene radiance recorded across angular dimensions, find wide applications in 3D reconstruction, virtual reality, and computational photography.However, the LF acquisition is inevitably time-consuming and resource-intensive due to the mainstream acquisition strategy involving manual capture or laborious software synthesis.Given such a challenge, we introduce LFdiff, a straightforward yet effective diffusion-based generative framework tailored for LF synthesis, which adopts only a single RGB image as input.LFdiff leverages disparity estimated by a monocular depth estimation network and incorporates two distinctive components: a novel condition scheme and a noise estimation network tailored for LF data.Specifically, we design a position-aware warping condition scheme, enhancing inter-view geometry learning via a robust conditional signal.We then propose DistgUnet, a disentanglement-based noise estimation network, to harness comprehensive LF representations.Extensive experiments demonstrate that LFdiff excels in synthesizing visually pleasing and disparity-controllable light fields with enhanced generalization capability.Additionally, comprehensive results affirm the broad applicability of the generated LF data, spanning applications like LF super-resolution and refocusing.

Title: Tropical Decision Boundaries for Neural Networks Are Robust Against Adversarial Attacks

• Authors: Kurt Pasque, Christopher Teska, Ruriko Yoshida, Keiji Miura, Jefferson Huang
• Subjects: cs.LG, cs.CR, cs.CV, math.CO
• Abstract URL: https://arxiv.org/abs/2402.00576
• Pdf URL: https://arxiv.org/pdf/2402.00576
• Copy Paste: [[2402.00576]] Tropical Decision Boundaries for Neural Networks Are Robust Against Adversarial Attacks(https://arxiv.org/abs/2402.00576)
• Keywords: attack, robust
• Abstract: We introduce a simple, easy to implement, and computationally efficient tropical convolutional neural network architecture that is robust against adversarial attacks. We exploit the tropical nature of piece-wise linear neural networks by embedding the data in the tropical projective torus in a single hidden layer which can be added to any model. We study the geometry of its decision boundary theoretically and show its robustness against adversarial attacks on image datasets using computational experiments.

Title: Dynamic Texture Transfer using PatchMatch and Transformers

• Authors: Guo Pu, Shiyao Xu, Xixin Cao, Zhouhui Lian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00606
• Pdf URL: https://arxiv.org/pdf/2402.00606
• Copy Paste: [[2402.00606]] Dynamic Texture Transfer using PatchMatch and Transformers(https://arxiv.org/abs/2402.00606)
• Keywords: transformer
• Abstract: How to automatically transfer the dynamic texture of a given video to the target still image is a challenging and ongoing problem. In this paper, we propose to handle this task via a simple yet effective model that utilizes both PatchMatch and Transformers. The key idea is to decompose the task of dynamic texture transfer into two stages, where the start frame of the target video with the desired dynamic texture is synthesized in the first stage via a distance map guided texture transfer module based on the PatchMatch algorithm. Then, in the second stage, the synthesized image is decomposed into structure-agnostic patches, according to which their corresponding subsequent patches can be predicted by exploiting the powerful capability of Transformers equipped with VQ-VAE for processing long discrete sequences. After getting all those patches, we apply a Gaussian weighted average merging strategy to smoothly assemble them into each frame of the target stylized video. Experimental results demonstrate the effectiveness and superiority of the proposed method in dynamic texture transfer compared to the state of the art.

Title: Are Synthetic Time-series Data Really not as Good as Real Data?

• Authors: Fanzhe Fu, Junru Chen, Jing Zhang, Carl Yang, Lvbin Ma, Yang Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00607
• Pdf URL: https://arxiv.org/pdf/2402.00607
• Copy Paste: [[2402.00607]] Are Synthetic Time-series Data Really not as Good as Real Data?(https://arxiv.org/abs/2402.00607)
• Keywords: extraction
• Abstract: Time-series data presents limitations stemming from data quality issues, bias and vulnerabilities, and generalization problem. Integrating universal data synthesis methods holds promise in improving generalization. However, current methods cannot guarantee that the generator's output covers all unseen real data. In this paper, we introduce InfoBoost -- a highly versatile cross-domain data synthesizing framework with time series representation learning capability. We have developed a method based on synthetic data that enables model training without the need for real data, surpassing the performance of models trained with real data. Additionally, we have trained a universal feature extractor based on our synthetic data that is applicable to all time-series data. Our approach overcomes interference from multiple sources rhythmic signal, noise interference, and long-period features that exceed sampling window capabilities. Through experiments, our non-deep-learning synthetic data enables models to achieve superior reconstruction performance and universal explicit representation extraction without the need for real data.

Title: Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks

• Authors: Maan Qraitem, Nazia Tasnim, Kate Saenko, Bryan A. Plummer
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00626
• Pdf URL: https://arxiv.org/pdf/2402.00626
• Copy Paste: [[2402.00626]] Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks(https://arxiv.org/abs/2402.00626)
• Keywords: attack
• Abstract: Recently, significant progress has been made on Large Vision-Language Models (LVLMs); a new class of VL models that make use of large pre-trained language models. Yet, their vulnerability to Typographic attacks, which involve superimposing misleading text onto an image remain unstudied. Furthermore, prior work typographic attacks rely on sampling a random misleading class from a predefined set of classes. However, the random chosen class might not be the most effective attack. To address these issues, we first introduce a novel benchmark uniquely designed to test LVLMs vulnerability to typographic attacks. Furthermore, we introduce a new and more effective typographic attack: Self-Generated typographic attacks. Indeed, our method, given an image, make use of the strong language capabilities of models like GPT-4V by simply prompting them to recommend a typographic attack. Using our novel benchmark, we uncover that typographic attacks represent a significant threat against LVLM(s). Furthermore, we uncover that typographic attacks recommended by GPT-4V using our new method are not only more effective against GPT-4V itself compared to prior work attacks, but also against a host of less capable yet popular open source models like LLaVA, InstructBLIP, and MiniGPT4.

Title: CapHuman: Capture Your Moments in Parallel Universes

• Authors: Chao Liang, Fan Ma, Linchao Zhu, Yingying Deng, Yi Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00627
• Pdf URL: https://arxiv.org/pdf/2402.00627
• Copy Paste: [[2402.00627]] CapHuman: Capture Your Moments in Parallel Universes(https://arxiv.org/abs/2402.00627)
• Keywords: diffusion, generative
• Abstract: We concentrate on a novel human-centric image synthesis task, that is, given only one reference facial photograph, it is expected to generate specific individual images with diverse head positions, poses, and facial expressions in different contexts. To accomplish this goal, we argue that our generative model should be capable of the following favorable characteristics: (1) a strong visual and semantic understanding of our world and human society for basic object and human image generation. (2) generalizable identity preservation ability. (3) flexible and fine-grained head control. Recently, large pre-trained text-to-image diffusion models have shown remarkable results, serving as a powerful generative foundation. As a basis, we aim to unleash the above two capabilities of the pre-trained model. In this work, we present a new framework named CapHuman. We embrace the ``encode then learn to align" paradigm, which enables generalizable identity preservation for new individuals without cumbersome tuning at inference. CapHuman encodes identity features and then learns to align them into the latent space. Moreover, we introduce the 3D facial prior to equip our model with control over the human head in a flexible and 3D-consistent manner. Extensive qualitative and quantitative analyses demonstrate our CapHuman can produce well-identity-preserved, photo-realistic, and high-fidelity portraits with content-rich representations and various head renditions, superior to established baselines. Code and checkpoint will be released at https://github.com/VamosC/CapHuman.

Title: SeFi-IDE: Semantic-Fidelity Identity Embedding for Personalized Diffusion-Based Generation

• Authors: Yang Li, Songlin Yang, Wei Wang, Jing Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00631
• Pdf URL: https://arxiv.org/pdf/2402.00631
• Copy Paste: [[2402.00631]] SeFi-IDE: Semantic-Fidelity Identity Embedding for Personalized Diffusion-Based Generation(https://arxiv.org/abs/2402.00631)
• Keywords: diffusion, generative
• Abstract: Advanced diffusion-based Text-to-Image (T2I) models, such as the Stable Diffusion Model, have made significant progress in generating diverse and high-quality images using text prompts alone. However, T2I models are unable to accurately map identities (IDs) when non-famous users require personalized image generation. The main problem is that existing T2I models do not learn the ID-image alignments of new users. The previous methods either failed to accurately fit the face region or lost the interactive generative ability with other existing concepts in T2I models (i.e., unable to generate other concepts described in given prompts such as scenes, actions, and facial attributes). In this paper, we focus on accurate and semantic-fidelity ID embedding into the Stable Diffusion Model for personalized generation. We address this challenge from two perspectives: face-wise region fitting, and semantic-fidelity token optimization. Specifically, we first visualize the attention overfit problem, and propose a face-wise attention loss to fit the face region instead of the whole target image. This key trick significantly enhances the ID accuracy and interactive generative ability with other existing concepts. Then, we optimize one ID representation as multiple per-stage tokens where each token contains two disentangled features. This expansion of the textual conditioning space enhances semantic-fidelity control. Extensive experiments validate that our results exhibit superior ID accuracy and manipulation ability compared to previous methods.

Title: Fisheye Camera and Ultrasonic Sensor Fusion For Near-Field Obstacle Perception in Bird's-Eye-View

• Authors: Arindam Das, Sudarshan Paul, Niko Scholz, Akhilesh Kumar Malviya, Ganesh Sistu, Ujjwal Bhattacharya, Ciarán Eising
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00637
• Pdf URL: https://arxiv.org/pdf/2402.00637
• Copy Paste: [[2402.00637]] Fisheye Camera and Ultrasonic Sensor Fusion For Near-Field Obstacle Perception in Bird's-Eye-View(https://arxiv.org/abs/2402.00637)
• Keywords: robust
• Abstract: Accurate obstacle identification represents a fundamental challenge within the scope of near-field perception for autonomous driving. Conventionally, fisheye cameras are frequently employed for comprehensive surround-view perception, including rear-view obstacle localization. However, the performance of such cameras can significantly deteriorate in low-light conditions, during nighttime, or when subjected to intense sun glare. Conversely, cost-effective sensors like ultrasonic sensors remain largely unaffected under these conditions. Therefore, we present, to our knowledge, the first end-to-end multimodal fusion model tailored for efficient obstacle perception in a bird's-eye-view (BEV) perspective, utilizing fisheye cameras and ultrasonic sensors. Initially, ResNeXt-50 is employed as a set of unimodal encoders to extract features specific to each modality. Subsequently, the feature space associated with the visible spectrum undergoes transformation into BEV. The fusion of these two modalities is facilitated via concatenation. At the same time, the ultrasonic spectrum-based unimodal feature maps pass through content-aware dilated convolution, applied to mitigate the sensor misalignment between two sensors in the fused feature space. Finally, the fused features are utilized by a two-stage semantic occupancy decoder to generate grid-wise predictions for precise obstacle perception. We conduct a systematic investigation to determine the optimal strategy for multimodal fusion of both sensors. We provide insights into our dataset creation procedures, annotation guidelines, and perform a thorough data analysis to ensure adequate coverage of all scenarios. When applied to our dataset, the experimental results underscore the robustness and effectiveness of our proposed multimodal fusion approach.

Title: Testing side-channel security of cryptographic implementations against future microarchitectures

• Authors: Gilles Barthe, Marcel Böhme, Sunjay Cauligi, Chitchanok Chuengsatiansup, Daniel Genkin, Marco Guarnieri, David Mateos Romero, Peter Schwabe, David Wu, Yuval Yarom
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.00641
• Pdf URL: https://arxiv.org/pdf/2402.00641
• Copy Paste: [[2402.00641]] Testing side-channel security of cryptographic implementations against future microarchitectures(https://arxiv.org/abs/2402.00641)
• Keywords: security, attack
• Abstract: How will future microarchitectures impact the security of existing cryptographic implementations? As we cannot keep reducing the size of transistors, chip vendors have started developing new microarchitectural optimizations to speed up computation. A recent study (Sanchez Vicarte et al., ISCA 2021) suggests that these optimizations might open the Pandora's box of microarchitectural attacks. However, there is little guidance on how to evaluate the security impact of future optimization proposals. To help chip vendors explore the impact of microarchitectural optimizations on cryptographic implementations, we develop (i) an expressive domain-specific language, called LmSpec, that allows them to specify the leakage model for the given optimization and (ii) a testing framework, called LmTest, to automatically detect leaks under the specified leakage model within the given implementation. Using this framework, we conduct an empirical study of 18 proposed microarchitectural optimizations on 25 implementations of eight cryptographic primitives in five popular libraries. We find that every implementation would contain secret-dependent leaks, sometimes sufficient to recover a victim's secret key, if these optimizations were realized. Ironically, some leaks are possible only because of coding idioms used to prevent leaks under the standard constant-time model.

Title: Improving the accuracy of freight mode choice models: A case study using the 2017 CFS PUF data set and ensemble learning techniques

• Authors: Diyi Liu, Hyeonsup Lim, Majbah Uddin, Yuandong Liu, Lee D. Han, Ho-ling Hwang, Shih-Miao Chin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00654
• Pdf URL: https://arxiv.org/pdf/2402.00654
• Copy Paste: [[2402.00654]] Improving the accuracy of freight mode choice models: A case study using the 2017 CFS PUF data set and ensemble learning techniques(https://arxiv.org/abs/2402.00654)
• Keywords: interpretability
• Abstract: The US Census Bureau has collected two rounds of experimental data from the Commodity Flow Survey, providing shipment-level characteristics of nationwide commodity movements, published in 2012 (i.e., Public Use Microdata) and in 2017 (i.e., Public Use File). With this information, data-driven methods have become increasingly valuable for understanding detailed patterns in freight logistics. In this study, we used the 2017 Commodity Flow Survey Public Use File data set to explore building a high-performance freight mode choice model, considering three main improvements: (1) constructing local models for each separate commodity/industry category; (2) extracting useful geographical features, particularly the derived distance of each freight mode between origin/destination zones; and (3) applying additional ensemble learning methods such as stacking or voting to combine results from local and unified models for improved performance. The proposed method achieved over 92% accuracy without incorporating external information, an over 19% increase compared to directly fitting Random Forests models over 10,000 samples. Furthermore, SHAP (Shapely Additive Explanations) values were computed to explain the outputs and major patterns obtained from the proposed model. The model framework could enhance the performance and interpretability of existing freight mode choice models.

Title: An Investigation of Hardware Security Bug Characteristics in Open-Source Projects

• Authors: Joey Ah-kiow, Benjamin Tan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.00684
• Pdf URL: https://arxiv.org/pdf/2402.00684
• Copy Paste: [[2402.00684]] An Investigation of Hardware Security Bug Characteristics in Open-Source Projects(https://arxiv.org/abs/2402.00684)
• Keywords: security
• Abstract: Hardware security is an important concern of system security as vulnerabilities can arise from design errors introduced throughout the development lifecycle. Recent works have proposed techniques to detect hardware security bugs, such as static analysis, fuzzing, and symbolic execution. However, the fundamental properties of hardware security bugs remain relatively unexplored. To gain a better understanding of hardware security bugs, we perform a deep dive into the popular OpenTitan project, including its bug reports and bug fixes. We manually classify the bugs as relevant to functionality or security and analyze characteristics, such as the impact and location of security bugs, and the size of their bug fixes. We also investigate relationships between security impact and bug management during development. Finally, we propose an abstract syntax tree-based analysis to identify the syntactic characteristics of bug fixes. Our results show that 53% of the bugs in OpenTitan have potential security implications and that 55% of all bug fixes modify only one file. Our findings underscore the importance of security-aware development practices and tools and motivate the development of techniques that leverage the highly localized nature of hardware bugs.

Title: A Review on the Use of Blockchain for the Internet of Things

• Authors: Tiago M. Fernandez-Carames, Paula Fraga-Lamas
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.00687
• Pdf URL: https://arxiv.org/pdf/2402.00687
• Copy Paste: [[2402.00687]] A Review on the Use of Blockchain for the Internet of Things(https://arxiv.org/abs/2402.00687)
• Keywords: security, privacy, attack, robust
• Abstract: The paradigm of Internet of Things (IoT) is paving the way for a world, where many of our daily objects will be interconnected and will interact with their environment in order to collect information and automate certain tasks. Such a vision requires, among other things, seamless authentication, data privacy, security, robustness against attacks, easy deployment, and self-maintenance. Such features can be brought by blockchain, a technology born with a cryptocurrency called Bitcoin. In this paper, a thorough review on how to adapt blockchain to the specific needs of IoT in order to develop Blockchain-based IoT (BIoT) applications is presented. After describing the basics of blockchain, the most relevant BIoT applications are described with the objective of emphasizing how blockchain can impact traditional cloud-centered IoT applications. Then, the current challenges and possible optimizations are detailed regarding many aspects that affect the design, development, and deployment of a BIoT application. Finally, some recommendations are enumerated with the aim of guiding future BIoT researchers and developers on some of the issues that will have to be tackled before deploying the next generation of BIoT applications.

Title: Ocassionally Secure: A Comparative Analysis of Code Generation Assistants

• Authors: Ran Elgedawy, John Sadik, Senjuti Dutta, Anuj Gautam, Konstantinos Georgiou, Farzin Gholamrezae, Fujiao Ji, Kyungchan Lim, Qian Liu, Scott Ruoti
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00689
• Pdf URL: https://arxiv.org/pdf/2402.00689
• Copy Paste: [[2402.00689]] Ocassionally Secure: A Comparative Analysis of Code Generation Assistants(https://arxiv.org/abs/2402.00689)
• Keywords: secure, security, large language model
• Abstract: $ $Large Language Models (LLMs) are being increasingly utilized in various applications, with code generations being a notable example. While previous research has shown that LLMs have the capability to generate both secure and insecure code, the literature does not take into account what factors help generate secure and effective code. Therefore in this paper we focus on identifying and understanding the conditions and contexts in which LLMs can be effectively and safely deployed in real-world scenarios to generate quality code. We conducted a comparative analysis of four advanced LLMs--GPT-3.5 and GPT-4 using ChatGPT and Bard and Gemini from Google--using 9 separate tasks to assess each model's code generation capabilities. We contextualized our study to represent the typical use cases of a real-life developer employing LLMs for everyday tasks as work. Additionally, we place an emphasis on security awareness which is represented through the use of two distinct versions of our developer persona. In total, we collected 61 code outputs and analyzed them across several aspects: functionality, security, performance, complexity, and reliability. These insights are crucial for understanding the models' capabilities and limitations, guiding future development and practical applications in the field of automated code generation.

Title: A Framework for Building Point Cloud Cleaning, Plane Detection and Semantic Segmentation

• Authors: Ilyass Abouelaziz, Youssef Mourchid
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00692
• Pdf URL: https://arxiv.org/pdf/2402.00692
• Copy Paste: [[2402.00692]] A Framework for Building Point Cloud Cleaning, Plane Detection and Semantic Segmentation(https://arxiv.org/abs/2402.00692)
• Keywords: robust, segmentation
• Abstract: This paper presents a framework to address the challenges involved in building point cloud cleaning, plane detection, and semantic segmentation, with the ultimate goal of enhancing building modeling. We focus in the cleaning stage on removing outliers from the acquired point cloud data by employing an adaptive threshold technique based on z-score measure. Following the cleaning process, we perform plane detection using the robust RANSAC paradigm. The goal is to carry out multiple plane segmentations, and to classify segments into distinct categories, such as floors, ceilings, and walls. The resulting segments can generate accurate and detailed point clouds representing the building's architectural elements. Moreover, we address the problem of semantic segmentation, which plays a vital role in the identification and classification of different components within the building, such as walls, windows, doors, roofs, and objects. Inspired by the PointNet architecture, we propose a deep learning architecture for efficient semantic segmentation in buildings. The results demonstrate the effectiveness of the proposed framework in handling building modeling tasks, paving the way for improved accuracy and efficiency in the field of building modelization.

Title: Approximating Optimal Morphing Attacks using Template Inversion

• Authors: Laurent Colbois, Hatef Otroshi Shahreza, Sébastien Marcel
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00695
• Pdf URL: https://arxiv.org/pdf/2402.00695
• Copy Paste: [[2402.00695]] Approximating Optimal Morphing Attacks using Template Inversion(https://arxiv.org/abs/2402.00695)
• Keywords: attack
• Abstract: Recent works have demonstrated the feasibility of inverting face recognition systems, enabling to recover convincing face images using only their embeddings. We leverage such template inversion models to develop a novel type ofdeep morphing attack based on inverting a theoretical optimal morph embedding, which is obtained as an average of the face embeddings of source images. We experiment with two variants of this approach: the first one exploits a fully self-contained embedding-to-image inversion model, while the second leverages the synthesis network of a pretrained StyleGAN network for increased morph realism. We generate morphing attacks from several source datasets and study the effectiveness of those attacks against several face recognition networks. We showcase that our method can compete with and regularly beat the previous state of the art for deep-learning based morph generation in terms of effectiveness, both in white-box and black-box attack scenarios, and is additionally much faster to run. We hope this might facilitate the development of large scale deep morph datasets for training detection models.

Title: In-Bed Pose Estimation: A Review

• Authors: Ziya Ata Yazıcı, Sara Colantonio, Hazım Kemal Ekenel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00700
• Pdf URL: https://arxiv.org/pdf/2402.00700
• Copy Paste: [[2402.00700]] In-Bed Pose Estimation: A Review(https://arxiv.org/abs/2402.00700)
• Keywords: privacy
• Abstract: Human pose estimation, the process of identifying joint positions in a person's body from images or videos, represents a widely utilized technology across diverse fields, including healthcare. One such healthcare application involves in-bed pose estimation, where the body pose of an individual lying under a blanket is analyzed. This task, for instance, can be used to monitor a person's sleep behavior and detect symptoms early for potential disease diagnosis in homes and hospitals. Several studies have utilized unimodal and multimodal methods to estimate in-bed human poses. The unimodal studies generally employ RGB images, whereas the multimodal studies use modalities including RGB, long-wavelength infrared, pressure map, and depth map. Multimodal studies have the advantage of using modalities in addition to RGB that might capture information useful to cope with occlusions. Moreover, some multimodal studies exclude RGB and, this way, better suit privacy preservation. To expedite advancements in this domain, we conduct a review of existing datasets and approaches. Our objectives are to show the limitations of the previous studies, current challenges, and provide insights for future works on the in-bed human pose estimation field.

Title: Towards an autonomous industry 4.0 warehouse: A UAV and blockchain-based system for inventory and traceability applications in big data-driven supply chain management

• Authors: Tiago M. Fernandez-Carames, Oscar Blanco-Novoa, Ivan Froiz-Miguez, Paula Fraga-Lamas
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.00709
• Pdf URL: https://arxiv.org/pdf/2402.00709
• Copy Paste: [[2402.00709]] Towards an autonomous industry 4.0 warehouse: A UAV and blockchain-based system for inventory and traceability applications in big data-driven supply chain management(https://arxiv.org/abs/2402.00709)
• Keywords: security
• Abstract: In this paper we present the design and evaluation of a UAV-based system aimed at automating inventory tasks and keeping the traceability of industrial items attached to Radio-Frequency IDentification (RFID) tags. To confront current shortcomings, such a system is developed under a versatile, modular and scalable architecture aimed to reinforce cyber security and decentralization while fostering external audits and big data analytics. Therefore, the system uses a blockchain and a distributed ledger to store certain inventory data collected by UAVs, validate them, ensure their trustworthiness and make them available to the interested parties. In order to show the performance of the proposed system, different tests were performed in a real industrial warehouse, concluding that the system is able to obtain the inventory data really fast in comparison to traditional manual tasks, while being also able to estimate the position of the items when hovering over them thanks to their tag's signal strength. In addition, the performance of the proposed blockchain-based architecture was evaluated in different scenarios.

Title: ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction

• Authors: Juan Nathaniel, Yongquan Qu, Tung Nguyen, Sungduk Yu, Julius Busecke, Aditya Grover, Pierre Gentine
• Subjects: cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.00712
• Pdf URL: https://arxiv.org/pdf/2402.00712
• Copy Paste: [[2402.00712]] ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction(https://arxiv.org/abs/2402.00712)
• Keywords: explainability
• Abstract: Accurate prediction of climate in the subseasonal-to-seasonal scale is crucial for disaster readiness, reduced economic risk, and improved policy-making amidst climate change. Yet, S2S prediction remains challenging due to the chaotic nature of the system. At present, existing benchmarks for weather and climate applications, tend to (1) have shorter forecasting range of up-to 14 days, (2) do not include a wide range of operational baseline forecasts, and (3) lack physics-based constraints for explainability. Thus, we propose ChaosBench, a large-scale, multi-channel, physics-based benchmark for S2S prediction. ChaosBench has over 460K frames of real-world observations and simulations, each with 60 variable-channels and spanning for up-to 45 years. We also propose several physics-based, in addition to vision-based metrics, that enables for a more physically-consistent model. Furthermore, we include a diverse set of physics-based forecasts from 4 national weather agencies as baselines to our data-driven counterpart. We establish two tasks that vary in complexity: full and sparse dynamics prediction. Our benchmark is one of the first to perform large-scale evaluation on existing models including PanguWeather, FourCastNetV2, GraphCast, and ClimaX, and finds methods originally developed for weather-scale applications fails on S2S task. We release our benchmark code and datasets at https://leap-stc.github.io/ChaosBench.

Title: Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders

• Authors: Yingji Zhang, Danilo S. Carvalho, Marco Valentino, Ian Pratt-Hartmann, Andre Freitas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00723
• Pdf URL: https://arxiv.org/pdf/2402.00723
• Copy Paste: [[2402.00723]] Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders(https://arxiv.org/abs/2402.00723)
• Keywords: transformer, generative
• Abstract: Achieving precise semantic control over the latent spaces of Variational AutoEncoders (VAEs) holds significant value for downstream tasks in NLP as the underlying generative mechanisms could be better localised, explained and improved upon. Recent research, however, has struggled to achieve consistent results, primarily due to the inevitable loss of semantic information in the variational bottleneck and limited control over the decoding mechanism. To overcome these challenges, we investigate discrete latent spaces in Vector Quantized Variational AutoEncoders (VQVAEs) to improve semantic control and generation in Transformer-based VAEs. In particular, We propose T5VQVAE, a novel model that leverages the controllability of VQVAEs to guide the self-attention mechanism in T5 at the token-level, exploiting its full generalization capabilities. Experimental results indicate that T5VQVAE outperforms existing state-of-the-art VAE models, including Optimus, in terms of controllability and preservation of semantic information across different tasks such as auto-encoding of sentences and mathematical expressions, text transfer, and inference. Moreover, T5VQVAE exhibits improved inference capabilities, suggesting potential applications for downstream natural language and symbolic reasoning tasks.

Title: Automatic Segmentation of the Spinal Cord Nerve Rootlets

• Authors: Jan Valosek, Theo Mathieu, Raphaelle Schlienger, Olivia S. Kowalczyk, Julien Cohen-Adad
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00724
• Pdf URL: https://arxiv.org/pdf/2402.00724
• Copy Paste: [[2402.00724]] Automatic Segmentation of the Spinal Cord Nerve Rootlets(https://arxiv.org/abs/2402.00724)
• Keywords: segmentation
• Abstract: Precise identification of spinal nerve rootlets is relevant to delineate spinal levels for the study of functional activity in the spinal cord. The goal of this study was to develop an automatic method for the semantic segmentation of spinal nerve rootlets from T2-weighted magnetic resonance imaging (MRI) scans. Images from two open-access MRI datasets were used to train a 3D multi-class convolutional neural network using an active learning approach to segment C2-C8 dorsal nerve rootlets. Each output class corresponds to a spinal level. The method was tested on 3T T2-weighted images from datasets unseen during training to assess inter-site, inter-session, and inter-resolution variability. The test Dice score was 0.67 +- 0.16 (mean +- standard deviation across rootlets levels), suggesting a good performance. The method also demonstrated low inter-vendor and inter-site variability (coefficient of variation <= 1.41 %), as well as low inter-session variability (coefficient of variation <= 1.30 %) indicating stable predictions across different MRI vendors, sites, and sessions. The proposed methodology is open-source and readily available in the Spinal Cord Toolbox (SCT) v6.2 and higher.

Title: Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation

• Authors: Hsiang Hsu, Guihong Li, Shaohan Hu, Chun-Fu (Richard)Chen
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.00728
• Pdf URL: https://arxiv.org/pdf/2402.00728
• Copy Paste: [[2402.00728]] Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation(https://arxiv.org/abs/2402.00728)
• Keywords: fair
• Abstract: Predictive multiplicity refers to the phenomenon in which classification tasks may admit multiple competing models that achieve almost-equally-optimal performance, yet generate conflicting outputs for individual samples. This presents significant concerns, as it can potentially result in systemic exclusion, inexplicable discrimination, and unfairness in practical applications. Measuring and mitigating predictive multiplicity, however, is computationally challenging due to the need to explore all such almost-equally-optimal models, known as the Rashomon set, in potentially huge hypothesis spaces. To address this challenge, we propose a novel framework that utilizes dropout techniques for exploring models in the Rashomon set. We provide rigorous theoretical derivations to connect the dropout parameters to properties of the Rashomon set, and empirically evaluate our framework through extensive experimentation. Numerical results show that our technique consistently outperforms baselines in terms of the effectiveness of predictive multiplicity metric estimation, with runtime speedup up to $20\times \sim 5000\times$. With efficient Rashomon set exploration and metric estimation, mitigation of predictive multiplicity is then achieved through dropout ensemble and model selection.

Title: Transforming and Combining Rewards for Aligning Large Language Models

• Authors: Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D'Amour, Sanmi Koyejo, Victor Veitch
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00742
• Pdf URL: https://arxiv.org/pdf/2402.00742
• Copy Paste: [[2402.00742]] Transforming and Combining Rewards for Aligning Large Language Models(https://arxiv.org/abs/2402.00742)
• Keywords: large language model
• Abstract: A common approach for aligning language models to human preferences is to first learn a reward model from preference data, and then use this reward model to update the language model. We study two closely related problems that arise in this approach. First, any monotone transformation of the reward model preserves preference ranking; is there a choice that is ``better'' than others? Second, we often wish to align language models to multiple properties: how should we combine multiple reward models? Using a probabilistic interpretation of the alignment procedure, we identify a natural choice for transformation for (the common case of) rewards learned from Bradley-Terry preference models. This derived transformation has two important properties. First, it emphasizes improving poorly-performing outputs, rather than outputs that already score well. This mitigates both underfitting (where some prompts are not improved) and reward hacking (where the model learns to exploit misspecification of the reward model). Second, it enables principled aggregation of rewards by linking summation to logical conjunction: the sum of transformed rewards corresponds to the probability that the output is ``good'' in all measured properties, in a sense we make precise. Experiments aligning language models to be both helpful and harmless using RLHF show substantial improvements over the baseline (non-transformed) approach.

Title: Benefits of Transformer: In-Context Learning in Linear Regression Tasks with Unstructured Data

• Authors: Yue Xing, Xiaofeng Lin, Namjoon Suh, Qifan Song, Guang Cheng
• Subjects: cs.LG, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.00743
• Pdf URL: https://arxiv.org/pdf/2402.00743
• Copy Paste: [[2402.00743]] Benefits of Transformer: In-Context Learning in Linear Regression Tasks with Unstructured Data(https://arxiv.org/abs/2402.00743)
• Keywords: transformer
• Abstract: In practice, it is observed that transformer-based models can learn concepts in context in the inference stage. While existing literature, e.g., \citet{zhang2023trained,huang2023context}, provide theoretical explanations on this in-context learning ability, they assume the input $x_i$ and the output $y_i$ for each sample are embedded in the same token (i.e., structured data). However, in reality, they are presented in two tokens (i.e., unstructured data \cite{wibisono2023role}). In this case, this paper conducts experiments in linear regression tasks to study the benefits of the architecture of transformers and provides some corresponding theoretical intuitions to explain why the transformer can learn from unstructured data. We study the exact components in a transformer that facilitate the in-context learning. In particular, we observe that (1) a transformer with two layers of softmax (self-)attentions with look-ahead attention mask can learn from the prompt if $y_i$ is in the token next to $x_i$ for each example; (2) positional encoding can further improve the performance; and (3) multi-head attention with a high input embedding dimension has a better prediction performance than single-head attention.

Title: Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement

• Authors: Xin Quan, Marco Valentino, Louise A. Dennis, André Freitas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00745
• Pdf URL: https://arxiv.org/pdf/2402.00745
• Copy Paste: [[2402.00745]] Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement(https://arxiv.org/abs/2402.00745)
• Keywords: interpretability, large language model
• Abstract: An increasing amount of research in Natural Language Inference (NLI) focuses on the application and evaluation of Large Language Models (LLMs) and their reasoning capabilities. Despite their success, however, LLMs are still prone to factual errors and inconsistencies in their explanations, offering limited control and interpretability for inference in complex domains. In this paper, we focus on ethical NLI, investigating how hybrid neuro-symbolic techniques can enhance the logical validity and alignment of ethical explanations produced by LLMs. Specifically, we present an abductive-deductive framework named Logic-Explainer, which integrates LLMs with an external backward-chaining solver to refine step-wise natural language explanations and jointly verify their correctness, reduce incompleteness and minimise redundancy. An extensive empirical analysis demonstrates that Logic-Explainer can improve explanations generated via in-context learning methods and Chain-of-Thought (CoT) on challenging ethical NLI tasks, while, at the same time, producing formal proofs describing and supporting models' reasoning. As ethical NLI requires commonsense reasoning to identify underlying moral violations, our results suggest the effectiveness of neuro-symbolic methods for multi-step NLI more broadly, opening new opportunities to enhance the logical consistency, reliability, and alignment of LLMs.

Title: Health-LLM: Personalized Retrieval-Augmented Disease Prediction Model

• Authors: Mingyu Jin, Qinkai Yu, Chong Zhang, Dong Shu, Suiyuan Zhu, Mengnan Du, Yongfeng Zhang, Yanda Meng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00746
• Pdf URL: https://arxiv.org/pdf/2402.00746
• Copy Paste: [[2402.00746]] Health-LLM: Personalized Retrieval-Augmented Disease Prediction Model(https://arxiv.org/abs/2402.00746)
• Keywords: extraction
• Abstract: Artificial intelligence (AI) in healthcare has significantly advanced intelligent medical treatment. However, traditional intelligent healthcare is limited by static data and unified standards, preventing full integration with individual situations and other challenges. Hence, a more professional and detailed intelligent healthcare method is needed for development. To this end, we propose an innovative framework named Heath-LLM, which combines large-scale feature extraction and medical knowledge trade-off scoring. Compared to traditional health management methods, our approach has three main advantages. First, our method integrates health reports into a large model to provide detailed task information. Second, professional medical expertise is used to adjust the weighted scores of health characteristics. Third, we use a semi-automated feature extraction framework to enhance the analytical power of language models and incorporate expert insights to improve the accuracy of disease prediction. We have conducted disease prediction experiments on a large number of health reports to assess the effectiveness of Health-LLM. The results of the experiments indicate that the proposed method surpasses traditional methods and has the potential to revolutionize disease prediction and personalized health management. The code is available at https://github.com/jmyissb/HealthLLM.

Title: Unlearnable Algorithms for In-context Learning

• Authors: Andrei Muresanu, Anvith Thudi, Michael R. Zhang, Nicolas Papernot
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.00751
• Pdf URL: https://arxiv.org/pdf/2402.00751
• Copy Paste: [[2402.00751]] Unlearnable Algorithms for In-context Learning(https://arxiv.org/abs/2402.00751)
• Keywords: large language model
• Abstract: Machine unlearning is a desirable operation as models get increasingly deployed on data with unknown provenance. However, achieving exact unlearning -- obtaining a model that matches the model distribution when the data to be forgotten was never used -- is challenging or inefficient, often requiring significant retraining. In this paper, we focus on efficient unlearning methods for the task adaptation phase of a pretrained large language model (LLM). We observe that an LLM's ability to do in-context learning for task adaptation allows for efficient exact unlearning of task adaptation training data. We provide an algorithm for selecting few-shot training examples to prepend to the prompt given to an LLM (for task adaptation), ERASE, whose unlearning operation cost is independent of model and dataset size, meaning it scales to large models and datasets. We additionally compare our approach to fine-tuning approaches and discuss the trade-offs between the two approaches. This leads us to propose a new holistic measure of unlearning cost which accounts for varying inference costs, and conclude that in-context learning can often be more favourable than fine-tuning for deployments involving unlearning requests.

Title: GS++: Error Analyzing and Optimal Gaussian Splatting

• Authors: Letian Huang, Jiayang Bai, Jie Guo, Yanwen Guo
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2402.00752
• Pdf URL: https://arxiv.org/pdf/2402.00752
• Copy Paste: [[2402.00752]] GS++: Error Analyzing and Optimal Gaussian Splatting(https://arxiv.org/abs/2402.00752)
• Keywords: robust
• Abstract: 3D Gaussian Splatting has garnered extensive attention and application in real-time neural rendering. Concurrently, concerns have been raised about the limitations of this technology in aspects such as point cloud storage, performance , and robustness in sparse viewpoints , leading to various improvements. However, there has been a notable lack of attention to the projection errors introduced by the local affine approximation inherent in the splatting itself, and the consequential impact of these errors on the quality of photo-realistic rendering. This paper addresses the projection error function of 3D Gaussian Splatting, commencing with the residual error from the first-order Taylor expansion of the projection function $\phi$. The analysis establishes a correlation between the error and the Gaussian mean position. Subsequently, leveraging function optimization theory, this paper analyzes the function's minima to provide an optimal projection strategy for Gaussian Splatting referred to Optimal Gaussian Splatting. Experimental validation further confirms that this projection methodology reduces artifacts, resulting in a more convincingly realistic rendering.

Title: Building Expressive and Tractable Probabilistic Generative Models: A Review

• Authors: Sahil Sidheekh, Sriraam Natarajan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00759
• Pdf URL: https://arxiv.org/pdf/2402.00759
• Copy Paste: [[2402.00759]] Building Expressive and Tractable Probabilistic Generative Models: A Review(https://arxiv.org/abs/2402.00759)
• Keywords: generative
• Abstract: We present a comprehensive survey of the advancements and techniques in the field of tractable probabilistic generative modeling, primarily focusing on Probabilistic Circuits (PCs). We provide a unified perspective on the inherent trade-offs between expressivity and the tractability, highlighting the design principles and algorithmic extensions that have enabled building expressive and efficient PCs, and provide a taxonomy of the field. We also discuss recent efforts to build deep and hybrid PCs by fusing notions from deep neural models, and outline the challenges and open questions that can guide future research in this evolving field.

Title: AnimateLCM: Accelerating the Animation of Personalized Diffusion Models and Adapters with Decoupled Consistency Learning

• Authors: Fu-Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Guanglu Song, Yu Liu, Hongsheng Li
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00769
• Pdf URL: https://arxiv.org/pdf/2402.00769
• Copy Paste: [[2402.00769]] AnimateLCM: Accelerating the Animation of Personalized Diffusion Models and Adapters with Decoupled Consistency Learning(https://arxiv.org/abs/2402.00769)
• Keywords: diffusion
• Abstract: Video diffusion models has been gaining increasing attention for its ability to produce videos that are both coherent and of high fidelity. However, the iterative denoising process makes it computationally intensive and time-consuming, thus limiting its applications. Inspired by the Consistency Model (CM) that distills pretrained image diffusion models to accelerate the sampling with minimal steps and its successful extension Latent Consistency Model (LCM) on conditional image generation, we propose AnimateLCM, allowing for high-fidelity video generation within minimal steps. Instead of directly conducting consistency learning on the raw video dataset, we propose a decoupled consistency learning strategy that decouples the distillation of image generation priors and motion generation priors, which improves the training efficiency and enhance the generation visual quality. Additionally, to enable the combination of plug-and-play adapters in stable diffusion community to achieve various functions (e.g., ControlNet for controllable generation). we propose an efficient strategy to adapt existing adapters to our distilled text-conditioned video consistency model or train adapters from scratch without harming the sampling speed. We validate the proposed strategy in image-conditioned video generation and layout-conditioned video generation, all achieving top-performing results. Experimental results validate the effectiveness of our proposed method. Code and weights will be made public. More details are available at https://github.com/G-U-N/AnimateLCM.

Title: Dense Reward for Free in Reinforcement Learning from Human Feedback

• Authors: Alex J. Chan, Hao Sun, Samuel Holt, Mihaela van der Schaar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00782
• Pdf URL: https://arxiv.org/pdf/2402.00782
• Copy Paste: [[2402.00782]] Dense Reward for Free in Reinforcement Learning from Human Feedback(https://arxiv.org/abs/2402.00782)
• Keywords: transformer, large language model
• Abstract: Reinforcement Learning from Human Feedback (RLHF) has been credited as the key advance that has allowed Large Language Models (LLMs) to effectively follow instructions and produce useful assistance. Classically, this involves generating completions from the LLM in response to a query before using a separate reward model to assign a score to the full completion. As an auto-regressive process, the LLM has to take many "actions" (selecting individual tokens) and only receives a single, sparse reward at the end of an episode, a setup that is known to be difficult to optimise in traditional reinforcement learning. In this work we leverage the fact that the reward model contains more information than just its scalar output, in particular, it calculates an attention map over tokens as part of the transformer architecture. We use these attention weights to redistribute the reward along the whole completion, effectively densifying the signal and highlighting the most important tokens, all without incurring extra computational cost or requiring any additional modelling. We demonstrate that, theoretically, this approach is equivalent to potential-based reward shaping, ensuring that the optimal policy remains unchanged. Empirically, we show that it stabilises training, accelerates the rate of learning, and, in practical cases, may lead to better local optima.

Title: CroissantLLM: A Truly Bilingual French-English Language Model

• Authors: Manuel Faysse, Patrick Fernandes, Nuno Guerreiro, António Loison, Duarte Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro Martins, Antoni Bigata Casademunt, François Yvon, André Martins, Gautier Viaud, Céline Hudelot, Pierre Colombo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00786
• Pdf URL: https://arxiv.org/pdf/2402.00786
• Copy Paste: [[2402.00786]] CroissantLLM: A Truly Bilingual French-English Language Model(https://arxiv.org/abs/2402.00786)
• Keywords: large language model
• Abstract: We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local hardware. To that end, we pioneer the approach of training an intrinsically bilingual model with a 1:1 English-to-French pretraining data ratio, a custom tokenizer, and bilingual finetuning datasets. We release the training dataset, notably containing a French split with manually curated, high-quality, and varied data sources. To assess performance outside of English, we craft a novel benchmark, FrenchBench, consisting of an array of classification and generation tasks, covering various orthogonal aspects of model performance in the French Language. Additionally, rooted in transparency and to foster further Large Language Model research, we release codebases, and dozens of checkpoints across various model sizes, training data distributions, and training steps, as well as fine-tuned Chat models, and strong translation models. We evaluate our model through the FMTI framework, and validate 81 % of the transparency criteria, far beyond the scores of even most open initiatives. This work enriches the NLP landscape, breaking away from previous English-centric work in order to strengthen our understanding of multilinguality in language models.

Title: Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spaces

• Authors: Chloe Wang, Oleksii Tsepa, Jun Ma, Bo Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00789
• Pdf URL: https://arxiv.org/pdf/2402.00789
• Copy Paste: [[2402.00789]] Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spaces(https://arxiv.org/abs/2402.00789)
• Keywords: transformer
• Abstract: Attention mechanisms have been widely used to capture long-range dependencies among nodes in Graph Transformers. Bottlenecked by the quadratic computational cost, attention mechanisms fail to scale in large graphs. Recent improvements in computational efficiency are mainly achieved by attention sparsification with random or heuristic-based graph subsampling, which falls short in data-dependent context reasoning. State space models (SSMs), such as Mamba, have gained prominence for their effectiveness and efficiency in modeling long-range dependencies in sequential data. However, adapting SSMs to non-sequential graph data presents a notable challenge. In this work, we introduce Graph-Mamba, the first attempt to enhance long-range context modeling in graph networks by integrating a Mamba block with the input-dependent node selection mechanism. Specifically, we formulate graph-centric node prioritization and permutation strategies to enhance context-aware reasoning, leading to a substantial improvement in predictive performance. Extensive experiments on ten benchmark datasets demonstrate that Graph-Mamba outperforms state-of-the-art methods in long-range graph prediction tasks, with a fraction of the computational cost in both FLOPs and GPU memory consumption. The code and models are publicly available at https://github.com/bowang-lab/Graph-Mamba.

Title: From Pre-Quantum to Post-Quantum IoT Security: A Survey on Quantum-Resistant Cryptosystems for the Internet of Things

• Authors: Tiago M. Fernandez-Carames
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.00790
• Pdf URL: https://arxiv.org/pdf/2402.00790
• Copy Paste: [[2402.00790]] From Pre-Quantum to Post-Quantum IoT Security: A Survey on Quantum-Resistant Cryptosystems for the Internet of Things(https://arxiv.org/abs/2402.00790)
• Keywords: security, protect, attack
• Abstract: This article provides a survey on what can be called post-quantum IoT systems (IoT systems protected from the currently known quantum computing attacks): the main post-quantum cryptosystems and initiatives are reviewed, the most relevant IoT architectures and challenges are analyzed, and the expected future trends are indicated. Thus, this paper is aimed at providing a wide view of post-quantum IoT security and give useful guidelines to the future post-quantum IoT developers.

Title: ReAGent: Towards A Model-agnostic Feature Attribution Method for Generative Language Models

• Authors: Zhixue Zhao, Boxuan Shan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00794
• Pdf URL: https://arxiv.org/pdf/2402.00794
• Copy Paste: [[2402.00794]] ReAGent: Towards A Model-agnostic Feature Attribution Method for Generative Language Models(https://arxiv.org/abs/2402.00794)
• Keywords: generative
• Abstract: Feature attribution methods (FAs), such as gradients and attention, are widely employed approaches to derive the importance of all input features to the model predictions. Existing work in natural language processing has mostly focused on developing and testing FAs for encoder-only language models (LMs) in classification tasks. However, it is unknown if it is faithful to use these FAs for decoder-only models on text generation, due to the inherent differences between model architectures and task settings respectively. Moreover, previous work has demonstrated that there is no `one-wins-all' FA across models and tasks. This makes the selection of a FA computationally expensive for large LMs since input importance derivation often requires multiple forward and backward passes including gradient computations that might be prohibitive even with access to large compute. To address these issues, we present a model-agnostic FA for generative LMs called Recursive Attribution Generator (ReAGent). Our method updates the token importance distribution in a recursive manner. For each update, we compute the difference in the probability distribution over the vocabulary for predicting the next token between using the original input and using a modified version where a part of the input is replaced with RoBERTa predictions. Our intuition is that replacing an important token in the context should have resulted in a larger change in the model's confidence in predicting the token than replacing an unimportant token. Our method can be universally applied to any generative LM without accessing internal model weights or additional training and fine-tuning, as most other FAs require. We extensively compare the faithfulness of ReAGent with seven popular FAs across six decoder-only LMs of various sizes. The results show that our method consistently provides more faithful token importance distributions.

Title: LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law

• Authors: Toni J.B. Liu, Nicolas Boullé, Raphaël Sarfati, Christopher J. Earls
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00795
• Pdf URL: https://arxiv.org/pdf/2402.00795
• Copy Paste: [[2402.00795]] LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law(https://arxiv.org/abs/2402.00795)
• Keywords: large language model
• Abstract: Pretrained large language models (LLMs) are surprisingly effective at performing zero-shot tasks, including time-series forecasting. However, understanding the mechanisms behind such capabilities remains highly challenging due to the complexity of the models. In this paper, we study LLMs' ability to extrapolate the behavior of dynamical systems whose evolution is governed by principles of physical interest. Our results show that LLaMA 2, a language model trained primarily on texts, achieves accurate predictions of dynamical system time series without fine-tuning or prompt engineering. Moreover, the accuracy of the learned physical rules increases with the length of the input context window, revealing an in-context version of neural scaling law. Along the way, we present a flexible and efficient algorithm for extracting probability density functions of multi-digit numbers directly from LLMs.

Title: Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents

• Authors: Zelong Li, Wenyue Hua, Hao Wang, He Zhu, Yongfeng Zhang
• Subjects: cs.LG, cs.AI, cs.CL, cs.FL
• Abstract URL: https://arxiv.org/abs/2402.00798
• Pdf URL: https://arxiv.org/pdf/2402.00798
• Copy Paste: [[2402.00798]] Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents(https://arxiv.org/abs/2402.00798)
• Keywords: large language model
• Abstract: Recent advancements on Large Language Models (LLMs) enable AI Agents to automatically generate and execute multi-step plans to solve complex tasks. However, since LLM's content generation process is hardly controllable, current LLM-based agents frequently generate invalid or non-executable plans, which jeopardizes the performance of the generated plans and corrupts users' trust in LLM-based agents. In response, this paper proposes a novel ``Formal-LLM'' framework for LLM-based agents by integrating the expressiveness of natural language and the precision of formal language. Specifically, the framework allows human users to express their requirements or constraints for the planning process as an automaton. A stack-based LLM plan generation process is then conducted under the supervision of the automaton to ensure that the generated plan satisfies the constraints, making the planning process controllable. We conduct experiments on both benchmark tasks and practical real-life tasks, and our framework achieves over 50% overall performance increase, which validates the feasibility and effectiveness of employing Formal-LLM to guide the plan generation of agents, preventing the agents from generating invalid and unsuccessful plans. Further, more controllable LLM-based agents can facilitate the broader utilization of LLM in application scenarios where high validity of planning is essential. The work is open-sourced at https://github.com/agiresearch/Formal-LLM.

Title: Distilling Conditional Diffusion Models for Offline Reinforcement Learning through Trajectory Stitching

• Authors: Shangzhe Li, Xinhua Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00807
• Pdf URL: https://arxiv.org/pdf/2402.00807
• Copy Paste: [[2402.00807]] Distilling Conditional Diffusion Models for Offline Reinforcement Learning through Trajectory Stitching(https://arxiv.org/abs/2402.00807)
• Keywords: diffusion, generative
• Abstract: Deep generative models have recently emerged as an effective approach to offline reinforcement learning. However, their large model size poses challenges in computation. We address this issue by proposing a knowledge distillation method based on data augmentation. In particular, high-return trajectories are generated from a conditional diffusion model, and they are blended with the original trajectories through a novel stitching algorithm that leverages a new reward generator. Applying the resulting dataset to behavioral cloning, the learned shallow policy whose size is much smaller outperforms or nearly matches deep generative planners on several D4RL benchmarks.

Title: ALISON: Fast and Effective Stylometric Authorship Obfuscation

• Authors: Eric Xing, Saranya Venkatraman, Thai Le, Dongwon Lee
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00835
• Pdf URL: https://arxiv.org/pdf/2402.00835
• Copy Paste: [[2402.00835]] ALISON: Fast and Effective Stylometric Authorship Obfuscation(https://arxiv.org/abs/2402.00835)
• Keywords: privacy, attack, transformer
• Abstract: Authorship Attribution (AA) and Authorship Obfuscation (AO) are two competing tasks of increasing importance in privacy research. Modern AA leverages an author's consistent writing style to match a text to its author using an AA classifier. AO is the corresponding adversarial task, aiming to modify a text in such a way that its semantics are preserved, yet an AA model cannot correctly infer its authorship. To address privacy concerns raised by state-of-the-art (SOTA) AA methods, new AO methods have been proposed but remain largely impractical to use due to their prohibitively slow training and obfuscation speed, often taking hours. To this challenge, we propose a practical AO method, ALISON, that (1) dramatically reduces training/obfuscation time, demonstrating more than 10x faster obfuscation than SOTA AO methods, (2) achieves better obfuscation success through attacking three transformer-based AA methods on two benchmark datasets, typically performing 15% better than competing methods, (3) does not require direct signals from a target AA classifier during obfuscation, and (4) utilizes unique stylometric features, allowing sound model interpretation for explainable obfuscation. We also demonstrate that ALISON can effectively prevent four SOTA AA methods from accurately determining the authorship of ChatGPT-generated texts, all while minimally changing the original text semantics. To ensure the reproducibility of our findings, our code and data are available at: https://github.com/EricX003/ALISON.

Title: X-CBA: Explainability Aided CatBoosted Anomal-E for Intrusion Detection System

• Authors: Kiymet Kaya, Elif Ak, Sumeyye Bas, Berk Canberk, Sule Gunduz Oguducu
• Subjects: cs.CR, cs.AI, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2402.00839
• Pdf URL: https://arxiv.org/pdf/2402.00839
• Copy Paste: [[2402.00839]] X-CBA: Explainability Aided CatBoosted Anomal-E for Intrusion Detection System(https://arxiv.org/abs/2402.00839)
• Keywords: security, defense, attack, explainability
• Abstract: The effectiveness of Intrusion Detection Systems (IDS) is critical in an era where cyber threats are becoming increasingly complex. Machine learning (ML) and deep learning (DL) models provide an efficient and accurate solution for identifying attacks and anomalies in computer networks. However, using ML and DL models in IDS has led to a trust deficit due to their non-transparent decision-making. This transparency gap in IDS research is significant, affecting confidence and accountability. To address, this paper introduces a novel Explainable IDS approach, called X-CBA, that leverages the structural advantages of Graph Neural Networks (GNNs) to effectively process network traffic data, while also adapting a new Explainable AI (XAI) methodology. Unlike most GNN-based IDS that depend on labeled network traffic and node features, thereby overlooking critical packet-level information, our approach leverages a broader range of traffic data through network flows, including edge attributes, to improve detection capabilities and adapt to novel threats. Through empirical testing, we establish that our approach not only achieves high accuracy with 99.47% in threat detection but also advances the field by providing clear, actionable explanations of its analytical outcomes. This research also aims to bridge the current gap and facilitate the broader integration of ML/DL technologies in cybersecurity defenses by offering a local and global explainability solution that is both precise and interpretable.

Title: Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?

• Authors: Xue-Yong Fu, Md Tahmid Rahman Laskar, Elena Khasanova, Cheng Chen, Shashi Bhushan TN
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00841
• Pdf URL: https://arxiv.org/pdf/2402.00841
• Copy Paste: [[2402.00841]] Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?(https://arxiv.org/abs/2402.00841)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities to solve a wide range of tasks without being explicitly fine-tuned on task-specific datasets. However, deploying LLMs in the real world is not trivial, as it requires substantial computing resources. In this paper, we investigate whether smaller, compact LLMs are a good alternative to the comparatively Larger LLMs2 to address significant costs associated with utilizing LLMs in the real world. In this regard, we study the meeting summarization task in a real-world industrial environment and conduct extensive experiments by comparing the performance of fine-tuned compact LLMs (e.g., FLAN-T5, TinyLLaMA, LiteLLaMA) with zero-shot larger LLMs (e.g., LLaMA-2, GPT-3.5, PaLM-2). We observe that most smaller LLMs, even after fine-tuning, fail to outperform larger zero-shot LLMs in meeting summarization datasets. However, a notable exception is FLAN-T5 (780M parameters), which performs on par or even better than many zero-shot Larger LLMs (from 7B to above 70B parameters), while being significantly smaller. This makes compact LLMs like FLAN-T5 a suitable cost-efficient solution for real-world industrial deployment.

Title: Data Augmentation Scheme for Raman Spectra with Highly Correlated Annotations

• Authors: Christoph Lange, Isabel Thiele, Lara Santolin, Sebastian L. Riedel, Maxim Borisyak, Peter Neubauer, M. Nicolas Cruz Bournazou
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2402.00851
• Pdf URL: https://arxiv.org/pdf/2402.00851
• Copy Paste: [[2402.00851]] Data Augmentation Scheme for Raman Spectra with Highly Correlated Annotations(https://arxiv.org/abs/2402.00851)
• Keywords: robust
• Abstract: In biotechnology Raman Spectroscopy is rapidly gaining popularity as a process analytical technology (PAT) that measures cell densities, substrate- and product concentrations. As it records vibrational modes of molecules it provides that information non-invasively in a single spectrum. Typically, partial least squares (PLS) is the model of choice to infer information about variables of interest from the spectra. However, biological processes are known for their complexity where convolutional neural networks (CNN) present a powerful alternative. They can handle non-Gaussian noise and account for beam misalignment, pixel malfunctions or the presence of additional substances. However, they require a lot of data during model training, and they pick up non-linear dependencies in the process variables. In this work, we exploit the additive nature of spectra in order to generate additional data points from a given dataset that have statistically independent labels so that a network trained on such data exhibits low correlations between the model predictions. We show that training a CNN on these generated data points improves the performance on datasets where the annotations do not bear the same correlation as the dataset that was used for model training. This data augmentation technique enables us to reuse spectra as training data for new contexts that exhibit different correlations. The additional data allows for building a better and more robust model. This is of interest in scenarios where large amounts of historical data are available but are currently not used for model training. We demonstrate the capabilities of the proposed method using synthetic spectra of Ralstonia eutropha batch cultivations to monitor substrate, biomass and polyhydroxyalkanoate (PHA) biopolymer concentrations during of the experiments.

Title: SymbolicAI: A framework for logic-based approaches combining generative models and solvers

• Authors: Marius-Constantin Dinu, Claudiu Leoveanu-Condrei, Markus Holzleitner, Werner Zellinger, Sepp Hochreiter
• Subjects: cs.LG, cs.AI, cs.SC, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.00854
• Pdf URL: https://arxiv.org/pdf/2402.00854
• Copy Paste: [[2402.00854]] SymbolicAI: A framework for logic-based approaches combining generative models and solvers(https://arxiv.org/abs/2402.00854)
• Keywords: generative, large language model
• Abstract: We introduce SymbolicAI, a versatile and modular framework employing a logic-based approach to concept learning and flow management in generative processes. SymbolicAI enables the seamless integration of generative models with a diverse range of solvers by treating large language models (LLMs) as semantic parsers that execute tasks based on both natural and formal language instructions, thus bridging the gap between symbolic reasoning and generative AI. We leverage probabilistic programming principles to tackle complex tasks, and utilize differentiable and classical programming paradigms with their respective strengths. The framework introduces a set of polymorphic, compositional, and self-referential operations for data stream manipulation, aligning LLM outputs with user objectives. As a result, we can transition between the capabilities of various foundation models endowed with zero- and few-shot learning capabilities and specialized, fine-tuned models or solvers proficient in addressing specific problems. In turn, the framework facilitates the creation and evaluation of explainable computational graphs. We conclude by introducing a quality measure and its empirical score for evaluating these computational graphs, and propose a benchmark that compares various state-of-the-art LLMs across a set of complex workflows. We refer to the empirical score as the "Vector Embedding for Relational Trajectory Evaluation through Cross-similarity", or VERTEX score for short. The framework codebase and benchmark are linked below.

Title: Can Large Language Models Understand Context?

• Authors: Yilun Zhu, Joel Ruben Antony Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Site Li, Yuan Zhang, Hong Yu, Bo-Hsiang Tseng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00858
• Pdf URL: https://arxiv.org/pdf/2402.00858
• Copy Paste: [[2402.00858]] Can Large Language Models Understand Context?(https://arxiv.org/abs/2402.00858)
• Keywords: generative, large language model
• Abstract: Understanding context is key to understanding human language, an ability which Large Language Models (LLMs) have been increasingly seen to demonstrate to an impressive extent. However, though the evaluation of LLMs encompasses various domains within the realm of Natural Language Processing, limited attention has been paid to probing their linguistic capability of understanding contextual features. This paper introduces a context understanding benchmark by adapting existing datasets to suit the evaluation of generative models. This benchmark comprises of four distinct tasks and nine datasets, all featuring prompts designed to assess the models' ability to understand context. First, we evaluate the performance of LLMs under the in-context learning pretraining scenario. Experimental results indicate that pre-trained dense models struggle with understanding more nuanced contextual features when compared to state-of-the-art fine-tuned models. Second, as LLM compression holds growing significance in both research and real-world applications, we assess the context understanding of quantized models under in-context-learning settings. We find that 3-bit post-training quantization leads to varying degrees of performance reduction on our benchmark. We conduct an extensive analysis of these scenarios to substantiate our experimental results.

Title: Evaluating Large Language Models for Generalization and Robustness via Data Compression

• Authors: Yucheng Li, Yunhao Guo, Frank Guerin, Chenghua Lin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00861
• Pdf URL: https://arxiv.org/pdf/2402.00861
• Copy Paste: [[2402.00861]] Evaluating Large Language Models for Generalization and Robustness via Data Compression(https://arxiv.org/abs/2402.00861)
• Keywords: robust, large language model
• Abstract: Existing methods for evaluating large language models face challenges such as data contamination, sensitivity to prompts, and the high cost of benchmark creation. To address this, we propose a lossless data compression based evaluation approach that tests how models' predictive abilities generalize after their training cutoff. Specifically, we collect comprehensive test data spanning 83 months from 2017 to 2023 and split the data into training and testing periods according to models' training data cutoff. We measure: 1) the compression performance on the testing period as a measure of generalization on unseen data; and 2) the performance gap between the training and testing period as a measure of robustness. Our experiments test 14 representative large language models with various sizes on sources including Wikipedia, news articles, code, arXiv papers, and multi-modal data. We find that the compression rate of many models reduces significantly after their cutoff date, but models such as Mistral and Llama-2 demonstrate a good balance between performance and robustness. Results also suggest that models struggle to generalize on news and code data, but work especially well on arXiv papers. We also find the context size and tokenization implementation have a big impact of on the overall compression performance.

Title: ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields

• Authors: Jiahua Dong, Yu-Xiong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00864
• Pdf URL: https://arxiv.org/pdf/2402.00864
• Copy Paste: [[2402.00864]] ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields(https://arxiv.org/abs/2402.00864)
• Keywords: diffusion
• Abstract: We introduce ViCA-NeRF, the first view-consistency-aware method for 3D editing with text instructions. In addition to the implicit neural radiance field (NeRF) modeling, our key insight is to exploit two sources of regularization that explicitly propagate the editing information across different views, thus ensuring multi-view consistency. For geometric regularization, we leverage the depth information derived from NeRF to establish image correspondences between different views. For learned regularization, we align the latent codes in the 2D diffusion model between edited and unedited images, enabling us to edit key views and propagate the update throughout the entire scene. Incorporating these two strategies, our ViCA-NeRF operates in two stages. In the initial stage, we blend edits from different views to create a preliminary 3D edit. This is followed by a second stage of NeRF training, dedicated to further refining the scene's appearance. Experimental results demonstrate that ViCA-NeRF provides more flexible, efficient (3 times faster) editing with higher levels of consistency and details, compared with the state of the art. Our code is publicly available.

Title: AToM: Amortized Text-to-Mesh using 2D Diffusion

• Authors: Guocheng Qian, Junli Cao, Aliaksandr Siarohin, Yash Kant, Chaoyang Wang, Michael Vasilkovsky, Hsin-Ying Lee, Yuwei Fang, Ivan Skorokhodov, Peiye Zhuang, Igor Gilitschenski, Jian Ren, Bernard Ghanem, Kfir Aberman, Sergey Tulyakov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00867
• Pdf URL: https://arxiv.org/pdf/2402.00867
• Copy Paste: [[2402.00867]] AToM: Amortized Text-to-Mesh using 2D Diffusion(https://arxiv.org/abs/2402.00867)
• Keywords: diffusion
• Abstract: We introduce Amortized Text-to-Mesh (AToM), a feed-forward text-to-mesh framework optimized across multiple text prompts simultaneously. In contrast to existing text-to-3D methods that often entail time-consuming per-prompt optimization and commonly output representations other than polygonal meshes, AToM directly generates high-quality textured meshes in less than 1 second with around 10 times reduction in the training cost, and generalizes to unseen prompts. Our key idea is a novel triplane-based text-to-mesh architecture with a two-stage amortized optimization strategy that ensures stable training and enables scalability. Through extensive experiments on various prompt benchmarks, AToM significantly outperforms state-of-the-art amortized approaches with over 4 times higher accuracy (in DF415 dataset) and produces more distinguishable and higher-quality 3D outputs. AToM demonstrates strong generalizability, offering finegrained 3D assets for unseen interpolated prompts without further optimization during inference, unlike per-prompt solutions.

Title: We're Not Using Videos Effectively: An Updated Domain Adaptive Video Segmentation Baseline

• Authors: Simar Kareer, Vivek Vijaykumar, Harsh Maheshwari, Prithvijit Chattopadhyay, Judy Hoffman, Viraj Prabhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00868
• Pdf URL: https://arxiv.org/pdf/2402.00868
• Copy Paste: [[2402.00868]] We're Not Using Videos Effectively: An Updated Domain Adaptive Video Segmentation Baseline(https://arxiv.org/abs/2402.00868)
• Keywords: segmentation
• Abstract: There has been abundant work in unsupervised domain adaptation for semantic segmentation (DAS) seeking to adapt a model trained on images from a labeled source domain to an unlabeled target domain. While the vast majority of prior work has studied this as a frame-level Image-DAS problem, a few Video-DAS works have sought to additionally leverage the temporal signal present in adjacent frames. However, Video-DAS works have historically studied a distinct set of benchmarks from Image-DAS, with minimal cross-benchmarking. In this work, we address this gap. Surprisingly, we find that (1) even after carefully controlling for data and model architecture, state-of-the-art Image-DAS methods (HRDA and HRDA+MIC)} outperform Video-DAS methods on established Video-DAS benchmarks (+14.5 mIoU on Viper$\rightarrow$CityscapesSeq, +19.0 mIoU on Synthia$\rightarrow$CityscapesSeq), and (2) naive combinations of Image-DAS and Video-DAS techniques only lead to marginal improvements across datasets. To avoid siloed progress between Image-DAS and Video-DAS, we open-source our codebase with support for a comprehensive set of Video-DAS and Image-DAS methods on a common benchmark. Code available at https://github.com/SimarKareer/UnifiedVideoDA