2024-01-22

Title: Resolution Chromatography of Diffusion Models

• Authors: Juno Hwang, Yong-Hyun Park, Junghyo Jo
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10247
• Pdf URL: https://arxiv.org/pdf/2401.10247
• Copy Paste: [[2401.10247]] Resolution Chromatography of Diffusion Models(https://arxiv.org/abs/2401.10247)
• Keywords: diffusion
• Abstract: Diffusion models generate high-resolution images through iterative stochastic processes. In particular, the denoising method is one of the most popular approaches that predicts the noise in samples and denoises it at each time step. It has been commonly observed that the resolution of generated samples changes over time, starting off blurry and coarse, and becoming sharper and finer. In this paper, we introduce "resolution chromatography" that indicates the signal generation rate of each resolution, which is very helpful concept to mathematically explain this coarse-to-fine behavior in generation process, to understand the role of noise schedule, and to design time-dependent modulation. Using resolution chromatography, we determine which resolution level becomes dominant at a specific time step, and experimentally verify our theory with text-to-image diffusion models. We also propose some direct applications utilizing the concept: upscaling pre-trained models to higher resolutions and time-dependent prompt composing. Our theory not only enables a better understanding of numerous pre-existing techniques for manipulating image generation, but also suggests the potential for designing better noise schedules.

Title: Multi-Source Collaborative Gradient Discrepancy Minimization for Federated Domain Generalization

• Authors: Yikang Wei, Yahong Han
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10272
• Pdf URL: https://arxiv.org/pdf/2401.10272
• Copy Paste: [[2401.10272]] Multi-Source Collaborative Gradient Discrepancy Minimization for Federated Domain Generalization(https://arxiv.org/abs/2401.10272)
• Keywords: privacy, federate
• Abstract: Federated Domain Generalization aims to learn a domain-invariant model from multiple decentralized source domains for deployment on unseen target domain. Due to privacy concerns, the data from different source domains are kept isolated, which poses challenges in bridging the domain gap. To address this issue, we propose a Multi-source Collaborative Gradient Discrepancy Minimization (MCGDM) method for federated domain generalization. Specifically, we propose intra-domain gradient matching between the original images and augmented images to avoid overfitting the domain-specific information within isolated domains. Additionally, we propose inter-domain gradient matching with the collaboration of other domains, which can further reduce the domain shift across decentralized domains. Combining intra-domain and inter-domain gradient matching, our method enables the learned model to generalize well on unseen domains. Furthermore, our method can be extended to the federated domain adaptation task by fine-tuning the target model on the pseudo-labeled target domain. The extensive experiments on federated domain generalization and adaptation indicate that our method outperforms the state-of-the-art methods significantly.

Title: Top in Chinese Data Processing: English Code Models

• Authors: Linghan Zheng, Hui Liu, Xiaojun Lin, Jiayuan Dong, Yue Sheng, Gang Shi, Zhiwei Liu, Hongwei Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10286
• Pdf URL: https://arxiv.org/pdf/2401.10286
• Copy Paste: [[2401.10286]] Top in Chinese Data Processing: English Code Models(https://arxiv.org/abs/2401.10286)
• Keywords: large language model
• Abstract: While the alignment between tasks and training corpora is a fundamental consensus in the application of language models, our series of experiments and the metrics we designed reveal that code-based Large Language Models (LLMs) significantly outperform models trained on data that is closely matched to the tasks in non-coding Chinese tasks. Moreover, in tasks high sensitivity to Chinese hallucinations, models exhibiting fewer linguistic features of the Chinese language achieve better performance. Our experimental results can be easily replicated in Chinese data processing tasks, such as preparing data for Retrieval-Augmented Generation (RAG), by simply replacing the base model with a code-based model. Additionally, our research offers a distinct perspective for discussion on the philosophical "Chinese Room" thought experiment.

Title: CLAN: A Contrastive Learning based Novelty Detection Framework for Human Activity Recognition

• Authors: Hyunju Kim, Dongman Lee
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.10288
• Pdf URL: https://arxiv.org/pdf/2401.10288
• Copy Paste: [[2401.10288]] CLAN: A Contrastive Learning based Novelty Detection Framework for Human Activity Recognition(https://arxiv.org/abs/2401.10288)
• Keywords: robust
• Abstract: In ambient assisted living, human activity recognition from time series sensor data mainly focuses on predefined activities, often overlooking new activity patterns. We propose CLAN, a two-tower contrastive learning-based novelty detection framework with diverse types of negative pairs for human activity recognition. It is tailored to challenges with human activity characteristics, including the significance of temporal and frequency features, complex activity dynamics, shared features across activities, and sensor modality variations. The framework aims to construct invariant representations of known activity robust to the challenges. To generate suitable negative pairs, it selects data augmentation methods according to the temporal and frequency characteristics of each dataset. It derives the key representations against meaningless dynamics by contrastive and classification losses-based representation learning and score function-based novelty detection that accommodate dynamic numbers of the different types of augmented samples. The proposed two-tower model extracts the representations in terms of time and frequency, mutually enhancing expressiveness for distinguishing between new and known activities, even when they share common features. Experiments on four real-world human activity datasets show that CLAN surpasses the best performance of existing novelty detection methods, improving by 8.3%, 13.7%, and 53.3% in AUROC, balanced accuracy, and FPR@TPR0.95 metrics respectively.

Title: An attempt to generate new bridge types from latent space of generative flow

• Authors: Hongjun Zhang
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10299
• Pdf URL: https://arxiv.org/pdf/2401.10299
• Copy Paste: [[2401.10299]] An attempt to generate new bridge types from latent space of generative flow(https://arxiv.org/abs/2401.10299)
• Keywords: generative
• Abstract: Through examples of coordinate and probability transformation between different distributions, the basic principle of normalizing flow is introduced in a simple and concise manner. From the perspective of the distribution of random variable function, the essence of probability transformation is explained, and the scaling factor Jacobian determinant of probability transformation is introduced. Treating the dataset as a sample from the population, obtaining normalizing flow is essentially through sampling surveys to statistically infer the numerical features of the population, and then the loss function is established by using the maximum likelihood estimation method. This article introduces how normalizing flow cleverly solves the two major application challenges of high-dimensional matrix determinant calculation and neural network reversible transformation. Using symmetric structured image dataset of three-span beam bridge, arch bridge, cable-stayed bridge and suspension bridge, constructing and training normalizing flow based on the Glow API in the TensorFlow Probability library. The model can smoothly transform the complex distribution of the bridge dataset into a standard normal distribution, and from the obtained latent space sampling, it can generate new bridge types that are different from the training dataset.

Title: On the Readiness of Scientific Data for a Fair and Transparent Use in Machine Learning

• Authors: Joan Giner-Miguelez, Abel Gómez, Jordi Cabot
• Subjects: cs.LG, cs.AI, cs.DL
• Abstract URL: https://arxiv.org/abs/2401.10304
• Pdf URL: https://arxiv.org/pdf/2401.10304
• Copy Paste: [[2401.10304]] On the Readiness of Scientific Data for a Fair and Transparent Use in Machine Learning(https://arxiv.org/abs/2401.10304)
• Keywords: fair
• Abstract: To ensure the fairness and trustworthiness of machine learning (ML) systems, recent legislative initiatives and relevant research in the ML community have pointed out the need to document the data used to train ML models. Besides, data-sharing practices in many scientific domains have evolved in recent years for reproducibility purposes. In this sense, the adoption of these practices by academic institutions has encouraged researchers to publish their data and technical documentation in peer-reviewed publications such as data papers. In this study, we analyze how this scientific data documentation meets the needs of the ML community and regulatory bodies for its use in ML technologies. We examine a sample of 4041 data papers of different domains, assessing their completeness and coverage of the requested dimensions, and trends in recent years, putting special emphasis on the most and least documented dimensions. As a result, we propose a set of recommendation guidelines for data creators and scientific data publishers to increase their data's preparedness for its transparent and fairer use in ML technologies.

Title: Mathematical Algorithm Design for Deep Learning under Societal and Judicial Constraints: The Algorithmic Transparency Requirement

• Authors: Holger Boche, Adalbert Fono, Gitta Kutyniok
• Subjects: cs.LG, cs.AI, cs.CC
• Abstract URL: https://arxiv.org/abs/2401.10310
• Pdf URL: https://arxiv.org/pdf/2401.10310
• Copy Paste: [[2401.10310]] Mathematical Algorithm Design for Deep Learning under Societal and Judicial Constraints: The Algorithmic Transparency Requirement(https://arxiv.org/abs/2401.10310)
• Keywords: fair
• Abstract: Deep learning still has drawbacks in terms of trustworthiness, which describes a comprehensible, fair, safe, and reliable method. To mitigate the potential risk of AI, clear obligations associated to trustworthiness have been proposed via regulatory guidelines, e.g., in the European AI Act. Therefore, a central question is to what extent trustworthy deep learning can be realized. Establishing the described properties constituting trustworthiness requires that the factors influencing an algorithmic computation can be retraced, i.e., the algorithmic implementation is transparent. Motivated by the observation that the current evolution of deep learning models necessitates a change in computing technology, we derive a mathematical framework which enables us to analyze whether a transparent implementation in a computing model is feasible. We exemplarily apply our trustworthiness framework to analyze deep learning approaches for inverse problems in digital and analog computing models represented by Turing and Blum-Shub-Smale Machines, respectively. Based on previous results, we find that Blum-Shub-Smale Machines have the potential to establish trustworthy solvers for inverse problems under fairly general conditions, whereas Turing machines cannot guarantee trustworthiness to the same degree.

Title: Hacking Predictors Means Hacking Cars: Using Sensitivity Analysis to Identify Trajectory Prediction Vulnerabilities for Autonomous Driving Security

• Authors: Marsalis Gibson, David Babazadeh, Claire Tomlin, Shankar Sastry
• Subjects: cs.CR, cs.LG, cs.RO, eess.SY
• Abstract URL: https://arxiv.org/abs/2401.10313
• Pdf URL: https://arxiv.org/pdf/2401.10313
• Copy Paste: [[2401.10313]] Hacking Predictors Means Hacking Cars: Using Sensitivity Analysis to Identify Trajectory Prediction Vulnerabilities for Autonomous Driving Security(https://arxiv.org/abs/2401.10313)
• Keywords: security, attack, robust
• Abstract: Adversarial attacks on learning-based trajectory predictors have already been demonstrated. However, there are still open questions about the effects of perturbations on trajectory predictor inputs other than state histories, and how these attacks impact downstream planning and control. In this paper, we conduct a sensitivity analysis on two trajectory prediction models, Trajectron++ and AgentFormer. We observe that between all inputs, almost all of the perturbation sensitivities for Trajectron++ lie only within the most recent state history time point, while perturbation sensitivities for AgentFormer are spread across state histories over time. We additionally demonstrate that, despite dominant sensitivity on state history perturbations, an undetectable image map perturbation made with the Fast Gradient Sign Method can induce large prediction error increases in both models. Even though image maps may contribute slightly to the prediction output of both models, this result reveals that rather than being robust to adversarial image perturbations, trajectory predictors are susceptible to image attacks. Using an optimization-based planner and example perturbations crafted from sensitivity results, we show how this vulnerability can cause a vehicle to come to a sudden stop from moderate driving speeds.

Title: Noise Contrastive Estimation-based Matching Framework for Low-resource Security Attack Pattern Recognition

• Authors: Tu Nguyen, Nedim Srndic, Alexander Neth
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.10337
• Pdf URL: https://arxiv.org/pdf/2401.10337
• Copy Paste: [[2401.10337]] Noise Contrastive Estimation-based Matching Framework for Low-resource Security Attack Pattern Recognition(https://arxiv.org/abs/2401.10337)
• Keywords: security, attack
• Abstract: Tactics, Techniques and Procedures (TTPs) represent sophisticated attack patterns in the cybersecurity domain, described encyclopedically in textual knowledge bases. Identifying TTPs in cybersecurity writing, often called TTP mapping, is an important and challenging task. Conventional learning approaches often target the problem in the classical multi-class or multilabel classification setting. This setting hinders the learning ability of the model due to a large number of classes (i.e., TTPs), the inevitable skewness of the label distribution and the complex hierarchical structure of the label space. We formulate the problem in a different learning paradigm, where the assignment of a text to a TTP label is decided by the direct semantic similarity between the two, thus reducing the complexity of competing solely over the large labeling space. To that end, we propose a neural matching architecture with an effective sampling-based learn-to-compare mechanism, facilitating the learning process of the matching model despite constrained resources.

Title: MELODY: Robust Semi-Supervised Hybrid Model for Entity-Level Online Anomaly Detection with Multivariate Time Series

• Authors: Jingchao Ni, Gauthier Guinet, Peihong Jiang, Laurent Callot, Andrey Kan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10338
• Pdf URL: https://arxiv.org/pdf/2401.10338
• Copy Paste: [[2401.10338]] MELODY: Robust Semi-Supervised Hybrid Model for Entity-Level Online Anomaly Detection with Multivariate Time Series(https://arxiv.org/abs/2401.10338)
• Keywords: robust
• Abstract: In large IT systems, software deployment is a crucial process in online services as their code is regularly updated. However, a faulty code change may degrade the target service's performance and cause cascading outages in downstream services. Thus, software deployments should be comprehensively monitored, and their anomalies should be detected timely. In this paper, we study the problem of anomaly detection for deployments. We begin by identifying the challenges unique to this anomaly detection problem, which is at entity-level (e.g., deployments), relative to the more typical problem of anomaly detection in multivariate time series (MTS). The unique challenges include the heterogeneity of deployments, the low latency tolerance, the ambiguous anomaly definition, and the limited supervision. To address them, we propose a novel framework, semi-supervised hybrid Model for Entity-Level Online Detection of anomalY (MELODY). MELODY first transforms the MTS of different entities to the same feature space by an online feature extractor, then uses a newly proposed semi-supervised deep one-class model for detecting anomalous entities. We evaluated MELODY on real data of cloud services with 1.2M+ time series. The relative F1 score improvement of MELODY over the state-of-the-art methods ranges from 7.6% to 56.5%. The user evaluation suggests MELODY is suitable for monitoring deployments in large online systems.

Title: Inconsistent dialogue responses and how to recover from them

• Authors: Mian Zhang, Lifeng Jin, Linfeng Song, Haitao Mi, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10353
• Pdf URL: https://arxiv.org/pdf/2401.10353
• Copy Paste: [[2401.10353]] Inconsistent dialogue responses and how to recover from them(https://arxiv.org/abs/2401.10353)
• Keywords: large language model
• Abstract: One critical issue for chat systems is to stay consistent about preferences, opinions, beliefs and facts of itself, which has been shown a difficult problem. In this work, we study methods to assess and bolster utterance consistency of chat systems. A dataset is first developed for studying the inconsistencies, where inconsistent dialogue responses, explanations of the inconsistencies, and recovery utterances are authored by annotators. This covers the life span of inconsistencies, namely introduction, understanding, and resolution. Building on this, we introduce a set of tasks centered on dialogue consistency, specifically focused on its detection and resolution. Our experimental findings indicate that our dataset significantly helps the progress in identifying and resolving conversational inconsistencies, and current popular large language models like ChatGPT which are good at resolving inconsistencies however still struggle with detection.

Title: Excuse me, sir? Your language model is leaking (information)

• Authors: Or Zamir
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10360
• Pdf URL: https://arxiv.org/pdf/2401.10360
• Copy Paste: [[2401.10360]] Excuse me, sir? Your language model is leaking (information)(https://arxiv.org/abs/2401.10360)
• Keywords: watermark, large language model
• Abstract: We introduce a cryptographic method to hide an arbitrary secret payload in the response of a Large Language Model (LLM). A secret key is required to extract the payload from the model's response, and without the key it is provably impossible to distinguish between the responses of the original LLM and the LLM that hides a payload. In particular, the quality of generated text is not affected by the payload. Our approach extends a recent result of Christ, Gunn and Zamir (2023) who introduced an undetectable watermarking scheme for LLMs.

Title: Hierarchical Federated Learning in Multi-hop Cluster-Based VANETs

• Authors: M. Saeid HaghighiFard, Sinem Coleri
• Subjects: cs.LG, cs.AI, cs.DC, cs.NI, eess.SY
• Abstract URL: https://arxiv.org/abs/2401.10361
• Pdf URL: https://arxiv.org/pdf/2401.10361
• Copy Paste: [[2401.10361]] Hierarchical Federated Learning in Multi-hop Cluster-Based VANETs(https://arxiv.org/abs/2401.10361)
• Keywords: privacy, protect, federate
• Abstract: The usage of federated learning (FL) in Vehicular Ad hoc Networks (VANET) has garnered significant interest in research due to the advantages of reducing transmission overhead and protecting user privacy by communicating local dataset gradients instead of raw data. However, implementing FL in VANETs faces challenges, including limited communication resources, high vehicle mobility, and the statistical diversity of data distributions. In order to tackle these issues, this paper introduces a novel framework for hierarchical federated learning (HFL) over multi-hop clustering-based VANET. The proposed method utilizes a weighted combination of the average relative speed and cosine similarity of FL model parameters as a clustering metric to consider both data diversity and high vehicle mobility. This metric ensures convergence with minimum changes in cluster heads while tackling the complexities associated with non-independent and identically distributed (non-IID) data scenarios. Additionally, the framework includes a novel mechanism to manage seamless transitions of cluster heads (CHs), followed by transferring the most recent FL model parameter to the designated CH. Furthermore, the proposed approach considers the option of merging CHs, aiming to reduce their count and, consequently, mitigate associated overhead. Through extensive simulations, the proposed hierarchical federated learning over clustered VANET has been demonstrated to improve accuracy and convergence time significantly while maintaining an acceptable level of packet overhead compared to previously proposed clustering algorithms and non-clustered VANET.

Title: Using LLM such as ChatGPT for Designing and Implementing a RISC Processor: Execution,Challenges and Limitations

• Authors: Shadeeb Hossain, Aayush Gohil, Yizhou Wang
• Subjects: cs.LG, cs.AR, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.10364
• Pdf URL: https://arxiv.org/pdf/2401.10364
• Copy Paste: [[2401.10364]] Using LLM such as ChatGPT for Designing and Implementing a RISC Processor: Execution,Challenges and Limitations(https://arxiv.org/abs/2401.10364)
• Keywords: large language model
• Abstract: This paper discusses the feasibility of using Large Language Models LLM for code generation with a particular application in designing an RISC. The paper also reviews the associated steps such as parsing, tokenization, encoding, attention mechanism, sampling the tokens and iterations during code generation. The generated code for the RISC components is verified through testbenches and hardware implementation on a FPGA board. Four metric parameters Correct output on the first iteration, Number of errors embedded in the code, Number of trials required to achieve the code and Failure to generate the code after three iterations, are used to compare the efficiency of using LLM in programming. In all the cases, the generated code had significant errors and human intervention was always required to fix the bugs. LLM can therefore be used to complement a programmer code design.

Title: Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning

• Authors: Eli Chien, Haoyu Wang, Ziang Chen, Pan Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10371
• Pdf URL: https://arxiv.org/pdf/2401.10371
• Copy Paste: [[2401.10371]] Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning(https://arxiv.org/abs/2401.10371)
• Keywords: privacy
• Abstract: Machine unlearning has raised significant interest with the adoption of laws ensuring the ``right to be forgotten''. Researchers have provided a probabilistic notion of approximate unlearning under a similar definition of Differential Privacy (DP), where privacy is defined as statistical indistinguishability to retraining from scratch. We propose Langevin unlearning, an unlearning framework based on noisy gradient descent with privacy guarantees for approximate unlearning problems. Langevin unlearning unifies the DP learning process and the privacy-certified unlearning process with many algorithmic benefits. These include approximate certified unlearning for non-convex problems, complexity saving compared to retraining, sequential and batch unlearning for multiple unlearning requests. We verify the practicality of Langevin unlearning by studying its privacy-utility-complexity trade-off via experiments on benchmark datasets, and also demonstrate its superiority against gradient-decent-plus-output-perturbation based approximate unlearning.

Title: Vulnerabilities of Foundation Model Integrated Federated Learning Under Adversarial Threats

• Authors: Chen Wu, Xi Li, Jiaqi Wang
• Subjects: cs.CR, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10375
• Pdf URL: https://arxiv.org/pdf/2401.10375
• Copy Paste: [[2401.10375]] Vulnerabilities of Foundation Model Integrated Federated Learning Under Adversarial Threats(https://arxiv.org/abs/2401.10375)
• Keywords: security, privacy, protect, defense, attack, federate
• Abstract: Federated Learning (FL) addresses critical issues in machine learning related to data privacy and security, yet suffering from data insufficiency and imbalance under certain circumstances. The emergence of foundation models (FMs) offers potential solutions to the limitations of existing FL frameworks, e.g., by generating synthetic data for model initialization. However, due to the inherent safety concerns of FMs, integrating FMs into FL could introduce new risks, which remains largely unexplored. To address this gap, we conduct the first investigation on the vulnerability of FM integrated FL (FM-FL) under adversarial threats. Based on a unified framework of FM-FL, we introduce a novel attack strategy that exploits safety issues of FM to compromise FL client models. Through extensive experiments with well-known models and benchmark datasets in both image and text domains, we reveal the high susceptibility of the FM-FL to this new threat under various FL configurations. Furthermore, we find that existing FL defense strategies offer limited protection against this novel attack approach. This research highlights the critical need for enhanced security measures in FL in the era of FMs.

Title: Bypassing a Reactive Jammer via NOMA-Based Transmissions in Critical Missions

• Authors: Mohammadreza Amini, Ghazal Asemian, Michel Kulhandjian, Burak Kantarci, Claude D'Amours, Melike Erol-Kantarci
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2401.10387
• Pdf URL: https://arxiv.org/pdf/2401.10387
• Copy Paste: [[2401.10387]] Bypassing a Reactive Jammer via NOMA-Based Transmissions in Critical Missions(https://arxiv.org/abs/2401.10387)
• Keywords: attack
• Abstract: Wireless networks can be vulnerable to radio jamming attacks. The quality of service under a jamming attack is not guaranteed and the service requirements such as reliability, latency, and effective rate, specifically in mission-critical military applications, can be deeply affected by the jammer's actions. This paper analyzes the effect of a reactive jammer. Particularly, reliability, average transmission delay, and the effective sum rate (ESR) for a NOMA-based scheme with finite blocklength transmissions are mathematically derived taking the detection probability of the jammer into account. Furthermore, the effect of UEs' allocated power and blocklength on the network metrics is explored. Contrary to the existing literature, results show that gNB can mitigate the impact of reactive jamming by decreasing transmit power, making the transmissions covert at the jammer side. Finally, an optimization problem is formulated to maximize the ESR under reliability, delay, and transmit power constraints. It is shown that by adjusting the allocated transmit power to UEs by gNB, the gNB can bypass the jammer effect to fulfill the 0.99999 reliability and the latency of 5ms without the need for packet re-transmission.

Title: Catastrophic Interference is Mitigated in Naturalistic Power-Law Learning Environments

• Authors: Atith Gandhi, Raj Sanjay Shah, Vijay Marupudi, Sashank Varma
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10393
• Pdf URL: https://arxiv.org/pdf/2401.10393
• Copy Paste: [[2401.10393]] Catastrophic Interference is Mitigated in Naturalistic Power-Law Learning Environments(https://arxiv.org/abs/2401.10393)
• Keywords: robust, generative
• Abstract: Neural networks often suffer from catastrophic interference (CI): performance on previously learned tasks drops off significantly when learning a new task. This contrasts strongly with humans, who can sequentially learn new tasks without appreciably forgetting previous tasks. Prior work has explored various techniques for mitigating CI such as regularization, rehearsal, generative replay, and distillation methods. The current work takes a different approach, one guided by cognitive science research showing that in naturalistic environments, the probability of encountering a task decreases as a power-law of the time since it was last performed. We argue that a realistic evaluation of techniques for the mitigation of CI should be performed in simulated naturalistic learning environments. Thus, we evaluate the extent of mitigation of CI when training simple rehearsal-based methods in power-law environments similar to the ones humans face. Our work explores this novel rehearsal-based approach for a domain-incremental task: learning permutations in the MNIST task. We compare our rehearsal environment with other baselines to show its efficacy in promoting continual learning. Additionally, we investigate whether this environment shows forward facilitation, i.e., faster learning of later tasks. Next, we explore the robustness of our learning environment to the number of tasks, model size, and amount of data rehearsed after each task. Notably, our results show that the performance is comparable or superior to that of models trained using popular regularization methods and also to rehearsals in non-power-law environments. The benefits of this training paradigm include simplicity and the lack of a need for extra neural circuitry. In addition, because our method is orthogonal to other methods, future research can combine training in power-law environments with other continual learning mechanisms.

Title: Analyzing and Mitigating Bias for Vulnerable Classes: Towards Balanced Representation in Dataset

• Authors: Dewant Katare, David Solans Noguero, Souneil Park, Nicolas Kourtellis, Marijn Janssen, Aaron Yi Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10397
• Pdf URL: https://arxiv.org/pdf/2401.10397
• Copy Paste: [[2401.10397]] Analyzing and Mitigating Bias for Vulnerable Classes: Towards Balanced Representation in Dataset(https://arxiv.org/abs/2401.10397)
• Keywords: fair, transformer
• Abstract: The accuracy and fairness of perception systems in autonomous driving are crucial, particularly for vulnerable road users. Mainstream research has looked into improving the performance metrics for classification accuracy. However, the hidden traits of bias inheritance in the AI models, class imbalances and disparities in the datasets are often overlooked. In this context, our study examines the class imbalances for vulnerable road users by focusing on class distribution analysis, performance evaluation, and bias impact assessment. We identify the concern of imbalances in class representation, leading to potential biases in detection accuracy. Utilizing popular CNN models and Vision Transformers (ViTs) with the nuScenes dataset, our performance evaluation reveals detection disparities for underrepresented classes. We propose a methodology for model optimization and bias mitigation, which includes data augmentation, resampling, and metric-specific learning. Using the proposed mitigation approaches, we see improvement in IoU(%) and NDS(%) metrics from 71.3 to 75.6 and 80.6 to 83.7 respectively, for the CNN model. Similarly, for ViT, we observe improvement in IoU and NDS metrics from 74.9 to 79.2 and 83.8 to 87.1 respectively. This research contributes to developing more reliable models and datasets, enhancing inclusiveness for minority classes.

Title: Reconstructing the Invisible: Video Frame Restoration through Siamese Masked Conditional Variational Autoencoder

• Authors: Yongchen Zhou, Richard Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10402
• Pdf URL: https://arxiv.org/pdf/2401.10402
• Copy Paste: [[2401.10402]] Reconstructing the Invisible: Video Frame Restoration through Siamese Masked Conditional Variational Autoencoder(https://arxiv.org/abs/2401.10402)
• Keywords: robust, transformer
• Abstract: In the domain of computer vision, the restoration of missing information in video frames is a critical challenge, particularly in applications such as autonomous driving and surveillance systems. This paper introduces the Siamese Masked Conditional Variational Autoencoder (SiamMCVAE), leveraging a siamese architecture with twin encoders based on vision transformers. This innovative design enhances the model's ability to comprehend lost content by capturing intrinsic similarities between paired frames. SiamMCVAE proficiently reconstructs missing elements in masked frames, effectively addressing issues arising from camera malfunctions through variational inferences. Experimental results robustly demonstrate the model's effectiveness in restoring missing information, thus enhancing the resilience of computer vision systems. The incorporation of Siamese Vision Transformer (SiamViT) encoders in SiamMCVAE exemplifies promising potential for addressing real-world challenges in computer vision, reinforcing the adaptability of autonomous systems in dynamic environments.

Title: Inflation with Diffusion: Efficient Temporal Adaptation for Text-to-Video Super-Resolution

• Authors: Xin Yuan, Jinoo Baek, Keyang Xu, Omer Tov, Hongliang Fei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10404
• Pdf URL: https://arxiv.org/pdf/2401.10404
• Copy Paste: [[2401.10404]] Inflation with Diffusion: Efficient Temporal Adaptation for Text-to-Video Super-Resolution(https://arxiv.org/abs/2401.10404)
• Keywords: diffusion
• Abstract: We propose an efficient diffusion-based text-to-video super-resolution (SR) tuning approach that leverages the readily learned capacity of pixel level image diffusion model to capture spatial information for video generation. To accomplish this goal, we design an efficient architecture by inflating the weightings of the text-to-image SR model into our video generation framework. Additionally, we incorporate a temporal adapter to ensure temporal coherence across video frames. We investigate different tuning approaches based on our inflated architecture and report trade-offs between computational costs and super-resolution quality. Empirical evaluation, both quantitative and qualitative, on the Shutterstock video dataset, demonstrates that our approach is able to perform text-to-video SR generation with good visual quality and temporal consistency. To evaluate temporal coherence, we also present visualizations in video format in https://drive.google.com/drive/folders/1YVc-KMSJqOrEUdQWVaI-Yfu8Vsfu_1aO?usp=sharing .

Title: Differentially Private and Adversarially Robust Machine Learning: An Empirical Evaluation

• Authors: Janvi Thakkar, Giulio Zizzo, Sergio Maffeis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10405
• Pdf URL: https://arxiv.org/pdf/2401.10405
• Copy Paste: [[2401.10405]] Differentially Private and Adversarially Robust Machine Learning: An Empirical Evaluation(https://arxiv.org/abs/2401.10405)
• Keywords: security, privacy, defense, attack, robust, membership infer
• Abstract: Malicious adversaries can attack machine learning models to infer sensitive information or damage the system by launching a series of evasion attacks. Although various work addresses privacy and security concerns, they focus on individual defenses, but in practice, models may undergo simultaneous attacks. This study explores the combination of adversarial training and differentially private training to defend against simultaneous attacks. While differentially-private adversarial training, as presented in DP-Adv, outperforms the other state-of-the-art methods in performance, it lacks formal privacy guarantees and empirical validation. Thus, in this work, we benchmark the performance of this technique using a membership inference attack and empirically show that the resulting approach is as private as non-robust private models. This work also highlights the need to explore privacy guarantees in dynamic training paradigms.

Title: Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?

• Authors: Marcio Fonseca, Shay B. Cohen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10415
• Pdf URL: https://arxiv.org/pdf/2401.10415
• Copy Paste: [[2401.10415]] Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?(https://arxiv.org/abs/2401.10415)
• Keywords: large language model
• Abstract: In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries such as paper reviews, abstracts, and lay summaries. By controlling stylistic features, we find that non-fine-tuned LLMs outperform humans in the MuP review generation task, both in terms of similarity to reference summaries and human preferences. Also, we show that we can improve the controllability of LLMs with keyword-based classifier-free guidance (CFG) while achieving lexical overlap comparable to strong fine-tuned baselines on arXiv and PubMed. However, our results also indicate that LLMs cannot consistently generate long summaries with more than 8 sentences. Furthermore, these models exhibit limited capacity to produce highly abstractive lay summaries. Although LLMs demonstrate strong generic summarization competency, sophisticated content control without costly fine-tuning remains an open problem for domain-specific applications.

Title: Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

• Authors: Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen, EnSiong Chng
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.10446
• Pdf URL: https://arxiv.org/pdf/2401.10446
• Copy Paste: [[2401.10446]] Large Language Models are Efficient Learners of Noise-Robust Speech Recognition(https://arxiv.org/abs/2401.10446)
• Keywords: robust, generative, large language model
• Abstract: Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which leverages the rich linguistic knowledge and powerful reasoning ability of LLMs to improve recognition results. The latest work proposes a GER benchmark with HyPoradise dataset to learn the mapping from ASR N-best hypotheses to ground-truth transcription by efficient LLM finetuning, which shows great effectiveness but lacks specificity on noise-robust ASR. In this work, we extend the benchmark to noisy conditions and investigate if we can teach LLMs to perform denoising for GER just like what robust ASR do}, where one solution is introducing noise information as a conditioner into LLM. However, directly incorporating noise embeddings from audio encoder could harm the LLM tuning due to cross-modality gap. To this end, we propose to extract a language-space noise embedding from the N-best list to represent the noise conditions of source speech, which can promote the denoising process in GER. Furthermore, in order to enhance its representation ability of audio noise, we design a knowledge distillation (KD) approach via mutual information estimation to distill the real noise information in audio embeddings to our language embedding. Experiments on various latest LLMs demonstrate our approach achieves a new breakthrough with up to 53.9% correction improvement in terms of word error rate while with limited training data. Analysis shows that our language-space noise embedding can well represent the noise conditions of source speech, under which off-the-shelf LLMs show strong ability of language-space denoising.

Title: Investigating Training Strategies and Model Robustness of Low-Rank Adaptation for Language Modeling in Speech Recognition

• Authors: Yu Yu, Chao-Han Huck Yang, Tuan Dinh, Sungho Ryu, Jari Kolehmainen, Roger Ren, Denis Filimonov, Prashanth G. Shivakumar, Ankur Gandhe, Ariya Rastow, Jia Xu, Ivan Bulyko, Andreas Stolcke
• Subjects: cs.CL, cs.AI, cs.LG, cs.NE, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.10447
• Pdf URL: https://arxiv.org/pdf/2401.10447
• Copy Paste: [[2401.10447]] Investigating Training Strategies and Model Robustness of Low-Rank Adaptation for Language Modeling in Speech Recognition(https://arxiv.org/abs/2401.10447)
• Keywords: robust
• Abstract: The use of low-rank adaptation (LoRA) with frozen pretrained language models (PLMs) has become increasing popular as a mainstream, resource-efficient modeling approach for memory-constrained hardware. In this study, we first explore how to enhance model performance by introducing various LoRA training strategies, achieving relative word error rate reductions of 3.50\% on the public Librispeech dataset and of 3.67\% on an internal dataset in the messaging domain. To further characterize the stability of LoRA-based second-pass speech recognition models, we examine robustness against input perturbations. These perturbations are rooted in homophone replacements and a novel metric called N-best Perturbation-based Rescoring Robustness (NPRR), both designed to measure the relative degradation in the performance of rescoring models. Our experimental results indicate that while advanced variants of LoRA, such as dynamic rank-allocated LoRA, lead to performance degradation in $1$-best perturbation, they alleviate the degradation in $N$-best perturbation. This finding is in comparison to fully-tuned models and vanilla LoRA tuning baselines, suggesting that a comprehensive selection is needed when using LoRA-based adaptation for compute-cost savings and robust language modeling.

Title: Learning to Robustly Reconstruct Low-light Dynamic Scenes from Spike Streams

• Authors: Liwen Hu, Ziluo Ding, Mianzhi Liu, Lei Ma, Tiejun Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10461
• Pdf URL: https://arxiv.org/pdf/2401.10461
• Copy Paste: [[2401.10461]] Learning to Robustly Reconstruct Low-light Dynamic Scenes from Spike Streams(https://arxiv.org/abs/2401.10461)
• Keywords: robust
• Abstract: As a neuromorphic sensor with high temporal resolution, spike camera can generate continuous binary spike streams to capture per-pixel light intensity. We can use reconstruction methods to restore scene details in high-speed scenarios. However, due to limited information in spike streams, low-light scenes are difficult to effectively reconstruct. In this paper, we propose a bidirectional recurrent-based reconstruction framework, including a Light-Robust Representation (LR-Rep) and a fusion module, to better handle such extreme conditions. LR-Rep is designed to aggregate temporal information in spike streams, and a fusion module is utilized to extract temporal features. Additionally, we have developed a reconstruction benchmark for high-speed low-light scenes. Light sources in the scenes are carefully aligned to real-world conditions. Experimental results demonstrate the superiority of our method, which also generalizes well to real spike streams. Related codes and proposed datasets will be released after publication.

Title: DeepEdit: Knowledge Editing as Decoding with Constraints

• Authors: Yiwei Wang, Muhao Chen, Nanyun Peng, Kai-Wei Chang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10471
• Pdf URL: https://arxiv.org/pdf/2401.10471
• Copy Paste: [[2401.10471]] DeepEdit: Knowledge Editing as Decoding with Constraints(https://arxiv.org/abs/2401.10471)
• Keywords: large language model
• Abstract: We develop a new perspective of knowledge editing for large language models (LLMs) as decoding with constraints. We propose DeepEdit (Depth-first Search based Progressive Decoding for Knowledge Editing), a neuro-symbolic method that improves knowledge editing with better coherence of reasoning, relevance to the question, and awareness of updated knowledge. DeepEdit can be flexibly applied to all black-box LLMs: it does not require any access to the model parameters, representations, or output vocabulary distributions. DeepEdit progressively produces the high-quality reasoning steps towards effective knowledge editing. It utilizes a depth-first search to revise the LLMs' output, which improves the output's informativeness to the input question and awareness of the updated knowledge. Qualitatively, DeepEdit effectively controls LLMs to produce more succinct reasoning in accord with knowledge editing. Quantitatively, DeepEdit yields significant gains on MQuaKE, a challenging multi-hop question-answering dataset with knowledge editing. We release the source code at https://github.com/wangywUST/DeepEdit.

Title: Name Tagging Under Domain Shift via Metric Learning for Life Sciences

• Authors: Hongyi Liu, Qingyun Wang, Payam Karisani, Heng Ji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10472
• Pdf URL: https://arxiv.org/pdf/2401.10472
• Copy Paste: [[2401.10472]] Name Tagging Under Domain Shift via Metric Learning for Life Sciences(https://arxiv.org/abs/2401.10472)
• Keywords: extraction, large language model
• Abstract: Name tagging is a key component of Information Extraction (IE), particularly in scientific domains such as biomedicine and chemistry, where large language models (LLMs), e.g., ChatGPT, fall short. We investigate the applicability of transfer learning for enhancing a name tagging model trained in the biomedical domain (the source domain) to be used in the chemical domain (the target domain). A common practice for training such a model in a few-shot learning setting is to pretrain the model on the labeled source data, and then, to finetune it on a hand-full of labeled target examples. In our experiments we observed that such a model is prone to mis-labeling the source entities, which can often appear in the text, as the target entities. To alleviate this problem, we propose a model to transfer the knowledge from the source domain to the target domain, however, at the same time, to project the source entities and target entities into separate regions of the feature space. This diminishes the risk of mis-labeling the source entities as the target entities. Our model consists of two stages: 1) entity grouping in the source domain, which incorporates knowledge from annotated events to establish relations between entities, and 2) entity discrimination in the target domain, which relies on pseudo labeling and contrastive learning to enhance discrimination between the entities in the two domains. We carry out our extensive experiments across three source and three target datasets, and demonstrate that our method outperforms the baselines, in some scenarios by 5\% absolute value.

Title: Budgeted Online Model Selection and Fine-Tuning via Federated Learning

• Authors: Pouya M. Ghari, Yanning Shen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10478
• Pdf URL: https://arxiv.org/pdf/2401.10478
• Copy Paste: [[2401.10478]] Budgeted Online Model Selection and Fine-Tuning via Federated Learning(https://arxiv.org/abs/2401.10478)
• Keywords: federate
• Abstract: Online model selection involves selecting a model from a set of candidate models 'on the fly' to perform prediction on a stream of data. The choice of candidate models henceforth has a crucial impact on the performance. Although employing a larger set of candidate models naturally leads to more flexibility in model selection, this may be infeasible in cases where prediction tasks are performed on edge devices with limited memory. Faced with this challenge, the present paper proposes an online federated model selection framework where a group of learners (clients) interacts with a server with sufficient memory such that the server stores all candidate models. However, each client only chooses to store a subset of models that can be fit into its memory and performs its own prediction task using one of the stored models. Furthermore, employing the proposed algorithm, clients and the server collaborate to fine-tune models to adapt them to a non-stationary environment. Theoretical analysis proves that the proposed algorithm enjoys sub-linear regret with respect to the best model in hindsight. Experiments on real datasets demonstrate the effectiveness of the proposed algorithm.

Title: Knowledge Fusion of Large Language Models

• Authors: Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, Shuming Shi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10491
• Pdf URL: https://arxiv.org/pdf/2401.10491
• Copy Paste: [[2401.10491]] Knowledge Fusion of Large Language Models(https://arxiv.org/abs/2401.10491)
• Keywords: generative, large language model
• Abstract: While training large language models (LLMs) from scratch can generate models with distinct functionalities and strengths, it comes at significant costs and may result in redundant capabilities. Alternatively, a cost-effective and compelling approach is to merge existing pre-trained LLMs into a more potent model. However, due to the varying architectures of these LLMs, directly blending their weights is impractical. In this paper, we introduce the notion of knowledge fusion for LLMs, aimed at combining the capabilities of existing LLMs and transferring them into a single LLM. By leveraging the generative distributions of source LLMs, we externalize their collective knowledge and unique strengths, thereby potentially elevating the capabilities of the target model beyond those of any individual source LLM. We validate our approach using three popular LLMs with different architectures--Llama-2, MPT, and OpenLLaMA--across various benchmarks and tasks. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation. Our code, model weights, and data are public at \url{https://github.com/fanqiwan/FuseLLM}.

Title: Enhancing medical vision-language contrastive learning via inter-matching relation modelling

• Authors: Mingjian Li, Mingyuan Meng, Michael Fulham, David Dagan Feng, Lei Bi, Jinman Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10501
• Pdf URL: https://arxiv.org/pdf/2401.10501
• Copy Paste: [[2401.10501]] Enhancing medical vision-language contrastive learning via inter-matching relation modelling(https://arxiv.org/abs/2401.10501)
• Keywords: segmentation
• Abstract: Medical image representations can be learned through medical vision-language contrastive learning (mVLCL) where medical imaging reports are used as weak supervision through image-text alignment. These learned image representations can be transferred to and benefit various downstream medical vision tasks such as disease classification and segmentation. Recent mVLCL methods attempt to align image sub-regions and the report keywords as local-matchings. However, these methods aggregate all local-matchings via simple pooling operations while ignoring the inherent relations between them. These methods therefore fail to reason between local-matchings that are semantically related, e.g., local-matchings that correspond to the disease word and the location word (semantic-relations), and also fail to differentiate such clinically important local-matchings from others that correspond to less meaningful words, e.g., conjunction words (importance-relations). Hence, we propose a mVLCL method that models the inter-matching relations between local-matchings via a relation-enhanced contrastive learning framework (RECLF). In RECLF, we introduce a semantic-relation reasoning module (SRM) and an importance-relation reasoning module (IRM) to enable more fine-grained report supervision for image representation learning. We evaluated our method using four public benchmark datasets on four downstream tasks, including segmentation, zero-shot classification, supervised classification, and cross-modal retrieval. Our results demonstrated the superiority of our RECLF over the state-of-the-art mVLCL methods with consistent improvements across single-modal and cross-modal tasks. These results suggest that our RECLF, by modelling the inter-matching relations, can learn improved medical image representations with better generalization capabilities.

Title: FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysis

• Authors: Chao Zhang, Yuren Mao, Yijiang Fan, Yu Mi, Yunjun Gao, Lu Chen, Dongfang Lou, Jinshu Lin
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2401.10506
• Pdf URL: https://arxiv.org/pdf/2401.10506
• Copy Paste: [[2401.10506]] FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysis(https://arxiv.org/abs/2401.10506)
• Keywords: large language model
• Abstract: Text-to-SQL, which provides zero-code interface for operating relational databases, has gained much attention in financial analysis; because, financial professionals may not well-skilled in SQL programming. However, until now, there is no practical Text-to-SQL benchmark dataset for financial analysis, and existing Text-to-SQL methods have not considered the unique characteristics of databases in financial applications, such as commonly existing wide tables. To address these issues, we collect a practical Text-to-SQL benchmark dataset and propose a model-agnostic Large Language Model (LLMs)-based Text-to-SQL framework for financial analysis. The benchmark dataset, BULL, is collected from the practical financial analysis business of Hundsun Technologies Inc., including databases for fund, stock, and macro economy. Besides, the proposed LLMs-based Text-to-SQL framework, FinSQL, provides a systematic treatment for financial Text-to-SQL from the perspectives of prompt construction, parameter-efficient fine-tuning and output calibration. Extensive experimental results on BULL demonstrate that FinSQL achieves the state-of-the-art Text-to-SQL performance at a small cost; furthermore, FinSQL can bring up to 36.64% performance improvement in scenarios requiring few-shot cross-database model transfer.

Title: GMC-IQA: Exploiting Global-correlation and Mean-opinion Consistency for No-reference Image Quality Assessment

• Authors: Zewen Chen, Juan Wang, Bing Li, Chunfeng Yuan, Weiming Hu, Junxian Liu, Peng Li, Yan Wang, Youqun Zhang, Congxuan Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10511
• Pdf URL: https://arxiv.org/pdf/2401.10511
• Copy Paste: [[2401.10511]] GMC-IQA: Exploiting Global-correlation and Mean-opinion Consistency for No-reference Image Quality Assessment(https://arxiv.org/abs/2401.10511)
• Keywords: robust
• Abstract: Due to the subjective nature of image quality assessment (IQA), assessing which image has better quality among a sequence of images is more reliable than assigning an absolute mean opinion score for an image. Thus, IQA models are evaluated by global correlation consistency (GCC) metrics like PLCC and SROCC, rather than mean opinion consistency (MOC) metrics like MAE and MSE. However, most existing methods adopt MOC metrics to define their loss functions, due to the infeasible computation of GCC metrics during training. In this work, we construct a novel loss function and network to exploit Global-correlation and Mean-opinion Consistency, forming a GMC-IQA framework. Specifically, we propose a novel GCC loss by defining a pairwise preference-based rank estimation to solve the non-differentiable problem of SROCC and introducing a queue mechanism to reserve previous data to approximate the global results of the whole data. Moreover, we propose a mean-opinion network, which integrates diverse opinion features to alleviate the randomness of weight learning and enhance the model robustness. Experiments indicate that our method outperforms SOTA methods on multiple authentic datasets with higher accuracy and generalization. We also adapt the proposed loss to various networks, which brings better performance and more stable training.

Title: Exploring Color Invariance through Image-Level Ensemble Learning

• Authors: Yunpeng Gong, Jiaquan Li, Lifei Chen, Min Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10512
• Pdf URL: https://arxiv.org/pdf/2401.10512
• Copy Paste: [[2401.10512]] Exploring Color Invariance through Image-Level Ensemble Learning(https://arxiv.org/abs/2401.10512)
• Keywords: robust, interpretability, segmentation
• Abstract: In the field of computer vision, the persistent presence of color bias, resulting from fluctuations in real-world lighting and camera conditions, presents a substantial challenge to the robustness of models. This issue is particularly pronounced in complex wide-area surveillance scenarios, such as person re-identification and industrial dust segmentation, where models often experience a decline in performance due to overfitting on color information during training, given the presence of environmental variations. Consequently, there is a need to effectively adapt models to cope with the complexities of camera conditions. To address this challenge, this study introduces a learning strategy named Random Color Erasing, which draws inspiration from ensemble learning. This strategy selectively erases partial or complete color information in the training data without disrupting the original image structure, thereby achieving a balanced weighting of color features and other features within the neural network. This approach mitigates the risk of overfitting and enhances the model's ability to handle color variation, thereby improving its overall robustness. The approach we propose serves as an ensemble learning strategy, characterized by robust interpretability. A comprehensive analysis of this methodology is presented in this paper. Across various tasks such as person re-identification and semantic segmentation, our approach consistently improves strong baseline methods. Notably, in comparison to existing methods that prioritize color robustness, our strategy significantly enhances performance in cross-domain scenarios. The code available at \url{https://github.com/layumi/Person\_reID\_baseline\_pytorch/blob/master/random\_erasing.py} or \url{https://github.com/finger-monkey/Data-Augmentation}.

Title: Cross-lingual Editing in Multilingual Language Models

• Authors: Himanshu Beniwal, Kowsik Nandagopan D, Mayank Singh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10521
• Pdf URL: https://arxiv.org/pdf/2401.10521
• Copy Paste: [[2401.10521]] Cross-lingual Editing in Multilingual Language Models(https://arxiv.org/abs/2401.10521)
• Keywords: large language model
• Abstract: The training of large language models (LLMs) necessitates substantial data and computational resources, and updating outdated LLMs entails significant efforts and resources. While numerous model editing techniques (METs) have emerged to efficiently update model outputs without retraining, their effectiveness in multilingual LLMs, where knowledge is stored in diverse languages, remains an underexplored research area. This research paper introduces the cross-lingual model editing (\textbf{XME}) paradigm, wherein a fact is edited in one language, and the subsequent update propagation is observed across other languages. To investigate the XME paradigm, we conducted experiments using BLOOM, mBERT, and XLM-RoBERTa using the two writing scripts: \textit{Latin} (English, French, and Spanish) and \textit{Indic} (Hindi, Gujarati, and Bengali). The results reveal notable performance limitations of state-of-the-art METs under the XME setting, mainly when the languages involved belong to two distinct script families. These findings highlight the need for further research and development of XME techniques to address these challenges. For more comprehensive information, the dataset used in this research and the associated code are publicly available at the following URL\url{https://github.com/lingo-iitgn/XME}.

Title: On mitigating stability-plasticity dilemma in CLIP-guided image morphing via geodesic distillation loss

• Authors: Yeongtak Oh, Saehyung Lee, Uiwon Hwang, Sungroh Yoon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10526
• Pdf URL: https://arxiv.org/pdf/2401.10526
• Copy Paste: [[2401.10526]] On mitigating stability-plasticity dilemma in CLIP-guided image morphing via geodesic distillation loss(https://arxiv.org/abs/2401.10526)
• Keywords: generative
• Abstract: Large-scale language-vision pre-training models, such as CLIP, have achieved remarkable text-guided image morphing results by leveraging several unconditional generative models. However, existing CLIP-guided image morphing methods encounter difficulties when morphing photorealistic images. Specifically, existing guidance fails to provide detailed explanations of the morphing regions within the image, leading to misguidance. In this paper, we observed that such misguidance could be effectively mitigated by simply using a proper regularization loss. Our approach comprises two key components: 1) a geodesic cosine similarity loss that minimizes inter-modality features (i.e., image and text) on a projected subspace of CLIP space, and 2) a latent regularization loss that minimizes intra-modality features (i.e., image and image) on the image manifold. By replacing the na\"ive directional CLIP loss in a drop-in replacement manner, our method achieves superior morphing results on both images and videos for various benchmarks, including CLIP-inversion.

Title: Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences

• Authors: Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, Huaxiu Yao, Furong Huang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10529
• Pdf URL: https://arxiv.org/pdf/2401.10529
• Copy Paste: [[2401.10529]] Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences(https://arxiv.org/abs/2401.10529)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have demonstrated proficiency in handling a variety of visual-language tasks. However, current MLLM benchmarks are predominantly designed to evaluate reasoning based on static information about a single image, and the ability of modern MLLMs to extrapolate from image sequences, which is essential for understanding our ever-changing world, has been less investigated. To address this challenge, this paper introduces Mementos, a new benchmark designed to assess MLLMs' sequential image reasoning abilities. Mementos features 4,761 diverse image sequences with varying lengths. We also employ a GPT-4 assisted method to evaluate MLLM reasoning performance. Through a careful evaluation of nine recent MLLMs on Mementos, including GPT-4V and Gemini, we find that they struggle to accurately describe dynamic information about given image sequences, often leading to hallucinations/misrepresentations of objects and their corresponding behaviors. Our quantitative analysis and case studies identify three key factors impacting MLLMs' sequential image reasoning: the correlation between object and behavioral hallucinations, the influence of cooccurring behaviors, and the compounding impact of behavioral hallucinations. Our dataset is available at https://github.com/umd-huang-lab/Mementos.

Title: Speech Swin-Transformer: Exploring a Hierarchical Transformer with Shifted Windows for Speech Emotion Recognition

• Authors: Yong Wang, Cheng Lu, Hailun Lian, Yan Zhao, Björn Schuller, Yuan Zong, Wenming Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10536
• Pdf URL: https://arxiv.org/pdf/2401.10536
• Copy Paste: [[2401.10536]] Speech Swin-Transformer: Exploring a Hierarchical Transformer with Shifted Windows for Speech Emotion Recognition(https://arxiv.org/abs/2401.10536)
• Keywords: transformer
• Abstract: Swin-Transformer has demonstrated remarkable success in computer vision by leveraging its hierarchical feature representation based on Transformer. In speech signals, emotional information is distributed across different scales of speech features, e.\,g., word, phrase, and utterance. Drawing above inspiration, this paper presents a hierarchical speech Transformer with shifted windows to aggregate multi-scale emotion features for speech emotion recognition (SER), called Speech Swin-Transformer. Specifically, we first divide the speech spectrogram into segment-level patches in the time domain, composed of multiple frame patches. These segment-level patches are then encoded using a stack of Swin blocks, in which a local window Transformer is utilized to explore local inter-frame emotional information across frame patches of each segment patch. After that, we also design a shifted window Transformer to compensate for patch correlations near the boundaries of segment patches. Finally, we employ a patch merging operation to aggregate segment-level emotional features for hierarchical speech representation by expanding the receptive field of Transformer from frame-level to segment-level. Experimental results demonstrate that our proposed Speech Swin-Transformer outperforms the state-of-the-art methods.

Title: PhoGAD: Graph-based Anomaly Behavior Detection with Persistent Homology Optimization

• Authors: Ziqi Yuan, Haoyi Zhou, Tianyu Chen, Jianxin Li
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.10547
• Pdf URL: https://arxiv.org/pdf/2401.10547
• Copy Paste: [[2401.10547]] PhoGAD: Graph-based Anomaly Behavior Detection with Persistent Homology Optimization(https://arxiv.org/abs/2401.10547)
• Keywords: attack, robust
• Abstract: A multitude of toxic online behaviors, ranging from network attacks to anonymous traffic and spam, have severely disrupted the smooth operation of networks. Due to the inherent sender-receiver nature of network behaviors, graph-based frameworks are commonly used for detecting anomalous behaviors. However, in real-world scenarios, the boundary between normal and anomalous behaviors tends to be ambiguous. The local heterophily of graphs interferes with the detection, and existing methods based on nodes or edges introduce unwanted noise into representation results, thereby impacting the effectiveness of detection. To address these issues, we propose PhoGAD, a graph-based anomaly detection framework. PhoGAD leverages persistent homology optimization to clarify behavioral boundaries. Building upon this, the weights of adjacent edges are designed to mitigate the effects of local heterophily. Subsequently, to tackle the noise problem, we conduct a formal analysis and propose a disentangled representation-based explicit embedding method, ultimately achieving anomaly behavior detection. Experiments on intrusion, traffic, and spam datasets verify that PhoGAD has surpassed the performance of state-of-the-art (SOTA) frameworks in detection efficacy. Notably, PhoGAD demonstrates robust detection even with diminished anomaly proportions, highlighting its applicability to real-world scenarios. The analysis of persistent homology demonstrates its effectiveness in capturing the topological structure formed by normal edge features. Additionally, ablation experiments validate the effectiveness of the innovative mechanisms integrated within PhoGAD.

Title: Symbol as Points: Panoptic Symbol Spotting via Point-based Representation

• Authors: Wenlong Liu, Tianyu Yang, Yuhan Wang, Qizhi Yu, Lei Zhang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.10556
• Pdf URL: https://arxiv.org/pdf/2401.10556
• Copy Paste: [[2401.10556]] Symbol as Points: Panoptic Symbol Spotting via Point-based Representation(https://arxiv.org/abs/2401.10556)
• Keywords: transformer, segmentation
• Abstract: This work studies the problem of panoptic symbol spotting, which is to spot and parse both countable object instances (windows, doors, tables, etc.) and uncountable stuff (wall, railing, etc.) from computer-aided design (CAD) drawings. Existing methods typically involve either rasterizing the vector graphics into images and using image-based methods for symbol spotting, or directly building graphs and using graph neural networks for symbol recognition. In this paper, we take a different approach, which treats graphic primitives as a set of 2D points that are locally connected and use point cloud segmentation methods to tackle it. Specifically, we utilize a point transformer to extract the primitive features and append a mask2former-like spotting head to predict the final output. To better use the local connection information of primitives and enhance their discriminability, we further propose the attention with connection module (ACM) and contrastive connection learning scheme (CCL). Finally, we propose a KNN interpolation mechanism for the mask attention module of the spotting head to better handle primitive mask downsampling, which is primitive-level in contrast to pixel-level for the image. Our approach, named SymPoint, is simple yet effective, outperforming recent state-of-the-art method GAT-CADNet by an absolute increase of 9.6% PQ and 10.4% RQ on the FloorPlanCAD dataset. The source code and models will be available at https://github.com/nicehuster/SymPoint.

Title: Dream360: Diverse and Immersive Outdoor Virtual Scene Creation via Transformer-Based 360 Image Outpainting

• Authors: Hao Ai, Zidong Cao, Haonan Lu, Chen Chen, Jian Ma, Pengyuan Zhou, Tae-Kyun Kim, Pan Hui, Lin Wang
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.10564
• Pdf URL: https://arxiv.org/pdf/2401.10564
• Copy Paste: [[2401.10564]] Dream360: Diverse and Immersive Outdoor Virtual Scene Creation via Transformer-Based 360 Image Outpainting(https://arxiv.org/abs/2401.10564)
• Keywords: transformer
• Abstract: 360 images, with a field-of-view (FoV) of 180x360, provide immersive and realistic environments for emerging virtual reality (VR) applications, such as virtual tourism, where users desire to create diverse panoramic scenes from a narrow FoV photo they take from a viewpoint via portable devices. It thus brings us to a technical challenge: `How to allow the users to freely create diverse and immersive virtual scenes from a narrow FoV image with a specified viewport?' To this end, we propose a transformer-based 360 image outpainting framework called Dream360, which can generate diverse, high-fidelity, and high-resolution panoramas from user-selected viewports, considering the spherical properties of 360 images. Compared with existing methods, e.g., [3], which primarily focus on inputs with rectangular masks and central locations while overlooking the spherical property of 360 images, our Dream360 offers higher outpainting flexibility and fidelity based on the spherical representation. Dream360 comprises two key learning stages: (I) codebook-based panorama outpainting via Spherical-VQGAN (S-VQGAN), and (II) frequency-aware refinement with a novel frequency-aware consistency loss. Specifically, S-VQGAN learns a sphere-specific codebook from spherical harmonic (SH) values, providing a better representation of spherical data distribution for scene modeling. The frequency-aware refinement matches the resolution and further improves the semantic consistency and visual fidelity of the generated results. Our Dream360 achieves significantly lower Frechet Inception Distance (FID) scores and better visual fidelity than existing methods. We also conducted a user study involving 15 participants to interactively evaluate the quality of the generated results in VR, demonstrating the flexibility and superiority of our Dream360 framework.

Title: Robust Multi-Modal Density Estimation

• Authors: Anna Mészáros, Julian F. Schumann, Javier Alonso-Mora, Arkady Zgonnikov, Jens Kober
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.10566
• Pdf URL: https://arxiv.org/pdf/2401.10566
• Copy Paste: [[2401.10566]] Robust Multi-Modal Density Estimation(https://arxiv.org/abs/2401.10566)
• Keywords: robust
• Abstract: Development of multi-modal, probabilistic prediction models has lead to a need for comprehensive evaluation metrics. While several metrics can characterize the accuracy of machine-learned models (e.g., negative log-likelihood, Jensen-Shannon divergence), these metrics typically operate on probability densities. Applying them to purely sample-based prediction models thus requires that the underlying density function is estimated. However, common methods such as kernel density estimation (KDE) have been demonstrated to lack robustness, while more complex methods have not been evaluated in multi-modal estimation problems. In this paper, we present ROME (RObust Multi-modal density Estimator), a non-parametric approach for density estimation which addresses the challenge of estimating multi-modal, non-normal, and highly correlated distributions. ROME utilizes clustering to segment a multi-modal set of samples into multiple uni-modal ones and then combines simple KDE estimates obtained for individual clusters in a single multi-modal estimate. We compared our approach to state-of-the-art methods for density estimation as well as ablations of ROME, showing that it not only outperforms established methods but is also more robust to a variety of distributions. Our results demonstrate that ROME can overcome the issues of over-fitting and over-smoothing exhibited by other estimators, promising a more robust evaluation of probabilistic machine learning models.

Title: PHOENIX: Open-Source Language Adaption for Direct Preference Optimization

• Authors: Matthias Uhlig, Sigurd Schacht, Sudarshan Kamath Barkur
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10580
• Pdf URL: https://arxiv.org/pdf/2401.10580
• Copy Paste: [[2401.10580]] PHOENIX: Open-Source Language Adaption for Direct Preference Optimization(https://arxiv.org/abs/2401.10580)
• Keywords: large language model
• Abstract: Large language models have gained immense importance in recent years and have demonstrated outstanding results in solving various tasks. However, despite these achievements, many questions remain unanswered in the context of large language models. Besides the optimal use of the models for inference and the alignment of the results to the desired specifications, the transfer of models to other languages is still an underdeveloped area of research. The recent publication of models such as Llama-2 and Zephyr has provided new insights into architectural improvements and the use of human feedback. However, insights into adapting these techniques to other languages remain scarce. In this paper, we build on latest improvements and apply the Direct Preference Optimization(DPO) approach to the German language. The model is available at https://huggingface.co/DRXD1000/Phoenix.

Title: Exploiting Kubernetes' Image Pull Implementation to Deny Node Availability

• Authors: Luis Augusto Dias Knob, Matteo Franzil, Domenico Siracusa
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.10582
• Pdf URL: https://arxiv.org/pdf/2401.10582
• Copy Paste: [[2401.10582]] Exploiting Kubernetes' Image Pull Implementation to Deny Node Availability(https://arxiv.org/abs/2401.10582)
• Keywords: security, attack
• Abstract: Kubernetes (K8s) has grown in popularity over the past few years to become the de-facto standard for container orchestration in cloud-native environments. While research is not new to topics such as containerization and access control security, the Application Programming Interface (API) interactions between K8s and its runtime interfaces have not been studied thoroughly. In particular, the CRI-API is responsible for abstracting the container runtime, managing the creation and lifecycle of containers along with the downloads of the respective images. However, this decoupling of concerns and the abstraction of the container runtime renders K8s unaware of the status of the downloading process of the container images, obstructing the monitoring of the resources allocated to such process. In this paper, we discuss how this lack of status information can be exploited as a Denial of Service attack in a K8s cluster. We show that such attacks can generate up to 95% average CPU usage, prevent downloading new container images, and increase I/O and network usage for a potentially unlimited amount of time. Finally, we propose two possible mitigation strategies: one, implemented as a stopgap solution, and another, requiring more radical architectural changes in the relationship between K8s and the CRI-API.

Title: PuriDefense: Randomized Local Implicit Adversarial Purification for Defending Black-box Query-based Attacks

• Authors: Ping Guo, Zhiyuan Yang, Xi Lin, Qingchuan Zhao, Qingfu Zhang
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10586
• Pdf URL: https://arxiv.org/pdf/2401.10586
• Copy Paste: [[2401.10586]] PuriDefense: Randomized Local Implicit Adversarial Purification for Defending Black-box Query-based Attacks(https://arxiv.org/abs/2401.10586)
• Keywords: defense, attack, robust
• Abstract: Black-box query-based attacks constitute significant threats to Machine Learning as a Service (MLaaS) systems since they can generate adversarial examples without accessing the target model's architecture and parameters. Traditional defense mechanisms, such as adversarial training, gradient masking, and input transformations, either impose substantial computational costs or compromise the test accuracy of non-adversarial inputs. To address these challenges, we propose an efficient defense mechanism, PuriDefense, that employs random patch-wise purifications with an ensemble of lightweight purification models at a low level of inference cost. These models leverage the local implicit function and rebuild the natural image manifold. Our theoretical analysis suggests that this approach slows down the convergence of query-based attacks by incorporating randomness into purifications. Extensive experiments on CIFAR-10 and ImageNet validate the effectiveness of our proposed purifier-based defense mechanism, demonstrating significant improvements in robustness against query-based attacks.

Title: Adversarially Robust Signed Graph Contrastive Learning from Balance Augmentation

• Authors: Jialong Zhou, Xing Ai, Yuni Lai, Kai Zhou
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.10590
• Pdf URL: https://arxiv.org/pdf/2401.10590
• Copy Paste: [[2401.10590]] Adversarially Robust Signed Graph Contrastive Learning from Balance Augmentation(https://arxiv.org/abs/2401.10590)
• Keywords: defense, attack, robust
• Abstract: Signed graphs consist of edges and signs, which can be separated into structural information and balance-related information, respectively. Existing signed graph neural networks (SGNNs) typically rely on balance-related information to generate embeddings. Nevertheless, the emergence of recent adversarial attacks has had a detrimental impact on the balance-related information. Similar to how structure learning can restore unsigned graphs, balance learning can be applied to signed graphs by improving the balance degree of the poisoned graph. However, this approach encounters the challenge "Irreversibility of Balance-related Information" - while the balance degree improves, the restored edges may not be the ones originally affected by attacks, resulting in poor defense effectiveness. To address this challenge, we propose a robust SGNN framework called Balance Augmented-Signed Graph Contrastive Learning (BA-SGCL), which combines Graph Contrastive Learning principles with balance augmentation techniques. Experimental results demonstrate that BA-SGCL not only enhances robustness against existing adversarial attacks but also achieves superior performance on link sign prediction task across various datasets.

Title: M2ORT: Many-To-One Regression Transformer for Spatial Transcriptomics Prediction from Histopathology Images

• Authors: Hongyi Wang, Xiuju Du, Jing Liu, Shuyi Ouyang, Yen-Wei Chen, Lanfen Lin
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2401.10608
• Pdf URL: https://arxiv.org/pdf/2401.10608
• Copy Paste: [[2401.10608]] M2ORT: Many-To-One Regression Transformer for Spatial Transcriptomics Prediction from Histopathology Images(https://arxiv.org/abs/2401.10608)
• Keywords: transformer
• Abstract: The advancement of Spatial Transcriptomics (ST) has facilitated the spatially-aware profiling of gene expressions based on histopathology images. Although ST data offers valuable insights into the micro-environment of tumors, its acquisition cost remains expensive. Therefore, directly predicting the ST expressions from digital pathology images is desired. Current methods usually adopt existing regression backbones for this task, which ignore the inherent multi-scale hierarchical data structure of digital pathology images. To address this limit, we propose M2ORT, a many-to-one regression Transformer that can accommodate the hierarchical structure of the pathology images through a decoupled multi-scale feature extractor. Different from traditional models that are trained with one-to-one image-label pairs, M2ORT accepts multiple pathology images of different magnifications at a time to jointly predict the gene expressions at their corresponding common ST spot, aiming at learning a many-to-one relationship through training. We have tested M2ORT on three public ST datasets and the experimental results show that M2ORT can achieve state-of-the-art performance with fewer parameters and floating-point operations (FLOPs). The code is available at: https://github.com/Dootmaan/M2ORT/.

Title: Interventional Fairness on Partially Known Causal Graphs: A Constrained Optimization Approach

• Authors: Aoqi Zuo, Yiqing Li, Susan Wei, Mingming Gong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10632
• Pdf URL: https://arxiv.org/pdf/2401.10632
• Copy Paste: [[2401.10632]] Interventional Fairness on Partially Known Causal Graphs: A Constrained Optimization Approach(https://arxiv.org/abs/2401.10632)
• Keywords: fair
• Abstract: Fair machine learning aims to prevent discrimination against individuals or sub-populations based on sensitive attributes such as gender and race. In recent years, causal inference methods have been increasingly used in fair machine learning to measure unfairness by causal effects. However, current methods assume that the true causal graph is given, which is often not true in real-world applications. To address this limitation, this paper proposes a framework for achieving causal fairness based on the notion of interventions when the true causal graph is partially known. The proposed approach involves modeling fair prediction using a Partially Directed Acyclic Graph (PDAG), specifically, a class of causal DAGs that can be learned from observational data combined with domain knowledge. The PDAG is used to measure causal fairness, and a constrained optimization problem is formulated to balance between fairness and accuracy. Results on both simulated and real-world datasets demonstrate the effectiveness of this method.

Title: Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Models

• Authors: Rima Hazra, Sayan Layek, Somnath Banerjee, Soujanya Poria
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10647
• Pdf URL: https://arxiv.org/pdf/2401.10647
• Copy Paste: [[2401.10647]] Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Models(https://arxiv.org/abs/2401.10647)
• Keywords: robust, large language model
• Abstract: In the rapidly advancing field of artificial intelligence, the concept of Red-Teaming or Jailbreaking large language models (LLMs) has emerged as a crucial area of study. This approach is especially significant in terms of assessing and enhancing the safety and robustness of these models. This paper investigates the intricate consequences of such modifications through model editing, uncovering a complex relationship between enhancing model accuracy and preserving its ethical integrity. Our in-depth analysis reveals a striking paradox: while injecting accurate information is crucial for model reliability, it can paradoxically destabilize the model's foundational framework, resulting in unpredictable and potentially unsafe behaviors. Additionally, we propose a benchmark dataset NicheHazardQA to investigate this unsafe behavior both within the same and cross topical domain. This aspect of our research sheds light on how the edits, impact the model's safety metrics and guardrails. Our findings show that model editing serves as a cost-effective tool for topical red-teaming by methodically applying targeted edits and evaluating the resultant model behavior

Title: Attentive Fusion: A Transformer-based Approach to Multimodal Hate Speech Detection

• Authors: Atanu Mandal, Gargi Roy, Amit Barman, Indranil Dutta, Sudip Kumar Naskar
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.10653
• Pdf URL: https://arxiv.org/pdf/2401.10653
• Copy Paste: [[2401.10653]] Attentive Fusion: A Transformer-based Approach to Multimodal Hate Speech Detection(https://arxiv.org/abs/2401.10653)
• Keywords: transformer
• Abstract: With the recent surge and exponential growth of social media usage, scrutinizing social media content for the presence of any hateful content is of utmost importance. Researchers have been diligently working since the past decade on distinguishing between content that promotes hatred and content that does not. Traditionally, the main focus has been on analyzing textual content. However, recent research attempts have also commenced into the identification of audio-based content. Nevertheless, studies have shown that relying solely on audio or text-based content may be ineffective, as recent upsurge indicates that individuals often employ sarcasm in their speech and writing. To overcome these challenges, we present an approach to identify whether a speech promotes hate or not utilizing both audio and textual representations. Our methodology is based on the Transformer framework that incorporates both audio and text sampling, accompanied by our very own layer called "Attentive Fusion". The results of our study surpassed previous state-of-the-art techniques, achieving an impressive macro F1 score of 0.927 on the Test Set.

Title: FIMBA: Evaluating the Robustness of AI in Genomics via Feature Importance Adversarial Attacks

• Authors: Heorhii Skovorodnikov, Hoda Alkhzaimi
• Subjects: cs.LG, cs.CR, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2401.10657
• Pdf URL: https://arxiv.org/pdf/2401.10657
• Copy Paste: [[2401.10657]] FIMBA: Evaluating the Robustness of AI in Genomics via Feature Importance Adversarial Attacks(https://arxiv.org/abs/2401.10657)
• Keywords: attack, robust
• Abstract: With the steady rise of the use of AI in bio-technical applications and the widespread adoption of genomics sequencing, an increasing amount of AI-based algorithms and tools is entering the research and production stage affecting critical decision-making streams like drug discovery and clinical outcomes. This paper demonstrates the vulnerability of AI models often utilized downstream tasks on recognized public genomics datasets. We undermine model robustness by deploying an attack that focuses on input transformation while mimicking the real data and confusing the model decision-making, ultimately yielding a pronounced deterioration in model performance. Further, we enhance our approach by generating poisoned data using a variational autoencoder-based model. Our empirical findings unequivocally demonstrate a decline in model performance, underscored by diminished accuracy and an upswing in false positives and false negatives. Furthermore, we analyze the resulting adversarial samples via spectral analysis yielding conclusions for countermeasures against such attacks.

Title: A Simple Framework to Accelerate Multilingual Language Model for Monolingual Text Generation

• Authors: Jimin Hong, Gibbeum Lee, Jaewoong Cho
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10660
• Pdf URL: https://arxiv.org/pdf/2401.10660
• Copy Paste: [[2401.10660]] A Simple Framework to Accelerate Multilingual Language Model for Monolingual Text Generation(https://arxiv.org/abs/2401.10660)
• Keywords: large language model
• Abstract: Recent advancements in large language models have facilitated the execution of complex language tasks, not only in English but also in non-English languages. However, the tokenizers of most language models, such as Llama, trained on English-centric corpora, tend to excessively fragment tokens in non-English languages. This issue is especially pronounced in non-roman alphabetic languages, which are often divided at a character or even Unicode level, leading to slower text generation. To address this, our study introduces a novel framework designed to expedite text generation in these languages. This framework predicts larger linguistic units than those of conventional multilingual tokenizers and is specifically tailored to the target language, thereby reducing the number of decoding steps required. Our empirical results demonstrate that the proposed framework increases the generation speed by a factor of 1.9 compared to standard decoding while maintaining the performance of a pre-trained multilingual model on monolingual tasks.

Title: PTPsec: Securing the Precision Time Protocol Against Time Delay Attacks Using Cyclic Path Asymmetry Analysis

• Authors: Andreas Finkenzeller, Oliver Butowski, Emanuel Regnath, Mohammad Hamad, Sebastian Steinhorst
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2401.10664
• Pdf URL: https://arxiv.org/pdf/2401.10664
• Copy Paste: [[2401.10664]] PTPsec: Securing the Precision Time Protocol Against Time Delay Attacks Using Cyclic Path Asymmetry Analysis(https://arxiv.org/abs/2401.10664)
• Keywords: secure, attack
• Abstract: High-precision time synchronization is a vital prerequisite for many modern applications and technologies, including Smart Grids, Time-Sensitive Networking (TSN), and 5G networks. Although the Precision Time Protocol (PTP) can accomplish this requirement in trusted environments, it becomes unreliable in the presence of specific cyber attacks. Mainly, time delay attacks pose the highest threat to the protocol, enabling attackers to diverge targeted clocks undetected. With the increasing danger of cyber attacks, especially against critical infrastructure, there is a great demand for effective countermeasures to secure both time synchronization and the applications that depend on it. However, current solutions are not sufficiently capable of mitigating sophisticated delay attacks. For example, they lack proper integration into the PTP protocol, scalability, or sound evaluation with the required microsecond-level accuracy. This work proposes an approach to detect and counteract delay attacks against PTP based on cyclic path asymmetry measurements over redundant paths. For that, we provide a method to find redundant paths in arbitrary networks and show how this redundancy can be exploited to reveal and mitigate undesirable asymmetries on the synchronization path that cause the malicious clock divergence. Furthermore, we propose PTPsec, a secure PTP protocol and its implementation based on the latest IEEE 1588-2019 standard. With PTPsec, we advance the conventional PTP to support reliable delay attack detection and mitigation. We validate our approach on a hardware testbed, which includes an attacker capable of performing static and incremental delay attacks at a microsecond precision. Our experimental results show that all attack scenarios can be reliably detected and mitigated with minimal detection time.

Title: Deep Learning-based Embedded Intrusion Detection System for Automotive CAN

• Authors: Shashwat Khandelwal, Eashan Wadhwa, Shreejith Shanker
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10674
• Pdf URL: https://arxiv.org/pdf/2401.10674
• Copy Paste: [[2401.10674]] Deep Learning-based Embedded Intrusion Detection System for Automotive CAN(https://arxiv.org/abs/2401.10674)
• Keywords: security, attack
• Abstract: Rising complexity of in-vehicle electronics is enabling new capabilities like autonomous driving and active safety. However, rising automation also increases risk of security threats which is compounded by lack of in-built security measures in legacy networks like CAN, allowing attackers to observe, tamper and modify information shared over such broadcast networks. Various intrusion detection approaches have been proposed to detect and tackle such threats, with machine learning models proving highly effective. However, deploying machine learning models will require high processing power through high-end processors or GPUs to perform them close to line rate. In this paper, we propose a hybrid FPGA-based ECU approach that can transparently integrate IDS functionality through a dedicated off-the-shelf hardware accelerator that implements a deep-CNN intrusion detection model. Our results show that the proposed approach provides an average accuracy of over 99% across multiple attack datasets with 0.64% false detection rates while consuming 94% less energy and achieving 51.8% reduction in per-message processing latency when compared to IDS implementations on GPUs.

Title: A Lightweight Multi-Attack CAN Intrusion Detection System on Hybrid FPGAs

• Authors: Shashwat Khandelwal, Shreejith Shanker
• Subjects: cs.CR, cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2401.10689
• Pdf URL: https://arxiv.org/pdf/2401.10689
• Copy Paste: [[2401.10689]] A Lightweight Multi-Attack CAN Intrusion Detection System on Hybrid FPGAs(https://arxiv.org/abs/2401.10689)
• Keywords: security, attack
• Abstract: Rising connectivity in vehicles is enabling new capabilities like connected autonomous driving and advanced driver assistance systems (ADAS) for improving the safety and reliability of next-generation vehicles. This increased access to in-vehicle functions compromises critical capabilities that use legacy invehicle networks like Controller Area Network (CAN), which has no inherent security or authentication mechanism. Intrusion detection and mitigation approaches, particularly using machine learning models, have shown promising results in detecting multiple attack vectors in CAN through their ability to generalise to new vectors. However, most deployments require dedicated computing units like GPUs to perform line-rate detection, consuming much higher power. In this paper, we present a lightweight multi-attack quantised machine learning model that is deployed using Xilinx's Deep Learning Processing Unit IP on a Zynq Ultrascale+ (XCZU3EG) FPGA, which is trained and validated using the public CAN Intrusion Detection dataset. The quantised model detects denial of service and fuzzing attacks with an accuracy of above 99 % and a false positive rate of 0.07%, which are comparable to the state-of-the-art techniques in the literature. The Intrusion Detection System (IDS) execution consumes just 2.0 W with software tasks running on the ECU and achieves a 25 % reduction in per-message processing latency over the state-of-the-art implementations. This deployment allows the ECU function to coexist with the IDS with minimal changes to the tasks, making it ideal for real-time IDS in in-vehicle systems.

Title: Beyond RMSE and MAE: Introducing EAUC to unmask hidden bias and unfairness in dyadic regression models

• Authors: Jorge Paz-Ruza, Amparo Alonso-Betanzos, Bertha Guijarro-Berdiñas, Brais Cancela, Carlos Eiras-Franco
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.10690
• Pdf URL: https://arxiv.org/pdf/2401.10690
• Copy Paste: [[2401.10690]] Beyond RMSE and MAE: Introducing EAUC to unmask hidden bias and unfairness in dyadic regression models(https://arxiv.org/abs/2401.10690)
• Keywords: fair
• Abstract: Dyadic regression models, which predict real-valued outcomes for pairs of entities, are fundamental in many domains (e.g. predicting the rating of a user to a product in Recommender Systems) and promising and under exploration in many others (e.g. approximating the adequate dosage of a drug for a patient in personalized pharmacology). In this work, we demonstrate that non-uniformity in the observed value distributions of individual entities leads to severely biased predictions in state-of-the-art models, skewing predictions towards the average of observed past values for the entity and providing worse-than-random predictive power in eccentric yet equally important cases. We show that the usage of global error metrics like Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) is insufficient to capture this phenomenon, which we name eccentricity bias, and we introduce Eccentricity-Area Under the Curve (EAUC) as a new complementary metric that can quantify it in all studied models and datasets. We also prove the adequateness of EAUC by using naive de-biasing corrections to demonstrate that a lower model bias correlates with a lower EAUC and vice-versa. This work contributes a bias-aware evaluation of dyadic regression models to avoid potential unfairness and risks in critical real-world applications of such systems.

Title: Explainable and Transferable Adversarial Attack for ML-Based Network Intrusion Detectors

• Authors: Hangsheng Zhang, Dongqi Han, Yinlong Liu, Zhiliang Wang, Jiyan Sun, Shangyuan Zhuang, Jiqiang Liu, Jinsong Dong
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.10691
• Pdf URL: https://arxiv.org/pdf/2401.10691
• Copy Paste: [[2401.10691]] Explainable and Transferable Adversarial Attack for ML-Based Network Intrusion Detectors(https://arxiv.org/abs/2401.10691)
• Keywords: attack
• Abstract: espite being widely used in network intrusion detection systems (NIDSs), machine learning (ML) has proven to be highly vulnerable to adversarial attacks. White-box and black-box adversarial attacks of NIDS have been explored in several studies. However, white-box attacks unrealistically assume that the attackers have full knowledge of the target NIDSs. Meanwhile, existing black-box attacks can not achieve high attack success rate due to the weak adversarial transferability between models (e.g., neural networks and tree models). Additionally, neither of them explains why adversarial examples exist and why they can transfer across models. To address these challenges, this paper introduces ETA, an Explainable Transfer-based Black-Box Adversarial Attack framework. ETA aims to achieve two primary objectives: 1) create transferable adversarial examples applicable to various ML models and 2) provide insights into the existence of adversarial examples and their transferability within NIDSs. Specifically, we first provide a general transfer-based adversarial attack method applicable across the entire ML space. Following that, we exploit a unique insight based on cooperative game theory and perturbation interpretations to explain adversarial examples and adversarial transferability. On this basis, we propose an Important-Sensitive Feature Selection (ISFS) method to guide the search for adversarial examples, achieving stronger transferability and ensuring traffic-space constraints.

Title: Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model

• Authors: Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, Jingjing Liu
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2401.10700
• Pdf URL: https://arxiv.org/pdf/2401.10700
• Copy Paste: [[2401.10700]] Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model(https://arxiv.org/abs/2401.10700)
• Keywords: diffusion
• Abstract: Safe offline RL is a promising way to bypass risky online interactions towards safe policy learning. Most existing methods only enforce soft constraints, i.e., constraining safety violations in expectation below thresholds predetermined. This can lead to potentially unsafe outcomes, thus unacceptable in safety-critical scenarios. An alternative is to enforce the hard constraint of zero violation. However, this can be challenging in offline setting, as it needs to strike the right balance among three highly intricate and correlated aspects: safety constraint satisfaction, reward maximization, and behavior regularization imposed by offline datasets. Interestingly, we discover that via reachability analysis of safe-control theory, the hard safety constraint can be equivalently translated to identifying the largest feasible region given the offline dataset. This seamlessly converts the original trilogy problem to a feasibility-dependent objective, i.e., maximizing reward value within the feasible region while minimizing safety risks in the infeasible region. Inspired by these, we propose FISOR (FeasIbility-guided Safe Offline RL), which allows safety constraint adherence, reward maximization, and offline policy learning to be realized via three decoupled processes, while offering strong safety performance and stability. In FISOR, the optimal policy for the translated optimization problem can be derived in a special form of weighted behavior cloning. Thus, we propose a novel energy-guided diffusion model that does not require training a complicated time-dependent classifier to extract the policy, greatly simplifying the training. We compare FISOR against baselines on DSRL benchmark for safe offline RL. Evaluation results show that FISOR is the only method that can guarantee safety satisfaction in all tasks, while achieving top returns in most tasks.

Title: Q&A Prompts: Discovering Rich Visual Clues through Mining Question-Answer Prompts for VQA requiring Diverse World Knowledge

• Authors: Haibi Wang, Weifeng Ge
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10712
• Pdf URL: https://arxiv.org/pdf/2401.10712
• Copy Paste: [[2401.10712]] Q&A Prompts: Discovering Rich Visual Clues through Mining Question-Answer Prompts for VQA requiring Diverse World Knowledge(https://arxiv.org/abs/2401.10712)
• Keywords: robust, large language model
• Abstract: With the breakthrough of multi-modal large language models, answering complex visual questions that demand advanced reasoning abilities and world knowledge has become a much more important testbed for developing AI models than ever. However, equipping AI models with robust cross-modality reasoning ability remains challenging since the cognition scheme of humans has not been understood systematically. In this paper, we believe that if we can collect visual clues in the given image as much as possible, we will recognize the image more accurately, understand the question better, recall relevant knowledge more easily, and finally reason out the answer. We discover these rich visual clues by mining question-answer pairs in images and sending them into multi-modal large language models as prompts. We call the proposed method Q&A Prompts. Specifically, we first use the image-answer pairs and the corresponding questions in the training set as inputs and outputs to train a visual question generation model. Then, we use an image tagging model to identify various instances and send packaged image-tag pairs into the visual question generation model to generate relevant questions with the extracted image tags as answers. Finally, we encode these generated question-answer pairs as prompts with a visual-aware prompting module and send them into pre-trained multi-modal large language models to reason out the final answers. Experimental results show that, compared with state-of-the-art methods, our Q&A Prompts achieves substantial improvements on the challenging visual question answering datasets requiring reasoning over diverse world knowledge, such as OK-VQA and A-OKVQA.

Title: Real-Time Zero-Day Intrusion Detection System for Automotive Controller Area Network on FPGAs

• Authors: Shashwat Khandelwal, Shreejith Shanker
• Subjects: cs.CR, cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2401.10724
• Pdf URL: https://arxiv.org/pdf/2401.10724
• Copy Paste: [[2401.10724]] Real-Time Zero-Day Intrusion Detection System for Automotive Controller Area Network on FPGAs(https://arxiv.org/abs/2401.10724)
• Keywords: attack
• Abstract: Increasing automation in vehicles enabled by increased connectivity to the outside world has exposed vulnerabilities in previously siloed automotive networks like controller area networks (CAN). Attributes of CAN such as broadcast-based communication among electronic control units (ECUs) that lowered deployment costs are now being exploited to carry out active injection attacks like denial of service (DoS), fuzzing, and spoofing attacks. Research literature has proposed multiple supervised machine learning models deployed as Intrusion detection systems (IDSs) to detect such malicious activity; however, these are largely limited to identifying previously known attack vectors. With the ever-increasing complexity of active injection attacks, detecting zero-day (novel) attacks in these networks in real-time (to prevent propagation) becomes a problem of particular interest. This paper presents an unsupervised-learning-based convolutional autoencoder architecture for detecting zero-day attacks, which is trained only on benign (attack-free) CAN messages. We quantise the model using Vitis-AI tools from AMD/Xilinx targeting a resource-constrained Zynq Ultrascale platform as our IDS-ECU system for integration. The proposed model successfully achieves equal or higher classification accuracy (> 99.5%) on unseen DoS, fuzzing, and spoofing attacks from a publicly available attack dataset when compared to the state-of-the-art unsupervised learning-based IDSs. Additionally, by cleverly overlapping IDS operation on a window of CAN messages with the reception, the model is able to meet line-rate detection (0.43 ms per window) of high-speed CAN, which when coupled with the low energy consumption per inference, makes this architecture ideally suited for detecting zero-day attacks on critical CAN networks.

Title: Tool-LMM: A Large Multi-Modal Model for Tool Agent Learning

• Authors: Chenyu Wang, Weixin Luo, Qianyu Chen, Haonan Mai, Jindi Guo, Sixun Dong, Xiaohua (Michael)Xuan, Zhengxin Li, Lin Ma, Shenghua Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10727
• Pdf URL: https://arxiv.org/pdf/2401.10727
• Copy Paste: [[2401.10727]] Tool-LMM: A Large Multi-Modal Model for Tool Agent Learning(https://arxiv.org/abs/2401.10727)
• Keywords: large language model
• Abstract: Recently, the astonishing performance of large language models (LLMs) in natural language comprehension and generation tasks triggered lots of exploration of using them as central controllers to build agent systems. Multiple studies focus on bridging the LLMs to external tools to extend the application scenarios. However, the current LLMs' perceiving tool-use ability is limited to a single text query, which may result in ambiguity in understanding the users' real intentions. LLMs are expected to eliminate that by perceiving the visual- or auditory-grounded instructions' information. Therefore, in this paper, we propose Tool-LMM, a system incorporating open-source LLMs and multi-modal encoders so that the learnt LLMs can be conscious of multi-modal input instruction and then select the function-matched tool correctly. To facilitate the evaluation of the model's capability, we collect a dataset featured by consisting of multi-modal input tools from HuggingFace. Another important feature of our dataset is that our dataset also contains multiple potential choices for the same instruction due to the existence of identical functions and synonymous functions, which provides more potential solutions for the same query. The experiments reveal that our LMM is capable of recommending appropriate tools for multi-modal instructions. Codes and data are available at https://github.com/Tool-LMM/Tool-LMM.

Title: Removal and Selection: Improving RGB-Infrared Object Detection via Coarse-to-Fine Fusion

• Authors: Tianyi Zhao, Maoxun Yuan, Xingxing Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10731
• Pdf URL: https://arxiv.org/pdf/2401.10731
• Copy Paste: [[2401.10731]] Removal and Selection: Improving RGB-Infrared Object Detection via Coarse-to-Fine Fusion(https://arxiv.org/abs/2401.10731)
• Keywords: robust
• Abstract: Object detection in visible (RGB) and infrared (IR) images has been widely applied in recent years. Leveraging the complementary characteristics of RGB and IR images, the object detector provides reliable and robust object localization from day to night. Existing fusion strategies directly inject RGB and IR images into convolution neural networks, leading to inferior detection performance. Since the RGB and IR features have modality-specific noise, these strategies will worsen the fused features along with the propagation. Inspired by the mechanism of human brain processing multimodal information, this work introduces a new coarse-to-fine perspective to purify and fuse two modality features. Specifically, following this perspective, we design a Redundant Spectrum Removal module to coarsely remove interfering information within each modality and a Dynamic Feature Selection module to finely select the desired features for feature fusion. To verify the effectiveness of the coarse-to-fine fusion strategy, we construct a new object detector called Removal and Selection Detector (RSDet). Extensive experiments on three RGB-IR object detection datasets verify the superior performance of our method.

Title: A Survey and Comparative Analysis of Security Properties of CAN Authentication Protocols

• Authors: Alessandro Lotto, Francesco Marchiori, Alessandro Brighente, Mauro Conti
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.10736
• Pdf URL: https://arxiv.org/pdf/2401.10736
• Copy Paste: [[2401.10736]] A Survey and Comparative Analysis of Security Properties of CAN Authentication Protocols(https://arxiv.org/abs/2401.10736)
• Keywords: secure, security, attack
• Abstract: The large number of Electronic Control Units (ECUs) mounted on modern cars and their expansive communication capabilities create a substantial attack surface for potential exploitation. Despite the evolution of automotive technology, the continued use of the originally insecure Controller Area Network (CAN) bus leaves in-vehicle communications inherently non-secure. In response to the absence of standardized authentication protocols within the automotive domain, researchers propose diverse solutions, each with unique strengths and vulnerabilities. However, the continuous influx of new protocols and potential oversights in meeting security requirements and essential operational features further complicate the implementability of these protocols. This paper comprehensively reviews and compares the 15 most prominent authentication protocols for the CAN bus. Our analysis emphasizes their strengths and weaknesses, evaluating their alignment with critical security requirements for automotive authentication. Additionally, we evaluate protocols based on essential operational criteria that contribute to ease of implementation in predefined infrastructures, enhancing overall reliability and reducing the probability of successful attacks. Our study reveals a prevalent focus on defending against external attackers in existing protocols, exposing vulnerabilities to internal threats. Notably, authentication protocols employing hash chains, Mixed Message Authentication Codes, and asymmetric encryption techniques emerge as the most effective approaches. Through our comparative study, we classify the considered protocols based on their security attributes and suitability for implementation, providing valuable insights for future developments in the field.

Title: Starlit: Privacy-Preserving Federated Learning to Enhance Financial Fraud Detection

• Authors: Aydin Abadi, Bradley Doyle, Francesco Gini, Kieron Guinamard, Sasi Kumar Murakonda, Jack Liddell, Paul Mellor, Steven J. Murdoch, Mohammad Naseri, Hector Page, George Theodorakopoulos, Suzanne Weller
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.10765
• Pdf URL: https://arxiv.org/pdf/2401.10765
• Copy Paste: [[2401.10765]] Starlit: Privacy-Preserving Federated Learning to Enhance Financial Fraud Detection(https://arxiv.org/abs/2401.10765)
• Keywords: security, privacy, federate
• Abstract: Federated Learning (FL) is a data-minimization approach enabling collaborative model training across diverse clients with local data, avoiding direct data exchange. However, state-of-the-art FL solutions to identify fraudulent financial transactions exhibit a subset of the following limitations. They (1) lack a formal security definition and proof, (2) assume prior freezing of suspicious customers' accounts by financial institutions (limiting the solutions' adoption), (3) scale poorly, involving either $O(n^2)$ computationally expensive modular exponentiation (where $n$ is the total number of financial institutions) or highly inefficient fully homomorphic encryption, (4) assume the parties have already completed the identity alignment phase, hence excluding it from the implementation, performance evaluation, and security analysis, and (5) struggle to resist clients' dropouts. This work introduces Starlit, a novel scalable privacy-preserving FL mechanism that overcomes these limitations. It has various applications, such as enhancing financial fraud detection, mitigating terrorism, and enhancing digital health. We implemented Starlit and conducted a thorough performance analysis using synthetic data from a key player in global financial transactions. The evaluation indicates Starlit's scalability, efficiency, and accuracy.

Title: Mitigating Hallucinations of Large Language Models via Knowledge Consistent Alignment

• Authors: Fanqi Wan, Xinting Huang, Leyang Cui, Xiaojun Quan, Wei Bi, Shuming Shi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10768
• Pdf URL: https://arxiv.org/pdf/2401.10768
• Copy Paste: [[2401.10768]] Mitigating Hallucinations of Large Language Models via Knowledge Consistent Alignment(https://arxiv.org/abs/2401.10768)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) have proven to be exceptional on a variety of tasks after alignment, they may still produce responses that contradict the context or world knowledge confidently, a phenomenon known as ``hallucination''. In this paper, we demonstrate that reducing the inconsistency between the external knowledge encapsulated in the training data and the intrinsic knowledge inherited in the pretraining corpus could mitigate hallucination in alignment. Specifically, we introduce a novel knowledge consistent alignment (KCA) approach, which involves automatically formulating examinations based on external knowledge for accessing the comprehension of LLMs. For data encompassing knowledge inconsistency, KCA implements several simple yet efficient strategies for processing. We illustrate the superior performance of the proposed KCA approach in mitigating hallucinations across six benchmarks using LLMs of different backbones and scales. Furthermore, we confirm the correlation between knowledge inconsistency and hallucination, signifying the effectiveness of reducing knowledge inconsistency in alleviating hallucinations. Our code, model weights, and data are public at \url{https://github.com/fanqiwan/KCA}.

Title: Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

• Authors: Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10774
• Pdf URL: https://arxiv.org/pdf/2401.10774
• Copy Paste: [[2401.10774]] Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads(https://arxiv.org/abs/2401.10774)
• Keywords: large language model
• Abstract: The inference process in Large Language Models (LLMs) is often limited due to the absence of parallelism in the auto-regressive decoding process, resulting in most operations being restricted by the memory bandwidth of accelerators. While methods such as speculative decoding have been suggested to address this issue, their implementation is impeded by the challenges associated with acquiring and maintaining a separate draft model. In this paper, we present Medusa, an efficient method that augments LLM inference by adding extra decoding heads to predict multiple subsequent tokens in parallel. Using a tree-based attention mechanism, Medusa constructs multiple candidate continuations and verifies them simultaneously in each decoding step. By leveraging parallel processing, Medusa introduces only minimal overhead in terms of single-step latency while substantially reducing the number of decoding steps required. We present two levels of fine-tuning procedures for Medusa to meet the needs of different use cases: Medusa-1: Medusa is directly fine-tuned on top of a frozen backbone LLM, enabling lossless inference acceleration. Medusa-2: Medusa is fine-tuned together with the backbone LLM, enabling better prediction accuracy of Medusa heads and higher speedup but needing a special training recipe that preserves the backbone model's capabilities. Moreover, we propose several extensions that improve or expand the utility of Medusa, including a self-distillation to handle situations where no training data is available and a typical acceptance scheme to boost the acceptance rate while maintaining generation quality. We evaluate Medusa on models of various sizes and training procedures. Our experiments demonstrate that Medusa-1 can achieve over 2.2x speedup without compromising generation quality, while Medusa-2 further improves the speedup to 2.3-3.6x.

Title: Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusion

• Authors: Zuoyue Li, Zhenqiang Li, Zhaopeng Cui, Marc Pollefeys, Martin R. Oswald
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10786
• Pdf URL: https://arxiv.org/pdf/2401.10786
• Copy Paste: [[2401.10786]] Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusion(https://arxiv.org/abs/2401.10786)
• Keywords: diffusion
• Abstract: Directly generating scenes from satellite imagery offers exciting possibilities for integration into applications like games and map services. However, challenges arise from significant view changes and scene scale. Previous efforts mainly focused on image or video generation, lacking exploration into the adaptability of scene generation for arbitrary views. Existing 3D generation works either operate at the object level or are difficult to utilize the geometry obtained from satellite imagery. To overcome these limitations, we propose a novel architecture for direct 3D scene generation by introducing diffusion models into 3D sparse representations and combining them with neural rendering techniques. Specifically, our approach generates texture colors at the point level for a given geometry using a 3D diffusion model first, which is then transformed into a scene representation in a feed-forward manner. The representation can be utilized to render arbitrary views which would excel in both single-frame quality and inter-frame consistency. Experiments in two city-scale datasets show that our model demonstrates proficiency in generating photo-realistic street-view image sequences and cross-view urban scenes from satellite imagery.

Title: Hybrid Online Certificate Status Protocol with Certificate Revocation List for Smart Grid Public Key Infrastructure

• Authors: Hong-Sheng Huang, Zhe-Yi Jiang, Hsuan-Tung Cheng, Hung-Min Sun
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.10787
• Pdf URL: https://arxiv.org/pdf/2401.10787
• Copy Paste: [[2401.10787]] Hybrid Online Certificate Status Protocol with Certificate Revocation List for Smart Grid Public Key Infrastructure(https://arxiv.org/abs/2401.10787)
• Keywords: secure, security, robust
• Abstract: Hsu et al. (2022) proposed a cryptographic scheme within the public key infrastructure to bolster the security of smart grid meters. Their proposal involved developing the Certificate Management over CMS mechanism to establish Simple Certificate Enrollment Protocol and Enrollment over Secure Transport protocol. Additionally, they implemented Online Certificate Status Protocol (OCSP) services to independently query the status of certificates. However, their implementation featured a single OCSP server handling all query requests. Considering the typical scenario in smart grid PKI environments with over tens of thousands of end-meters, we introduced a Hybrid Online Certificate Status Protocol mechanism. This approach decreases demand of query resources from the client to OCSP servers collaborating with Certificate Revocation Lists. Our simulations, mimicking meter behavior, demonstrated increased efficiency, creating a more robust architecture tailored to the smart grid meter landscape.

Title: Measuring the Impact of Scene Level Objects on Object Detection: Towards Quantitative Explanations of Detection Decisions

• Authors: Lynn Vonder Haar, Timothy Elvira, Luke Newcomb, Omar Ochoa
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10790
• Pdf URL: https://arxiv.org/pdf/2401.10790
• Copy Paste: [[2401.10790]] Measuring the Impact of Scene Level Objects on Object Detection: Towards Quantitative Explanations of Detection Decisions(https://arxiv.org/abs/2401.10790)
• Keywords: explainability
• Abstract: Although accuracy and other common metrics can provide a useful window into the performance of an object detection model, they lack a deeper view of the model's decision process. Regardless of the quality of the training data and process, the features that an object detection model learns cannot be guaranteed. A model may learn a relationship between certain background context, i.e., scene level objects, and the presence of the labeled classes. Furthermore, standard performance verification and metrics would not identify this phenomenon. This paper presents a new black box explainability method for additional verification of object detection models by finding the impact of scene level objects on the identification of the objects within the image. By comparing the accuracies of a model on test data with and without certain scene level objects, the contributions of these objects to the model's performance becomes clearer. The experiment presented here will assess the impact of buildings and people in image context on the detection of emergency road vehicles by a fine-tuned YOLOv8 model. A large increase in accuracy in the presence of a scene level object will indicate the model's reliance on that object to make its detections. The results of this research lead to providing a quantitative explanation of the object detection model's decision process, enabling a deeper understanding of the model's performance.

Title: RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision

• Authors: Fernando Pérez-García, Harshita Sharma, Sam Bond-Taylor, Kenza Bouzid, Valentina Salvatelli, Maximilian Ilse, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Matthew P. Lungren, Maria Wetscherek, Noel Codella, Stephanie L. Hyland, Javier Alvarez-Valle, Ozan Oktay
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10815
• Pdf URL: https://arxiv.org/pdf/2401.10815
• Copy Paste: [[2401.10815]] RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision(https://arxiv.org/abs/2401.10815)
• Keywords: segmentation
• Abstract: Language-supervised pre-training has proven to be a valuable method for extracting semantically meaningful features from images, serving as a foundational element in multimodal systems within the computer vision and medical imaging domains. However, resulting features are limited by the information contained within the text. This is particularly problematic in medical imaging, where radiologists' written findings focus on specific observations; a challenge compounded by the scarcity of paired imaging-text data due to concerns over leakage of personal health information. In this work, we fundamentally challenge the prevailing reliance on language supervision for learning general purpose biomedical imaging encoders. We introduce RAD-DINO, a biomedical image encoder pre-trained solely on unimodal biomedical imaging data that obtains similar or greater performance than state-of-the-art biomedical language supervised models on a diverse range of benchmarks. Specifically, the quality of learned representations is evaluated on standard imaging tasks (classification and semantic segmentation), and a vision-language alignment task (text report generation from images). To further demonstrate the drawback of language supervision, we show that features from RAD-DINO correlate with other medical records (e.g., sex or age) better than language-supervised models, which are generally not mentioned in radiology reports. Finally, we conduct a series of ablations determining the factors in RAD-DINO's performance; notably, we observe that RAD-DINO's downstream performance scales well with the quantity and diversity of training data, demonstrating that image-only supervision is a scalable approach for training a foundational biomedical image encoder.

Title: ActAnywhere: Subject-Aware Video Background Generation

• Authors: Boxiao Pan, Zhan Xu, Chun-Hao Paul Huang, Krishna Kumar Singh, Yang Zhou, Leonidas J. Guibas, Jimei Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10822
• Pdf URL: https://arxiv.org/pdf/2401.10822
• Copy Paste: [[2401.10822]] ActAnywhere: Subject-Aware Video Background Generation(https://arxiv.org/abs/2401.10822)
• Keywords: diffusion, generative, segmentation
• Abstract: Generating video background that tailors to foreground subject motion is an important problem for the movie industry and visual effects community. This task involves synthesizing background that aligns with the motion and appearance of the foreground subject, while also complies with the artist's creative intention. We introduce ActAnywhere, a generative model that automates this process which traditionally requires tedious manual efforts. Our model leverages the power of large-scale video diffusion models, and is specifically tailored for this task. ActAnywhere takes a sequence of foreground subject segmentation as input and an image that describes the desired scene as condition, to produce a coherent video with realistic foreground-background interactions while adhering to the condition frame. We train our model on a large-scale dataset of human-scene interaction videos. Extensive evaluations demonstrate the superior performance of our model, significantly outperforming baselines. Moreover, we show that ActAnywhere generalizes to diverse out-of-distribution samples, including non-human subjects. Please visit our project webpage at https://actanywhere.github.io.

Title: A survey on recent advances in named entity recognition

• Authors: Imed Keraghel, Stanislas Morbieu, Mohamed Nadif
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10825
• Pdf URL: https://arxiv.org/pdf/2401.10825
• Copy Paste: [[2401.10825]] A survey on recent advances in named entity recognition(https://arxiv.org/abs/2401.10825)
• Keywords: transformer, large language model
• Abstract: Named Entity Recognition seeks to extract substrings within a text that name real-world objects and to determine their type (for example, whether they refer to persons or organizations). In this survey, we first present an overview of recent popular approaches, but we also look at graph- and transformer- based methods including Large Language Models (LLMs) that have not had much coverage in other surveys. Second, we focus on methods designed for datasets with scarce annotations. Third, we evaluate the performance of the main NER implementations on a variety of datasets with differing characteristics (as regards their domain, their size, and their number of classes). We thus provide a deep comparison of algorithms that are never considered together. Our experiments shed some light on how the characteristics of datasets affect the behavior of the methods that we compare.

Title: Understanding Video Transformers via Universal Concept Discovery

• Authors: Matthew Kowal, Achal Dave, Rares Ambrus, Adrien Gaidon, Konstantinos G. Derpanis, Pavel Tokmakov
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2401.10831
• Pdf URL: https://arxiv.org/pdf/2401.10831
• Copy Paste: [[2401.10831]] Understanding Video Transformers via Universal Concept Discovery(https://arxiv.org/abs/2401.10831)
• Keywords: interpretability, transformer
• Abstract: This paper studies the problem of concept-based interpretability of transformer representations for videos. Concretely, we seek to explain the decision-making process of video transformers based on high-level, spatiotemporal concepts that are automatically discovered. Prior research on concept-based interpretability has concentrated solely on image-level tasks. Comparatively, video models deal with the added temporal dimension, increasing complexity and posing challenges in identifying dynamic concepts over time. In this work, we systematically address these challenges by introducing the first Video Transformer Concept Discovery (VTCD) algorithm. To this end, we propose an efficient approach for unsupervised identification of units of video transformer representations - concepts, and ranking their importance to the output of a model. The resulting concepts are highly interpretable, revealing spatio-temporal reasoning mechanisms and object-centric representations in unstructured video models. Performing this analysis jointly over a diverse set of supervised and self-supervised representations, we discover that some of these mechanism are universal in video transformers. Finally, we demonstrate that VTCDcan be used to improve model performance for fine-grained tasks.

Title: Using LLMs to discover emerging coded antisemitic hate-speech emergence in extremist social media

• Authors: Dhanush Kikkisetti, Raza Ul Mustafa, Wendy Melillo, Roberto Corizzo, Zois Boukouvalas, Jeff Gill, Nathalie Japkowicz
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10841
• Pdf URL: https://arxiv.org/pdf/2401.10841
• Copy Paste: [[2401.10841]] Using LLMs to discover emerging coded antisemitic hate-speech emergence in extremist social media(https://arxiv.org/abs/2401.10841)
• Keywords: large language model
• Abstract: Online hate speech proliferation has created a difficult problem for social media platforms. A particular challenge relates to the use of coded language by groups interested in both creating a sense of belonging for its users and evading detection. Coded language evolves quickly and its use varies over time. This paper proposes a methodology for detecting emerging coded hate-laden terminology. The methodology is tested in the context of online antisemitic discourse. The approach considers posts scraped from social media platforms, often used by extremist users. The posts are scraped using seed expressions related to previously known discourse of hatred towards Jews. The method begins by identifying the expressions most representative of each post and calculating their frequency in the whole corpus. It filters out grammatically incoherent expressions as well as previously encountered ones so as to focus on emergent well-formed terminology. This is followed by an assessment of semantic similarity to known antisemitic terminology using a fine-tuned large language model, and subsequent filtering out of the expressions that are too distant from known expressions of hatred. Emergent antisemitic expressions containing terms clearly relating to Jewish topics are then removed to return only coded expressions of hatred.

Title: Source-Free and Image-Only Unsupervised Domain Adaptation for Category Level Object Pose Estimation

• Authors: Prakhar Kaushik, Aayush Mishra, Adam Kortylewski, Alan Yuille
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10848
• Pdf URL: https://arxiv.org/pdf/2401.10848
• Copy Paste: [[2401.10848]] Source-Free and Image-Only Unsupervised Domain Adaptation for Category Level Object Pose Estimation(https://arxiv.org/abs/2401.10848)
• Keywords: robust, generative
• Abstract: We consider the problem of source-free unsupervised category-level pose estimation from only RGB images to a target domain without any access to source domain data or 3D annotations during adaptation. Collecting and annotating real-world 3D data and corresponding images is laborious, expensive, yet unavoidable process, since even 3D pose domain adaptation methods require 3D data in the target domain. We introduce 3DUDA, a method capable of adapting to a nuisance-ridden target domain without 3D or depth data. Our key insight stems from the observation that specific object subparts remain stable across out-of-domain (OOD) scenarios, enabling strategic utilization of these invariant subcomponents for effective model updates. We represent object categories as simple cuboid meshes, and harness a generative model of neural feature activations modeled at each mesh vertex learnt using differential rendering. We focus on individual locally robust mesh vertex features and iteratively update them based on their proximity to corresponding features in the target domain even when the global pose is not correct. Our model is then trained in an EM fashion, alternating between updating the vertex features and the feature extractor. We show that our method simulates fine-tuning on a global pseudo-labeled dataset under mild assumptions, which converges to the target domain asymptotically. Through extensive empirical validation, including a complex extreme UDA setup which combines real nuisances, synthetic noise, and occlusion, we demonstrate the potency of our simple approach in addressing the domain shift challenge and significantly improving pose estimation accuracy.

Title: Ensembler: Combating model inversion attacks using model ensemble during collaborative inference

• Authors: Dancheng Liu, Jinjun Xiong
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10859
• Pdf URL: https://arxiv.org/pdf/2401.10859
• Copy Paste: [[2401.10859]] Ensembler: Combating model inversion attacks using model ensemble during collaborative inference(https://arxiv.org/abs/2401.10859)
• Keywords: privacy, attack
• Abstract: Deep learning models have exhibited remarkable performance across various domains. Nevertheless, the burgeoning model sizes compel edge devices to offload a significant portion of the inference process to the cloud. While this practice offers numerous advantages, it also raises critical concerns regarding user data privacy. In scenarios where the cloud server's trustworthiness is in question, the need for a practical and adaptable method to safeguard data privacy becomes imperative. In this paper, we introduce Ensembler, an extensible framework designed to substantially increase the difficulty of conducting model inversion attacks for adversarial parties. Ensembler leverages model ensembling on the adversarial server, running in parallel with existing approaches that introduce perturbations to sensitive data during colloborative inference. Our experiments demonstrate that when combined with even basic Gaussian noise, Ensembler can effectively shield images from reconstruction attacks, achieving recognition levels that fall below human performance in some strict settings, significantly outperforming baseline methods lacking the Ensembler framework.

Title: Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning

• Authors: Adib Hasan, Ileana Rugina, Alex Wang
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.10862
• Pdf URL: https://arxiv.org/pdf/2401.10862
• Copy Paste: [[2401.10862]] Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning(https://arxiv.org/abs/2401.10862)
• Keywords: protect, attack, large language model
• Abstract: Large Language Models (LLMs) are vulnerable to `Jailbreaking' prompts, a type of attack that can coax these models into generating harmful and illegal content. In this paper, we show that pruning up to 20% of LLM parameters markedly increases their resistance to such attacks without additional training and without sacrificing their performance in standard benchmarks. Intriguingly, we discovered that the enhanced safety observed post-pruning correlates to the initial safety training level of the model, hinting that the effect of pruning could be more general and may hold for other LLM behaviors beyond safety. Additionally, we introduce a curated dataset of 225 harmful tasks across five categories, inserted into ten different Jailbreaking prompts, showing that pruning aids LLMs in concentrating attention on task-relevant tokens in jailbreaking prompts. Lastly, our experiments reveal that the prominent chat models, such as LLaMA-2 Chat, Vicuna, and Mistral Instruct exhibit high susceptibility to jailbreaking attacks, with some categories achieving nearly 70-100% success rate. These insights underline the potential of pruning as a generalizable approach for improving LLM safety, reliability, and potentially other desired behaviors.

Title: Reinforcement learning for question answering in programming domain using public community scoring as a human feedback

• Authors: Alexey Gorbatovski, Sergey Kovalchuk
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.10882
• Pdf URL: https://arxiv.org/pdf/2401.10882
• Copy Paste: [[2401.10882]] Reinforcement learning for question answering in programming domain using public community scoring as a human feedback(https://arxiv.org/abs/2401.10882)
• Keywords: large language model
• Abstract: In this study, we investigate the enhancement of the GPT Neo 125M performance in Community Question Answering (CQA) with a focus on programming, through the integration of Reinforcement Learning from Human Feedback (RLHF) and the utilization of scores from Stack Overflow. Two distinct reward model training strategies are employed for fine-tuning with Proximal Policy Optimization (PPO). Notably, the improvements in performance achieved through this method are comparable to those of GPT Neo 2.7B parameter variant. Additionally, an auxiliary scoring mechanism is introduced, which demonstrates the limitations of conventional linguistic metrics in evaluating responses in the programming domain. Through accurate analysis, this paper looks at the divergence between traditional linguistic metrics and our human-preferences-based reward model, underscoring the imperative for domain-specific evaluation methods. By elucidating the complexities involved in applying RLHF to programming CQA and accentuating the significance of context-aware evaluation, this study contributes to the ongoing efforts in refining Large Language Models through focused human feedback.

Title: Synthesizing Moving People with 3D Control

• Authors: Boyi Li, Jathushan Rajasegaran, Yossi Gandelsman, Alexei A. Efros, Jitendra Malik
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10889
• Pdf URL: https://arxiv.org/pdf/2401.10889
• Copy Paste: [[2401.10889]] Synthesizing Moving People with 3D Control(https://arxiv.org/abs/2401.10889)
• Keywords: diffusion
• Abstract: In this paper, we present a diffusion model-based framework for animating people from a single image for a given target 3D motion sequence. Our approach has two core components: a) learning priors about invisible parts of the human body and clothing, and b) rendering novel body poses with proper clothing and texture. For the first part, we learn an in-filling diffusion model to hallucinate unseen parts of a person given a single image. We train this model on texture map space, which makes it more sample-efficient since it is invariant to pose and viewpoint. Second, we develop a diffusion-based rendering pipeline, which is controlled by 3D human poses. This produces realistic renderings of novel poses of the person, including clothing, hair, and plausible in-filling of unseen regions. This disentangled approach allows our method to generate a sequence of images that are faithful to the target motion in the 3D pose and, to the input image in terms of visual similarity. In addition to that, the 3D control allows various synthetic camera trajectories to render a person. Our experiments show that our method is resilient in generating prolonged motions and varied challenging and complex poses compared to prior methods. Please check our website for more details: https://boyiliee.github.io/3DHM.github.io/.

Title: Event detection from novel data sources: Leveraging satellite imagery alongside GPS traces

• Authors: Ekin Ugurel, Steffen Coenen, Minda Zhou Chen, Cynthia Chen
• Subjects: cs.CV, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.10890
• Pdf URL: https://arxiv.org/pdf/2401.10890
• Copy Paste: [[2401.10890]] Event detection from novel data sources: Leveraging satellite imagery alongside GPS traces(https://arxiv.org/abs/2401.10890)
• Keywords: privacy
• Abstract: Rapid identification and response to breaking events, particularly those that pose a threat to human life such as natural disasters or conflicts, is of paramount importance. The prevalence of mobile devices and the ubiquity of network connectivity has generated a massive amount of temporally- and spatially-stamped data. Numerous studies have used mobile data to derive individual human mobility patterns for various applications. Similarly, the increasing number of orbital satellites has made it easier to gather high-resolution images capturing a snapshot of a geographical area in sub-daily temporal frequency. We propose a novel data fusion methodology integrating satellite imagery with privacy-enhanced mobile data to augment the event inference task, whether in real-time or historical. In the absence of boots on the ground, mobile data is able to give an approximation of human mobility, proximity to one another, and the built environment. On the other hand, satellite imagery can provide visual information on physical changes to the built and natural environment. The expected use cases for our methodology include small-scale disaster detection (i.e., tornadoes, wildfires, and floods) in rural regions, search and rescue operation augmentation for lost hikers in remote wilderness areas, and identification of active conflict areas and population displacement in war-torn states. Our implementation is open-source on GitHub: https://github.com/ekinugurel/SatMobFusion.

Title: Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data

• Authors: Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10891
• Pdf URL: https://arxiv.org/pdf/2401.10891
• Copy Paste: [[2401.10891]] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data(https://arxiv.org/abs/2401.10891)
• Keywords: robust
• Abstract: This work presents Depth Anything, a highly practical solution for robust monocular depth estimation. Without pursuing novel technical modules, we aim to build a simple yet powerful foundation model dealing with any images under any circumstances. To this end, we scale up the dataset by designing a data engine to collect and automatically annotate large-scale unlabeled data (~62M), which significantly enlarges the data coverage and thus is able to reduce the generalization error. We investigate two simple yet effective strategies that make data scaling-up promising. First, a more challenging optimization target is created by leveraging data augmentation tools. It compels the model to actively seek extra visual knowledge and acquire robust representations. Second, an auxiliary supervision is developed to enforce the model to inherit rich semantic priors from pre-trained encoders. We evaluate its zero-shot capabilities extensively, including six public datasets and randomly captured photos. It demonstrates impressive generalization ability. Further, through fine-tuning it with metric depth information from NYUv2 and KITTI, new SOTAs are set. Our better depth model also results in a better depth-conditioned ControlNet. Our models are released at https://github.com/LiheYoung/Depth-Anything.