2024-04-30

Title: CoSD: Collaborative Stance Detection with Contrastive Heterogeneous Topic Graph Learning

• Authors: Yinghan Cheng, Qi Zhang, Chongyang Shi, Liang Xiao, Shufeng Hao, Liang Hu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.17609
• Pdf URL: https://arxiv.org/pdf/2404.17609
• Copy Paste: [[2404.17609]] CoSD: Collaborative Stance Detection with Contrastive Heterogeneous Topic Graph Learning(https://arxiv.org/abs/2404.17609)
• Keywords: explainability
• Abstract: Stance detection seeks to identify the viewpoints of individuals either in favor or against a given target or a controversial topic. Current advanced neural models for stance detection typically employ fully parametric softmax classifiers. However, these methods suffer from several limitations, including lack of explainability, insensitivity to the latent data structure, and unimodality, which greatly restrict their performance and applications. To address these challenges, we present a novel collaborative stance detection framework called (CoSD) which leverages contrastive heterogeneous topic graph learning to learn topic-aware semantics and collaborative signals among texts, topics, and stance labels for enhancing stance detection. During training, we construct a heterogeneous graph to structurally organize texts and stances through implicit topics via employing latent Dirichlet allocation. We then perform contrastive graph learning to learn heterogeneous node representations, aggregating informative multi-hop collaborative signals via an elaborate Collaboration Propagation Aggregation (CPA) module. During inference, we introduce a hybrid similarity scoring module to enable the comprehensive incorporation of topic-aware semantics and collaborative signals for stance detection. Extensive experiments on two benchmark datasets demonstrate the state-of-the-art detection performance of CoSD, verifying the effectiveness and explainability of our collaborative framework.

Title: Beyond Traditional Threats: A Persistent Backdoor Attack on Federated Learning

• Authors: Tao Liu, Yuhang Zhang, Zhu Feng, Zhiqin Yang, Chen Xu, Dapeng Man, Wu Yang
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.17617
• Pdf URL: https://arxiv.org/pdf/2404.17617
• Copy Paste: [[2404.17617]] Beyond Traditional Threats: A Persistent Backdoor Attack on Federated Learning(https://arxiv.org/abs/2404.17617)
• Keywords: attack, federate
• Abstract: Backdoors on federated learning will be diluted by subsequent benign updates. This is reflected in the significant reduction of attack success rate as iterations increase, ultimately failing. We use a new metric to quantify the degree of this weakened backdoor effect, called attack persistence. Given that research to improve this performance has not been widely noted,we propose a Full Combination Backdoor Attack (FCBA) method. It aggregates more combined trigger information for a more complete backdoor pattern in the global model. Trained backdoored global model is more resilient to benign updates, leading to a higher attack success rate on the test set. We test on three datasets and evaluate with two models across various settings. FCBA's persistence outperforms SOTA federated learning backdoor attacks. On GTSRB, postattack 120 rounds, our attack success rate rose over 50% from baseline. The core code of our method is available at https://github.com/PhD-TaoLiu/FCBA.

Title: Alice's Adventures in a Differentiable Wonderland -- Volume I, A Tour of the Land

• Authors: Simone Scardapane
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17625
• Pdf URL: https://arxiv.org/pdf/2404.17625
• Copy Paste: [[2404.17625]] Alice's Adventures in a Differentiable Wonderland -- Volume I, A Tour of the Land(https://arxiv.org/abs/2404.17625)
• Keywords: explainability, transformer, generative
• Abstract: This book is a self-contained introduction to the design of modern (deep) neural networks. Because the term "neural" comes with a lot of historical baggage, I prefer the simpler term "differentiable models" in the text. The focus of this 250-pages volume is on building efficient blocks for processing $n$D data, including convolutions, transformers, graph layers, and modern recurrent models (including linearized transformers and structured state-space models). Because the field is evolving quickly, I have tried to strike a good balance between theory and code, historical considerations and recent trends. I assume the reader has some exposure to machine learning and linear algebra, but I try to cover the preliminaries when necessary. The volume is a refined draft from a set of lecture notes for a course called Neural Networks for Data Science Applications that I teach in Sapienza. I do not cover many advanced topics (generative modeling, explainability, prompting, agents), which will be published over time in the companion website.

Title: Empowering Large Language Models for Textual Data Augmentation

• Authors: Yichuan Li, Kaize Ding, Jianling Wang, Kyumin Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17642
• Pdf URL: https://arxiv.org/pdf/2404.17642
• Copy Paste: [[2404.17642]] Empowering Large Language Models for Textual Data Augmentation(https://arxiv.org/abs/2404.17642)
• Keywords: large language model
• Abstract: With the capabilities of understanding and executing natural language instructions, Large language models (LLMs) can potentially act as a powerful tool for textual data augmentation. However, the quality of augmented data depends heavily on the augmentation instructions provided, and the effectiveness can fluctuate across different downstream tasks. While manually crafting and selecting instructions can offer some improvement, this approach faces scalability and consistency issues in practice due to the diversity of downstream tasks. In this work, we address these limitations by proposing a new solution, which can automatically generate a large pool of augmentation instructions and select the most suitable task-informed instructions, thereby empowering LLMs to create high-quality augmented data for different downstream tasks. Empirically, the proposed approach consistently generates augmented data with better quality compared to non-LLM and LLM-based data augmentation methods, leading to the best performance on 26 few-shot learning tasks sourced from a wide range of application domains.

Title: PLAYER*: Enhancing LLM-based Multi-Agent Communication and Interaction in Murder Mystery Games

• Authors: Qinglin Zhu, Runcong Zhao, Jinhua Du, Lin Gui, Yulan He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.17662
• Pdf URL: https://arxiv.org/pdf/2404.17662
• Copy Paste: [[2404.17662]] PLAYER*: Enhancing LLM-based Multi-Agent Communication and Interaction in Murder Mystery Games(https://arxiv.org/abs/2404.17662)
• Keywords: large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have enhanced the efficacy of agent communication and social interactions. Despite these advancements, building LLM-based agents for reasoning in dynamic environments involving competition and collaboration remains challenging due to the limitations of informed graph-based search methods. We propose PLAYER*, a novel framework based on an anytime sampling-based planner, which utilises sensors and pruners to enable a purely question-driven searching framework for complex reasoning tasks. We also introduce a quantifiable evaluation method using multiple-choice questions and construct the WellPlay dataset with 1,482 QA pairs. Experiments demonstrate PLAYER*'s efficiency and performance enhancements compared to existing methods in complex, dynamic environments with quantifiable results.

Title: Center-Based Relaxed Learning Against Membership Inference Attacks

• Authors: Xingli Fang, Jung-Eun Kim
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.17674
• Pdf URL: https://arxiv.org/pdf/2404.17674
• Copy Paste: [[2404.17674]] Center-Based Relaxed Learning Against Membership Inference Attacks(https://arxiv.org/abs/2404.17674)
• Keywords: privacy, defense, attack, membership infer
• Abstract: Membership inference attacks (MIAs) are currently considered one of the main privacy attack strategies, and their defense mechanisms have also been extensively explored. However, there is still a gap between the existing defense approaches and ideal models in performance and deployment costs. In particular, we observed that the privacy vulnerability of the model is closely correlated with the gap between the model's data-memorizing ability and generalization ability. To address this, we propose a new architecture-agnostic training paradigm called center-based relaxed learning (CRL), which is adaptive to any classification model and provides privacy preservation by sacrificing a minimal or no loss of model generalizability. We emphasize that CRL can better maintain the model's consistency between member and non-member data. Through extensive experiments on standard classification datasets, we empirically show that this approach exhibits comparable performance without requiring additional model capacity or data costs.

Title: Deep Learning for Melt Pool Depth Contour Prediction From Surface Thermal Images via Vision Transformers

• Authors: Francis Ogoke, Peter Myung-Won Pak, Alexander Myers, Guadalupe Quirarte, Jack Beuth, Jonathan Malen, Amir Barati Farimani
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17699
• Pdf URL: https://arxiv.org/pdf/2404.17699
• Copy Paste: [[2404.17699]] Deep Learning for Melt Pool Depth Contour Prediction From Surface Thermal Images via Vision Transformers(https://arxiv.org/abs/2404.17699)
• Keywords: transformer
• Abstract: Insufficient overlap between the melt pools produced during Laser Powder Bed Fusion (L-PBF) can lead to lack-of-fusion defects and deteriorated mechanical and fatigue performance. In-situ monitoring of the melt pool subsurface morphology requires specialized equipment that may not be readily accessible or scalable. Therefore, we introduce a machine learning framework to correlate in-situ two-color thermal images observed via high-speed color imaging to the two-dimensional profile of the melt pool cross-section. Specifically, we employ a hybrid CNN-Transformer architecture to establish a correlation between single bead off-axis thermal image sequences and melt pool cross-section contours measured via optical microscopy. In this architecture, a ResNet model embeds the spatial information contained within the thermal images to a latent vector, while a Transformer model correlates the sequence of embedded vectors to extract temporal information. Our framework is able to model the curvature of the subsurface melt pool structure, with improved performance in high energy density regimes compared to analytical melt pool models. The performance of this model is evaluated through dimensional and geometric comparisons to the corresponding experimental melt pool observations.

Title: CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solving

• Authors: Pei Chen, Boran Han, Shuai Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.17729
• Pdf URL: https://arxiv.org/pdf/2404.17729
• Copy Paste: [[2404.17729]] CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solving(https://arxiv.org/abs/2404.17729)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown great ability in solving traditional natural language tasks and elementary reasoning tasks with appropriate prompting techniques. However, their ability is still limited in solving complicated science problems. In this work, we aim to push the upper bound of the reasoning capability of LLMs by proposing a collaborative multi-agent, multi-reasoning-path (CoMM) prompting framework. Specifically, we prompt LLMs to play different roles in a problem-solving team, and encourage different role-play agents to collaboratively solve the target task. In particular, we discover that applying different reasoning paths for different roles is an effective strategy to implement few-shot prompting approaches in the multi-agent scenarios. Empirical results demonstrate the effectiveness of the proposed methods on two college-level science problems over competitive baselines. Our further analysis shows the necessity of prompting LLMs to play different roles or experts independently. We release the code at: https://github.com/amazon-science/comm-prompt

Title: Generative Dataset Distillation: Balancing Global Structure and Local Details

• Authors: Longzhen Li, Guang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.17732
• Pdf URL: https://arxiv.org/pdf/2404.17732
• Copy Paste: [[2404.17732]] Generative Dataset Distillation: Balancing Global Structure and Local Details(https://arxiv.org/abs/2404.17732)
• Keywords: generative
• Abstract: In this paper, we propose a new dataset distillation method that considers balancing global structure and local details when distilling the information from a large dataset into a generative model. Dataset distillation has been proposed to reduce the size of the required dataset when training models. The conventional dataset distillation methods face the problem of long redeployment time and poor cross-architecture performance. Moreover, previous methods focused too much on the high-level semantic attributes between the synthetic dataset and the original dataset while ignoring the local features such as texture and shape. Based on the above understanding, we propose a new method for distilling the original image dataset into a generative model. Our method involves using a conditional generative adversarial network to generate the distilled dataset. Subsequently, we ensure balancing global structure and local details in the distillation process, continuously optimizing the generator for more information-dense dataset generation.

Title: Building a Large Japanese Web Corpus for Large Language Models

• Authors: Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Rio Yokota, Sakae Mizuki
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17733
• Pdf URL: https://arxiv.org/pdf/2404.17733
• Copy Paste: [[2404.17733]] Building a Large Japanese Web Corpus for Large Language Models(https://arxiv.org/abs/2404.17733)
• Keywords: large language model
• Abstract: Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were not created for the quality of Japanese texts. This study builds a large Japanese web corpus by extracting and refining text from the Common Crawl archive (21 snapshots of approximately 63.4 billion pages crawled between 2020 and 2023). This corpus consists of approximately 312.1 billion characters (approximately 173 million pages), which is the largest of all available training corpora for Japanese LLMs, surpassing CC-100 (approximately 25.8 billion characters), mC4 (approximately 239.7 billion characters) and OSCAR 23.10 (approximately 74 billion characters). To confirm the quality of the corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct as base LLMs and gained consistent (6.6-8.1 points) improvements on Japanese benchmark datasets. We also demonstrate that the improvement on Llama 2 13B brought from the presented corpus was the largest among those from other existing corpora.

Title: Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models

• Authors: Aneesh Komanduri, Chen Zhao, Feng Chen, Xintao Wu
• Subjects: cs.LG, cs.AI, stat.ME
• Abstract URL: https://arxiv.org/abs/2404.17735
• Pdf URL: https://arxiv.org/pdf/2404.17735
• Copy Paste: [[2404.17735]] Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models(https://arxiv.org/abs/2404.17735)
• Keywords: diffusion
• Abstract: Diffusion probabilistic models (DPMs) have become the state-of-the-art in high-quality image generation. However, DPMs have an arbitrary noisy latent space with no interpretable or controllable semantics. Although there has been significant research effort to improve image sample quality, there is little work on representation-controlled generation using diffusion models. Specifically, causal modeling and controllable counterfactual generation using DPMs is an underexplored area. In this work, we propose CausalDiffAE, a diffusion-based causal representation learning framework to enable counterfactual generation according to a specified causal model. Our key idea is to use an encoder to extract high-level semantically meaningful causal variables from high-dimensional data and model stochastic variation using reverse diffusion. We propose a causal encoding mechanism that maps high-dimensional data to causally related latent factors and parameterize the causal mechanisms among latent factors using neural networks. To enforce the disentanglement of causal variables, we formulate a variational objective and leverage auxiliary label information in a prior to regularize the latent space. We propose a DDIM-based counterfactual generation procedure subject to do-interventions. Finally, to address the limited label supervision scenario, we also study the application of CausalDiffAE when a part of the training data is unlabeled, which also enables granular control over the strength of interventions in generating counterfactuals during inference. We empirically show that CausalDiffAE learns a disentangled latent space and is capable of generating high-quality counterfactual images.

Title: Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification

• Authors: Chao Yi, Lu Ren, De-Chuan Zhan, Han-Jia Ye
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17753
• Pdf URL: https://arxiv.org/pdf/2404.17753
• Copy Paste: [[2404.17753]] Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification(https://arxiv.org/abs/2404.17753)
• Keywords: robust, extraction, data-free
• Abstract: CLIP showcases exceptional cross-modal matching capabilities due to its training on image-text contrastive learning tasks. However, without specific optimization for unimodal scenarios, its performance in single-modality feature extraction might be suboptimal. Despite this, some studies have directly used CLIP's image encoder for tasks like few-shot classification, introducing a misalignment between its pre-training objectives and feature extraction methods. This inconsistency can diminish the quality of the image's feature representation, adversely affecting CLIP's effectiveness in target tasks. In this paper, we view text features as precise neighbors of image features in CLIP's space and present a novel CrOss-moDal nEighbor Representation(CODER) based on the distance structure between images and their neighbor texts. This feature extraction method aligns better with CLIP's pre-training objectives, thereby fully leveraging CLIP's robust cross-modal capabilities. The key to construct a high-quality CODER lies in how to create a vast amount of high-quality and diverse texts to match with images. We introduce the Auto Text Generator(ATG) to automatically generate the required texts in a data-free and training-free manner. We apply CODER to CLIP's zero-shot and few-shot image classification tasks. Experiment results across various datasets and models confirm CODER's effectiveness. Code is available at:https://github.com/YCaigogogo/CVPR24-CODER.

Title: Adversarial Examples: Generation Proposal in the Context of Facial Recognition Systems

• Authors: Marina Fuster, Ignacio Vidaurreta
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.17760
• Pdf URL: https://arxiv.org/pdf/2404.17760
• Copy Paste: [[2404.17760]] Adversarial Examples: Generation Proposal in the Context of Facial Recognition Systems(https://arxiv.org/abs/2404.17760)
• Keywords: attack
• Abstract: In this paper we investigate the vulnerability that facial recognition systems present to adversarial examples by introducing a new methodology from the attacker perspective. The technique is based on the use of the autoencoder latent space, organized with principal component analysis. We intend to analyze the potential to craft adversarial examples suitable for both dodging and impersonation attacks, against state-of-the-art systems. Our initial hypothesis, which was not strongly favoured by the results, stated that it would be possible to separate between the "identity" and "facial expression" features to produce high-quality examples. Despite the findings not supporting it, the results sparked insights into adversarial examples generation and opened new research avenues in the area.

Title: Large Multi-modality Model Assisted AI-Generated Image Quality Assessment

• Authors: Puyi Wang, Wei Sun, Zicheng Zhang, Jun Jia, Yanwei Jiang, Zhichao Zhang, Xiongkuo Min, Guangtao Zhai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17762
• Pdf URL: https://arxiv.org/pdf/2404.17762
• Copy Paste: [[2404.17762]] Large Multi-modality Model Assisted AI-Generated Image Quality Assessment(https://arxiv.org/abs/2404.17762)
• Keywords: transformer
• Abstract: Traditional deep neural network (DNN)-based image quality assessment (IQA) models leverage convolutional neural networks (CNN) or Transformer to learn the quality-aware feature representation, achieving commendable performance on natural scene images. However, when applied to AI-Generated images (AGIs), these DNN-based IQA models exhibit subpar performance. This situation is largely due to the semantic inaccuracies inherent in certain AGIs caused by uncontrollable nature of the generation process. Thus, the capability to discern semantic content becomes crucial for assessing the quality of AGIs. Traditional DNN-based IQA models, constrained by limited parameter complexity and training data, struggle to capture complex fine-grained semantic features, making it challenging to grasp the existence and coherence of semantic content of the entire image. To address the shortfall in semantic content perception of current IQA models, we introduce a large Multi-modality model Assisted AI-Generated Image Quality Assessment (MA-AGIQA) model, which utilizes semantically informed guidance to sense semantic information and extract semantic vectors through carefully designed text prompts. Moreover, it employs a mixture of experts (MoE) structure to dynamically integrate the semantic information with the quality-aware features extracted by traditional DNN-based IQA models. Comprehensive experiments conducted on two AI-generated content datasets, AIGCQA-20k and AGIQA-3k show that MA-AGIQA achieves state-of-the-art performance, and demonstrate its superior generalization capabilities on assessing the quality of AGIs. Code is available at https://github.com/wangpuyi/MA-AGIQA.

Title: RFL-CDNet: Towards Accurate Change Detection via Richer Feature Learning

• Authors: Yuhang Gan, Wenjie Xuan, Hang Chen, Juhua Liu, Bo Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17765
• Pdf URL: https://arxiv.org/pdf/2404.17765
• Copy Paste: [[2404.17765]] RFL-CDNet: Towards Accurate Change Detection via Richer Feature Learning(https://arxiv.org/abs/2404.17765)
• Keywords: extraction
• Abstract: Change Detection is a crucial but extremely challenging task of remote sensing image analysis, and much progress has been made with the rapid development of deep learning. However, most existing deep learning-based change detection methods mainly focus on intricate feature extraction and multi-scale feature fusion, while ignoring the insufficient utilization of features in the intermediate stages, thus resulting in sub-optimal results. To this end, we propose a novel framework, named RFL-CDNet, that utilizes richer feature learning to boost change detection performance. Specifically, we first introduce deep multiple supervision to enhance intermediate representations, thus unleashing the potential of backbone feature extractor at each stage. Furthermore, we design the Coarse-To-Fine Guiding (C2FG) module and the Learnable Fusion (LF) module to further improve feature learning and obtain more discriminative feature representations. The C2FG module aims to seamlessly integrate the side prediction from the previous coarse-scale into the current fine-scale prediction in a coarse-to-fine manner, while LF module assumes that the contribution of each stage and each spatial location is independent, thus designing a learnable module to fuse multiple predictions. Experiments on several benchmark datasets show that our proposed RFL-CDNet achieves state-of-the-art performance on WHU cultivated land dataset and CDD dataset, and the second-best performance on WHU building dataset. The source code and models are publicly available at https://github.com/Hhaizee/RFL-CDNet.

Title: MRScore: Evaluating Radiology Report Generation with LLM-based Reward System

• Authors: Yunyi Liu, Zhanyu Wang, Yingshu Li, Xinyu Liang, Lingqiao Liu, Lei Wang, Luping Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17778
• Pdf URL: https://arxiv.org/pdf/2404.17778
• Copy Paste: [[2404.17778]] MRScore: Evaluating Radiology Report Generation with LLM-based Reward System(https://arxiv.org/abs/2404.17778)
• Keywords: large language model
• Abstract: In recent years, automated radiology report generation has experienced significant growth. This paper introduces MRScore, an automatic evaluation metric tailored for radiology report generation by leveraging Large Language Models (LLMs). Conventional NLG (natural language generation) metrics like BLEU are inadequate for accurately assessing the generated radiology reports, as systematically demonstrated by our observations within this paper. To address this challenge, we collaborated with radiologists to develop a framework that guides LLMs for radiology report evaluation, ensuring alignment with human analysis. Our framework includes two key components: i) utilizing GPT to generate large amounts of training data, i.e., reports with different qualities, and ii) pairing GPT-generated reports as accepted and rejected samples and training LLMs to produce MRScore as the model reward. Our experiments demonstrate MRScore's higher correlation with human judgments and superior performance in model selection compared to traditional metrics. Our code and datasets will be available on GitHub.

Title: Temporal Scaling Law for Large Language Models

• Authors: Yizhe Xiong, Xiansheng Chen, Xin Ye, Hui Chen, Zijia Lin, Haoran Lian, Jianwei Niu, Guiguang Ding
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.17785
• Pdf URL: https://arxiv.org/pdf/2404.17785
• Copy Paste: [[2404.17785]] Temporal Scaling Law for Large Language Models(https://arxiv.org/abs/2404.17785)
• Keywords: generative, large language model
• Abstract: Recently, Large Language Models (LLMs) are widely adopted in a wide range of tasks, leading to increasing attention towards the research on how scaling LLMs affects their performance. Existing works, termed as Scaling Laws, have discovered that the loss of LLMs scales as power laws with model size, computational budget, and dataset size. However, the performance of LLMs throughout the training process remains untouched. In this paper, we propose the novel concept of Temporal Scaling Law and study the loss of LLMs from the temporal dimension. We first investigate the imbalance of loss on each token positions and develop a reciprocal-law across model scales and training stages. We then derive the temporal scaling law by studying the temporal patterns of the reciprocal-law parameters. Results on both in-distribution (IID) data and out-of-distribution (OOD) data demonstrate that our temporal scaling law accurately predicts the performance of LLMs in future training stages. Moreover, the temporal scaling law reveals that LLMs learn uniformly on different token positions, despite the loss imbalance. Experiments on pre-training LLMs in various scales show that this phenomenon verifies the default training paradigm for generative language models, in which no re-weighting strategies are attached during training. Overall, the temporal scaling law provides deeper insight into LLM pre-training.

Title: BiLO: Bilevel Local Operator Learning for PDE inverse problems

• Authors: Ray Zirui Zhang, Xiaohui Xie, John Lowengrub
• Subjects: cs.LG, math.NA, math.OC
• Abstract URL: https://arxiv.org/abs/2404.17789
• Pdf URL: https://arxiv.org/pdf/2404.17789
• Copy Paste: [[2404.17789]] BiLO: Bilevel Local Operator Learning for PDE inverse problems(https://arxiv.org/abs/2404.17789)
• Keywords: robust
• Abstract: We propose a new neural network based method for solving inverse problems for partial differential equations (PDEs) by formulating the PDE inverse problem as a bilevel optimization problem. At the upper level, we minimize the data loss with respect to the PDE parameters. At the lower level, we train a neural network to locally approximate the PDE solution operator in the neighborhood of a given set of PDE parameters, which enables an accurate approximation of the descent direction for the upper level optimization problem. The lower level loss function includes the L2 norms of both the residual and its derivative with respect to the PDE parameters. We apply gradient descent simultaneously on both the upper and lower level optimization problems, leading to an effective and fast algorithm. The method, which we refer to as BiLO (Bilevel Local Operator learning), is also able to efficiently infer unknown functions in the PDEs through the introduction of an auxiliary variable. We demonstrate that our method enforces strong PDE constraints, is robust to sparse and noisy data, and eliminates the need to balance the residual and the data loss, which is inherent to soft PDE constraints.

Title: Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities

• Authors: Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, Naoaki Okazaki
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17790
• Pdf URL: https://arxiv.org/pdf/2404.17790
• Copy Paste: [[2404.17790]] Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities(https://arxiv.org/abs/2404.17790)
• Keywords: large language model
• Abstract: Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. In this study, we constructed Swallow, an LLM with enhanced Japanese capability, by extending the vocabulary of Llama 2 to include Japanese characters and conducting continual pre-training on a large Japanese web corpus. Experimental results confirmed that the performance on Japanese tasks drastically improved through continual pre-training, and the performance monotonically increased with the amount of training data up to 100B tokens. Consequently, Swallow achieved superior performance compared to other LLMs that were trained from scratch in English and Japanese. An analysis of the effects of continual pre-training revealed that it was particularly effective for Japanese question answering tasks. Furthermore, to elucidate effective methodologies for cross-lingual continual pre-training from English to Japanese, we investigated the impact of vocabulary expansion and the effectiveness of incorporating parallel corpora. The results showed that the efficiency gained through vocabulary expansion had no negative impact on performance, except for the summarization task, and that the combined use of parallel corpora enhanced translation ability.

Title: CLFT: Camera-LiDAR Fusion Transformer for Semantic Segmentation in Autonomous Driving

• Authors: Junyi Gu, Mauro Bellone, Tomáš Pivoňka, Raivo Sell
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.17793
• Pdf URL: https://arxiv.org/pdf/2404.17793
• Copy Paste: [[2404.17793]] CLFT: Camera-LiDAR Fusion Transformer for Semantic Segmentation in Autonomous Driving(https://arxiv.org/abs/2404.17793)
• Keywords: robust, transformer, segmentation
• Abstract: Critical research about camera-and-LiDAR-based semantic object segmentation for autonomous driving significantly benefited from the recent development of deep learning. Specifically, the vision transformer is the novel ground-breaker that successfully brought the multi-head-attention mechanism to computer vision applications. Therefore, we propose a vision-transformer-based network to carry out camera-LiDAR fusion for semantic segmentation applied to autonomous driving. Our proposal uses the novel progressive-assemble strategy of vision transformers on a double-direction network and then integrates the results in a cross-fusion strategy over the transformer decoder layers. Unlike other works in the literature, our camera-LiDAR fusion transformers have been evaluated in challenging conditions like rain and low illumination, showing robust performance. The paper reports the segmentation results over the vehicle and human classes in different modalities: camera-only, LiDAR-only, and camera-LiDAR fusion. We perform coherent controlled benchmark experiments of CLFT against other networks that are also designed for semantic segmentation. The experiments aim to evaluate the performance of CLFT independently from two perspectives: multimodal sensor fusion and backbone architectures. The quantitative assessments show our CLFT networks yield an improvement of up to 10\% for challenging dark-wet conditions when comparing with Fully-Convolutional-Neural-Network-based (FCN) camera-LiDAR fusion neural network. Contrasting to the network with transformer backbone but using single modality input, the all-around improvement is 5-10\%.

Title: Personalized Federated Learning via Sequential Layer Expansion in Representation Learning

• Authors: Jaewon Jang, Bonjun Choi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17799
• Pdf URL: https://arxiv.org/pdf/2404.17799
• Copy Paste: [[2404.17799]] Personalized Federated Learning via Sequential Layer Expansion in Representation Learning(https://arxiv.org/abs/2404.17799)
• Keywords: privacy, federate
• Abstract: Federated learning ensures the privacy of clients by conducting distributed training on individual client devices and sharing only the model weights with a central server. However, in real-world scenarios, the heterogeneity of data among clients necessitates appropriate personalization methods. In this paper, we aim to address this heterogeneity using a form of parameter decoupling known as representation learning. Representation learning divides deep learning models into 'base' and 'head' components. The base component, capturing common features across all clients, is shared with the server, while the head component, capturing unique features specific to individual clients, remains local. We propose a new representation learning-based approach that suggests decoupling the entire deep learning model into more densely divided parts with the application of suitable scheduling methods, which can benefit not only data heterogeneity but also class heterogeneity. In this paper, we compare and analyze two layer scheduling approaches, namely forward (\textit{Vanilla}) and backward (\textit{Anti}), in the context of data and class heterogeneity among clients. Our experimental results show that the proposed algorithm, when compared to existing personalized federated learning algorithms, achieves increased accuracy, especially under challenging conditions, while reducing computation costs.

Title: Dynamical Mode Recognition of Coupled Flame Oscillators by Supervised and Unsupervised Learning Approaches

• Authors: Weiming Xu, Tao Yang, Peng Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.17801
• Pdf URL: https://arxiv.org/pdf/2404.17801
• Copy Paste: [[2404.17801]] Dynamical Mode Recognition of Coupled Flame Oscillators by Supervised and Unsupervised Learning Approaches(https://arxiv.org/abs/2404.17801)
• Keywords: diffusion
• Abstract: Combustion instability in gas turbines and rocket engines, as one of the most challenging problems in combustion research, arises from the complex interactions among flames, which are also influenced by chemical reactions, heat and mass transfer, and acoustics. Identifying and understanding combustion instability is essential to ensure the safe and reliable operation of many combustion systems, where exploring and classifying the dynamical behaviors of complex flame systems is a core take. To facilitate fundamental studies, the present work concerns dynamical mode recognition of coupled flame oscillators made of flickering buoyant diffusion flames, which have gained increasing attention in recent years but are not sufficiently understood. The time series data of flame oscillators are generated by fully validated reacting flow simulations. Due to limitations of expertise-based models, a data-driven approach is adopted. In this study, a nonlinear dimensional reduction model of variational autoencoder (VAE) is used to project the simulation data onto a 2-dimensional latent space. Based on the phase trajectories in latent space, both supervised and unsupervised classifiers are proposed for datasets with well known labeling and without, respectively. For labeled datasets, we establish the Wasserstein-distance-based classifier (WDC) for mode recognition; for unlabeled datasets, we develop a novel unsupervised classifier (GMM-DTWC) combining dynamic time warping (DTW) and Gaussian mixture model (GMM). Through comparing with conventional approaches for dimensionality reduction and classification, the proposed supervised and unsupervised VAE-based approaches exhibit a prominent performance for distinguishing dynamical modes, implying their potential extension to dynamical mode recognition of complex combustion problems.

Title: Empirical Analysis of Dialogue Relation Extraction with Large Language Models

• Authors: Guozheng Li, Zijie Xu, Ziyu Shang, Jiajun Liu, Ke Ji, Yikai Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17802
• Pdf URL: https://arxiv.org/pdf/2404.17802
• Copy Paste: [[2404.17802]] Empirical Analysis of Dialogue Relation Extraction with Large Language Models(https://arxiv.org/abs/2404.17802)
• Keywords: extraction, large language model
• Abstract: Dialogue relation extraction (DRE) aims to extract relations between two arguments within a dialogue, which is more challenging than standard RE due to the higher person pronoun frequency and lower information density in dialogues. However, existing DRE methods still suffer from two serious issues: (1) hard to capture long and sparse multi-turn information, and (2) struggle to extract golden relations based on partial dialogues, which motivates us to discover more effective methods that can alleviate the above issues. We notice that the rise of large language models (LLMs) has sparked considerable interest in evaluating their performance across diverse tasks. To this end, we initially investigate the capabilities of different LLMs in DRE, considering both proprietary models and open-source models. Interestingly, we discover that LLMs significantly alleviate two issues in existing DRE methods. Generally, we have following findings: (1) scaling up model size substantially boosts the overall DRE performance and achieves exceptional results, tackling the difficulty of capturing long and sparse multi-turn information; (2) LLMs encounter with much smaller performance drop from entire dialogue setting to partial dialogue setting compared to existing methods; (3) LLMs deliver competitive or superior performances under both full-shot and few-shot settings compared to current state-of-the-art; (4) LLMs show modest performances on inverse relations but much stronger improvements on general relations, and they can handle dialogues of various lengths especially for longer sequences.

Title: From Optimization to Generalization: Fair Federated Learning against Quality Shift via Inter-Client Sharpness Matching

• Authors: Nannan Wu, Zhuo Kuang, Zengqiang Yan, Li Yu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17805
• Pdf URL: https://arxiv.org/pdf/2404.17805
• Copy Paste: [[2404.17805]] From Optimization to Generalization: Fair Federated Learning against Quality Shift via Inter-Client Sharpness Matching(https://arxiv.org/abs/2404.17805)
• Keywords: privacy, federate, fair
• Abstract: Due to escalating privacy concerns, federated learning has been recognized as a vital approach for training deep neural networks with decentralized medical data. In practice, it is challenging to ensure consistent imaging quality across various institutions, often attributed to equipment malfunctions affecting a minority of clients. This imbalance in image quality can cause the federated model to develop an inherent bias towards higher-quality images, thus posing a severe fairness issue. In this study, we pioneer the identification and formulation of this new fairness challenge within the context of the imaging quality shift. Traditional methods for promoting fairness in federated learning predominantly focus on balancing empirical risks across diverse client distributions. This strategy primarily facilitates fair optimization across different training data distributions, yet neglects the crucial aspect of generalization. To address this, we introduce a solution termed Federated learning with Inter-client Sharpness Matching (FedISM). FedISM enhances both local training and global aggregation by incorporating sharpness-awareness, aiming to harmonize the sharpness levels across clients for fair generalization. Our empirical evaluations, conducted using the widely-used ICH and ISIC 2019 datasets, establish FedISM's superiority over current state-of-the-art federated learning methods in promoting fairness. Code is available at https://github.com/wnn2000/FFL4MIA.

Title: Meta In-Context Learning Makes Large Language Models Better Zero and Few-Shot Relation Extractors

• Authors: Guozheng Li, Peng Wang, Jiajun Liu, Yikai Guo, Ke Ji, Ziyu Shang, Zijie Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17807
• Pdf URL: https://arxiv.org/pdf/2404.17807
• Copy Paste: [[2404.17807]] Meta In-Context Learning Makes Large Language Models Better Zero and Few-Shot Relation Extractors(https://arxiv.org/abs/2404.17807)
• Keywords: extraction, large language model
• Abstract: Relation extraction (RE) is an important task that aims to identify the relationships between entities in texts. While large language models (LLMs) have revealed remarkable in-context learning (ICL) capability for general zero and few-shot learning, recent studies indicate that current LLMs still struggle with zero and few-shot RE. Previous studies are mainly dedicated to design prompt formats and select good examples for improving ICL-based RE. Although both factors are vital for ICL, if one can fundamentally boost the ICL capability of LLMs in RE, the zero and few-shot RE performance via ICL would be significantly improved. To this end, we introduce \textsc{Micre} (\textbf{M}eta \textbf{I}n-\textbf{C}ontext learning of LLMs for \textbf{R}elation \textbf{E}xtraction), a new meta-training framework for zero and few-shot RE where an LLM is tuned to do ICL on a diverse collection of RE datasets (i.e., learning to learn in context for RE). Through meta-training, the model becomes more effectively to learn a new RE task in context by conditioning on a few training examples with no parameter updates or task-specific templates at inference time, enabling better zero and few-shot task generalization. We experiment \textsc{Micre} on various LLMs with different model scales and 12 public RE datasets, and then evaluate it on unseen RE benchmarks under zero and few-shot settings. \textsc{Micre} delivers comparable or superior performance compared to a range of baselines including supervised fine-tuning and typical in-context learning methods. We find that the gains are particular significant for larger model scales, and using a diverse set of the meta-training RE datasets is key to improvements. Empirically, we show that \textsc{Micre} can transfer the relation semantic knowledge via relation label name during inference on target RE datasets.

Title: Recall, Retrieve and Reason: Towards Better In-Context Relation Extraction

• Authors: Guozheng Li, Peng Wang, Wenjun Ke, Yikai Guo, Ke Ji, Ziyu Shang, Jiajun Liu, Zijie Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17809
• Pdf URL: https://arxiv.org/pdf/2404.17809
• Copy Paste: [[2404.17809]] Recall, Retrieve and Reason: Towards Better In-Context Relation Extraction(https://arxiv.org/abs/2404.17809)
• Keywords: extraction, large language model
• Abstract: Relation extraction (RE) aims to identify relations between entities mentioned in texts. Although large language models (LLMs) have demonstrated impressive in-context learning (ICL) abilities in various tasks, they still suffer from poor performances compared to most supervised fine-tuned RE methods. Utilizing ICL for RE with LLMs encounters two challenges: (1) retrieving good demonstrations from training examples, and (2) enabling LLMs exhibit strong ICL abilities in RE. On the one hand, retrieving good demonstrations is a non-trivial process in RE, which easily results in low relevance regarding entities and relations. On the other hand, ICL with an LLM achieves poor performance in RE while RE is different from language modeling in nature or the LLM is not large enough. In this work, we propose a novel recall-retrieve-reason RE framework that synergizes LLMs with retrieval corpora (training examples) to enable relevant retrieving and reliable in-context reasoning. Specifically, we distill the consistently ontological knowledge from training datasets to let LLMs generate relevant entity pairs grounded by retrieval corpora as valid queries. These entity pairs are then used to retrieve relevant training examples from the retrieval corpora as demonstrations for LLMs to conduct better ICL via instruction tuning. Extensive experiments on different LLMs and RE datasets demonstrate that our method generates relevant and valid entity pairs and boosts ICL abilities of LLMs, achieving competitive or new state-of-the-art performance on sentence-level RE compared to previous supervised fine-tuning methods and ICL-based methods.

Title: VANER: Leveraging Large Language Model for Versatile and Adaptive Biomedical Named Entity Recognition

• Authors: Junyi Biana, Weiqi Zhai, Xiaodi Huang, Jiaxuan Zheng, Shanfeng Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.17835
• Pdf URL: https://arxiv.org/pdf/2404.17835
• Copy Paste: [[2404.17835]] VANER: Leveraging Large Language Model for Versatile and Adaptive Biomedical Named Entity Recognition(https://arxiv.org/abs/2404.17835)
• Keywords: extraction, generative, large language model
• Abstract: Prevalent solution for BioNER involves using representation learning techniques coupled with sequence labeling. However, such methods are inherently task-specific, demonstrate poor generalizability, and often require dedicated model for each dataset. To leverage the versatile capabilities of recently remarkable large language models (LLMs), several endeavors have explored generative approaches to entity extraction. Yet, these approaches often fall short of the effectiveness of previouly sequence labeling approaches. In this paper, we utilize the open-sourced LLM LLaMA2 as the backbone model, and design specific instructions to distinguish between different types of entities and datasets. By combining the LLM's understanding of instructions with sequence labeling techniques, we use mix of datasets to train a model capable of extracting various types of entities. Given that the backbone LLMs lacks specialized medical knowledge, we also integrate external entity knowledge bases and employ instruction tuning to compel the model to densely recognize carefully curated entities. Our model VANER, trained with a small partition of parameters, significantly outperforms previous LLMs-based models and, for the first time, as a model based on LLM, surpasses the majority of conventional state-of-the-art BioNER systems, achieving the highest F1 scores across three datasets.

Title: Improving Smart Contract Security with Contrastive Learning-based Vulnerability Detection

• Authors: Yizhou Chen, Zeyu Sun, Zhihao Gong, Dan Hao
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.17839
• Pdf URL: https://arxiv.org/pdf/2404.17839
• Copy Paste: [[2404.17839]] Improving Smart Contract Security with Contrastive Learning-based Vulnerability Detection(https://arxiv.org/abs/2404.17839)
• Keywords: security
• Abstract: Currently, smart contract vulnerabilities (SCVs) have emerged as a major factor threatening the transaction security of blockchain. Existing state-of-the-art methods rely on deep learning to mitigate this threat. They treat each input contract as an independent entity and feed it into a deep learning model to learn vulnerability patterns by fitting vulnerability labels. It is a pity that they disregard the correlation between contracts, failing to consider the commonalities between contracts of the same type and the differences among contracts of different types. As a result, the performance of these methods falls short of the desired level. To tackle this problem, we propose a novel Contrastive Learning Enhanced Automated Recognition Approach for Smart Contract Vulnerabilities, named Clear. In particular, Clear employs a contrastive learning (CL) model to capture the fine-grained correlation information among contracts and generates correlation labels based on the relationships between contracts to guide the training process of the CL model. Finally, it combines the correlation and the semantic information of the contract to detect SCVs. Through an empirical evaluation of a large-scale real-world dataset of over 40K smart contracts and compare 13 state-of-the-art baseline methods. We show that Clear achieves (1) optimal performance over all baseline methods; (2) 9.73%-39.99% higher F1-score than existing deep learning methods.

Title: Toxicity Classification in Ukrainian

• Authors: Daryna Dementieva, Valeriia Khylenko, Nikolay Babakov, Georg Groh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.17841
• Pdf URL: https://arxiv.org/pdf/2404.17841
• Copy Paste: [[2404.17841]] Toxicity Classification in Ukrainian(https://arxiv.org/abs/2404.17841)
• Keywords: robust, fair
• Abstract: The task of toxicity detection is still a relevant task, especially in the context of safe and fair LMs development. Nevertheless, labeled binary toxicity classification corpora are not available for all languages, which is understandable given the resource-intensive nature of the annotation process. Ukrainian, in particular, is among the languages lacking such resources. To our knowledge, there has been no existing toxicity classification corpus in Ukrainian. In this study, we aim to fill this gap by investigating cross-lingual knowledge transfer techniques and creating labeled corpora by: (i)~translating from an English corpus, (ii)~filtering toxic samples using keywords, and (iii)~annotating with crowdsourcing. We compare LLMs prompting and other cross-lingual transfer approaches with and without fine-tuning offering insights into the most robust and efficient baselines.

Title: pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learning

• Authors: Liping Yi, Han Yu, Chao Ren, Heng Zhang, Gang Wang, Xiaoguang Liu, Xiaoxiao Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.17847
• Pdf URL: https://arxiv.org/pdf/2404.17847
• Copy Paste: [[2404.17847]] pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learning(https://arxiv.org/abs/2404.17847)
• Keywords: federate
• Abstract: Model-heterogeneous personalized federated learning (MHPFL) enables FL clients to train structurally different personalized models on non-independent and identically distributed (non-IID) local data. Existing MHPFL methods focus on achieving client-level personalization, but cannot address batch-level data heterogeneity. To bridge this important gap, we propose a model-heterogeneous personalized Federated learning approach with Adaptive Feature Mixture (pFedAFM) for supervised learning tasks. It consists of three novel designs: 1) A sharing global homogeneous small feature extractor is assigned alongside each client's local heterogeneous model (consisting of a heterogeneous feature extractor and a prediction header) to facilitate cross-client knowledge fusion. The two feature extractors share the local heterogeneous model's prediction header containing rich personalized prediction knowledge to retain personalized prediction capabilities. 2) An iterative training strategy is designed to alternately train the global homogeneous small feature extractor and the local heterogeneous large model for effective global-local knowledge exchange. 3) A trainable weight vector is designed to dynamically mix the features extracted by both feature extractors to adapt to batch-level data heterogeneity. Theoretical analysis proves that pFedAFM can converge over time. Extensive experiments on 2 benchmark datasets demonstrate that it significantly outperforms 7 state-of-the-art MHPFL methods, achieving up to 7.93% accuracy improvement while incurring low communication and computation costs.

Title: GLIMS: Attention-Guided Lightweight Multi-Scale Hybrid Network for Volumetric Semantic Segmentation

• Authors: Ziya Ata Yazıcı, İlkay Öksüz, Hazım Kemal Ekenel
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17854
• Pdf URL: https://arxiv.org/pdf/2404.17854
• Copy Paste: [[2404.17854]] GLIMS: Attention-Guided Lightweight Multi-Scale Hybrid Network for Volumetric Semantic Segmentation(https://arxiv.org/abs/2404.17854)
• Keywords: robust, transformer, segmentation
• Abstract: Convolutional Neural Networks (CNNs) have become widely adopted for medical image segmentation tasks, demonstrating promising performance. However, the inherent inductive biases in convolutional architectures limit their ability to model long-range dependencies and spatial correlations. While recent transformer-based architectures address these limitations by leveraging self-attention mechanisms to encode long-range dependencies and learn expressive representations, they often struggle to extract low-level features and are highly dependent on data availability. This motivated us for the development of GLIMS, a data-efficient attention-guided hybrid volumetric segmentation network. GLIMS utilizes Dilated Feature Aggregator Convolutional Blocks (DACB) to capture local-global feature correlations efficiently. Furthermore, the incorporated Swin Transformer-based bottleneck bridges the local and global features to improve the robustness of the model. Additionally, GLIMS employs an attention-guided segmentation approach through Channel and Spatial-Wise Attention Blocks (CSAB) to localize expressive features for fine-grained border segmentation. Quantitative and qualitative results on glioblastoma and multi-organ CT segmentation tasks demonstrate GLIMS' effectiveness in terms of complexity and accuracy. GLIMS demonstrated outstanding performance on BraTS2021 and BTCV datasets, surpassing the performance of Swin UNETR. Notably, GLIMS achieved this high performance with a significantly reduced number of trainable parameters. Specifically, GLIMS has 47.16M trainable parameters and 72.30G FLOPs, while Swin UNETR has 61.98M trainable parameters and 394.84G FLOPs. The code is publicly available on https://github.com/yaziciz/GLIMS.

Title: Revisiting Multi-modal Emotion Learning with Broad State Space Models and Probability-guidance Fusion

• Authors: Yuntao Shou, Tao Meng, Fuchen Zhang, Nan Yin, Keqin Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.17858
• Pdf URL: https://arxiv.org/pdf/2404.17858
• Copy Paste: [[2404.17858]] Revisiting Multi-modal Emotion Learning with Broad State Space Models and Probability-guidance Fusion(https://arxiv.org/abs/2404.17858)
• Keywords: transformer
• Abstract: Multi-modal Emotion Recognition in Conversation (MERC) has received considerable attention in various fields, e.g., human-computer interaction and recommendation systems. Most existing works perform feature disentanglement and fusion to extract emotional contextual information from multi-modal features and emotion classification. After revisiting the characteristic of MERC, we argue that long-range contextual semantic information should be extracted in the feature disentanglement stage and the inter-modal semantic information consistency should be maximized in the feature fusion stage. Inspired by recent State Space Models (SSMs), Mamba can efficiently model long-distance dependencies. Therefore, in this work, we fully consider the above insights to further improve the performance of MERC. Specifically, on the one hand, in the feature disentanglement stage, we propose a Broad Mamba, which does not rely on a self-attention mechanism for sequence modeling, but uses state space models to compress emotional representation, and utilizes broad learning systems to explore the potential data distribution in broad space. Different from previous SSMs, we design a bidirectional SSM convolution to extract global context information. On the other hand, we design a multi-modal fusion strategy based on probability guidance to maximize the consistency of information between modalities. Experimental results show that the proposed method can overcome the computational and memory limitations of Transformer when modeling long-distance contexts, and has great potential to become a next-generation general architecture in MERC.

Title: Solvent: liquidity verification of smart contracts

• Authors: Massimo Bartoletti, Angelo Ferrando, Enrico Lipparini, Vadim Malvone
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2404.17864
• Pdf URL: https://arxiv.org/pdf/2404.17864
• Copy Paste: [[2404.17864]] Solvent: liquidity verification of smart contracts(https://arxiv.org/abs/2404.17864)
• Keywords: security, attack
• Abstract: Smart contracts are programs executed by blockchains networks to regulate the exchange of crypto-assets between untrusted users. Due to their immutability, public accessibility and high value at stake, smart contracts are an attractive target for attackers, as evidenced by a long history of security incidents. This has been a driving factor for the application of formal methods to Ethereum, the leading smart contract platform, and Solidity, its main smart contract language, which have become the target of dozens of verification tools with varying objectives. A current limitation of these tools is that they are not really effective in expressing and verifying liquidity properties regarding the exchange of crypto-assets: for example, is it true that in every reachable state a user can fire a sequence of transactions to withdraw a given amount of crypto-assets? We propose Solvent, a tool aimed at verifying these kinds of properties, which are beyond the reach of existing verification tools for Solidity. We evaluate the effectiveness and performance of Solvent through a common benchmark of smart contracts.

Title: Are Watermarks Bugs for Deepfake Detectors? Rethinking Proactive Forensics

• Authors: Xiaoshuai Wu, Xin Liao, Bo Ou, Yuling Liu, Zheng Qin
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.17867
• Pdf URL: https://arxiv.org/pdf/2404.17867
• Copy Paste: [[2404.17867]] Are Watermarks Bugs for Deepfake Detectors? Rethinking Proactive Forensics(https://arxiv.org/abs/2404.17867)
• Keywords: robust, watermark
• Abstract: AI-generated content has accelerated the topic of media synthesis, particularly Deepfake, which can manipulate our portraits for positive or malicious purposes. Before releasing these threatening face images, one promising forensics solution is the injection of robust watermarks to track their own provenance. However, we argue that current watermarking models, originally devised for genuine images, may harm the deployed Deepfake detectors when directly applied to forged images, since the watermarks are prone to overlap with the forgery signals used for detection. To bridge this gap, we thus propose AdvMark, on behalf of proactive forensics, to exploit the adversarial vulnerability of passive detectors for good. Specifically, AdvMark serves as a plug-and-play procedure for fine-tuning any robust watermarking into adversarial watermarking, to enhance the forensic detectability of watermarked images; meanwhile, the watermarks can still be extracted for provenance tracking. Extensive experiments demonstrate the effectiveness of the proposed AdvMark, leveraging robust watermarking to fool Deepfake detectors, which can help improve the accuracy of downstream Deepfake detection without tuning the in-the-wild detectors. We believe this work will shed some light on the harmless proactive forensics against Deepfake.

Title: Feature graphs for interpretable unsupervised tree ensembles: centrality, interaction, and application in disease subtyping

• Authors: Christel Sirocchi, Martin Urschler, Bastian Pfeifer
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17886
• Pdf URL: https://arxiv.org/pdf/2404.17886
• Copy Paste: [[2404.17886]] Feature graphs for interpretable unsupervised tree ensembles: centrality, interaction, and application in disease subtyping(https://arxiv.org/abs/2404.17886)
• Keywords: interpretability
• Abstract: Interpretable machine learning has emerged as central in leveraging artificial intelligence within high-stakes domains such as healthcare, where understanding the rationale behind model predictions is as critical as achieving high predictive accuracy. In this context, feature selection assumes a pivotal role in enhancing model interpretability by identifying the most important input features in black-box models. While random forests are frequently used in biomedicine for their remarkable performance on tabular datasets, the accuracy gained from aggregating decision trees comes at the expense of interpretability. Consequently, feature selection for enhancing interpretability in random forests has been extensively explored in supervised settings. However, its investigation in the unsupervised regime remains notably limited. To address this gap, the study introduces novel methods to construct feature graphs from unsupervised random forests and feature selection strategies to derive effective feature combinations from these graphs. Feature graphs are constructed for the entire dataset as well as individual clusters leveraging the parent-child node splits within the trees, such that feature centrality captures their relevance to the clustering task, while edge weights reflect the discriminating power of feature pairs. Graph-based feature selection methods are extensively evaluated on synthetic and benchmark datasets both in terms of their ability to reduce dimensionality while improving clustering performance, as well as to enhance model interpretability. An application on omics data for disease subtyping identifies the top features for each cluster, showcasing the potential of the proposed approach to enhance interpretability in clustering analyses and its utility in a real-world biomedical application.

Title: A Hybrid Approach for Document Layout Analysis in Document images

• Authors: Tahira Shehzadi, Didier Stricker, Muhammad Zeshan Afzal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17888
• Pdf URL: https://arxiv.org/pdf/2404.17888
• Copy Paste: [[2404.17888]] A Hybrid Approach for Document Layout Analysis in Document images(https://arxiv.org/abs/2404.17888)
• Keywords: extraction, transformer
• Abstract: Document layout analysis involves understanding the arrangement of elements within a document. This paper navigates the complexities of understanding various elements within document images, such as text, images, tables, and headings. The approach employs an advanced Transformer-based object detection network as an innovative graphical page object detector for identifying tables, figures, and displayed elements. We introduce a query encoding mechanism to provide high-quality object queries for contrastive learning, enhancing efficiency in the decoder phase. We also present a hybrid matching scheme that integrates the decoder's original one-to-one matching strategy with the one-to-many matching strategy during the training phase. This approach aims to improve the model's accuracy and versatility in detecting various graphical elements on a page. Our experiments on PubLayNet, DocLayNet, and PubTables benchmarks show that our approach outperforms current state-of-the-art methods. It achieves an average precision of 97.3% on PubLayNet, 81.6% on DocLayNet, and 98.6 on PubTables, demonstrating its superior performance in layout analysis. These advancements not only enhance the conversion of document images into editable and accessible formats but also streamline information retrieval and data extraction processes.

Title: Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models

• Authors: Zhongzhen Huang, Kui Xue, Yongqi Fan, Linjie Mu, Ruoyu Liu, Tong Ruan, Shaoting Zhang, Xiaofan Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.17897
• Pdf URL: https://arxiv.org/pdf/2404.17897
• Copy Paste: [[2404.17897]] Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models(https://arxiv.org/abs/2404.17897)
• Keywords: large language model
• Abstract: Large-scale language models (LLMs) have achieved remarkable success across various language tasks but suffer from hallucinations and temporal misalignment. To mitigate these shortcomings, Retrieval-augmented generation (RAG) has been utilized to provide external knowledge to facilitate the answer generation. However, applying such models to the medical domain faces several challenges due to the lack of domain-specific knowledge and the intricacy of real-world scenarios. In this study, we explore LLMs with RAG framework for knowledge-intensive tasks in the medical field. To evaluate the capabilities of LLMs, we introduce MedicineQA, a multi-round dialogue benchmark that simulates the real-world medication consultation scenario and requires LLMs to answer with retrieved evidence from the medicine database. MedicineQA contains 300 multi-round question-answering pairs, each embedded within a detailed dialogue history, highlighting the challenge posed by this knowledge-intensive task to current LLMs. We further propose a new \textit{Distill-Retrieve-Read} framework instead of the previous \textit{Retrieve-then-Read}. Specifically, the distillation and retrieval process utilizes a tool calling mechanism to formulate search queries that emulate the keyword-based inquiries used by search engines. With experimental results, we show that our framework brings notable performance improvements and surpasses the previous counterparts in the evidence retrieval process in terms of evidence retrieval accuracy. This advancement sheds light on applying RAG to the medical domain.

Title: Unsupervised Anomaly Detection via Masked Diffusion Posterior Sampling

• Authors: Di Wu, Shicai Fan, Xue Zhou, Li Yu, Yuzhong Deng, Jianxiao Zou, Baihong Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17900
• Pdf URL: https://arxiv.org/pdf/2404.17900
• Copy Paste: [[2404.17900]] Unsupervised Anomaly Detection via Masked Diffusion Posterior Sampling(https://arxiv.org/abs/2404.17900)
• Keywords: diffusion, generative
• Abstract: Reconstruction-based methods have been commonly used for unsupervised anomaly detection, in which a normal image is reconstructed and compared with the given test image to detect and locate anomalies. Recently, diffusion models have shown promising applications for anomaly detection due to their powerful generative ability. However, these models lack strict mathematical support for normal image reconstruction and unexpectedly suffer from low reconstruction quality. To address these issues, this paper proposes a novel and highly-interpretable method named Masked Diffusion Posterior Sampling (MDPS). In MDPS, the problem of normal image reconstruction is mathematically modeled as multiple diffusion posterior sampling for normal images based on the devised masked noisy observation model and the diffusion-based normal image prior under Bayesian framework. Using a metric designed from pixel-level and perceptual-level perspectives, MDPS can effectively compute the difference map between each normal posterior sample and the given test image. Anomaly scores are obtained by averaging all difference maps for multiple posterior samples. Exhaustive experiments on MVTec and BTAD datasets demonstrate that MDPS can achieve state-of-the-art performance in normal image reconstruction quality as well as anomaly detection and localization.

Title: SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models

• Authors: Manav Nitin Kapadnis, Sohan Patnaik, Abhilash Nandy, Sourjyadip Ray, Pawan Goyal, Debdoot Sheet
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.17912
• Pdf URL: https://arxiv.org/pdf/2404.17912
• Copy Paste: [[2404.17912]] SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models(https://arxiv.org/abs/2404.17912)
• Keywords: robust, large language model
• Abstract: Radiology Report Generation (R2Gen) demonstrates how Multi-modal Large Language Models (MLLMs) can automate the creation of accurate and coherent radiological reports. Existing methods often hallucinate details in text-based reports that don't accurately reflect the image content. To mitigate this, we introduce a novel strategy, SERPENT-VLM (SElf Refining Radiology RePort GENeraTion using Vision Language Models), which improves the R2Gen task by integrating a self-refining mechanism into the MLLM framework. We employ a unique self-supervised loss that leverages similarity between pooled image representations and the contextual representations of the generated radiological text, alongside the standard Causal Language Modeling objective, to refine image-text representations. This allows the model to scrutinize and align the generated text through dynamic interaction between a given image and the generated text, therefore reducing hallucination and continuously enhancing nuanced report generation. SERPENT-VLM outperforms existing baselines such as LLaVA-Med, BiomedGPT, etc., achieving SoTA performance on the IU X-ray and Radiology Objects in COntext (ROCO) datasets, and also proves to be robust against noisy images. A qualitative case study emphasizes the significant advancements towards more sophisticated MLLM frameworks for R2Gen, opening paths for further research into self-supervised refinement in the medical imaging domain.

Title: FedCRL: Personalized Federated Learning with Contrastive Shared Representations for Label Heterogeneity in Non-IID Data

• Authors: Chenghao Huang, Xiaolu Chen, Yanru Zhang, Hao Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17916
• Pdf URL: https://arxiv.org/pdf/2404.17916
• Copy Paste: [[2404.17916]] FedCRL: Personalized Federated Learning with Contrastive Shared Representations for Label Heterogeneity in Non-IID Data(https://arxiv.org/abs/2404.17916)
• Keywords: federate
• Abstract: To deal with heterogeneity resulting from label distribution skew and data scarcity in distributed machine learning scenarios, this paper proposes a novel Personalized Federated Learning (PFL) algorithm, named Federated Contrastive Representation Learning (FedCRL). FedCRL introduces contrastive representation learning (CRL) on shared representations to facilitate knowledge acquisition of clients. Specifically, both local model parameters and averaged values of local representations are considered as shareable information to the server, both of which are then aggregated globally. CRL is applied between local representations and global representations to regularize personalized training by drawing similar representations closer and separating dissimilar ones, thereby enhancing local models with external knowledge and avoiding being harmed by label distribution skew. Additionally, FedCRL adopts local aggregation between each local model and the global model to tackle data scarcity. A loss-wise weighting mechanism is introduced to guide the local aggregation using each local model's contrastive loss to coordinate the global model involvement in each client, thus helping clients with scarce data. Our simulations demonstrate FedCRL's effectiveness in mitigating label heterogeneity by achieving accuracy improvements over existing methods on datasets with varying degrees of label heterogeneity.

Title: EvaNet: Elevation-Guided Flood Extent Mapping on Earth Imagery

• Authors: Mirza Tanzim Sami, Da Yan, Saugat Adhikari, Lyuheng Yuan, Jiao Han, Zhe Jiang, Jalal Khalil, Yang Zhou
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.17917
• Pdf URL: https://arxiv.org/pdf/2404.17917
• Copy Paste: [[2404.17917]] EvaNet: Elevation-Guided Flood Extent Mapping on Earth Imagery(https://arxiv.org/abs/2404.17917)
• Keywords: segmentation
• Abstract: Accurate and timely mapping of flood extent from high-resolution satellite imagery plays a crucial role in disaster management such as damage assessment and relief activities. However, current state-of-the-art solutions are based on U-Net, which can-not segment the flood pixels accurately due to the ambiguous pixels (e.g., tree canopies, clouds) that prevent a direct judgement from only the spectral features. Thanks to the digital elevation model (DEM) data readily available from sources such as United States Geological Survey (USGS), this work explores the use of an elevation map to improve flood extent mapping. We propose, EvaNet, an elevation-guided segmentation model based on the encoder-decoder architecture with two novel techniques: (1) a loss function encoding the physical law of gravity that if a location is flooded (resp. dry), then its adjacent locations with a lower (resp. higher) elevation must also be flooded (resp. dry); (2) a new (de)convolution operation that integrates the elevation map by a location sensitive gating mechanism to regulate how much spectral features flow through adjacent layers. Extensive experiments show that EvaNet significantly outperforms the U-Net baselines, and works as a perfect drop-in replacement for U-Net in existing solutions to flood extent mapping.

Title: Open-Set 3D Semantic Instance Maps for Vision Language Navigation -- O3D-SIM

• Authors: Laksh Nanwani, Kumaraditya Gupta, Aditya Mathur, Swayam Agrawal, A.H. Abdul Hafez, K. Madhava Krishna
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.17922
• Pdf URL: https://arxiv.org/pdf/2404.17922
• Copy Paste: [[2404.17922]] Open-Set 3D Semantic Instance Maps for Vision Language Navigation -- O3D-SIM(https://arxiv.org/abs/2404.17922)
• Keywords: robust, extraction, segmentation
• Abstract: Humans excel at forming mental maps of their surroundings, equipping them to understand object relationships and navigate based on language queries. Our previous work SI Maps [1] showed that having instance-level information and the semantic understanding of an environment helps significantly improve performance for language-guided tasks. We extend this instance-level approach to 3D while increasing the pipeline's robustness and improving quantitative and qualitative results. Our method leverages foundational models for object recognition, image segmentation, and feature extraction. We propose a representation that results in a 3D point cloud map with instance-level embeddings, which bring in the semantic understanding that natural language commands can query. Quantitatively, the work improves upon the success rate of language-guided tasks. At the same time, we qualitatively observe the ability to identify instances more clearly and leverage the foundational models and language and image-aligned embeddings to identify objects that, otherwise, a closed-set approach wouldn't be able to identify.

Title: Spatio-Temporal Side Tuning Pre-trained Foundation Models for Video-based Pedestrian Attribute Recognition

• Authors: Xiao Wang, Qian Zhu, Jiandong Jin, Jun Zhu, Futian Wang, Bo Jiang, Yaowei Wang, Yonghong Tian
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.17929
• Pdf URL: https://arxiv.org/pdf/2404.17929
• Copy Paste: [[2404.17929]] Spatio-Temporal Side Tuning Pre-trained Foundation Models for Video-based Pedestrian Attribute Recognition(https://arxiv.org/abs/2404.17929)
• Keywords: transformer
• Abstract: Existing pedestrian attribute recognition (PAR) algorithms are mainly developed based on a static image, however, the performance is unreliable in challenging scenarios, such as heavy occlusion, motion blur, etc. In this work, we propose to understand human attributes using video frames that can fully use temporal information by fine-tuning a pre-trained multi-modal foundation model efficiently. Specifically, we formulate the video-based PAR as a vision-language fusion problem and adopt a pre-trained foundation model CLIP to extract the visual features. More importantly, we propose a novel spatiotemporal side-tuning strategy to achieve parameter-efficient optimization of the pre-trained vision foundation model. To better utilize the semantic information, we take the full attribute list that needs to be recognized as another input and transform the attribute words/phrases into the corresponding sentence via split, expand, and prompt operations. Then, the text encoder of CLIP is utilized for embedding processed attribute descriptions. The averaged visual tokens and text tokens are concatenated and fed into a fusion Transformer for multi-modal interactive learning. The enhanced tokens will be fed into a classification head for pedestrian attribute prediction. Extensive experiments on two large-scale video-based PAR datasets fully validated the effectiveness of our proposed framework. The source code of this paper is available at https://github.com/Event-AHU/OpenPAR.

Title: Multi-Stream Cellular Test-Time Adaptation of Real-Time Models Evolving in Dynamic Environments

• Authors: Benoît Gérin, Anaïs Halin, Anthony Cioppa, Maxim Henry, Bernard Ghanem, Benoît Macq, Christophe De Vleeschouwer, Marc Van Droogenbroeck
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.17930
• Pdf URL: https://arxiv.org/pdf/2404.17930
• Copy Paste: [[2404.17930]] Multi-Stream Cellular Test-Time Adaptation of Real-Time Models Evolving in Dynamic Environments(https://arxiv.org/abs/2404.17930)
• Keywords: segmentation
• Abstract: In the era of the Internet of Things (IoT), objects connect through a dynamic network, empowered by technologies like 5G, enabling real-time data sharing. However, smart objects, notably autonomous vehicles, face challenges in critical local computations due to limited resources. Lightweight AI models offer a solution but struggle with diverse data distributions. To address this limitation, we propose a novel Multi-Stream Cellular Test-Time Adaptation (MSC-TTA) setup where models adapt on the fly to a dynamic environment divided into cells. Then, we propose a real-time adaptive student-teacher method that leverages the multiple streams available in each cell to quickly adapt to changing data distributions. We validate our methodology in the context of autonomous vehicles navigating across cells defined based on location and weather conditions. To facilitate future benchmarking, we release a new multi-stream large-scale synthetic semantic segmentation dataset, called DADE, and show that our multi-stream approach outperforms a single-stream baseline. We believe that our work will open research opportunities in the IoT and 5G eras, offering solutions for real-time model adaptation.

Title: FDCE-Net: Underwater Image Enhancement with Embedding Frequency and Dual Color Encoder

• Authors: Zheng Cheng, Guodong Fan, Jingchun Zhou, Min Gan, C. L. Philip Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17936
• Pdf URL: https://arxiv.org/pdf/2404.17936
• Copy Paste: [[2404.17936]] FDCE-Net: Underwater Image Enhancement with Embedding Frequency and Dual Color Encoder(https://arxiv.org/abs/2404.17936)
• Keywords: robust
• Abstract: Underwater images often suffer from various issues such as low brightness, color shift, blurred details, and noise due to light absorption and scattering caused by water and suspended particles. Previous underwater image enhancement (UIE) methods have primarily focused on spatial domain enhancement, neglecting the frequency domain information inherent in the images. However, the degradation factors of underwater images are closely intertwined in the spatial domain. Although certain methods focus on enhancing images in the frequency domain, they overlook the inherent relationship between the image degradation factors and the information present in the frequency domain. As a result, these methods frequently enhance certain attributes of the improved image while inadequately addressing or even exacerbating other attributes. Moreover, many existing methods heavily rely on prior knowledge to address color shift problems in underwater images, limiting their flexibility and robustness. In order to overcome these limitations, we propose the Embedding Frequency and Dual Color Encoder Network (FDCE-Net) in our paper. The FDCE-Net consists of two main structures: (1) Frequency Spatial Network (FS-Net) aims to achieve initial enhancement by utilizing our designed Frequency Spatial Residual Block (FSRB) to decouple image degradation factors in the frequency domain and enhance different attributes separately. (2) To tackle the color shift issue, we introduce the Dual-Color Encoder (DCE). The DCE establishes correlations between color and semantic representations through cross-attention and leverages multi-scale image features to guide the optimization of adaptive color query. The final enhanced images are generated by combining the outputs of FS-Net and DCE through a fusion network. These images exhibit rich details, clear textures, low noise and natural colors.

Title: DTization: A New Method for Supervised Feature Scaling

• Authors: Niful Islam
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17937
• Pdf URL: https://arxiv.org/pdf/2404.17937
• Copy Paste: [[2404.17937]] DTization: A New Method for Supervised Feature Scaling(https://arxiv.org/abs/2404.17937)
• Keywords: robust
• Abstract: Artificial intelligence is currently a dominant force in shaping various aspects of the world. Machine learning is a sub-field in artificial intelligence. Feature scaling is one of the data pre-processing techniques that improves the performance of machine learning algorithms. The traditional feature scaling techniques are unsupervised where they do not have influence of the dependent variable in the scaling process. In this paper, we have presented a novel feature scaling technique named DTization that employs decision tree and robust scaler for supervised feature scaling. The proposed method utilizes decision tree to measure the feature importance and based on the importance, different features get scaled differently with the robust scaler algorithm. The proposed method has been extensively evaluated on ten classification and regression datasets on various evaluation matrices and the results show a noteworthy performance improvement compared to the traditional feature scaling methods.

Title: Bounding the Expected Robustness of Graph Neural Networks Subject to Node Feature Attacks

• Authors: Yassine Abbahaddou, Sofiane Ennadir, Johannes F. Lutzeyer, Michalis Vazirgiannis, Henrik Boström
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.17947
• Pdf URL: https://arxiv.org/pdf/2404.17947
• Copy Paste: [[2404.17947]] Bounding the Expected Robustness of Graph Neural Networks Subject to Node Feature Attacks(https://arxiv.org/abs/2404.17947)
• Keywords: defense, attack, robust
• Abstract: Graph Neural Networks (GNNs) have demonstrated state-of-the-art performance in various graph representation learning tasks. Recently, studies revealed their vulnerability to adversarial attacks. In this work, we theoretically define the concept of expected robustness in the context of attributed graphs and relate it to the classical definition of adversarial robustness in the graph representation learning literature. Our definition allows us to derive an upper bound of the expected robustness of Graph Convolutional Networks (GCNs) and Graph Isomorphism Networks subject to node feature attacks. Building on these findings, we connect the expected robustness of GNNs to the orthonormality of their weight matrices and consequently propose an attack-independent, more robust variant of the GCN, called the Graph Convolutional Orthonormal Robust Networks (GCORNs). We further introduce a probabilistic method to estimate the expected robustness, which allows us to evaluate the effectiveness of GCORN on several real-world datasets. Experimental experiments showed that GCORN outperforms available defense methods. Our code is publicly available at: \href{https://github.com/Sennadir/GCORN}{https://github.com/Sennadir/GCORN}.

Title: Cauchy-Schwarz Divergence Information Bottleneck for Regression

• Authors: Shujian Yu, Xi Yu, Sigurd Løkse, Robert Jenssen, Jose C. Principe
• Subjects: cs.LG, cs.IT, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.17951
• Pdf URL: https://arxiv.org/pdf/2404.17951
• Copy Paste: [[2404.17951]] Cauchy-Schwarz Divergence Information Bottleneck for Regression(https://arxiv.org/abs/2404.17951)
• Keywords: robust, explainability
• Abstract: The information bottleneck (IB) approach is popular to improve the generalization, robustness and explainability of deep neural networks. Essentially, it aims to find a minimum sufficient representation $\mathbf{t}$ by striking a trade-off between a compression term $I(\mathbf{x};\mathbf{t})$ and a prediction term $I(y;\mathbf{t})$, where $I(\cdot;\cdot)$ refers to the mutual information (MI). MI is for the IB for the most part expressed in terms of the Kullback-Leibler (KL) divergence, which in the regression case corresponds to prediction based on mean squared error (MSE) loss with Gaussian assumption and compression approximated by variational inference. In this paper, we study the IB principle for the regression problem and develop a new way to parameterize the IB with deep neural networks by exploiting favorable properties of the Cauchy-Schwarz (CS) divergence. By doing so, we move away from MSE-based regression and ease estimation by avoiding variational approximations or distributional assumptions. We investigate the improved generalization ability of our proposed CS-IB and demonstrate strong adversarial robustness guarantees. We demonstrate its superior performance on six real-world regression tasks over other popular deep IB approaches. We additionally observe that the solutions discovered by CS-IB always achieve the best trade-off between prediction accuracy and compression ratio in the information plane. The code is available at \url{https://github.com/SJYuCNEL/Cauchy-Schwarz-Information-Bottleneck}.

Title: PhishGuard: A Convolutional Neural Network Based Model for Detecting Phishing URLs with Explainability Analysis

• Authors: Md Robiul Islam, Md Mahamodul Islam, Mst. Suraiya Afrin, Anika Antara, Nujhat Tabassum, Al Amin
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.17960
• Pdf URL: https://arxiv.org/pdf/2404.17960
• Copy Paste: [[2404.17960]] PhishGuard: A Convolutional Neural Network Based Model for Detecting Phishing URLs with Explainability Analysis(https://arxiv.org/abs/2404.17960)
• Keywords: security, attack, robust, explainability
• Abstract: Cybersecurity is one of the global issues because of the extensive dependence on cyber systems of individuals, industries, and organizations. Among the cyber attacks, phishing is increasing tremendously and affecting the global economy. Therefore, this phenomenon highlights the vital need for enhancing user awareness and robust support at both individual and organizational levels. Phishing URL identification is the best way to address the problem. Various machine learning and deep learning methods have been proposed to automate the detection of phishing URLs. However, these approaches often need more convincing accuracy and rely on datasets consisting of limited samples. Furthermore, these black box intelligent models decision to detect suspicious URLs needs proper explanation to understand the features affecting the output. To address the issues, we propose a 1D Convolutional Neural Network (CNN) and trained the model with extensive features and a substantial amount of data. The proposed model outperforms existing works by attaining an accuracy of 99.85%. Additionally, our explainability analysis highlights certain features that significantly contribute to identifying the phishing URL.

Title: Random Walk on Pixel Manifolds for Anomaly Segmentation of Complex Driving Scenes

• Authors: Zelong Zeng, Kaname Tomite
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17961
• Pdf URL: https://arxiv.org/pdf/2404.17961
• Copy Paste: [[2404.17961]] Random Walk on Pixel Manifolds for Anomaly Segmentation of Complex Driving Scenes(https://arxiv.org/abs/2404.17961)
• Keywords: segmentation
• Abstract: In anomaly segmentation for complex driving scenes, state-of-the-art approaches utilize anomaly scoring functions to calculate anomaly scores. For these functions, accurately predicting the logits of inlier classes for each pixel is crucial for precisely inferring the anomaly score. However, in real-world driving scenarios, the diversity of scenes often results in distorted manifolds of pixel embeddings in embedding space. This effect is not conducive to directly using the pixel embeddings for the logit prediction during inference, a concern overlooked by existing methods. To address this problem, we propose a novel method called Random Walk on Pixel Manifolds (RWPM). RWPM utilizes random walks to reveal the intrinsic relationships among pixels to refine the pixel embeddings. The refined pixel embeddings alleviate the distortion of manifolds, improving the accuracy of anomaly scores. Our extensive experiments show that RWPM consistently improve the performance of the existing anomaly segmentation methods and achieve the best results. Code: \url{https://github.com/ZelongZeng/RWPM}.

Title: Privacy-Preserving Aggregation for Decentralized Learning with Byzantine-Robustness

• Authors: Ali Reza Ghavamipour, Benjamin Zi Hao Zhao, Oguzhan Ersoy, Fatih Turkmen
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17970
• Pdf URL: https://arxiv.org/pdf/2404.17970
• Copy Paste: [[2404.17970]] Privacy-Preserving Aggregation for Decentralized Learning with Byzantine-Robustness(https://arxiv.org/abs/2404.17970)
• Keywords: secure, security, privacy, protect, defense, attack, robust, federate
• Abstract: Decentralized machine learning (DL) has been receiving an increasing interest recently due to the elimination of a single point of failure, present in Federated learning setting. Yet, it is threatened by the looming threat of Byzantine clients who intentionally disrupt the learning process by broadcasting arbitrary model updates to other clients, seeking to degrade the performance of the global model. In response, robust aggregation schemes have emerged as promising solutions to defend against such Byzantine clients, thereby enhancing the robustness of Decentralized Learning. Defenses against Byzantine adversaries, however, typically require access to the updates of other clients, a counterproductive privacy trade-off that in turn increases the risk of inference attacks on those same model updates. In this paper, we introduce SecureDL, a novel DL protocol designed to enhance the security and privacy of DL against Byzantine threats. SecureDL~facilitates a collaborative defense, while protecting the privacy of clients' model updates through secure multiparty computation. The protocol employs efficient computation of cosine similarity and normalization of updates to robustly detect and exclude model updates detrimental to model convergence. By using MNIST, Fashion-MNIST, SVHN and CIFAR-10 datasets, we evaluated SecureDL against various Byzantine attacks and compared its effectiveness with four existing defense mechanisms. Our experiments show that SecureDL is effective even in the case of attacks by the malicious majority (e.g., 80% Byzantine clients) while preserving high training accuracy.

Title: Automating Customer Needs Analysis: A Comparative Study of Large Language Models in the Travel Industry

• Authors: Simone Barandoni, Filippo Chiarello, Lorenzo Cascone, Emiliano Marrale, Salvatore Puccio
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.17975
• Pdf URL: https://arxiv.org/pdf/2404.17975
• Copy Paste: [[2404.17975]] Automating Customer Needs Analysis: A Comparative Study of Large Language Models in the Travel Industry(https://arxiv.org/abs/2404.17975)
• Keywords: extraction, large language model
• Abstract: In the rapidly evolving landscape of Natural Language Processing (NLP), Large Language Models (LLMs) have emerged as powerful tools for many tasks, such as extracting valuable insights from vast amounts of textual data. In this study, we conduct a comparative analysis of LLMs for the extraction of travel customer needs from TripAdvisor posts. Leveraging a diverse range of models, including both open-source and proprietary ones such as GPT-4 and Gemini, we aim to elucidate their strengths and weaknesses in this specialized domain. Through an evaluation process involving metrics such as BERTScore, ROUGE, and BLEU, we assess the performance of each model in accurately identifying and summarizing customer needs. Our findings highlight the efficacy of opensource LLMs, particularly Mistral 7B, in achieving comparable performance to larger closed models while offering affordability and customization benefits. Additionally, we underscore the importance of considering factors such as model size, resource requirements, and performance metrics when selecting the most suitable LLM for customer needs analysis tasks. Overall, this study contributes valuable insights for businesses seeking to leverage advanced NLP techniques to enhance customer experience and drive operational efficiency in the travel industry.

Title: A Method of Moments Embedding Constraint and its Application to Semi-Supervised Learning

• Authors: Michael Majurski, Sumeet Menon, Parniyan Farvardin, David Chapman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17978
• Pdf URL: https://arxiv.org/pdf/2404.17978
• Copy Paste: [[2404.17978]] A Method of Moments Embedding Constraint and its Application to Semi-Supervised Learning(https://arxiv.org/abs/2404.17978)
• Keywords: generative
• Abstract: Discriminative deep learning models with a linear+softmax final layer have a problem: the latent space only predicts the conditional probabilities $p(Y|X)$ but not the full joint distribution $p(Y,X)$, which necessitates a generative approach. The conditional probability cannot detect outliers, causing outlier sensitivity in softmax networks. This exacerbates model over-confidence impacting many problems, such as hallucinations, confounding biases, and dependence on large datasets. To address this we introduce a novel embedding constraint based on the Method of Moments (MoM). We investigate the use of polynomial moments ranging from 1st through 4th order hyper-covariance matrices. Furthermore, we use this embedding constraint to train an Axis-Aligned Gaussian Mixture Model (AAGMM) final layer, which learns not only the conditional, but also the joint distribution of the latent space. We apply this method to the domain of semi-supervised image classification by extending FlexMatch with our technique. We find our MoM constraint with the AAGMM layer is able to match the reported FlexMatch accuracy, while also modeling the joint distribution, thereby reducing outlier sensitivity. We also present a preliminary outlier detection strategy based on Mahalanobis distance and discuss future improvements to this strategy. Code is available at: \url{https://github.com/mmajurski/ssl-gmm}

Title: Privacy-Preserving, Dropout-Resilient Aggregation in Decentralized Learning

• Authors: Ali Reza Ghavamipour, Benjamin Zi Hao Zhao, Fatih Turkmen
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17984
• Pdf URL: https://arxiv.org/pdf/2404.17984
• Copy Paste: [[2404.17984]] Privacy-Preserving, Dropout-Resilient Aggregation in Decentralized Learning(https://arxiv.org/abs/2404.17984)
• Keywords: privacy, protect, attack, robust
• Abstract: Decentralized learning (DL) offers a novel paradigm in machine learning by distributing training across clients without central aggregation, enhancing scalability and efficiency. However, DL's peer-to-peer model raises challenges in protecting against inference attacks and privacy leaks. By forgoing central bottlenecks, DL demands privacy-preserving aggregation methods to protect data from 'honest but curious' clients and adversaries, maintaining network-wide privacy. Privacy-preserving DL faces the additional hurdle of client dropout, clients not submitting updates due to connectivity problems or unavailability, further complicating aggregation. This work proposes three secret sharing-based dropout resilience approaches for privacy-preserving DL. Our study evaluates the efficiency, performance, and accuracy of these protocols through experiments on datasets such as MNIST, Fashion-MNIST, SVHN, and CIFAR-10. We compare our protocols with traditional secret-sharing solutions across scenarios, including those with up to 1000 clients. Evaluations show that our protocols significantly outperform conventional methods, especially in scenarios with up to 30% of clients dropout and model sizes of up to $10^6$ parameters. Our approaches demonstrate markedly high efficiency with larger models, higher dropout rates, and extensive client networks, highlighting their effectiveness in enhancing decentralized learning systems' privacy and dropout robustness.

Title: Detection of Conspiracy Theories Beyond Keyword Bias in German-Language Telegram Using Large Language Models

• Authors: Milena Pustet, Elisabeth Steffen, Helena Mihaljević
• Subjects: cs.CL, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.17985
• Pdf URL: https://arxiv.org/pdf/2404.17985
• Copy Paste: [[2404.17985]] Detection of Conspiracy Theories Beyond Keyword Bias in German-Language Telegram Using Large Language Models(https://arxiv.org/abs/2404.17985)
• Keywords: large language model
• Abstract: The automated detection of conspiracy theories online typically relies on supervised learning. However, creating respective training data requires expertise, time and mental resilience, given the often harmful content. Moreover, available datasets are predominantly in English and often keyword-based, introducing a token-level bias into the models. Our work addresses the task of detecting conspiracy theories in German Telegram messages. We compare the performance of supervised fine-tuning approaches using BERT-like models with prompt-based approaches using Llama2, GPT-3.5, and GPT-4 which require little or no additional training data. We use a dataset of $\sim\!\! 4,000$ messages collected during the COVID-19 pandemic, without the use of keyword filters. Our findings demonstrate that both approaches can be leveraged effectively: For supervised fine-tuning, we report an F1 score of $\sim\!\! 0.8$ for the positive class, making our model comparable to recent models trained on keyword-focused English corpora. We demonstrate our model's adaptability to intra-domain temporal shifts, achieving F1 scores of $\sim\!\! 0.7$. Among prompting variants, the best model is GPT-4, achieving an F1 score of $\sim\!\! 0.8$ for the positive class in a zero-shot setting and equipped with a custom conspiracy theory definition.

Title: InfoSec.pptx: A Longitudinal Study of Speakers, Topics, and Sponsors at Security Conferences in Academia and Industry

• Authors: Lukas Walter, Clemens Sauerwein, Daniel W. Woods
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.17989
• Pdf URL: https://arxiv.org/pdf/2404.17989
• Copy Paste: [[2404.17989]] InfoSec.pptx: A Longitudinal Study of Speakers, Topics, and Sponsors at Security Conferences in Academia and Industry(https://arxiv.org/abs/2404.17989)
• Keywords: security, defense, attack
• Abstract: Security conferences are important venues at which academics and practitioners share knowledge about new attacks and state-of-the-art defenses. Despite this, researchers have not studied who shares information and about which security topics. To address this, our study characterizes the speakers, sponsors, and topics presented at the most prestigious academic and industry conferences. We collect a longitudinal data set that contains 9,728 abstracts and 1,686 sponsors across 4 academic and 6 industry conferences. There is limited knowledge sharing between industry and academia. Conferences vary significantly in the equality of how talks/authorship is distributed across individuals. The topics of academic and industry abstracts display consistent coverage of techniques within the MITRE ATT&CK framework. Top tier academic conferences, as well as DEFCON and Black Hat, inconsistently address the governance, response and recovery functions of the NIST Cybersecurity Framework. Commercial InfoSec and insurance conferences (RSA, Gartner, Advisen and NetDillgience) cover the framework more consistently. Prevention and detection remain the most common topic of talks, with no clear temporal trend.

Title: TabVFL: Improving Latent Representation in Vertical Federated Learning

• Authors: Mohamed Rashad, Zilong Zhao, Jeremie Decouchant, Lydia Y. Chen
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.17990
• Pdf URL: https://arxiv.org/pdf/2404.17990
• Copy Paste: [[2404.17990]] TabVFL: Improving Latent Representation in Vertical Federated Learning(https://arxiv.org/abs/2404.17990)
• Keywords: privacy, robust, federate
• Abstract: Autoencoders are popular neural networks that are able to compress high dimensional data to extract relevant latent information. TabNet is a state-of-the-art neural network model designed for tabular data that utilizes an autoencoder architecture for training. Vertical Federated Learning (VFL) is an emerging distributed machine learning paradigm that allows multiple parties to train a model collaboratively on vertically partitioned data while maintaining data privacy. The existing design of training autoencoders in VFL is to train a separate autoencoder in each participant and aggregate the latent representation later. This design could potentially break important correlations between feature data of participating parties, as each autoencoder is trained on locally available features while disregarding the features of others. In addition, traditional autoencoders are not specifically designed for tabular data, which is ubiquitous in VFL settings. Moreover, the impact of client failures during training on the model robustness is under-researched in the VFL scene. In this paper, we propose TabVFL, a distributed framework designed to improve latent representation learning using the joint features of participants. The framework (i) preserves privacy by mitigating potential data leakage with the addition of a fully-connected layer, (ii) conserves feature correlations by learning one latent representation vector, and (iii) provides enhanced robustness against client failures during training phase. Extensive experiments on five classification datasets show that TabVFL can outperform the prior work design, with 26.12% of improvement on f1-score.

Title: Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension

• Authors: Lin Ai, Zheng Hui, Zizhou Liu, Julia Hirschberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.17991
• Pdf URL: https://arxiv.org/pdf/2404.17991
• Copy Paste: [[2404.17991]] Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension(https://arxiv.org/abs/2404.17991)
• Keywords: extraction, generative, large language model
• Abstract: Machine Reading Comprehension (MRC) poses a significant challenge in the field of Natural Language Processing (NLP). While mainstream MRC methods predominantly leverage extractive strategies using encoder-only models such as BERT, generative approaches face the issue of out-of-control generation -- a critical problem where answers generated are often incorrect, irrelevant, or unfaithful to the source text. To address these limitations in generative models for MRC, we introduce the Question-Attended Span Extraction (QASE) module. Integrated during the fine-tuning phase of pre-trained generative language models (PLMs), QASE significantly enhances their performance, allowing them to surpass the extractive capabilities of advanced Large Language Models (LLMs) such as GPT-4. Notably, these gains in performance do not come with an increase in computational demands. The efficacy of the QASE module has been rigorously tested across various datasets, consistently achieving or even surpassing state-of-the-art (SOTA) results.

Title: MediFact at MEDIQA-CORR 2024: Why AI Needs a Human Touch

• Authors: Nadia Saeed
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.17999
• Pdf URL: https://arxiv.org/pdf/2404.17999
• Copy Paste: [[2404.17999]] MediFact at MEDIQA-CORR 2024: Why AI Needs a Human Touch(https://arxiv.org/abs/2404.17999)
• Keywords: extraction, large language model
• Abstract: Accurate representation of medical information is crucial for patient safety, yet artificial intelligence (AI) systems, such as Large Language Models (LLMs), encounter challenges in error-free clinical text interpretation. This paper presents a novel approach submitted to the MEDIQA-CORR 2024 shared task (Ben Abacha et al., 2024a), focusing on the automatic correction of single-word errors in clinical notes. Unlike LLMs that rely on extensive generic data, our method emphasizes extracting contextually relevant information from available clinical text data. Leveraging an ensemble of extractive and abstractive question-answering approaches, we construct a supervised learning framework with domain-specific feature engineering. Our methodology incorporates domain expertise to enhance error correction accuracy. By integrating domain expertise and prioritizing meaningful information extraction, our approach underscores the significance of a human-centric strategy in adapting AI for healthcare.

Title: Implicit Generative Prior for Bayesian Neural Networks

• Authors: Yijia Liu, Xiao Wang
• Subjects: cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2404.18008
• Pdf URL: https://arxiv.org/pdf/2404.18008
• Copy Paste: [[2404.18008]] Implicit Generative Prior for Bayesian Neural Networks(https://arxiv.org/abs/2404.18008)
• Keywords: generative
• Abstract: Predictive uncertainty quantification is crucial for reliable decision-making in various applied domains. Bayesian neural networks offer a powerful framework for this task. However, defining meaningful priors and ensuring computational efficiency remain significant challenges, especially for complex real-world applications. This paper addresses these challenges by proposing a novel neural adaptive empirical Bayes (NA-EB) framework. NA-EB leverages a class of implicit generative priors derived from low-dimensional distributions. This allows for efficient handling of complex data structures and effective capture of underlying relationships in real-world datasets. The proposed NA-EB framework combines variational inference with a gradient ascent algorithm. This enables simultaneous hyperparameter selection and approximation of the posterior distribution, leading to improved computational efficiency. We establish the theoretical foundation of the framework through posterior and classification consistency. We demonstrate the practical applications of our framework through extensive evaluations on a variety of tasks, including the two-spiral problem, regression, 10 UCI datasets, and image classification tasks on both MNIST and CIFAR-10 datasets. The results of our experiments highlight the superiority of our proposed framework over existing methods, such as sparse variational Bayesian and generative models, in terms of prediction accuracy and uncertainty quantification.

Title: DM-Align: Leveraging the Power of Natural Language Instructions to Make Changes to Images

• Authors: Maria Mihaela Trusca, Tinne Tuytelaars, Marie-Francine Moens
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18020
• Pdf URL: https://arxiv.org/pdf/2404.18020
• Copy Paste: [[2404.18020]] DM-Align: Leveraging the Power of Natural Language Instructions to Make Changes to Images(https://arxiv.org/abs/2404.18020)
• Keywords: diffusion
• Abstract: Text-based semantic image editing assumes the manipulation of an image using a natural language instruction. Although recent works are capable of generating creative and qualitative images, the problem is still mostly approached as a black box sensitive to generating unexpected outputs. Therefore, we propose a novel model to enhance the text-based control of an image editor by explicitly reasoning about which parts of the image to alter or preserve. It relies on word alignments between a description of the original source image and the instruction that reflects the needed updates, and the input image. The proposed Diffusion Masking with word Alignments (DM-Align) allows the editing of an image in a transparent and explainable way. It is evaluated on a subset of the Bison dataset and a self-defined dataset dubbed Dream. When comparing to state-of-the-art baselines, quantitative and qualitative results show that DM-Align has superior performance in image editing conditioned on language instructions, well preserves the background of the image and can better cope with long text instructions.

Title: Retrieval Robust to Object Motion Blur

• Authors: Rong Zou, Marc Pollefeys, Denys Rozumnyi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18025
• Pdf URL: https://arxiv.org/pdf/2404.18025
• Copy Paste: [[2404.18025]] Retrieval Robust to Object Motion Blur(https://arxiv.org/abs/2404.18025)
• Keywords: robust
• Abstract: Moving objects are frequently seen in daily life and usually appear blurred in images due to their motion. While general object retrieval is a widely explored area in computer vision, it primarily focuses on sharp and static objects, and retrieval of motion-blurred objects in large image collections remains unexplored. We propose a method for object retrieval in images that are affected by motion blur. The proposed method learns a robust representation capable of matching blurred objects to their deblurred versions and vice versa. To evaluate our approach, we present the first large-scale datasets for blurred object retrieval, featuring images with objects exhibiting varying degrees of blur in various poses and scales. We conducted extensive experiments, showing that our method outperforms state-of-the-art retrieval methods on the new blur-retrieval datasets, which validates the effectiveness of the proposed approach.

Title: Exposing Text-Image Inconsistency Using Diffusion Models

• Authors: Mingzhen Huang, Shan Jia, Zhou Zhou, Yan Ju, Jialing Cai, Siwei Lyu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18033
• Pdf URL: https://arxiv.org/pdf/2404.18033
• Copy Paste: [[2404.18033]] Exposing Text-Image Inconsistency Using Diffusion Models(https://arxiv.org/abs/2404.18033)
• Keywords: robust, diffusion
• Abstract: In the battle against widespread online misinformation, a growing problem is text-image inconsistency, where images are misleadingly paired with texts with different intent or meaning. Existing classification-based methods for text-image inconsistency can identify contextual inconsistencies but fail to provide explainable justifications for their decisions that humans can understand. Although more nuanced, human evaluation is impractical at scale and susceptible to errors. To address these limitations, this study introduces D-TIIL (Diffusion-based Text-Image Inconsistency Localization), which employs text-to-image diffusion models to localize semantic inconsistencies in text and image pairs. These models, trained on large-scale datasets act as ``omniscient" agents that filter out irrelevant information and incorporate background knowledge to identify inconsistencies. In addition, D-TIIL uses text embeddings and modified image regions to visualize these inconsistencies. To evaluate D-TIIL's efficacy, we introduce a new TIIL dataset containing 14K consistent and inconsistent text-image pairs. Unlike existing datasets, TIIL enables assessment at the level of individual words and image regions and is carefully designed to represent various inconsistencies. D-TIIL offers a scalable and evidence-based approach to identifying and localizing text-image inconsistency, providing a robust framework for future research combating misinformation.

Title: Fashion Recommendation: Outfit Compatibility using GNN

• Authors: Samaksh Gulati
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18040
• Pdf URL: https://arxiv.org/pdf/2404.18040
• Copy Paste: [[2404.18040]] Fashion Recommendation: Outfit Compatibility using GNN(https://arxiv.org/abs/2404.18040)
• Keywords: transformer
• Abstract: Numerous industries have benefited from the use of machine learning and fashion in industry is no exception. By gaining a better understanding of what makes a good outfit, companies can provide useful product recommendations to their users. In this project, we follow two existing approaches that employ graphs to represent outfits and use modified versions of the Graph neural network (GNN) frameworks. Both Node-wise Graph Neural Network (NGNN) and Hypergraph Neural Network aim to score a set of items according to the outfit compatibility of items. The data used is the Polyvore Dataset which consists of curated outfits with product images and text descriptions for each product in an outfit. We recreate the analysis on a subset of this data and compare the two existing models on their performance on two tasks Fill in the blank (FITB): finding an item that completes an outfit, and Compatibility prediction: estimating compatibility of different items grouped as an outfit. We can replicate the results directionally and find that HGNN does have a slightly better performance on both tasks. On top of replicating the results of the two papers we also tried to use embeddings generated from a vision transformer and witness enhanced prediction accuracy across the board

Title: Utilizing Large Language Models for Information Extraction from Real Estate Transactions

• Authors: Yu Zhao, Haoxiang Gao
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18043
• Pdf URL: https://arxiv.org/pdf/2404.18043
• Copy Paste: [[2404.18043]] Utilizing Large Language Models for Information Extraction from Real Estate Transactions(https://arxiv.org/abs/2404.18043)
• Keywords: extraction, transformer, large language model
• Abstract: Real estate sales contracts contain crucial information for property transactions, but manual extraction of data can be time-consuming and error-prone. This paper explores the application of large language models, specifically transformer-based architectures, for automated information extraction from real estate contracts. We discuss challenges, techniques, and future directions in leveraging these models to improve efficiency and accuracy in real estate contract analysis.

Title: Efficient LLM Inference with Kcache

• Authors: Qiaozhi He, Zhihua Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18057
• Pdf URL: https://arxiv.org/pdf/2404.18057
• Copy Paste: [[2404.18057]] Efficient LLM Inference with Kcache(https://arxiv.org/abs/2404.18057)
• Keywords: large language model
• Abstract: Large Language Models(LLMs) have had a profound impact on AI applications, particularly in the domains of long-text comprehension and generation. KV Cache technology is one of the most widely used techniques in the industry. It ensures efficient sequence generation by caching previously computed KV states. However, it also introduces significant memory overhead. We discovered that KV Cache is not necessary and proposed a novel KCache technique to alleviate the memory bottleneck issue during the LLMs inference process. KCache can be used directly for inference without any training process, Our evaluations show that KCache improves the throughput of popular LLMs by 40% with the baseline, while keeping accuracy.

Title: Grounded Compositional and Diverse Text-to-3D with Pretrained Multi-View Diffusion Model

• Authors: Xiaolong Li, Jiawei Mo, Ying Wang, Chethan Parameshwara, Xiaohan Fei, Ashwin Swaminathan, CJ Taylor, Zhuowen Tu, Paolo Favaro, Stefano Soatto
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18065
• Pdf URL: https://arxiv.org/pdf/2404.18065
• Copy Paste: [[2404.18065]] Grounded Compositional and Diverse Text-to-3D with Pretrained Multi-View Diffusion Model(https://arxiv.org/abs/2404.18065)
• Keywords: diffusion
• Abstract: In this paper, we propose an effective two-stage approach named Grounded-Dreamer to generate 3D assets that can accurately follow complex, compositional text prompts while achieving high fidelity by using a pre-trained multi-view diffusion model. Multi-view diffusion models, such as MVDream, have shown to generate high-fidelity 3D assets using score distillation sampling (SDS). However, applied naively, these methods often fail to comprehend compositional text prompts, and may often entirely omit certain subjects or parts. To address this issue, we first advocate leveraging text-guided 4-view images as the bottleneck in the text-to-3D pipeline. We then introduce an attention refocusing mechanism to encourage text-aligned 4-view image generation, without the necessity to re-train the multi-view diffusion model or craft a high-quality compositional 3D dataset. We further propose a hybrid optimization strategy to encourage synergy between the SDS loss and the sparse RGB reference images. Our method consistently outperforms previous state-of-the-art (SOTA) methods in generating compositional 3D assets, excelling in both quality and accuracy, and enabling diverse 3D from the same text prompt.

Title: Can Perplexity Predict Fine-Tuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali

• Authors: Nishant Luitel, Nirajan Bekoju, Anand Kumar Sah, Subarna Shakya
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18071
• Pdf URL: https://arxiv.org/pdf/2404.18071
• Copy Paste: [[2404.18071]] Can Perplexity Predict Fine-Tuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali(https://arxiv.org/abs/2404.18071)
• Keywords: transformer
• Abstract: Recent language models use subwording mechanisms to handle Out-of-Vocabulary(OOV) words seen during test time and, their generation capacity is generally measured using perplexity, an intrinsic metric. It is known that increasing the subword granularity results in a decrease of perplexity value. However, the study of how subwording affects the understanding capacity of language models has been very few and only limited to a handful of languages. To reduce this gap we used 6 different tokenization schemes to pretrain relatively small language models in Nepali and used the representations learned to finetune on several downstream tasks. Although byte-level BPE algorithm has been used in recent models like GPT, RoBERTa we show that on average they are sub-optimal in comparison to algorithms such as SentencePiece in finetuning performances for Nepali. Additionally, similar recent studies have focused on the Bert-based language model. We, however, pretrain and finetune sequential transformer-based language models.

Title: Contextual Spelling Correction with Language Model for Low-resource Setting

• Authors: Nishant Luitel, Nirajan Bekoju, Anand Kumar Sah, Subarna Shakya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18072
• Pdf URL: https://arxiv.org/pdf/2404.18072
• Copy Paste: [[2404.18072]] Contextual Spelling Correction with Language Model for Low-resource Setting(https://arxiv.org/abs/2404.18072)
• Keywords: transformer
• Abstract: The task of Spell Correction(SC) in low-resource languages presents a significant challenge due to the availability of only a limited corpus of data and no annotated spelling correction datasets. To tackle these challenges a small-scale word-based transformer LM is trained to provide the SC model with contextual understanding. Further, the probabilistic error rules are extracted from the corpus in an unsupervised way to model the tendency of error happening(error model). Then the combination of LM and error model is used to develop the SC model through the well-known noisy channel framework. The effectiveness of this approach is demonstrated through experiments on the Nepali language where there is access to just an unprocessed corpus of textual data.

Title: Cyber Security in Containerization Platforms: A Comparative Study of Security Challenges, Measures and Best Practices

• Authors: Sohome Adhikari, Sabur Baidya
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.18082
• Pdf URL: https://arxiv.org/pdf/2404.18082
• Copy Paste: [[2404.18082]] Cyber Security in Containerization Platforms: A Comparative Study of Security Challenges, Measures and Best Practices(https://arxiv.org/abs/2404.18082)
• Keywords: secure, security, defense
• Abstract: The paper reviews the comparative study of security measures, challenges, and best practices with a view to enhancing cyber safety in containerized platforms. This review is intended to give insight into the enhanced security posture of containerized environments, with a view to examining safety vulnerabilities in containerization platforms, exploring strategies for increasing containers isolation and assessing how encryption techniques play an important role in providing secure applications. The paper also provides practical guidance for organizations seeking to strengthen their cyber security defenses in the containerization area platforms.

Title: CRE-LLM: A Domain-Specific Chinese Relation Extraction Framework with Fine-tuned Large Language Model

• Authors: Zhengpeng Shi, Haoran Luo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18085
• Pdf URL: https://arxiv.org/pdf/2404.18085
• Copy Paste: [[2404.18085]] CRE-LLM: A Domain-Specific Chinese Relation Extraction Framework with Fine-tuned Large Language Model(https://arxiv.org/abs/2404.18085)
• Keywords: robust, extraction, generative, large language model
• Abstract: Domain-Specific Chinese Relation Extraction (DSCRE) aims to extract relations between entities from domain-specific Chinese text. Despite the rapid development of PLMs in recent years, especially LLMs, DSCRE still faces three core challenges: complex network structure design, poor awareness, and high consumption of fine-tuning. Given the impressive performance of large language models (LLMs) in natural language processing, we propose a new framework called CRE-LLM. This framework is based on fine-tuning open-source LLMs, such as Llama-2, ChatGLM2, and Baichuan2. CRE-LLM enhances the logic-awareness and generative capabilities of the model by constructing an appropriate prompt and utilizing open-source LLMs for instruction-supervised fine-tuning. And then it directly extracts the relations of the given entities in the input textual data, which improving the CRE approach. To demonstrate the effectiveness of the proposed framework, we conducted extensive experiments on two domain-specific CRE datasets, FinRE and SanWen. The experimental results show that CRE-LLM is significantly superior and robust, achieving state-of-the-art (SOTA) performance on the FinRE dataset. This paper introduces a novel approach to domain-specific relation extraction (DSCRE) tasks that are semantically more complex by combining LLMs with triples. Our code is publicly available.

Title: A Novel Classification of Attacks on Blockchain Layers: Vulnerabilities, Attacks, Mitigations, and Research Directions

• Authors: Kaustubh Dwivedi, Ankit Agrawal, Ashutosh Bhatia, Kamlesh Tiwari
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.18090
• Pdf URL: https://arxiv.org/pdf/2404.18090
• Copy Paste: [[2404.18090]] A Novel Classification of Attacks on Blockchain Layers: Vulnerabilities, Attacks, Mitigations, and Research Directions(https://arxiv.org/abs/2404.18090)
• Keywords: security, privacy, attack
• Abstract: The widespread adoption of blockchain technology has amplified the spectrum of potential threats to its integrity and security. The ongoing quest to exploit vulnerabilities emphasizes how critical it is to expand on current research initiatives. Thus, using a methodology based on discrete blockchain layers, our survey study aims to broaden the existing body of knowledge by thoroughly discussing both new and known attack vectors inside the blockchain ecosystem. This survey proposes a novel classification of blockchain attacks and an in-depth investigation of blockchain data security. In particular, the paper provides a thorough discussion of the attack techniques and vulnerabilities that are specific to each tier, along with a detailed look at mitigating techniques. We reveal the deep dynamics of these security concerns by closely investigating the fundamental causes of attacks at various blockchain tiers. We clarify mitigation methods for known vulnerabilities and offer new information on recently developed attack vectors. We also discuss the implications of quantum computing in blockchain and the weaknesses in the current technology that can be exploited in the future. Our study advances the field of blockchain security and privacy research while also contributing to our understanding of blockchain vulnerabilities and attacks. This survey paper is a useful tool for readers who want to learn more about the intricacies of blockchain security. It also invites researchers to help strengthen blockchain privacy and security, paving the way for further developments in this dynamic and ever-evolving field.

Title: Advancing Supervised Learning with the Wave Loss Function: A Robust and Smooth Approach

• Authors: Mushir Akhtar, M. Tanveer, Mohd. Arshad
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18101
• Pdf URL: https://arxiv.org/pdf/2404.18101
• Copy Paste: [[2404.18101]] Advancing Supervised Learning with the Wave Loss Function: A Robust and Smooth Approach(https://arxiv.org/abs/2404.18101)
• Keywords: robust
• Abstract: Loss function plays a vital role in supervised learning frameworks. The selection of the appropriate loss function holds the potential to have a substantial impact on the proficiency attained by the acquired model. The training of supervised learning algorithms inherently adheres to predetermined loss functions during the optimization process. In this paper, we present a novel contribution to the realm of supervised machine learning: an asymmetric loss function named wave loss. It exhibits robustness against outliers, insensitivity to noise, boundedness, and a crucial smoothness property. Theoretically, we establish that the proposed wave loss function manifests the essential characteristic of being classification-calibrated. Leveraging this breakthrough, we incorporate the proposed wave loss function into the least squares setting of support vector machines (SVM) and twin support vector machines (TSVM), resulting in two robust and smooth models termed Wave-SVM and Wave-TSVM, respectively. To address the optimization problem inherent in Wave-SVM, we utilize the adaptive moment estimation (Adam) algorithm. It is noteworthy that this paper marks the first instance of the Adam algorithm application to solve an SVM model. Further, we devise an iterative algorithm to solve the optimization problems of Wave-TSVM. To empirically showcase the effectiveness of the proposed Wave-SVM and Wave-TSVM, we evaluate them on benchmark UCI and KEEL datasets (with and without feature noise) from diverse domains. Moreover, to exemplify the applicability of Wave-SVM in the biomedical domain, we evaluate it on the Alzheimer Disease Neuroimaging Initiative (ADNI) dataset. The experimental outcomes unequivocally reveal the prowess of Wave-SVM and Wave-TSVM in achieving superior prediction accuracy against the baseline models.

Title: Semi-supervised Text-based Person Search

• Authors: Daming Gao, Yang Bai, Min Cao, Hao Dou, Mang Ye, Min Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18106
• Pdf URL: https://arxiv.org/pdf/2404.18106
• Copy Paste: [[2404.18106]] Semi-supervised Text-based Person Search(https://arxiv.org/abs/2404.18106)
• Keywords: robust
• Abstract: Text-based person search (TBPS) aims to retrieve images of a specific person from a large image gallery based on a natural language description. Existing methods rely on massive annotated image-text data to achieve satisfactory performance in fully-supervised learning. It poses a significant challenge in practice, as acquiring person images from surveillance videos is relatively easy, while obtaining annotated texts is challenging. The paper undertakes a pioneering initiative to explore TBPS under the semi-supervised setting, where only a limited number of person images are annotated with textual descriptions while the majority of images lack annotations. We present a two-stage basic solution based on generation-then-retrieval for semi-supervised TBPS. The generation stage enriches annotated data by applying an image captioning model to generate pseudo-texts for unannotated images. Later, the retrieval stage performs fully-supervised retrieval learning using the augmented data. Significantly, considering the noise interference of the pseudo-texts on retrieval learning, we propose a noise-robust retrieval framework that enhances the ability of the retrieval model to handle noisy data. The framework integrates two key strategies: Hybrid Patch-Channel Masking (PC-Mask) to refine the model architecture, and Noise-Guided Progressive Training (NP-Train) to enhance the training process. PC-Mask performs masking on the input data at both the patch-level and the channel-level to prevent overfitting noisy supervision. NP-Train introduces a progressive training schedule based on the noise level of pseudo-texts to facilitate noise-robust learning. Extensive experiments on multiple TBPS benchmarks show that the proposed framework achieves promising performance under the semi-supervised setting.

Title: Garbage Segmentation and Attribute Analysis by Robotic Dogs

• Authors: Nuo Xu, Jianfeng Liao, Qiwei Meng, Wei Song
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.18112
• Pdf URL: https://arxiv.org/pdf/2404.18112
• Copy Paste: [[2404.18112]] Garbage Segmentation and Attribute Analysis by Robotic Dogs(https://arxiv.org/abs/2404.18112)
• Keywords: segmentation
• Abstract: Efficient waste management and recycling heavily rely on garbage exploration and identification. In this study, we propose GSA2Seg (Garbage Segmentation and Attribute Analysis), a novel visual approach that utilizes quadruped robotic dogs as autonomous agents to address waste management and recycling challenges in diverse indoor and outdoor environments. Equipped with advanced visual perception system, including visual sensors and instance segmentators, the robotic dogs adeptly navigate their surroundings, diligently searching for common garbage items. Inspired by open-vocabulary algorithms, we introduce an innovative method for object attribute analysis. By combining garbage segmentation and attribute analysis techniques, the robotic dogs accurately determine the state of the trash, including its position and placement properties. This information enhances the robotic arm's grasping capabilities, facilitating successful garbage retrieval. Additionally, we contribute an image dataset, named GSA2D, to support evaluation. Through extensive experiments on GSA2D, this paper provides a comprehensive analysis of GSA2Seg's effectiveness. Dataset available: \href{https://www.kaggle.com/datasets/hellob/gsa2d-2024}{https://www.kaggle.com/datasets/hellob/gsa2d-2024}.

Title: Enhancing Fairness in Neural Networks Using FairVIC

• Authors: Charmaine Barker, Daniel Bethell, Dimitar Kazakov
• Subjects: cs.LG, cs.AI, cs.CY, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.18134
• Pdf URL: https://arxiv.org/pdf/2404.18134
• Copy Paste: [[2404.18134]] Enhancing Fairness in Neural Networks Using FairVIC(https://arxiv.org/abs/2404.18134)
• Keywords: protect, fair
• Abstract: Mitigating bias in automated decision-making systems, specifically deep learning models, is a critical challenge in achieving fairness. This complexity stems from factors such as nuanced definitions of fairness, unique biases in each dataset, and the trade-off between fairness and model accuracy. To address such issues, we introduce FairVIC, an innovative approach designed to enhance fairness in neural networks by addressing inherent biases at the training stage. FairVIC differs from traditional approaches that typically address biases at the data preprocessing stage. Instead, it integrates variance, invariance and covariance into the loss function to minimise the model's dependency on protected characteristics for making predictions, thus promoting fairness. Our experimentation and evaluation consists of training neural networks on three datasets known for their biases, comparing our results to state-of-the-art algorithms, evaluating on different sizes of model architectures, and carrying out sensitivity analysis to examine the fairness-accuracy trade-off. Through our implementation of FairVIC, we observed a significant improvement in fairness across all metrics tested, without compromising the model's accuracy to a detrimental extent. Our findings suggest that FairVIC presents a straightforward, out-of-the-box solution for the development of fairer deep learning models, thereby offering a generalisable solution applicable across many tasks and datasets.

Title: SafePaint: Anti-forensic Image Inpainting with Domain Adaptation

• Authors: Dunyun Chen, Xin Liao, Xiaoshuai Wu, Shiwei Chen
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.18136
• Pdf URL: https://arxiv.org/pdf/2404.18136
• Copy Paste: [[2404.18136]] SafePaint: Anti-forensic Image Inpainting with Domain Adaptation(https://arxiv.org/abs/2404.18136)
• Keywords: security
• Abstract: Existing image inpainting methods have achieved remarkable accomplishments in generating visually appealing results, often accompanied by a trend toward creating more intricate structural textures. However, while these models excel at creating more realistic image content, they often leave noticeable traces of tampering, posing a significant threat to security. In this work, we take the anti-forensic capabilities into consideration, firstly proposing an end-to-end training framework for anti-forensic image inpainting named SafePaint. Specifically, we innovatively formulated image inpainting as two major tasks: semantically plausible content completion and region-wise optimization. The former is similar to current inpainting methods that aim to restore the missing regions of corrupted images. The latter, through domain adaptation, endeavors to reconcile the discrepancies between the inpainted region and the unaltered area to achieve anti-forensic goals. Through comprehensive theoretical analysis, we validate the effectiveness of domain adaptation for anti-forensic performance. Furthermore, we meticulously crafted a region-wise separated attention (RWSA) module, which not only aligns with our objective of anti-forensics but also enhances the performance of the model. Extensive qualitative and quantitative evaluations show our approach achieves comparable results to existing image inpainting methods while offering anti-forensic capabilities not available in other methods.

Title: Tracking Transforming Objects: A Benchmark

• Authors: You Wu, Yuelong Wang, Yaxin Liao, Fuliang Wu, Hengzhou Ye, Shuiwang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18143
• Pdf URL: https://arxiv.org/pdf/2404.18143
• Copy Paste: [[2404.18143]] Tracking Transforming Objects: A Benchmark(https://arxiv.org/abs/2404.18143)
• Keywords: security, robust
• Abstract: Tracking transforming objects holds significant importance in various fields due to the dynamic nature of many real-world scenarios. By enabling systems accurately represent transforming objects over time, tracking transforming objects facilitates advancements in areas such as autonomous systems, human-computer interaction, and security applications. Moreover, understanding the behavior of transforming objects provides valuable insights into complex interactions or processes, contributing to the development of intelligent systems capable of robust and adaptive perception in dynamic environments. However, current research in the field mainly focuses on tracking generic objects. In this study, we bridge this gap by collecting a novel dedicated Dataset for Tracking Transforming Objects, called DTTO, which contains 100 sequences, amounting to approximately 9.3K frames. We provide carefully hand-annotated bounding boxes for each frame within these sequences, making DTTO the pioneering benchmark dedicated to tracking transforming objects. We thoroughly evaluate 20 state-of-the-art trackers on the benchmark, aiming to comprehend the performance of existing methods and provide a comparison for future research on DTTO. With the release of DTTO, our goal is to facilitate further research and applications related to tracking transforming objects.

Title: Generative AI for Visualization: State of the Art and Future Directions

• Authors: Yilin Ye, Jianing Hao, Yihan Hou, Zhan Wang, Shishi Xiao, Yuyu Luo, Wei Zeng
• Subjects: cs.LG, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.18144
• Pdf URL: https://arxiv.org/pdf/2404.18144
• Copy Paste: [[2404.18144]] Generative AI for Visualization: State of the Art and Future Directions(https://arxiv.org/abs/2404.18144)
• Keywords: diffusion, generative, large language model
• Abstract: Generative AI (GenAI) has witnessed remarkable progress in recent years and demonstrated impressive performance in various generation tasks in different domains such as computer vision and computational design. Many researchers have attempted to integrate GenAI into visualization framework, leveraging the superior generative capacity for different operations. Concurrently, recent major breakthroughs in GenAI like diffusion model and large language model have also drastically increase the potential of GenAI4VIS. From a technical perspective, this paper looks back on previous visualization studies leveraging GenAI and discusses the challenges and opportunities for future research. Specifically, we cover the applications of different types of GenAI methods including sequence, tabular, spatial and graph generation techniques for different tasks of visualization which we summarize into four major stages: data enhancement, visual mapping generation, stylization and interaction. For each specific visualization sub-task, we illustrate the typical data and concrete GenAI algorithms, aiming to provide in-depth understanding of the state-of-the-art GenAI4VIS techniques and their limitations. Furthermore, based on the survey, we discuss three major aspects of challenges and research opportunities including evaluation, dataset, and the gap between end-to-end GenAI and generative algorithms. By summarizing different generation algorithms, their current applications and limitations, this paper endeavors to provide useful insights for future GenAI4VIS research.

Title: Compressed Deepfake Video Detection Based on 3D Spatiotemporal Trajectories

• Authors: Zongmei Chen, Xin Liao, Xiaoshuai Wu, Yanxiang Chen
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.18149
• Pdf URL: https://arxiv.org/pdf/2404.18149
• Copy Paste: [[2404.18149]] Compressed Deepfake Video Detection Based on 3D Spatiotemporal Trajectories(https://arxiv.org/abs/2404.18149)
• Keywords: robust
• Abstract: The misuse of deepfake technology by malicious actors poses a potential threat to nations, societies, and individuals. However, existing methods for detecting deepfakes primarily focus on uncompressed videos, such as noise characteristics, local textures, or frequency statistics. When applied to compressed videos, these methods experience a decrease in detection performance and are less suitable for real-world scenarios. In this paper, we propose a deepfake video detection method based on 3D spatiotemporal trajectories. Specifically, we utilize a robust 3D model to construct spatiotemporal motion features, integrating feature details from both 2D and 3D frames to mitigate the influence of large head rotation angles or insufficient lighting within frames. Furthermore, we separate facial expressions from head movements and design a sequential analysis method based on phase space motion trajectories to explore the feature differences between genuine and fake faces in deepfake videos. We conduct extensive experiments to validate the performance of our proposed method on several compressed deepfake benchmarks. The robustness of the well-designed features is verified by calculating the consistent distribution of facial landmarks before and after video compression.Our method yields satisfactory results and showcases its potential for practical applications.

Title: Masked Attention as a Mechanism for Improving Interpretability of Vision Transformers

• Authors: Clément Grisi, Geert Litjens, Jeroen van der Laak
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.18152
• Pdf URL: https://arxiv.org/pdf/2404.18152
• Copy Paste: [[2404.18152]] Masked Attention as a Mechanism for Improving Interpretability of Vision Transformers(https://arxiv.org/abs/2404.18152)
• Keywords: robust, interpretability, transformer
• Abstract: Vision Transformers are at the heart of the current surge of interest in foundation models for histopathology. They process images by breaking them into smaller patches following a regular grid, regardless of their content. Yet, not all parts of an image are equally relevant for its understanding. This is particularly true in computational pathology where background is completely non-informative and may introduce artefacts that could mislead predictions. To address this issue, we propose a novel method that explicitly masks background in Vision Transformers' attention mechanism. This ensures tokens corresponding to background patches do not contribute to the final image representation, thereby improving model robustness and interpretability. We validate our approach using prostate cancer grading from whole-slide images as a case study. Our results demonstrate that it achieves comparable performance with plain self-attention while providing more accurate and clinically meaningful attention heatmaps.

Title: ShapeMoiré: Channel-Wise Shape-Guided Network for Image Demoiréing

• Authors: Jinming Cao, Sicheng Shen, Qiu Zhou, Yifang Yin, Yangyan Li, Roger Zimmermann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18155
• Pdf URL: https://arxiv.org/pdf/2404.18155
• Copy Paste: [[2404.18155]] ShapeMoiré: Channel-Wise Shape-Guided Network for Image Demoiréing(https://arxiv.org/abs/2404.18155)
• Keywords: robust
• Abstract: Photographing optoelectronic displays often introduces unwanted moir\'e patterns due to analog signal interference between the pixel grids of the display and the camera sensor arrays. This work identifies two problems that are largely ignored by existing image demoir\'eing approaches: 1) moir\'e patterns vary across different channels (RGB); 2) repetitive patterns are constantly observed. However, employing conventional convolutional (CNN) layers cannot address these problems. Instead, this paper presents the use of our recently proposed Shape concept. It was originally employed to model consistent features from fragmented regions, particularly when identical or similar objects coexist in an RGB-D image. Interestingly, we find that the Shape information effectively captures the moir\'e patterns in artifact images. Motivated by this discovery, we propose a ShapeMoir\'e method to aid in image demoir\'eing. Beyond modeling shape features at the patch-level, we further extend this to the global image-level and design a novel Shape-Architecture. Consequently, our proposed method, equipped with both ShapeConv and Shape-Architecture, can be seamlessly integrated into existing approaches without introducing additional parameters or computation overhead during inference. We conduct extensive experiments on four widely used datasets, and the results demonstrate that our ShapeMoir\'e achieves state-of-the-art performance, particularly in terms of the PSNR metric. We then apply our method across four popular architectures to showcase its generalization capabilities. Moreover, our ShapeMoir\'e is robust and viable under real-world demoir\'eing scenarios involving smartphone photographs.

Title: IMEX-Reg: Implicit-Explicit Regularization in the Function Space for Continual Learning

• Authors: Prashant Bhat, Bharath Renjith, Elahe Arani, Bahram Zonooz
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18161
• Pdf URL: https://arxiv.org/pdf/2404.18161
• Copy Paste: [[2404.18161]] IMEX-Reg: Implicit-Explicit Regularization in the Function Space for Continual Learning(https://arxiv.org/abs/2404.18161)
• Keywords: robust, fair
• Abstract: Continual learning (CL) remains one of the long-standing challenges for deep neural networks due to catastrophic forgetting of previously acquired knowledge. Although rehearsal-based approaches have been fairly successful in mitigating catastrophic forgetting, they suffer from overfitting on buffered samples and prior information loss, hindering generalization under low-buffer regimes. Inspired by how humans learn using strong inductive biases, we propose IMEX-Reg to improve the generalization performance of experience rehearsal in CL under low buffer regimes. Specifically, we employ a two-pronged implicit-explicit regularization approach using contrastive representation learning (CRL) and consistency regularization. To further leverage the global relationship between representations learned using CRL, we propose a regularization strategy to guide the classifier toward the activation correlations in the unit hypersphere of the CRL. Our results show that IMEX-Reg significantly improves generalization performance and outperforms rehearsal-based approaches in several CL scenarios. It is also robust to natural and adversarial corruptions with less task-recency bias. Additionally, we provide theoretical insights to support our design decisions further.

Title: Mamba-FETrack: Frame-Event Tracking via State Space Model

• Authors: Ju Huang, Shiao Wang, Shuai Wang, Zhe Wu, Xiao Wang, Bo Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18174
• Pdf URL: https://arxiv.org/pdf/2404.18174
• Copy Paste: [[2404.18174]] Mamba-FETrack: Frame-Event Tracking via State Space Model(https://arxiv.org/abs/2404.18174)
• Keywords: transformer
• Abstract: RGB-Event based tracking is an emerging research topic, focusing on how to effectively integrate heterogeneous multi-modal data (synchronized exposure video frames and asynchronous pulse Event stream). Existing works typically employ Transformer based networks to handle these modalities and achieve decent accuracy through input-level or feature-level fusion on multiple datasets. However, these trackers require significant memory consumption and computational complexity due to the use of self-attention mechanism. This paper proposes a novel RGB-Event tracking framework, Mamba-FETrack, based on the State Space Model (SSM) to achieve high-performance tracking while effectively reducing computational costs and realizing more efficient tracking. Specifically, we adopt two modality-specific Mamba backbone networks to extract the features of RGB frames and Event streams. Then, we also propose to boost the interactive learning between the RGB and Event features using the Mamba network. The fused features will be fed into the tracking head for target object localization. Extensive experiments on FELT and FE108 datasets fully validated the efficiency and effectiveness of our proposed tracker. Specifically, our Mamba-based tracker achieves 43.5/55.6 on the SR/PR metric, while the ViT-S based tracker (OSTrack) obtains 40.0/50.9. The GPU memory cost of ours and ViT-S based tracker is 13.98GB and 15.44GB, which decreased about $9.5\%$. The FLOPs and parameters of ours/ViT-S based OSTrack are 59GB/1076GB and 7MB/60MB, which decreased about $94.5\%$ and $88.3\%$, respectively. We hope this work can bring some new insights to the tracking field and greatly promote the application of the Mamba architecture in tracking. The source code of this work will be released on \url{https://github.com/Event-AHU/Mamba_FETrack}.

Title: Exploring the Robustness of In-Context Learning with Noisy Labels

• Authors: Chen Cheng, Xinzhi Yu, Haodong Wen, Jinsong Sun, Guanzhang Yue, Yihao Zhang, Zeming Wei
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2404.18191
• Pdf URL: https://arxiv.org/pdf/2404.18191
• Copy Paste: [[2404.18191]] Exploring the Robustness of In-Context Learning with Noisy Labels(https://arxiv.org/abs/2404.18191)
• Keywords: robust, transformer, large language model
• Abstract: Recently, the mysterious In-Context Learning (ICL) ability exhibited by Transformer architectures, especially in large language models (LLMs), has sparked significant research interest. However, the resilience of Transformers' in-context learning capabilities in the presence of noisy samples, prevalent in both training corpora and prompt demonstrations, remains underexplored. In this paper, inspired by prior research that studies ICL ability using simple function classes, we take a closer look at this problem by investigating the robustness of Transformers against noisy labels. Specifically, we first conduct a thorough evaluation and analysis of the robustness of Transformers against noisy labels during in-context learning and show that they exhibit notable resilience against diverse types of noise in demonstration labels. Furthermore, we delve deeper into this problem by exploring whether introducing noise into the training set, akin to a form of data augmentation, enhances such robustness during inference, and find that such noise can indeed improve the robustness of ICL. Overall, our fruitful analysis and findings provide a comprehensive understanding of the resilience of Transformer models against label noises during ICL and provide valuable insights into the research on Transformers in natural language processing. Our code is available at https://github.com/InezYu0928/in-context-learning.

Title: LMM-PCQA: Assisting Point Cloud Quality Assessment with LMM

• Authors: Zicheng Zhang, Haoning Wu, Yingjie Zhou, Chunyi Li, Wei Sun, Chaofeng Chen, Xiongkuo Min, Xiaohong Liu, Weisi Lin, Guangtao Zhai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18203
• Pdf URL: https://arxiv.org/pdf/2404.18203
• Copy Paste: [[2404.18203]] LMM-PCQA: Assisting Point Cloud Quality Assessment with LMM(https://arxiv.org/abs/2404.18203)
• Keywords: robust
• Abstract: Although large multi-modality models (LMMs) have seen extensive exploration and application in various quality assessment studies, their integration into Point Cloud Quality Assessment (PCQA) remains unexplored. Given LMMs' exceptional performance and robustness in low-level vision and quality assessment tasks, this study aims to investigate the feasibility of imparting PCQA knowledge to LMMs through text supervision. To achieve this, we transform quality labels into textual descriptions during the fine-tuning phase, enabling LMMs to derive quality rating logits from 2D projections of point clouds. To compensate for the loss of perception in the 3D domain, structural features are extracted as well. These quality logits and structural features are then combined and regressed into quality scores. Our experimental results affirm the effectiveness of our approach, showcasing a novel integration of LMMs into PCQA that enhances model understanding and assessment accuracy. We hope our contributions can inspire subsequent investigations into the fusion of LMMs with PCQA, fostering advancements in 3D visual quality analysis and beyond.

Title: Paint by Inpaint: Learning to Add Image Objects by Removing Them First

• Authors: Navve Wasserman, Noam Rotstein, Roy Ganz, Ron Kimmel
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18212
• Pdf URL: https://arxiv.org/pdf/2404.18212
• Copy Paste: [[2404.18212]] Paint by Inpaint: Learning to Add Image Objects by Removing Them First(https://arxiv.org/abs/2404.18212)
• Keywords: diffusion, large language model, segmentation
• Abstract: Image editing has advanced significantly with the introduction of text-conditioned diffusion models. Despite this progress, seamlessly adding objects to images based on textual instructions without requiring user-provided input masks remains a challenge. We address this by leveraging the insight that removing objects (Inpaint) is significantly simpler than its inverse process of adding them (Paint), attributed to the utilization of segmentation mask datasets alongside inpainting models that inpaint within these masks. Capitalizing on this realization, by implementing an automated and extensive pipeline, we curate a filtered large-scale image dataset containing pairs of images and their corresponding object-removed versions. Using these pairs, we train a diffusion model to inverse the inpainting process, effectively adding objects into images. Unlike other editing datasets, ours features natural target images instead of synthetic ones; moreover, it maintains consistency between source and target by construction. Additionally, we utilize a large Vision-Language Model to provide detailed descriptions of the removed objects and a Large Language Model to convert these descriptions into diverse, natural-language instructions. We show that the trained model surpasses existing ones both qualitatively and quantitatively, and release the large-scale dataset alongside the trained models for the community.

Title: S$^2$Mamba: A Spatial-spectral State Space Model for Hyperspectral Image Classification

• Authors: Guanchun Wang, Xiangrong Zhang, Zelin Peng, Tianyang Zhang, Xiuping Jia, Licheng Jiao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18213
• Pdf URL: https://arxiv.org/pdf/2404.18213
• Copy Paste: [[2404.18213]] S$^2$Mamba: A Spatial-spectral State Space Model for Hyperspectral Image Classification(https://arxiv.org/abs/2404.18213)
• Keywords: extraction, transformer
• Abstract: Land cover analysis using hyperspectral images (HSI) remains an open problem due to their low spatial resolution and complex spectral information. Recent studies are primarily dedicated to designing Transformer-based architectures for spatial-spectral long-range dependencies modeling, which is computationally expensive with quadratic complexity. Selective structured state space model (Mamba), which is efficient for modeling long-range dependencies with linear complexity, has recently shown promising progress. However, its potential in hyperspectral image processing that requires handling numerous spectral bands has not yet been explored. In this paper, we innovatively propose S$^2$Mamba, a spatial-spectral state space model for hyperspectral image classification, to excavate spatial-spectral contextual features, resulting in more efficient and accurate land cover analysis. In S$^2$Mamba, two selective structured state space models through different dimensions are designed for feature extraction, one for spatial, and the other for spectral, along with a spatial-spectral mixture gate for optimal fusion. More specifically, S$^2$Mamba first captures spatial contextual relations by interacting each pixel with its adjacent through a Patch Cross Scanning module and then explores semantic information from continuous spectral bands through a Bi-directional Spectral Scanning module. Considering the distinct expertise of the two attributes in homogenous and complicated texture scenes, we realize the Spatial-spectral Mixture Gate by a group of learnable matrices, allowing for the adaptive incorporation of representations learned across different dimensions. Extensive experiments conducted on HSI classification benchmarks demonstrate the superiority and prospect of S$^2$Mamba. The code will be available at: https://github.com/PURE-melo/S2Mamba.

Title: TextGram: Towards a better domain-adaptive pretraining

• Authors: Sharayu Hiwarkhedkar, Saloni Mittal, Vidula Magdum, Omkar Dhekane, Raviraj Joshi, Geetanjali Kale, Arnav Ladkat
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18228
• Pdf URL: https://arxiv.org/pdf/2404.18228
• Copy Paste: [[2404.18228]] TextGram: Towards a better domain-adaptive pretraining(https://arxiv.org/abs/2404.18228)
• Keywords: transformer, large language model
• Abstract: For green AI, it is crucial to measure and reduce the carbon footprint emitted during the training of large language models. In NLP, performing pre-training on Transformer models requires significant computational resources. This pre-training involves using a large amount of text data to gain prior knowledge for performing downstream tasks. Thus, it is important that we select the correct data in the form of domain-specific data from this vast corpus to achieve optimum results aligned with our domain-specific tasks. While training on large unsupervised data is expensive, it can be optimized by performing a data selection step before pretraining. Selecting important data reduces the space overhead and the substantial amount of time required to pre-train the model while maintaining constant accuracy. We investigate the existing selection strategies and propose our own domain-adaptive data selection method - TextGram - that effectively selects essential data from large corpora. We compare and evaluate the results of finetuned models for text classification task with and without data selection. We show that the proposed strategy works better compared to other selection methods.

Title: From Persona to Personalization: A Survey on Role-Playing Language Agents

• Authors: Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, Yanghua Xiao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18231
• Pdf URL: https://arxiv.org/pdf/2404.18231
• Copy Paste: [[2404.18231]] From Persona to Personalization: A Survey on Role-Playing Language Agents(https://arxiv.org/abs/2404.18231)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have significantly boosted the rise of Role-Playing Language Agents (RPLAs), i.e., specialized AI systems designed to simulate assigned personas. By harnessing multiple advanced abilities of LLMs, including in-context learning, instruction following, and social intelligence, RPLAs achieve a remarkable sense of human likeness and vivid role-playing performance. RPLAs can mimic a wide range of personas, ranging from historical figures and fictional characters to real-life individuals. Consequently, they have catalyzed numerous AI applications, such as emotional companions, interactive video games, personalized assistants and copilots, and digital clones. In this paper, we conduct a comprehensive survey of this field, illustrating the evolution and recent progress in RPLAs integrating with cutting-edge LLM technologies. We categorize personas into three types: 1) Demographic Persona, which leverages statistical stereotypes; 2) Character Persona, focused on well-established figures; and 3) Individualized Persona, customized through ongoing user interactions for personalized services. We begin by presenting a comprehensive overview of current methodologies for RPLAs, followed by the details for each persona type, covering corresponding data sourcing, agent construction, and evaluation. Afterward, we discuss the fundamental risks, existing limitations, and future prospects of RPLAs. Additionally, we provide a brief review of RPLAs in AI applications, which reflects practical user demands that shape and drive RPLA research. Through this work, we aim to establish a clear taxonomy of RPLA research and applications, and facilitate future research in this critical and ever-evolving field, and pave the way for a future where humans and RPLAs coexist in harmony.

Title: SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning

• Authors: Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, Sijia Liu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18239
• Pdf URL: https://arxiv.org/pdf/2404.18239
• Copy Paste: [[2404.18239]] SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning(https://arxiv.org/abs/2404.18239)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have highlighted the necessity of effective unlearning mechanisms to comply with data regulations and ethical AI practices. LLM unlearning aims at removing undesired data influences and associated model capabilities without compromising utility out of the scope of unlearning. While interest in studying LLM unlearning is growing,the impact of the optimizer choice for LLM unlearning remains under-explored. In this work, we shed light on the significance of optimizer selection in LLM unlearning for the first time, establishing a clear connection between {second-order optimization} and influence unlearning (a classical approach using influence functions to update the model for data influence removal). This insight propels us to develop a second-order unlearning framework, termed SOUL, built upon the second-order clipped stochastic optimization (Sophia)-based LLM training method. SOUL extends the static, one-shot model update using influence unlearning to a dynamic, iterative unlearning process. Our extensive experiments show that SOUL consistently outperforms conventional first-order methods across various unlearning tasks, models, and metrics, suggesting the promise of second-order optimization in providing a scalable and easily implementable solution for LLM unlearning.

Title: LEGENT: Open Platform for Embodied Agents

• Authors: Zhili Cheng, Zhitong Wang, Jinyi Hu, Shengding Hu, An Liu, Yuge Tu, Pengkai Li, Lei Shi, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18243
• Pdf URL: https://arxiv.org/pdf/2404.18243
• Copy Paste: [[2404.18243]] LEGENT: Open Platform for Embodied Agents(https://arxiv.org/abs/2404.18243)
• Keywords: large language model
• Abstract: Despite advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), their integration into language-grounded, human-like embodied agents remains incomplete, hindering complex real-life task performance in physical environments. Existing integrations often feature limited open sourcing, challenging collective progress in this field. We introduce LEGENT, an open, scalable platform for developing embodied agents using LLMs and LMMs. LEGENT offers a dual approach: a rich, interactive 3D environment with communicable and actionable agents, paired with a user-friendly interface, and a sophisticated data generation pipeline utilizing advanced algorithms to exploit supervision from simulated worlds at scale. In our experiments, an embryonic vision-language-action model trained on LEGENT-generated data surpasses GPT-4V in embodied tasks, showcasing promising generalization capabilities.

Title: Fisher Information Improved Training-Free Conditional Diffusion Model

• Authors: Kaiyu Song, Hanjiang Lai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18252
• Pdf URL: https://arxiv.org/pdf/2404.18252
• Copy Paste: [[2404.18252]] Fisher Information Improved Training-Free Conditional Diffusion Model(https://arxiv.org/abs/2404.18252)
• Keywords: diffusion
• Abstract: Recently, the diffusion model with the training-free methods has succeeded in conditional image generation tasks. However, there is an efficiency problem because it requires calculating the gradient with high computational cost, and previous methods make strong assumptions to solve it, sacrificing generalization. In this work, we propose the Fisher information guided diffusion model (FIGD). Concretely, we introduce the Fisher information to estimate the gradient without making any additional assumptions to reduce computation cost. Meanwhile, we demonstrate that the Fisher information ensures the generalization of FIGD and provides new insights for training-free methods based on the information theory. The experimental results demonstrate that FIGD could achieve different conditional generations more quickly while maintaining high quality.

Title: PatentGPT: A Large Language Model for Intellectual Property

• Authors: Zilong Bai, Ruiji Zhang, Linqing Chen, Qijun Cai, Yuan Zhong, Cong Wang Yan Fang, Jie Fang, Jing Sun, Weikuan Wang, Lizhi Zhou, Haoran Hua Tian Qiu, Chaochao Wang, Cheng Sun, Jianping Lu, Yixin Wang, Yubin Xia Meng Hu, Haowen Liu, Peng Xu, Licong Xu, Fu Bian, Xiaolong Gu, Lisha Zhang Weilei Wang, Changyang Tu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18255
• Pdf URL: https://arxiv.org/pdf/2404.18255
• Copy Paste: [[2404.18255]] PatentGPT: A Large Language Model for Intellectual Property(https://arxiv.org/abs/2404.18255)
• Keywords: privacy, protect, large language model
• Abstract: In recent years, large language models have attracted significant attention due to their exceptional performance across a multitude of natural language process tasks, and have been widely applied in various fields. However, the application of large language models in the Intellectual Property (IP) space is challenging due to the strong need for specialized knowledge, privacy protection, processing of extremely long text in this field. In this technical report, we present for the first time a low-cost, standardized procedure for training IP-oriented LLMs, meeting the unique requirements of the IP domain. Using this standard process, we have trained the PatentGPT series models based on open-source pretrained models. By evaluating them on the open-source IP-oriented benchmark MOZIP, our domain-specific LLMs outperforms GPT-4, indicating the effectiveness of the proposed training procedure and the expertise of the PatentGPT models in the IP demain. What is impressive is that our model significantly outperformed GPT-4 on the 2019 China Patent Agent Qualification Examination by achieving a score of 65, reaching the level of human experts. Additionally, the PatentGPT model, which utilizes the SMoE architecture, achieves performance comparable to that of GPT-4 in the IP domain and demonstrates a better cost-performance ratio on long-text tasks, potentially serving as an alternative to GPT-4 within the IP domain.

Title: Align, Minimize and Diversify: A Source-Free Unsupervised Domain Adaptation Method for Handwritten Text Recognition

• Authors: María Alfaro-Contreras, Jorge Calvo-Zaragoza
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18260
• Pdf URL: https://arxiv.org/pdf/2404.18260
• Copy Paste: [[2404.18260]] Align, Minimize and Diversify: A Source-Free Unsupervised Domain Adaptation Method for Handwritten Text Recognition(https://arxiv.org/abs/2404.18260)
• Keywords: robust
• Abstract: This paper serves to introduce the Align, Minimize and Diversify (AMD) method, a Source-Free Unsupervised Domain Adaptation approach for Handwritten Text Recognition (HTR). This framework decouples the adaptation process from the source data, thus not only sidestepping the resource-intensive retraining process but also making it possible to leverage the wealth of pre-trained knowledge encoded in modern Deep Learning architectures. Our method explicitly eliminates the need to revisit the source data during adaptation by incorporating three distinct regularization terms: the Align term, which reduces the feature distribution discrepancy between source and target data, ensuring the transferability of the pre-trained representation; the Minimize term, which encourages the model to make assertive predictions, pushing the outputs towards one-hot-like distributions in order to minimize prediction uncertainty, and finally, the Diversify term, which safeguards against the degeneracy in predictions by promoting varied and distinctive sequences throughout the target data, preventing informational collapse. Experimental results from several benchmarks demonstrated the effectiveness and robustness of AMD, showing it to be competitive and often outperforming DA methods in HTR.

Title: Parameter-Efficient Tuning Large Language Models for Graph Representation Learning

• Authors: Qi Zhu, Da Zheng, Xiang Song, Shichang Zhang, Bowen Jin, Yizhou Sun, George Karypis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18271
• Pdf URL: https://arxiv.org/pdf/2404.18271
• Copy Paste: [[2404.18271]] Parameter-Efficient Tuning Large Language Models for Graph Representation Learning(https://arxiv.org/abs/2404.18271)
• Keywords: large language model
• Abstract: Text-rich graphs, which exhibit rich textual information on nodes and edges, are prevalent across a wide range of real-world business applications. Large Language Models (LLMs) have demonstrated remarkable abilities in understanding text, which also introduced the potential for more expressive modeling in text-rich graphs. Despite these capabilities, efficiently applying LLMs to representation learning on graphs presents significant challenges. Recently, parameter-efficient fine-tuning methods for LLMs have enabled efficient new task generalization with minimal time and memory consumption. Inspired by this, we introduce Graph-aware Parameter-Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning with LLMs on text-rich graphs. Specifically, we utilize a graph neural network (GNN) to encode structural information from neighboring nodes into a graph prompt. This prompt is then inserted at the beginning of the text sequence. To improve the quality of graph prompts, we pre-trained the GNN to assist the frozen LLM in predicting the next token in the node text. Compared with existing joint GNN and LMs, our method directly generate the node embeddings from large language models with an affordable fine-tuning cost. We validate our approach through comprehensive experiments conducted on 8 different text-rich graphs, observing an average improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction evaluations. Our results demonstrate the efficacy and efficiency of our model, showing that it can be smoothly integrated with various large language models, including OPT, LLaMA and Falcon.

Title: Bias Neutralization Framework: Measuring Fairness in Large Language Models with Bias Intelligence Quotient (BiQ)

• Authors: Malur Narayan, John Pasmore, Elton Sampaio, Vijay Raghavan, Gabriella Waters
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18276
• Pdf URL: https://arxiv.org/pdf/2404.18276
• Copy Paste: [[2404.18276]] Bias Neutralization Framework: Measuring Fairness in Large Language Models with Bias Intelligence Quotient (BiQ)(https://arxiv.org/abs/2404.18276)
• Keywords: fair, large language model
• Abstract: The burgeoning influence of Large Language Models (LLMs) in shaping public discourse and decision-making underscores the imperative to address inherent biases within these AI systems. In the wake of AI's expansive integration across sectors, addressing racial bias in LLMs has never been more critical. This paper introduces a novel framework called Comprehensive Bias Neutralization Framework (CBNF) which embodies an innovative approach to quantifying and mitigating biases within LLMs. Our framework combines the Large Language Model Bias Index (LLMBI) [Oketunji, A., Anas, M., Saina, D., (2023)] and Bias removaL with No Demographics (BLIND) [Orgad, H., Belinkov, Y. (2023)] methodologies to create a new metric called Bias Intelligence Quotient (BiQ)which detects, measures, and mitigates racial bias in LLMs without reliance on demographic annotations. By introducing a new metric called BiQ that enhances LLMBI with additional fairness metrics, CBNF offers a multi-dimensional metric for bias assessment, underscoring the necessity of a nuanced approach to fairness in AI [Mehrabi et al., 2021]. This paper presents a detailed analysis of Latimer AI (a language model incrementally trained on black history and culture) in comparison to ChatGPT 3.5, illustrating Latimer AI's efficacy in detecting racial, cultural, and gender biases through targeted training and refined bias mitigation strategies [Latimer & Bender, 2023].

Title: Comparing LLM prompting with Cross-lingual transfer performance on Indigenous and Low-resource Brazilian Languages

• Authors: David Ifeoluwa Adelani, A. Seza Doğruöz, André Coneglian, Atul Kr. Ojha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18286
• Pdf URL: https://arxiv.org/pdf/2404.18286
• Copy Paste: [[2404.18286]] Comparing LLM prompting with Cross-lingual transfer performance on Indigenous and Low-resource Brazilian Languages(https://arxiv.org/abs/2404.18286)
• Keywords: large language model
• Abstract: Large Language Models are transforming NLP for a variety of tasks. However, how LLMs perform NLP tasks for low-resource languages (LRLs) is less explored. In line with the goals of the AmeicasNLP workshop, we focus on 12 LRLs from Brazil, 2 LRLs from Africa and 2 high-resource languages (HRLs) (e.g., English and Brazilian Portuguese). Our results indicate that the LLMs perform worse for the part of speech (POS) labeling of LRLs in comparison to HRLs. We explain the reasons behind this failure and provide an error analyses through examples observed in our data set.

Title: Joint Energy and Latency Optimization in Federated Learning over Cell-Free Massive MIMO Networks

• Authors: Afsaneh Mahmoudi, Mahmoud Zaher, Emil Björnson
• Subjects: cs.LG, cs.IT, cs.NI
• Abstract URL: https://arxiv.org/abs/2404.18287
• Pdf URL: https://arxiv.org/pdf/2404.18287
• Copy Paste: [[2404.18287]] Joint Energy and Latency Optimization in Federated Learning over Cell-Free Massive MIMO Networks(https://arxiv.org/abs/2404.18287)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) is a distributed learning paradigm wherein users exchange FL models with a server instead of raw datasets, thereby preserving data privacy and reducing communication overhead. However, the increased number of FL users may hinder completing large-scale FL over wireless networks due to high imposed latency. Cell-free massive multiple-input multiple-output~(CFmMIMO) is a promising architecture for implementing FL because it serves many users on the same time/frequency resources. While CFmMIMO enhances energy efficiency through spatial multiplexing and collaborative beamforming, it remains crucial to meticulously allocate uplink transmission powers to the FL users. In this paper, we propose an uplink power allocation scheme in FL over CFmMIMO by considering the effect of each user's power on the energy and latency of other users to jointly minimize the users' uplink energy and the latency of FL training. The proposed solution algorithm is based on the coordinate gradient descent method. Numerical results show that our proposed method outperforms the well-known max-sum rate by increasing up to~$27$\% and max-min energy efficiency of the Dinkelbach method by increasing up to~$21$\% in terms of test accuracy while having limited uplink energy and latency budget for FL over CFmMIMO.

Title: Panoptic Segmentation and Labelling of Lumbar Spine Vertebrae using Modified Attention Unet

• Authors: Rikathi Pal, Priya Saha, Somoballi Ghoshal, Amlan Chakrabarti, Susmita Sur-Kolay
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18291
• Pdf URL: https://arxiv.org/pdf/2404.18291
• Copy Paste: [[2404.18291]] Panoptic Segmentation and Labelling of Lumbar Spine Vertebrae using Modified Attention Unet(https://arxiv.org/abs/2404.18291)
• Keywords: segmentation
• Abstract: Segmentation and labeling of vertebrae in MRI images of the spine are critical for the diagnosis of illnesses and abnormalities. These steps are indispensable as MRI technology provides detailed information about the tissue structure of the spine. Both supervised and unsupervised segmentation methods exist, yet acquiring sufficient data remains challenging for achieving high accuracy. In this study, we propose an enhancing approach based on modified attention U-Net architecture for panoptic segmentation of 3D sliced MRI data of the lumbar spine. Our method achieves an impressive accuracy of 99.5\% by incorporating novel masking logic, thus significantly advancing the state-of-the-art in vertebral segmentation and labeling. This contributes to more precise and reliable diagnosis and treatment planning.

Title: Near-Term Enforcement of AI Chip Export Controls Using A Minimal Firmware-Based Design for Offline Licensing

• Authors: James Petrie
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.18308
• Pdf URL: https://arxiv.org/pdf/2404.18308
• Copy Paste: [[2404.18308]] Near-Term Enforcement of AI Chip Export Controls Using A Minimal Firmware-Based Design for Offline Licensing(https://arxiv.org/abs/2404.18308)
• Keywords: secure, security, protect, attack
• Abstract: Offline licensing is a technical mechanism for compute governance that could be used to prevent unregulated training of potentially dangerous frontier AI models. The mechanism works by disabling AI chips unless they have an up-to-date license from a regulator. In this report, we present a technical design for a minimal version of offline licensing that could be delivered via a firmware update. Existing AI chips could potentially support offline licensing within a year if they have the following (relatively common) hardware security features: firmware verification, firmware rollback protection, and secure non-volatile memory. Public documentation suggests that NVIDIA's H100 AI chip already has these security features. Without additional hardware modifications, the system is susceptible to physical hardware attacks. However, these attacks might require expensive equipment and could be difficult to reliably apply to thousands of AI chips. A firmware-based offline licensing design shares the same legal requirements and license approval mechanism as a hardware-based solution. Implementing a firmware-based solution now could accelerate the eventual deployment of a more secure hardware-based solution in the future. For AI chip manufacturers, implementing this security mechanism might allow chips to be sold to customers that would otherwise be prohibited by export restrictions. For governments, it may be important to be able to prevent unsafe or malicious actors from training frontier AI models in the next few years. Based on this initial analysis, firmware-based offline licensing could partially solve urgent security and trade problems and is technically feasible for AI chips that have common hardware security features.

Title: Trends and Challenges of Real-time Learning in Large Language Models: A Critical Review

• Authors: Mladjan Jovanovic, Peter Voss
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18311
• Pdf URL: https://arxiv.org/pdf/2404.18311
• Copy Paste: [[2404.18311]] Trends and Challenges of Real-time Learning in Large Language Models: A Critical Review(https://arxiv.org/abs/2404.18311)
• Keywords: large language model
• Abstract: Real-time learning concerns the ability of learning systems to acquire knowledge over time, enabling their adaptation and generalization to novel tasks. It is a critical ability for intelligent, real-world systems, especially when data may be insufficient or difficult to obtain. This review provides a comprehensive analysis of real-time learning in Large Language Models. It synthesizes the state-of-the-art real-time learning paradigms, including continual learning, meta-learning, parameter-efficient learning, and mixture-of-experts learning. We demonstrate their utility for real-time learning by describing specific achievements from these related topics and their critical factors. Finally, the paper highlights current problems and challenges for future research in the field. By consolidating the latest relevant research developments, this review offers a comprehensive understanding of real-time learning and its implications for designing and developing LLM-based learning systems addressing real-world problems.

Title: DIRESA, a distance-preserving nonlinear dimension reduction technique based on regularized autoencoders

• Authors: Geert De Paepe, Lesley De Cruz
• Subjects: cs.LG, nlin.CD, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2404.18314
• Pdf URL: https://arxiv.org/pdf/2404.18314
• Copy Paste: [[2404.18314]] DIRESA, a distance-preserving nonlinear dimension reduction technique based on regularized autoencoders(https://arxiv.org/abs/2404.18314)
• Keywords: robust
• Abstract: In meteorology, finding similar weather patterns or analogs in historical datasets can be useful for data assimilation, forecasting, and postprocessing. In climate science, analogs in historical and climate projection data are used for attribution and impact studies. However, most of the time, those large weather and climate datasets are nearline. They must be downloaded, which takes a lot of bandwidth and disk space, before the computationally expensive search can be executed. We propose a dimension reduction technique based on autoencoder (AE) neural networks to compress those datasets and perform the search in an interpretable, compressed latent space. A distance-regularized Siamese twin autoencoder (DIRESA) architecture is designed to preserve distance in latent space while capturing the nonlinearities in the datasets. Using conceptual climate models of different complexities, we show that the latent components thus obtained provide physical insight into the dominant modes of variability in the system. Compressing datasets with DIRESA reduces the online storage and keeps the latent components uncorrelated, while the distance (ordering) preservation and reconstruction fidelity robustly outperform Principal Component Analysis (PCA) and other dimension reduction techniques such as UMAP or variational autoencoders.

Title: SAFE-RL: Saliency-Aware Counterfactual Explainer for Deep Reinforcement Learning Policies

• Authors: Amir Samadi, Konstantinos Koufos, Kurt Debattista, Mehrdad Dianati
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18326
• Pdf URL: https://arxiv.org/pdf/2404.18326
• Copy Paste: [[2404.18326]] SAFE-RL: Saliency-Aware Counterfactual Explainer for Deep Reinforcement Learning Policies(https://arxiv.org/abs/2404.18326)
• Keywords: explainability, generative
• Abstract: While Deep Reinforcement Learning (DRL) has emerged as a promising solution for intricate control tasks, the lack of explainability of the learned policies impedes its uptake in safety-critical applications, such as automated driving systems (ADS). Counterfactual (CF) explanations have recently gained prominence for their ability to interpret black-box Deep Learning (DL) models. CF examples are associated with minimal changes in the input, resulting in a complementary output by the DL model. Finding such alternations, particularly for high-dimensional visual inputs, poses significant challenges. Besides, the temporal dependency introduced by the reliance of the DRL agent action on a history of past state observations further complicates the generation of CF examples. To address these challenges, we propose using a saliency map to identify the most influential input pixels across the sequence of past observed states by the agent. Then, we feed this map to a deep generative model, enabling the generation of plausible CFs with constrained modifications centred on the salient regions. We evaluate the effectiveness of our framework in diverse domains, including ADS, Atari Pong, Pacman and space-invaders games, using traditional performance metrics such as validity, proximity and sparsity. Experimental results demonstrate that this framework generates more informative and plausible CFs than the state-of-the-art for a wide range of environments and DRL agents. In order to foster research in this area, we have made our datasets and codes publicly available at https://github.com/Amir-Samadi/SAFE-RL.

Title: Multi-stage Attack Detection and Prediction Using Graph Neural Networks: An IoT Feasibility Study

• Authors: Hamdi Friji, Ioannis Mavromatis, Adrian Sanchez-Mompo, Pietro Carnelli, Alexis Olivereau, Aftab Khan
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18328
• Pdf URL: https://arxiv.org/pdf/2404.18328
• Copy Paste: [[2404.18328]] Multi-stage Attack Detection and Prediction Using Graph Neural Networks: An IoT Feasibility Study(https://arxiv.org/abs/2404.18328)
• Keywords: security, attack
• Abstract: With the ever-increasing reliance on digital networks for various aspects of modern life, ensuring their security has become a critical challenge. Intrusion Detection Systems play a crucial role in ensuring network security, actively identifying and mitigating malicious behaviours. However, the relentless advancement of cyber-threats has rendered traditional/classical approaches insufficient in addressing the sophistication and complexity of attacks. This paper proposes a novel 3-stage intrusion detection system inspired by a simplified version of the Lockheed Martin cyber kill chain to detect advanced multi-step attacks. The proposed approach consists of three models, each responsible for detecting a group of attacks with common characteristics. The detection outcome of the first two stages is used to conduct a feasibility study on the possibility of predicting attacks in the third stage. Using the ToN IoT dataset, we achieved an average of 94% F1-Score among different stages, outperforming the benchmark approaches based on Random-forest model. Finally, we comment on the feasibility of this approach to be integrated in a real-world system and propose various possible future work.

Title: L-DIT: A dApp for Live Detectability, Identifiability and Trackability for ASOs on the Behavioral Dynamics Blockchain

• Authors: Anirban Chowdhury, Yasir Latif, Ivan Aksenov, Moriba K. Jah, Samya Bagchi
• Subjects: cs.CR, astro-ph.IM
• Abstract URL: https://arxiv.org/abs/2404.18350
• Pdf URL: https://arxiv.org/pdf/2404.18350
• Copy Paste: [[2404.18350]] L-DIT: A dApp for Live Detectability, Identifiability and Trackability for ASOs on the Behavioral Dynamics Blockchain(https://arxiv.org/abs/2404.18350)
• Keywords: security
• Abstract: As the number of Anthropogenic Space Objects (ASOs) grows, there is an urgent need to ensure space safety, security, and sustainability (S3) for long-term space use. Currently, no globally effective method can quantify the safety, security, and Sustainability of all ASOs in orbit. Existing methods such as the Space Sustainability Rating (SSR) rely on volunteering private information to provide sustainability ratings. However, the need for such sensitive data might prove to be a barrier to adoption for space entities. For effective comparison of ASOs, the rating mechanism should apply to all ASOs, even retroactively, so that the sustainability of a single ASO can be assessed holistically. Lastly, geopolitical boundaries and alignments play a crucial and limiting role in a volunteered rating system, limiting the space safety, security, and sustainability. This work presents a Live Detectability, Identifiability, and Trackability (L-DIT) score through a distributed app (dApp) built on top of the Behavioral Dynamics blockchain (BDB). The BDB chain is a space situational awareness (SSA) chain that provides verified and cross-checked ASO data from multiple sources. This unique combination of consensus-based information from BDB and permissionless access to data allows the DIT scoring method presented here to be applied to all ASOs. While the underlying BDB chain collects, filters, and validates SSA data from various open (and closed if available) sources, the L-DIT dApp consumes the data from the chain to provide L-DIT score that can contribute towards an operator's, manufacturer's, or owner's sustainability practices. Our dApp provides data for all ASOs, allowing their sustainability score to be compared against other ASOs, regardless of geopolitical alignments, providing business value to entities such as space insurance providers and enabling compliance validation and enforcement.

Title: Post-hoc and manifold explanations analysis of facial expression data based on deep learning

• Authors: Yang Xiao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18352
• Pdf URL: https://arxiv.org/pdf/2404.18352
• Copy Paste: [[2404.18352]] Post-hoc and manifold explanations analysis of facial expression data based on deep learning(https://arxiv.org/abs/2404.18352)
• Keywords: explainability
• Abstract: The complex information processing system of humans generates a lot of objective and subjective evaluations, making the exploration of human cognitive products of great cutting-edge theoretical value. In recent years, deep learning technologies, which are inspired by biological brain mechanisms, have made significant strides in the application of psychological or cognitive scientific research, particularly in the memorization and recognition of facial data. This paper investigates through experimental research how neural networks process and store facial expression data and associate these data with a range of psychological attributes produced by humans. Researchers utilized deep learning model VGG16, demonstrating that neural networks can learn and reproduce key features of facial data, thereby storing image memories. Moreover, the experimental results reveal the potential of deep learning models in understanding human emotions and cognitive processes and establish a manifold visualization interpretation of cognitive products or psychological attributes from a non-Euclidean space perspective, offering new insights into enhancing the explainability of AI. This study not only advances the application of AI technology in the field of psychology but also provides a new psychological theoretical understanding the information processing of the AI. The code is available in here: https://github.com/NKUShaw/Psychoinformatics.

Title: Do Neutral Prompts Produce Insecure Code? FormAI-v2 Dataset: Labelling Vulnerabilities in Code Generated by Large Language Models

• Authors: Norbert Tihanyi, Tamas Bisztray, Mohamed Amine Ferrag, Ridhi Jain, Lucas C. Cordeiro
• Subjects: cs.CR, cs.AI, cs.PL
• Abstract URL: https://arxiv.org/abs/2404.18353
• Pdf URL: https://arxiv.org/pdf/2404.18353
• Copy Paste: [[2404.18353]] Do Neutral Prompts Produce Insecure Code? FormAI-v2 Dataset: Labelling Vulnerabilities in Code Generated by Large Language Models(https://arxiv.org/abs/2404.18353)
• Keywords: secure, security, robust, large language model
• Abstract: This study provides a comparative analysis of state-of-the-art large language models (LLMs), analyzing how likely they generate vulnerabilities when writing simple C programs using a neutral zero-shot prompt. We address a significant gap in the literature concerning the security properties of code produced by these models without specific directives. N. Tihanyi et al. introduced the FormAI dataset at PROMISE '23, containing 112,000 GPT-3.5-generated C programs, with over 51.24% identified as vulnerable. We expand that work by introducing the FormAI-v2 dataset comprising 265,000 compilable C programs generated using various LLMs, including robust models such as Google's GEMINI-pro, OpenAI's GPT-4, and TII's 180 billion-parameter Falcon, to Meta's specialized 13 billion-parameter CodeLLama2 and various other compact models. Each program in the dataset is labelled based on the vulnerabilities detected in its source code through formal verification using the Efficient SMT-based Context-Bounded Model Checker (ESBMC). This technique eliminates false positives by delivering a counterexample and ensures the exclusion of false negatives by completing the verification process. Our study reveals that at least 63.47% of the generated programs are vulnerable. The differences between the models are minor, as they all display similar coding errors with slight variations. Our research highlights that while LLMs offer promising capabilities for code generation, deploying their output in a production environment requires risk assessment and validation.

Title: FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Models

• Authors: Wei Li, Ren Ma, Jiang Wu, Chenya Gu, Jiahui Peng, Jinyang Len, Songyang Zhang, Hang Yan, Dahua Lin, Conghui He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18359
• Pdf URL: https://arxiv.org/pdf/2404.18359
• Copy Paste: [[2404.18359]] FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Models(https://arxiv.org/abs/2404.18359)
• Keywords: robust, large language model
• Abstract: In the burgeoning field of large language models (LLMs), the assessment of fundamental knowledge remains a critical challenge, particularly for models tailored to Chinese language and culture. This paper introduces FoundaBench, a pioneering benchmark designed to rigorously evaluate the fundamental knowledge capabilities of Chinese LLMs. FoundaBench encompasses a diverse array of 3354 multiple-choice questions across common sense and K-12 educational subjects, meticulously curated to reflect the breadth and depth of everyday and academic knowledge. We present an extensive evaluation of 12 state-of-the-art LLMs using FoundaBench, employing both traditional assessment methods and our CircularEval protocol to mitigate potential biases in model responses. Our results highlight the superior performance of models pre-trained on Chinese corpora, and reveal a significant disparity between models' reasoning and memory recall capabilities. The insights gleaned from FoundaBench evaluations set a new standard for understanding the fundamental knowledge of LLMs, providing a robust framework for future advancements in the field.

Title: "What Keeps People Secure is That They Met The Security Team": Deconstructing Drivers And Goals of Organizational Security Awareness

• Authors: Jonas Hielscher, Simon Parkin
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.18365
• Pdf URL: https://arxiv.org/pdf/2404.18365
• Copy Paste: [[2404.18365]] "What Keeps People Secure is That They Met The Security Team": Deconstructing Drivers And Goals of Organizational Security Awareness(https://arxiv.org/abs/2404.18365)
• Keywords: secure, security
• Abstract: Security awareness campaigns in organizations now collectively cost billions of dollars annually. There is increasing focus on ensuring certain security behaviors among employees. On the surface, this would imply a user-centered view of security in organizations. Despite this, the basis of what security awareness managers do and what decides this are unclear. We conducted n=15 semi-structured interviews with full-time security awareness managers, with experience across various national and international companies in European countries, with thousands of employees. Through thematic analysis, we identify that success in awareness management is fragile while having the potential to improve; there are a range of restrictions, and mismatched drivers and goals for security awareness, affecting how it is structured, delivered, measured, and improved. We find that security awareness as a practice is underspecified, and split between messaging around secure behaviors and connecting to employees, with a lack of recognition for the measures that awareness managers regard as important. We discuss ways forward, including alternative indicators of success, and security usability advocacy for employees.

Title: QANA: LLM-based Question Generation and Network Analysis for Zero-shot Key Point Analysis and Beyond

• Authors: Tomoki Fukuma, Koki Noda, Toshihide Ubukata Kousuke Hoso, Yoshiharu Ichikawa, Kyosuke Kambe, Yu Masubuch, Fujio Toriumi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18371
• Pdf URL: https://arxiv.org/pdf/2404.18371
• Copy Paste: [[2404.18371]] QANA: LLM-based Question Generation and Network Analysis for Zero-shot Key Point Analysis and Beyond(https://arxiv.org/abs/2404.18371)
• Keywords: large language model
• Abstract: The proliferation of social media has led to information overload and increased interest in opinion mining. We propose "Question-Answering Network Analysis" (QANA), a novel opinion mining framework that utilizes Large Language Models (LLMs) to generate questions from users' comments, constructs a bipartite graph based on the comments' answerability to the questions, and applies centrality measures to examine the importance of opinions. We investigate the impact of question generation styles, LLM selections, and the choice of embedding model on the quality of the constructed QA networks by comparing them with annotated Key Point Analysis datasets. QANA achieves comparable performance to previous state-of-the-art supervised models in a zero-shot manner for Key Point Matching task, also reducing the computational cost from quadratic to linear. For Key Point Generation, questions with high PageRank or degree centrality align well with manually annotated key points. Notably, QANA enables analysts to assess the importance of key points from various aspects according to their selection of centrality measure. QANA's primary contribution lies in its flexibility to extract key points from a wide range of perspectives, which enhances the quality and impartiality of opinion mining.

Title: SPECIAL: Synopsis Assisted Secure Collaborative Analytics

• Authors: Chenghong Wang, Lina Qiu, Johes Bater, Yukui Luo
• Subjects: cs.CR, cs.DB
• Abstract URL: https://arxiv.org/abs/2404.18388
• Pdf URL: https://arxiv.org/pdf/2404.18388
• Copy Paste: [[2404.18388]] SPECIAL: Synopsis Assisted Secure Collaborative Analytics(https://arxiv.org/abs/2404.18388)
• Keywords: secure, privacy
• Abstract: Secure collaborative analytics (SCA) enable the processing of analytical SQL queries across multiple owners' data, even when direct data sharing is not feasible. Although essential for strong privacy, the large overhead from data-oblivious primitives in traditional SCA has hindered its practical adoption. Recent SCA variants that permit controlled leakages under differential privacy (DP) show a better balance between privacy and efficiency. However, they still face significant challenges, such as potentially unbounded privacy loss, suboptimal query planning, and lossy processing. To address these challenges, we introduce SPECIAL, the first SCA system that simultaneously ensures bounded privacy loss, advanced query planning, and lossless processing. SPECIAL employs a novel synopsis-assisted secure processing model, where a one-time privacy cost is spent to acquire private synopses (table statistics) from owner data. These synopses then allow SPECIAL to estimate (compaction) sizes for secure operations (e.g., filter, join) and index encrypted data without extra privacy loss. Crucially, these estimates and indexes can be prepared before runtime, thereby facilitating efficient query planning and accurate cost estimations. Moreover, by using one-sided noise mechanisms and private upper bound techniques, SPECIAL ensures strict lossless processing for complex queries (e.g., multi-join). Through a comprehensive benchmark, we show that SPECIAL significantly outperforms cutting-edge SCAs, with up to 80X faster query times and over 900X smaller memory for complex queries. Moreover, it also achieves up to an 89X reduction in privacy loss under continual processing.

Title: Reconstructing Satellites in 3D from Amateur Telescope Images

• Authors: Zhiming Chang, Boyang Liu, Yifei Xia, Youming Guo, Boxin Shi, He Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18394
• Pdf URL: https://arxiv.org/pdf/2404.18394
• Copy Paste: [[2404.18394]] Reconstructing Satellites in 3D from Amateur Telescope Images(https://arxiv.org/abs/2404.18394)
• Keywords: robust, extraction
• Abstract: This paper proposes a framework for the 3D reconstruction of satellites in low-Earth orbit, utilizing videos captured by small amateur telescopes. The video data obtained from these telescopes differ significantly from data for standard 3D reconstruction tasks, characterized by intense motion blur, atmospheric turbulence, pervasive background light pollution, extended focal length and constrained observational perspectives. To address these challenges, our approach begins with a comprehensive pre-processing workflow that encompasses deep learning-based image restoration, feature point extraction and camera pose initialization. We proceed with the application of an improved 3D Gaussian splatting algorithm for reconstructing the 3D model. Our technique supports simultaneous 3D Gaussian training and pose estimation, enabling the robust generation of intricate 3D point clouds from sparse, noisy data. The procedure is further bolstered by a post-editing phase designed to eliminate noise points inconsistent with our prior knowledge of a satellite's geometric constraints. We validate our approach using both synthetic datasets and actual observations of China's Space Station, showcasing its significant advantages over existing methods in reconstructing 3D space objects from ground-based observations.

Title: DRAM-Profiler: An Experimental DRAM RowHammer Vulnerability Profiling Mechanism

• Authors: Ranyang Zhou, Jacqueline T. Liu, Nakul Kochar, Sabbir Ahmed, Adnan Siraj Rakin, Shaahin Angizi
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2404.18396
• Pdf URL: https://arxiv.org/pdf/2404.18396
• Copy Paste: [[2404.18396]] DRAM-Profiler: An Experimental DRAM RowHammer Vulnerability Profiling Mechanism(https://arxiv.org/abs/2404.18396)
• Keywords: security, attack
• Abstract: RowHammer stands out as a prominent example, potentially the pioneering one, showcasing how a failure mechanism at the circuit level can give rise to a significant and pervasive security vulnerability within systems. Prior research has approached RowHammer attacks within a static threat model framework. Nonetheless, it warrants consideration within a more nuanced and dynamic model. This paper presents a low-overhead DRAM RowHammer vulnerability profiling technique termed DRAM-Profiler, which utilizes innovative test vectors for categorizing memory cells into distinct security levels. The proposed test vectors intentionally weaken the spatial correlation between the aggressors and victim rows before an attack for evaluation, thus aiding designers in mitigating RowHammer vulnerabilities in the mapping phase. While there has been no previous research showcasing the impact of such profiling to our knowledge, our study methodically assesses 128 commercial DDR4 DRAM products. The results uncover the significant variability among chips from different manufacturers in the type and quantity of RowHammer attacks that can be exploited by adversaries.

Title: LLM-SR: Scientific Equation Discovery via Programming with Large Language Models

• Authors: Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, Chandan K Reddy
• Subjects: cs.LG, cs.AI, cs.CL, cs.NE
• Abstract URL: https://arxiv.org/abs/2404.18400
• Pdf URL: https://arxiv.org/pdf/2404.18400
• Copy Paste: [[2404.18400]] LLM-SR: Scientific Equation Discovery via Programming with Large Language Models(https://arxiv.org/abs/2404.18400)
• Keywords: robust, large language model
• Abstract: Mathematical equations have been unreasonably effective in describing complex natural phenomena across various scientific disciplines. However, discovering such insightful equations from data presents significant challenges due to the necessity of navigating extremely high-dimensional combinatorial and nonlinear hypothesis spaces. Traditional methods of equation discovery largely focus on extracting equations from data alone, often neglecting the rich domain-specific prior knowledge that scientists typically depend on. To bridge this gap, we introduce LLM-SR, a novel approach that leverages the extensive scientific knowledge and robust code generation capabilities of Large Language Models (LLMs) to discover scientific equations from data in an efficient manner. Specifically, LLM-SR treats equations as programs with mathematical operators and combines LLMs' scientific priors with evolutionary search over equation programs. The LLM iteratively proposes new equation skeletons, drawing from its physical understanding, which are then optimized against data to estimate skeleton parameters. We demonstrate LLM-SR's effectiveness across three diverse scientific domains, where it discovers physically accurate equations that provide significantly better fits to in-domain and out-of-domain data compared to the well-established equation discovery baselines

Title: Spectral-Spatial Mamba for Hyperspectral Image Classification

• Authors: Lingbo Huang, Yushi Chen, Xin He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18401
• Pdf URL: https://arxiv.org/pdf/2404.18401
• Copy Paste: [[2404.18401]] Spectral-Spatial Mamba for Hyperspectral Image Classification(https://arxiv.org/abs/2404.18401)
• Keywords: transformer
• Abstract: Recently, deep learning models have achieved excellent performance in hyperspectral image (HSI) classification. Among the many deep models, Transformer has gradually attracted interest for its excellence in modeling the long-range dependencies of spatial-spectral features in HSI. However, Transformer has the problem of quadratic computational complexity due to the self-attention mechanism, which is heavier than other models and thus has limited adoption in HSI processing. Fortunately, the recently emerging state space model-based Mamba shows great computational efficiency while achieving the modeling power of Transformers. Therefore, in this paper, we make a preliminary attempt to apply the Mamba to HSI classification, leading to the proposed spectral-spatial Mamba (SS-Mamba). Specifically, the proposed SS-Mamba mainly consists of spectral-spatial token generation module and several stacked spectral-spatial Mamba blocks. Firstly, the token generation module converts any given HSI cube to spatial and spectral tokens as sequences. And then these tokens are sent to stacked spectral-spatial mamba blocks (SS-MB). Each SS-MB block consists of two basic mamba blocks and a spectral-spatial feature enhancement module. The spatial and spectral tokens are processed separately by the two basic mamba blocks, respectively. Besides, the feature enhancement module modulates spatial and spectral tokens using HSI sample's center region information. In this way, the spectral and spatial tokens cooperate with each other and achieve information fusion within each block. The experimental results conducted on widely used HSI datasets reveal that the proposed model achieves competitive results compared with the state-of-the-art methods. The Mamba-based method opens a new window for HSI classification.

Title: ICMarks: A Robust Watermarking Framework for Integrated Circuit Physical Design IP Protection

• Authors: Ruisi Zhang, Rachel Selina Rajarathnam, David Z. Pan, Farinaz Koushanfar
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2404.18407
• Pdf URL: https://arxiv.org/pdf/2404.18407
• Copy Paste: [[2404.18407]] ICMarks: A Robust Watermarking Framework for Integrated Circuit Physical Design IP Protection(https://arxiv.org/abs/2404.18407)
• Keywords: protect, attack, robust, watermark
• Abstract: Physical design watermarking on contemporary integrated circuit (IC) layout encodes signatures without considering the dense connections and design constraints, which could lead to performance degradation on the watermarked products. This paper presents ICMarks, a quality-preserving and robust watermarking framework for modern IC physical design. ICMarks embeds unique watermark signatures during the physical design's placement stage, thereby authenticating the IC layout ownership. ICMarks's novelty lies in (i) strategically identifying a region of cells to watermark with minimal impact on the layout performance and (ii) a two-level watermarking framework for augmented robustness toward potential removal and forging attacks. Extensive evaluations on benchmarks of different design objectives and sizes validate that ICMarks incurs no wirelength and timing metrics degradation, while successfully proving ownership. Furthermore, we demonstrate ICMarks is robust against two major watermarking attack categories, namely, watermark removal and forging attacks; even if the adversaries have prior knowledge of the watermarking schemes, the signatures cannot be removed without significantly undermining the layout quality.

Title: PKU-AIGIQA-4K: A Perceptual Quality Assessment Database for Both Text-to-Image and Image-to-Image AI-Generated Images

• Authors: Jiquan Yuan, Fanyi Yang, Jihe Li, Xinyan Cao, Jinming Che, Jinlong Lin, Xixin Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18409
• Pdf URL: https://arxiv.org/pdf/2404.18409
• Copy Paste: [[2404.18409]] PKU-AIGIQA-4K: A Perceptual Quality Assessment Database for Both Text-to-Image and Image-to-Image AI-Generated Images(https://arxiv.org/abs/2404.18409)
• Keywords: generative
• Abstract: In recent years, image generation technology has rapidly advanced, resulting in the creation of a vast array of AI-generated images (AIGIs). However, the quality of these AIGIs is highly inconsistent, with low-quality AIGIs severely impairing the visual experience of users. Due to the widespread application of AIGIs, the AI-generated image quality assessment (AIGIQA), aimed at evaluating the quality of AIGIs from the perspective of human perception, has garnered increasing interest among scholars. Nonetheless, current research has not yet fully explored this field. We have observed that existing databases are limited to images generated from single scenario settings. Databases such as AGIQA-1K, AGIQA-3K, and AIGCIQA2023, for example, only include images generated by text-to-image generative models. This oversight highlights a critical gap in the current research landscape, underscoring the need for dedicated databases catering to image-to-image scenarios, as well as more comprehensive databases that encompass a broader range of AI-generated image scenarios. Addressing these issues, we have established a large scale perceptual quality assessment database for both text-to-image and image-to-image AIGIs, named PKU-AIGIQA-4K. We then conduct a well-organized subjective experiment to collect quality labels for AIGIs and perform a comprehensive analysis of the PKU-AIGIQA-4K database. Regarding the use of image prompts during the training process, we propose three image quality assessment (IQA) methods based on pre-trained models that include a no-reference method NR-AIGCIQA, a full-reference method FR-AIGCIQA, and a partial-reference method PR-AIGCIQA. Finally, leveraging the PKU-AIGIQA-4K database, we conduct extensive benchmark experiments and compare the performance of the proposed methods and the current IQA methods.

Title: Mixture-of-Instructions: Comprehensive Alignment of a Large Language Model through the Mixture of Diverse System Prompting Instructions

• Authors: Bowen Xu, Shaoyu Wu, Kai Liu, Lulu Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18410
• Pdf URL: https://arxiv.org/pdf/2404.18410
• Copy Paste: [[2404.18410]] Mixture-of-Instructions: Comprehensive Alignment of a Large Language Model through the Mixture of Diverse System Prompting Instructions(https://arxiv.org/abs/2404.18410)
• Keywords: robust, generative, large language model
• Abstract: With the proliferation of large language models (LLMs), the comprehensive alignment of such models across multiple tasks has emerged as a critical area of research. Existing alignment methodologies primarily address single task, such as multi-turn dialogue, coding, mathematical problem-solving, and tool usage. However, AI-driven products that leverage language models usually necessitate a fusion of these abilities to function effectively in real-world scenarios. Moreover, the considerable computational resources required for proper alignment of LLMs underscore the need for a more robust, efficient, and encompassing approach to multi-task alignment, ensuring improved generative performance. In response to these challenges, we introduce a novel technique termed Mixture-of-Instructions (MoI), which employs a strategy of instruction concatenation combined with diverse system prompts to boost the alignment efficiency of language models. We have also compiled a diverse set of seven benchmark datasets to rigorously evaluate the alignment efficacy of the MoI-enhanced language model. Our methodology was applied to the open-source Qwen-7B-chat model, culminating in the development of Qwen-SFT-MoI. This enhanced model demonstrates significant advancements in generative capabilities across coding, mathematics, and tool use tasks.

Title: Research on Intelligent Aided Diagnosis System of Medical Image Based on Computer Deep Learning

• Authors: Jiajie Yuan, Linxiao Wu, Yulu Gong, Zhou Yu, Ziang Liu, Shuyao He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18419
• Pdf URL: https://arxiv.org/pdf/2404.18419
• Copy Paste: [[2404.18419]] Research on Intelligent Aided Diagnosis System of Medical Image Based on Computer Deep Learning(https://arxiv.org/abs/2404.18419)
• Keywords: extraction, segmentation
• Abstract: This paper combines Struts and Hibernate two architectures together, using DAO (Data Access Object) to store and access data. Then a set of dual-mode humidity medical image library suitable for deep network is established, and a dual-mode medical image assisted diagnosis method based on the image is proposed. Through the test of various feature extraction methods, the optimal operating characteristic under curve product (AUROC) is 0.9985, the recall rate is 0.9814, and the accuracy is 0.9833. This method can be applied to clinical diagnosis, and it is a practical method. Any outpatient doctor can register quickly through the system, or log in to the platform to upload the image to obtain more accurate images. Through the system, each outpatient physician can quickly register or log in to the platform for image uploading, thus obtaining more accurate images. The segmentation of images can guide doctors in clinical departments. Then the image is analyzed to determine the location and nature of the tumor, so as to make targeted treatment.

Title: Unsupervised Dynamics Prediction with Object-Centric Kinematics

• Authors: Yeon-Ji Song, Suhyung Choi, Jaein Kim, Jin-Hwa Kim, Byoung-Tak Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18423
• Pdf URL: https://arxiv.org/pdf/2404.18423
• Copy Paste: [[2404.18423]] Unsupervised Dynamics Prediction with Object-Centric Kinematics(https://arxiv.org/abs/2404.18423)
• Keywords: transformer
• Abstract: Human perception involves discerning complex multi-object scenes into time-static object appearance (\ie, size, shape, color) and time-varying object motion (\ie, location, velocity, acceleration). This innate ability to unconsciously understand the environment is the motivation behind the success of dynamics modeling. Object-centric representations have emerged as a promising tool for dynamics prediction, yet they primarily focus on the objects' appearance, often overlooking other crucial attributes. In this paper, we propose Object-Centric Kinematics (OCK), a framework for dynamics prediction leveraging object-centric representations. Our model utilizes a novel component named object kinematics, which comprises low-level structured states of objects' position, velocity, and acceleration. The object kinematics are obtained via either implicit or explicit approaches, enabling comprehensive spatiotemporal object reasoning, and integrated through various transformer mechanisms, facilitating effective object-centric dynamics modeling. Our model demonstrates superior performance when handling objects and backgrounds in complex scenes characterized by a wide range of object attributes and dynamic movements. Moreover, our model demonstrates generalization capabilities across diverse synthetic environments, highlighting its potential for broad applicability in vision-related tasks.

Title: ShadowMaskFormer: Mask Augmented Patch Embeddings for Shadow Removal

• Authors: Zhuohao Li, Guoyang Xie, Guannan Jiang, Zhichao Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18433
• Pdf URL: https://arxiv.org/pdf/2404.18433
• Copy Paste: [[2404.18433]] ShadowMaskFormer: Mask Augmented Patch Embeddings for Shadow Removal(https://arxiv.org/abs/2404.18433)
• Keywords: transformer
• Abstract: Transformer recently emerged as the de facto model for computer vision tasks and has also been successfully applied to shadow removal. However, these existing methods heavily rely on intricate modifications to the attention mechanisms within the transformer blocks while using a generic patch embedding. As a result, it often leads to complex architectural designs requiring additional computation resources. In this work, we aim to explore the efficacy of incorporating shadow information within the early processing stage. Accordingly, we propose a transformer-based framework with a novel patch embedding that is tailored for shadow removal, dubbed ShadowMaskFormer. Specifically, we present a simple and effective mask-augmented patch embedding to integrate shadow information and promote the model's emphasis on acquiring knowledge for shadow regions. Extensive experiments conducted on the ISTD, ISTD+, and SRD benchmark datasets demonstrate the efficacy of our method against state-of-the-art approaches while using fewer model parameters.

Title: BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrievers

• Authors: Ran Xu, Wenqi Shi, Yue Yu, Yuchen Zhuang, Yanqiao Zhu, May D. Wang, Joyce C. Ho, Chao Zhang, Carl Yang
• Subjects: cs.CL, cs.AI, cs.IR, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2404.18443
• Pdf URL: https://arxiv.org/pdf/2404.18443
• Copy Paste: [[2404.18443]] BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrievers(https://arxiv.org/abs/2404.18443)
• Keywords: large language model
• Abstract: Developing effective biomedical retrieval models is important for excelling at knowledge-intensive biomedical tasks but still challenging due to the deficiency of sufficient publicly annotated biomedical data and computational resources. We present BMRetriever, a series of dense retrievers for enhancing biomedical retrieval via unsupervised pre-training on large biomedical corpora, followed by instruction fine-tuning on a combination of labeled datasets and synthetic pairs. Experiments on 5 biomedical tasks across 11 datasets verify BMRetriever's efficacy on various biomedical applications. BMRetriever also exhibits strong parameter efficiency, with the 410M variant outperforming baselines up to 11.7 times larger, and the 2B variant matching the performance of models with over 5B parameters. The training data and model checkpoints are released at \url{https://huggingface.co/BMRetriever} to ensure transparency, reproducibility, and application to new domains.

Title: U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Models

• Authors: Song Mei
• Subjects: cs.LG, cs.AI, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.18444
• Pdf URL: https://arxiv.org/pdf/2404.18444
• Copy Paste: [[2404.18444]] U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Models(https://arxiv.org/abs/2404.18444)
• Keywords: diffusion, generative, segmentation
• Abstract: U-Nets are among the most widely used architectures in computer vision, renowned for their exceptional performance in applications such as image segmentation, denoising, and diffusion modeling. However, a theoretical explanation of the U-Net architecture design has not yet been fully established. This paper introduces a novel interpretation of the U-Net architecture by studying certain generative hierarchical models, which are tree-structured graphical models extensively utilized in both language and image domains. With their encoder-decoder structure, long skip connections, and pooling and up-sampling layers, we demonstrate how U-Nets can naturally implement the belief propagation denoising algorithm in such generative hierarchical models, thereby efficiently approximating the denoising functions. This leads to an efficient sample complexity bound for learning the denoising function using U-Nets within these models. Additionally, we discuss the broader implications of these findings for diffusion models in generative hierarchical models. We also demonstrate that the conventional architecture of convolutional neural networks (ConvNets) is ideally suited for classification tasks within these models. This offers a unified view of the roles of ConvNets and U-Nets, highlighting the versatility of generative hierarchical models in modeling complex data distributions across language and image domains.

Title: MFP: Making Full Use of Probability Maps for Interactive Image Segmentation

• Authors: Chaewon Lee, Seon-Ho Lee, Chang-Su Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18448
• Pdf URL: https://arxiv.org/pdf/2404.18448
• Copy Paste: [[2404.18448]] MFP: Making Full Use of Probability Maps for Interactive Image Segmentation(https://arxiv.org/abs/2404.18448)
• Keywords: segmentation
• Abstract: In recent interactive segmentation algorithms, previous probability maps are used as network input to help predictions in the current segmentation round. However, despite the utilization of previous masks, useful information contained in the probability maps is not well propagated to the current predictions. In this paper, to overcome this limitation, we propose a novel and effective algorithm for click-based interactive image segmentation, called MFP, which attempts to make full use of probability maps. We first modulate previous probability maps to enhance their representations of user-specified objects. Then, we feed the modulated probability maps as additional input to the segmentation network. We implement the proposed MFP algorithm based on the ResNet-34, HRNet-18, and ViT-B backbones and assess the performance extensively on various datasets. It is demonstrated that MFP meaningfully outperforms the existing algorithms using identical backbones. The source codes are available at \href{https://github.com/cwlee00/MFP}{https://github.com/cwlee00/MFP}.

Title: Fostering Trust in Smart Inverters: A Framework for Firmware Update Management and Tracking in VPP Context

• Authors: Thusitha Dayaratne, Carsten Rudolph, Tom Shirley, Sol Levi, David Shirley
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2404.18453
• Pdf URL: https://arxiv.org/pdf/2404.18453
• Copy Paste: [[2404.18453]] Fostering Trust in Smart Inverters: A Framework for Firmware Update Management and Tracking in VPP Context(https://arxiv.org/abs/2404.18453)
• Keywords: security
• Abstract: Ensuring the reliability and security of smart inverters that provide the interface between distributed energy resources (DERs) and the power grid becomes paramount with the surge in integrating DERs into the (smart) power grid. Despite the importance of having updated firmware / software versions within a reasonable time frame, existing methods for establishing trust through firmware updates lack effective historical tracking and verification. This paper introduces a novel framework to manage and track firmware update history, leveraging verifiable credentials. By tracking the update history and implementing a trust cycle based on these verifiable updates, we aim to improve grid resilience, enhance cybersecurity, and increase transparency for stakeholders.

Title: Chameleon: A Data-Efficient Generalist for Dense Visual Prediction in the Wild

• Authors: Donggyun Kim, Seongwoong Cho, Semin Kim, Chong Luo, Seunghoon Hong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18459
• Pdf URL: https://arxiv.org/pdf/2404.18459
• Copy Paste: [[2404.18459]] Chameleon: A Data-Efficient Generalist for Dense Visual Prediction in the Wild(https://arxiv.org/abs/2404.18459)
• Keywords: large language model
• Abstract: Large language models have evolved data-efficient generalists, benefiting from the universal language interface and large-scale pre-training. However, constructing a data-efficient generalist for dense visual prediction presents a distinct challenge due to the variation in label structures across different tasks. Consequently, generalization to unseen dense prediction tasks in the low-data regime is not straightforward and has received less attention from previous vision generalists. In this study, we explore a universal model that can flexibly adapt to unseen dense label structures with a few examples, enabling it to serve as a data-efficient vision generalist in diverse real-world scenarios. To this end, we base our method on a powerful meta-learning framework and explore several axes to improve its performance and versatility for real-world problems, such as flexible adaptation mechanisms and scalability. We evaluate our model across a spectrum of unseen real-world scenarios where low-shot learning is desirable, including video, 3D, medical, biological, and user-interactive tasks. Equipped with a generic architecture and an effective adaptation mechanism, our model flexibly adapts to all of these tasks with at most 50 labeled images, showcasing a significant advancement over existing data-efficient generalist approaches. Codes are available at https://github.com/GitGyun/chameleon.

Title: Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them in

• Authors: Utkarsh Agarwal, Kumar Tanmay, Aditi Khandelwal, Monojit Choudhury
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18460
• Pdf URL: https://arxiv.org/pdf/2404.18460
• Copy Paste: [[2404.18460]] Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them in(https://arxiv.org/abs/2404.18460)
• Keywords: large language model
• Abstract: Ethical reasoning is a crucial skill for Large Language Models (LLMs). However, moral values are not universal, but rather influenced by language and culture. This paper explores how three prominent LLMs -- GPT-4, ChatGPT, and Llama2-70B-Chat -- perform ethical reasoning in different languages and if their moral judgement depend on the language in which they are prompted. We extend the study of ethical reasoning of LLMs by Rao et al. (2023) to a multilingual setup following their framework of probing LLMs with ethical dilemmas and policies from three branches of normative ethics: deontology, virtue, and consequentialism. We experiment with six languages: English, Spanish, Russian, Chinese, Hindi, and Swahili. We find that GPT-4 is the most consistent and unbiased ethical reasoner across languages, while ChatGPT and Llama2-70B-Chat show significant moral value bias when we move to languages other than English. Interestingly, the nature of this bias significantly vary across languages for all LLMs, including GPT-4.

Title: Clicks2Line: Using Lines for Interactive Image Segmentation

• Authors: Chaewon Lee, Chang-Su Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18461
• Pdf URL: https://arxiv.org/pdf/2404.18461
• Copy Paste: [[2404.18461]] Clicks2Line: Using Lines for Interactive Image Segmentation(https://arxiv.org/abs/2404.18461)
• Keywords: segmentation
• Abstract: For click-based interactive segmentation methods, reducing the number of clicks required to obtain a desired segmentation result is essential. Although recent click-based methods yield decent segmentation results, we observe that substantial amount of clicks are required to segment elongated regions. To reduce the amount of user-effort required, we propose using lines instead of clicks for such cases. In this paper, an interactive segmentation algorithm which adaptively adopts either clicks or lines as input is proposed. Experimental results demonstrate that using lines can generate better segmentation results than clicks for several cases.

Title: HFT: Half Fine-Tuning for Large Language Models

• Authors: Tingfeng Hui, Zhenyu Zhang, Shuohuan Wang, Weiran Xu, Yu Sun, Hua Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18466
• Pdf URL: https://arxiv.org/pdf/2404.18466
• Copy Paste: [[2404.18466]] HFT: Half Fine-Tuning for Large Language Models(https://arxiv.org/abs/2404.18466)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) with one or more fine-tuning phases have become a necessary step to unlock various capabilities, enabling LLMs to follow natural language instructions or align with human preferences. However, it carries the risk of catastrophic forgetting during sequential training, the parametric knowledge or the ability learned in previous stages may be overwhelmed by incoming training data. In this paper, we find that by regularly resetting partial parameters, LLMs can restore some of the original knowledge. Inspired by this, we introduce Half Fine-Tuning (HFT) for LLMs, as a substitute for full fine-tuning (FFT), to mitigate the forgetting issues, where half of the parameters are selected to learn new tasks while the other half are frozen to remain previous knowledge. We provide a feasibility analysis from the perspective of optimization and interpret the parameter selection operation as a regularization term. Without changing the model architecture, HFT could be seamlessly integrated into existing fine-tuning frameworks. Extensive experiments and analysis on supervised fine-tuning, direct preference optimization, and continual learning consistently demonstrate the effectiveness, robustness, and efficiency of HFT. Compared with FFT, HFT not only significantly alleviates the forgetting problem, but also achieves the best performance in a series of downstream benchmarks, with an approximately 30% reduction in training time.

Title: Explainability of Machine Learning Approaches in Forensic Linguistics: A Case Study in Geolinguistic Authorship Profiling

• Authors: Dana Roemling, Yves Scherrer, Aleksandra Miletic
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18510
• Pdf URL: https://arxiv.org/pdf/2404.18510
• Copy Paste: [[2404.18510]] Explainability of Machine Learning Approaches in Forensic Linguistics: A Case Study in Geolinguistic Authorship Profiling(https://arxiv.org/abs/2404.18510)
• Keywords: explainability
• Abstract: Forensic authorship profiling uses linguistic markers to infer characteristics about an author of a text. This task is paralleled in dialect classification, where a prediction is made about the linguistic variety of a text based on the text itself. While there have been significant advances in the last years in variety classification (Jauhiainen et al., 2019) and state-of-the-art approaches reach accuracies of up to 100% depending on the similarity of varieties and the scope of prediction (e.g., Milne et al., 2012; Blodgett et al., 2017), forensic linguistics rarely relies on these approaches due to their lack of transparency (see Nini, 2023), amongst other reasons. In this paper we therefore explore explainability of machine learning approaches considering the forensic context. We focus on variety classification as a means of geolinguistic profiling of unknown texts. For this we work with an approach proposed by Xie et al. (2024) to extract the lexical items most relevant to the variety classifications. We find that the extracted lexical features are indeed representative of their respective varieties and note that the trained models also rely on place names for classifications.

Title: On the Impact of Data Heterogeneity in Federated Learning Environments with Application to Healthcare Networks

• Authors: Usevalad Milasheuski. Luca Barbieri, Bernardo Camajori Tedeschini, Monica Nicoli, Stefano Savazzi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18519
• Pdf URL: https://arxiv.org/pdf/2404.18519
• Copy Paste: [[2404.18519]] On the Impact of Data Heterogeneity in Federated Learning Environments with Application to Healthcare Networks(https://arxiv.org/abs/2404.18519)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) allows multiple privacy-sensitive applications to leverage their dataset for a global model construction without any disclosure of the information. One of those domains is healthcare, where groups of silos collaborate in order to generate a global predictor with improved accuracy and generalization. However, the inherent challenge lies in the high heterogeneity of medical data, necessitating sophisticated techniques for assessment and compensation. This paper presents a comprehensive exploration of the mathematical formalization and taxonomy of heterogeneity within FL environments, focusing on the intricacies of medical data. In particular, we address the evaluation and comparison of the most popular FL algorithms with respect to their ability to cope with quantity-based, feature and label distribution-based heterogeneity. The goal is to provide a quantitative evaluation of the impact of data heterogeneity in FL systems for healthcare networks as well as a guideline on FL algorithm selection. Our research extends beyond existing studies by benchmarking seven of the most common FL algorithms against the unique challenges posed by medical data use cases. The paper targets the prediction of the risk of stroke recurrence through a set of tabular clinical reports collected by different federated hospital silos: data heterogeneity frequently encountered in this scenario and its impact on FL performance are discussed.

Title: Enabling Efficient and Flexible Interpretability of Data-driven Anomaly Detection in Industrial Processes with AcME-AD

• Authors: Valentina Zaccaria, Chiara Masiero, David Dandolo, Gian Antonio Susto
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18525
• Pdf URL: https://arxiv.org/pdf/2404.18525
• Copy Paste: [[2404.18525]] Enabling Efficient and Flexible Interpretability of Data-driven Anomaly Detection in Industrial Processes with AcME-AD(https://arxiv.org/abs/2404.18525)
• Keywords: interpretability
• Abstract: While Machine Learning has become crucial for Industry 4.0, its opaque nature hinders trust and impedes the transformation of valuable insights into actionable decision, a challenge exacerbated in the evolving Industry 5.0 with its human-centric focus. This paper addresses this need by testing the applicability of AcME-AD in industrial settings. This recently developed framework facilitates fast and user-friendly explanations for anomaly detection. AcME-AD is model-agnostic, offering flexibility, and prioritizes real-time efficiency. Thus, it seems suitable for seamless integration with industrial Decision Support Systems. We present the first industrial application of AcME-AD, showcasing its effectiveness through experiments. These tests demonstrate AcME-AD's potential as a valuable tool for explainable AD and feature-based root cause analysis within industrial environments, paving the way for trustworthy and actionable insights in the age of Industry 5.0.

Title: Bridging Data Barriers among Participants: Assessing the Potential of Geoenergy through Federated Learning

• Authors: Weike Peng, Jiaxin Gao, Yuntian Chen, Shengwei Wang
• Subjects: cs.LG, cs.AI, cs.CR, stat.AP
• Abstract URL: https://arxiv.org/abs/2404.18527
• Pdf URL: https://arxiv.org/pdf/2404.18527
• Copy Paste: [[2404.18527]] Bridging Data Barriers among Participants: Assessing the Potential of Geoenergy through Federated Learning(https://arxiv.org/abs/2404.18527)
• Keywords: privacy, federate
• Abstract: Machine learning algorithms emerge as a promising approach in energy fields, but its practical is hindered by data barriers, stemming from high collection costs and privacy concerns. This study introduces a novel federated learning (FL) framework based on XGBoost models, enabling safe collaborative modeling with accessible yet concealed data from multiple parties. Hyperparameter tuning of the models is achieved through Bayesian Optimization. To ascertain the merits of the proposed FL-XGBoost method, a comparative analysis is conducted between separate and centralized models to address a classical binary classification problem in geoenergy sector. The results reveal that the proposed FL framework strikes an optimal balance between privacy and accuracy. FL models demonstrate superior accuracy and generalization capabilities compared to separate models, particularly for participants with limited data or low correlation features and offers significant privacy benefits compared to centralized model. The aggregated optimization approach within the FL agreement proves effective in tuning hyperparameters. This study opens new avenues for assessing unconventional reservoirs through collaborative and privacy-preserving FL techniques.

Title: Generation of Uncorrelated Residual Variables for Chemical Process Fault Diagnosis via Transfer Learning-based Input-Output Decoupled Network

• Authors: Zhuofu Pan, Qingkai Sui, Yalin Wang, Jiang Luo, Jie Chen, Hongtian Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18528
• Pdf URL: https://arxiv.org/pdf/2404.18528
• Copy Paste: [[2404.18528]] Generation of Uncorrelated Residual Variables for Chemical Process Fault Diagnosis via Transfer Learning-based Input-Output Decoupled Network(https://arxiv.org/abs/2404.18528)
• Keywords: extraction
• Abstract: Structural decoupling has played an essential role in model-based fault isolation and estimation in past decades, which facilitates accurate fault localization and reconstruction thanks to the diagonal transfer matrix design. However, traditional methods exhibit limited effectiveness in modeling high-dimensional nonlinearity and big data, and the decoupling idea has not been well-valued in data-driven frameworks. Known for big data and complex feature extraction capabilities, deep learning has recently been used to develop residual generation models. Nevertheless, it lacks decoupling-related diagnostic designs. To this end, this paper proposes a transfer learning-based input-output decoupled network (TDN) for diagnostic purposes, which consists of an input-output decoupled network (IDN) and a pre-trained variational autocoder (VAE). In IDN, uncorrelated residual variables are generated by diagonalization and parallel computing operations. During the transfer learning phase, knowledge of normal status is provided according to VAE's loss and maximum mean discrepancy loss to guide the training of IDN. After training, IDN learns the mapping from faulty to normal, thereby serving as the fault detection index and the estimated fault signal simultaneously. At last, the effectiveness of the developed TDN is verified by a numerical example and a chemical simulation.

Title: MileBench: Benchmarking MLLMs in Long Context

• Authors: Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, Benyou Wang
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18532
• Pdf URL: https://arxiv.org/pdf/2404.18532
• Copy Paste: [[2404.18532]] MileBench: Benchmarking MLLMs in Long Context(https://arxiv.org/abs/2404.18532)
• Keywords: large language model
• Abstract: Despite the advancements and impressive performance of Multimodal Large Language Models (MLLMs) on benchmarks, their effectiveness in real-world, long-context, and multi-image tasks is unclear due to the benchmarks' limited scope. Existing benchmarks often focus on single-image and short-text samples, and when assessing multi-image tasks, they either limit the image count or focus on specific task (e.g time-series captioning), potentially obscuring the performance challenges of MLLMs. To address these limitations, we introduce MileBench, a pioneering benchmark designed to test the MultImodal Long-contExt capabilities of MLLMs. This benchmark comprises not only multimodal long contexts, but also multiple tasks requiring both comprehension and generation. We establish two distinct evaluation sets, diagnostic and realistic, to systematically assess MLLMs' long-context adaptation capacity and their ability to complete tasks in long-context scenarios. Our experimental results, obtained from testing 20 models, revealed that while the closed-source GPT-4(Vision) and Gemini 1.5 outperform others, most open-source MLLMs struggle in long-context situations. Interestingly, the performance gap tends to widen with an increase in the number of images. We strongly encourage an intensification of research efforts towards enhancing MLLMs' long-context capabilities, especially in scenarios involving multiple images.

Title: Evaluating and Mitigating Linguistic Discrimination in Large Language Models

• Authors: Guoliang Dong, Haoyu Wang, Jun Sun, Xinyu Wang
• Subjects: cs.CL, cs.AI, cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.18534
• Pdf URL: https://arxiv.org/pdf/2404.18534
• Copy Paste: [[2404.18534]] Evaluating and Mitigating Linguistic Discrimination in Large Language Models(https://arxiv.org/abs/2404.18534)
• Keywords: large language model
• Abstract: By training on text in various languages, large language models (LLMs) typically possess multilingual support and demonstrate remarkable capabilities in solving tasks described in different languages. However, LLMs can exhibit linguistic discrimination due to the uneven distribution of training data across languages. That is, LLMs are hard to keep the consistency of responses when faced with the same task but depicted in different languages. In this study, we first explore the consistency in the LLMs' outputs responding to queries in various languages from two aspects: safety and quality. We conduct this analysis with two datasets (AdvBench and NQ) based on four LLMs (Llama2-13b, Gemma-7b, GPT-3.5-turbo and Gemini-pro). The results show that LLMs exhibit stronger human alignment capabilities with queries in English, French, Russian, and Spanish (only 1.04\% of harmful queries successfully jailbreak on average) compared to queries in Bengali, Georgian, Nepali and Maithili (27.7\% of harmful queries jailbreak successfully on average). Moreover, for queries in English, Danish, Czech and Slovenian, LLMs tend to produce responses with a higher quality (with 0.1494 $F_1$ score on average) compared to the other languages. Upon these findings, we propose LDFighter, a similarity-based voting, to mitigate the linguistic discrimination in LLMs. LDFighter ensures consistent service for different language speakers. We evaluate LDFighter with both benign queries and harmful queries. The results show that LDFighter not only significantly reduces the jailbreak success rate but also improve the response quality on average, demonstrating its effectiveness.

Title: Enhancing Boundary Segmentation for Topological Accuracy with Skeleton-based Methods

• Authors: Chuni Liu, Boyuan Ma, Xiaojuan Ban, Yujie Xie, Hao Wang, Weihua Xue, Jingchao Ma, Ke Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18539
• Pdf URL: https://arxiv.org/pdf/2404.18539
• Copy Paste: [[2404.18539]] Enhancing Boundary Segmentation for Topological Accuracy with Skeleton-based Methods(https://arxiv.org/abs/2404.18539)
• Keywords: segmentation
• Abstract: Topological consistency plays a crucial role in the task of boundary segmentation for reticular images, such as cell membrane segmentation in neuron electron microscopic images, grain boundary segmentation in material microscopic images and road segmentation in aerial images. In these fields, topological changes in segmentation results have a serious impact on the downstream tasks, which can even exceed the misalignment of the boundary itself. To enhance the topology accuracy in segmentation results, we propose the Skea-Topo Aware loss, which is a novel loss function that takes into account the shape of each object and topological significance of the pixels. It consists of two components. First, the skeleton-aware weighted loss improves the segmentation accuracy by better modeling the object geometry with skeletons. Second, a boundary rectified term effectively identifies and emphasizes topological critical pixels in the prediction errors using both foreground and background skeletons in the ground truth and predictions. Experiments prove that our method improves topological consistency by up to 7 points in VI compared to 13 state-of-art methods, based on objective and subjective assessments across three different boundary segmentation datasets. The code is available at https://github.com/clovermini/Skea_topo.

Title: Machine Learning for Windows Malware Detection and Classification: Methods, Challenges and Ongoing Research

• Authors: Daniel Gibert
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18541
• Pdf URL: https://arxiv.org/pdf/2404.18541
• Copy Paste: [[2404.18541]] Machine Learning for Windows Malware Detection and Classification: Methods, Challenges and Ongoing Research(https://arxiv.org/abs/2404.18541)
• Keywords: defense, attack
• Abstract: In this chapter, readers will explore how machine learning has been applied to build malware detection systems designed for the Windows operating system. This chapter starts by introducing the main components of a Machine Learning pipeline, highlighting the challenges of collecting and maintaining up-to-date datasets. Following this introduction, various state-of-the-art malware detectors are presented, encompassing both feature-based and deep learning-based detectors. Subsequent sections introduce the primary challenges encountered by machine learning-based malware detectors, including concept drift and adversarial attacks. Lastly, this chapter concludes by providing a brief overview of the ongoing research on adversarial defenses.

Title: Time Machine GPT

• Authors: Felix Drinkall, Eghbal Rahimikia, Janet B. Pierrehumbert, Stefan Zohren
• Subjects: cs.CL, cs.CE, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18543
• Pdf URL: https://arxiv.org/pdf/2404.18543
• Copy Paste: [[2404.18543]] Time Machine GPT(https://arxiv.org/abs/2404.18543)
• Keywords: large language model
• Abstract: Large language models (LLMs) are often trained on extensive, temporally indiscriminate text corpora, reflecting the lack of datasets with temporal metadata. This approach is not aligned with the evolving nature of language. Conventional methods for creating temporally adapted language models often depend on further pre-training static models on time-specific data. This paper presents a new approach: a series of point-in-time LLMs called Time Machine GPT (TiMaGPT), specifically designed to be nonprognosticative. This ensures they remain uninformed about future factual information and linguistic changes. This strategy is beneficial for understanding language evolution and is of critical importance when applying models in dynamic contexts, such as time-series forecasting, where foresight of future information can prove problematic. We provide access to both the models and training datasets.

Title: IncidentResponseGPT: Generating Traffic Incident Response Plans with Generative Artificial Intelligence

• Authors: Artur Grigorev, Khaled Saleh, Yuming Ou
• Subjects: cs.LG, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.18550
• Pdf URL: https://arxiv.org/pdf/2404.18550
• Copy Paste: [[2404.18550]] IncidentResponseGPT: Generating Traffic Incident Response Plans with Generative Artificial Intelligence(https://arxiv.org/abs/2404.18550)
• Keywords: generative
• Abstract: Traffic congestion due to road incidents poses a significant challenge in urban environments, leading to increased pollution, economic losses, and traffic congestion. Efficiently managing these incidents is imperative for mitigating their adverse effects; however, the complexity of urban traffic systems and the variety of potential incidents represent a considerable obstacle. This paper introduces IncidentResponseGPT, an innovative solution designed to assist traffic management authorities by providing rapid, informed, and adaptable traffic incident response plans. By integrating a Generative AI platform with real-time traffic incident reports and operational guidelines, our system aims to streamline the decision-making process in responding to traffic incidents. The research addresses the critical challenges involved in deploying AI in traffic management, including overcoming the complexity of urban traffic networks, ensuring real-time decision-making capabilities, aligning with local laws and regulations, and securing public acceptance for AI-driven systems. Through a combination of text analysis of accident reports, validation of AI recommendations through traffic simulation, and implementation of transparent and validated AI systems, IncidentResponseGPT offers a promising approach to optimizing traffic flow and reducing congestion in the face of traffic incidents. The relevance of this work extends to traffic management authorities, emergency response teams, and municipal bodies, all integral stakeholders in urban traffic control and incident management. By proposing a novel solution to the identified challenges, this research aims to develop a framework that not only facilitates faster resolution of traffic incidents but also minimizes their overall impact on urban traffic systems.

Title: SIDBench: A Python Framework for Reliably Assessing Synthetic Image Detection Methods

• Authors: Manos Schinas, Symeon Papadopoulos
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18552
• Pdf URL: https://arxiv.org/pdf/2404.18552
• Copy Paste: [[2404.18552]] SIDBench: A Python Framework for Reliably Assessing Synthetic Image Detection Methods(https://arxiv.org/abs/2404.18552)
• Keywords: generative
• Abstract: The generative AI technology offers an increasing variety of tools for generating entirely synthetic images that are increasingly indistinguishable from real ones. Unlike methods that alter portions of an image, the creation of completely synthetic images presents a unique challenge and several Synthetic Image Detection (SID) methods have recently appeared to tackle it. Yet, there is often a large gap between experimental results on benchmark datasets and the performance of methods in the wild. To better address the evaluation needs of SID and help close this gap, this paper introduces a benchmarking framework that integrates several state-of-the-art SID models. Our selection of integrated models was based on the utilization of varied input features, and different network architectures, aiming to encompass a broad spectrum of techniques. The framework leverages recent datasets with a diverse set of generative models, high level of photo-realism and resolution, reflecting the rapid improvements in image synthesis technology. Additionally, the framework enables the study of how image transformations, common in assets shared online, such as JPEG compression, affect detection performance. SIDBench is available on https://github.com/mever-team/sidbench and is designed in a modular manner to enable easy inclusion of new datasets and SID models.

Title: Can GPT-4 do L2 analytic assessment?

• Authors: Stefano Bannò, Hari Krishna Vydana, Kate M. Knill, Mark J. F. Gales
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18557
• Pdf URL: https://arxiv.org/pdf/2404.18557
• Copy Paste: [[2404.18557]] Can GPT-4 do L2 analytic assessment?(https://arxiv.org/abs/2404.18557)
• Keywords: large language model
• Abstract: Automated essay scoring (AES) to evaluate second language (L2) proficiency has been a firmly established technology used in educational contexts for decades. Although holistic scoring has seen advancements in AES that match or even exceed human performance, analytic scoring still encounters issues as it inherits flaws and shortcomings from the human scoring process. The recent introduction of large language models presents new opportunities for automating the evaluation of specific aspects of L2 writing proficiency. In this paper, we perform a series of experiments using GPT-4 in a zero-shot fashion on a publicly available dataset annotated with holistic scores based on the Common European Framework of Reference and aim to extract detailed information about their underlying analytic components. We observe significant correlations between the automatically predicted analytic scores and multiple features associated with the individual proficiency components.

Title: Injecting Salesperson's Dialogue Strategies in Large Language Models with Chain-of-Thought Reasoning

• Authors: Wen-Yu Chang, Yun-Nung Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18564
• Pdf URL: https://arxiv.org/pdf/2404.18564
• Copy Paste: [[2404.18564]] Injecting Salesperson's Dialogue Strategies in Large Language Models with Chain-of-Thought Reasoning(https://arxiv.org/abs/2404.18564)
• Keywords: large language model
• Abstract: Recent research in dialogue systems and corpora has focused on two main categories: task-oriented (TOD) and open-domain (chit-chat) dialogues. TOD systems help users accomplish specific tasks, while open-domain systems aim to create engaging conversations. However, in real-world scenarios, user intents are often revealed during interactions. A recent study introduced SalesBot, which simulates dialogues transitioning from chit-chat to task-oriented scenarios to train sales agents. Unfortunately, the initial data lacked smooth transitions and coherent long-turn dialogues, resulting in poor naturalness in sales-customer interactions. To address these issues, this paper presents SalesBot 2.0, an improved dataset. It leverages commonsense knowledge from large language models (LLMs) through strategic prompting. Additionally, we introduce a novel model called SalesAgent, trained on salesperson's interactions, using chain-of-thought (CoT) reasoning. This model excels in transitioning topics, understanding user intents, and selecting appropriate strategies. Experiments using diverse user simulations validate the effectiveness of our method in controlling dialogue strategies in LLMs. Furthermore, SalesBot 2.0 enhances coherence and reduces aggression, facilitating better model learning for sales-customer interactions.

Title: Assessing Cybersecurity Vulnerabilities in Code Large Language Models

• Authors: Md Imran Hossen, Jianyi Zhang, Yinzhi Cao, Xiali Hei
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.18567
• Pdf URL: https://arxiv.org/pdf/2404.18567
• Copy Paste: [[2404.18567]] Assessing Cybersecurity Vulnerabilities in Code Large Language Models(https://arxiv.org/abs/2404.18567)
• Keywords: security, defense, attack, robust, large language model
• Abstract: Instruction-tuned Code Large Language Models (Code LLMs) are increasingly utilized as AI coding assistants and integrated into various applications. However, the cybersecurity vulnerabilities and implications arising from the widespread integration of these models are not yet fully understood due to limited research in this domain. To bridge this gap, this paper presents EvilInstructCoder, a framework specifically designed to assess the cybersecurity vulnerabilities of instruction-tuned Code LLMs to adversarial attacks. EvilInstructCoder introduces the Adversarial Code Injection Engine to automatically generate malicious code snippets and inject them into benign code to poison instruction tuning datasets. It incorporates practical threat models to reflect real-world adversaries with varying capabilities and evaluates the exploitability of instruction-tuned Code LLMs under these diverse adversarial attack scenarios. Through the use of EvilInstructCoder, we conduct a comprehensive investigation into the exploitability of instruction tuning for coding tasks using three state-of-the-art Code LLM models: CodeLlama, DeepSeek-Coder, and StarCoder2, under various adversarial attack scenarios. Our experimental results reveal a significant vulnerability in these models, demonstrating that adversaries can manipulate the models to generate malicious payloads within benign code contexts in response to natural language instructions. For instance, under the backdoor attack setting, by poisoning only 81 samples (0.5\% of the entire instruction dataset), we achieve Attack Success Rate at 1 (ASR@1) scores ranging from 76\% to 86\% for different model families. Our study sheds light on the critical cybersecurity vulnerabilities posed by instruction-tuned Code LLMs and emphasizes the urgent necessity for robust defense mechanisms to mitigate the identified vulnerabilities.

Title: Learning Governing Equations of Unobserved States in Dynamical Systems

• Authors: Gevik Grigorian, Sandip V. George, Simon Arridge
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18572
• Pdf URL: https://arxiv.org/pdf/2404.18572
• Copy Paste: [[2404.18572]] Learning Governing Equations of Unobserved States in Dynamical Systems(https://arxiv.org/abs/2404.18572)
• Keywords: robust
• Abstract: Data driven modelling and scientific machine learning have been responsible for significant advances in determining suitable models to describe data. Within dynamical systems, neural ordinary differential equations (ODEs), where the system equations are set to be governed by a neural network, have become a popular tool for this challenge in recent years. However, less emphasis has been placed on systems that are only partially-observed. In this work, we employ a hybrid neural ODE structure, where the system equations are governed by a combination of a neural network and domain-specific knowledge, together with symbolic regression (SR), to learn governing equations of partially-observed dynamical systems. We test this approach on two case studies: A 3-dimensional model of the Lotka-Volterra system and a 5-dimensional model of the Lorenz system. We demonstrate that the method is capable of successfully learning the true underlying governing equations of unobserved states within these systems, with robustness to measurement noise.

Title: FREB-TQA: A Fine-Grained Robustness Evaluation Benchmark for Table Question Answering

• Authors: Wei Zhou, Mohsen Mesgar, Heike Adel, Annemarie Friedrich
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18585
• Pdf URL: https://arxiv.org/pdf/2404.18585
• Copy Paste: [[2404.18585]] FREB-TQA: A Fine-Grained Robustness Evaluation Benchmark for Table Question Answering(https://arxiv.org/abs/2404.18585)
• Keywords: robust
• Abstract: Table Question Answering (TQA) aims at composing an answer to a question based on tabular data. While prior research has shown that TQA models lack robustness, understanding the underlying cause and nature of this issue remains predominantly unclear, posing a significant obstacle to the development of robust TQA systems. In this paper, we formalize three major desiderata for a fine-grained evaluation of robustness of TQA systems. They should (i) answer questions regardless of alterations in table structure, (ii) base their responses on the content of relevant cells rather than on biases, and (iii) demonstrate robust numerical reasoning capabilities. To investigate these aspects, we create and publish a novel TQA evaluation benchmark in English. Our extensive experimental analysis reveals that none of the examined state-of-the-art TQA systems consistently excels in these three aspects. Our benchmark is a crucial instrument for monitoring the behavior of TQA systems and paves the way for the development of robust TQA systems. We release our benchmark publicly.

Title: FashionSD-X: Multimodal Fashion Garment Synthesis using Latent Diffusion

• Authors: Abhishek Kumar Singh, Ioannis Patras
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18591
• Pdf URL: https://arxiv.org/pdf/2404.18591
• Copy Paste: [[2404.18591]] FashionSD-X: Multimodal Fashion Garment Synthesis using Latent Diffusion(https://arxiv.org/abs/2404.18591)
• Keywords: diffusion, generative
• Abstract: The rapid evolution of the fashion industry increasingly intersects with technological advancements, particularly through the integration of generative AI. This study introduces a novel generative pipeline designed to transform the fashion design process by employing latent diffusion models. Utilizing ControlNet and LoRA fine-tuning, our approach generates high-quality images from multimodal inputs such as text and sketches. We leverage and enhance state-of-the-art virtual try-on datasets, including Multimodal Dress Code and VITON-HD, by integrating sketch data. Our evaluation, utilizing metrics like FID, CLIP Score, and KID, demonstrates that our model significantly outperforms traditional stable diffusion models. The results not only highlight the effectiveness of our model in generating fashion-appropriate outputs but also underscore the potential of diffusion models in revolutionizing fashion design workflows. This research paves the way for more interactive, personalized, and technologically enriched methodologies in fashion design and representation, bridging the gap between creative vision and practical application.

Title: Anywhere: A Multi-Agent Framework for Reliable and Diverse Foreground-Conditioned Image Inpainting

• Authors: Tianyidan Xie, Rui Ma, Qian Wang, Xiaoqian Ye, Feixuan Liu, Ying Tai, Zhenyu Zhang, Zili Yi
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.18598
• Pdf URL: https://arxiv.org/pdf/2404.18598
• Copy Paste: [[2404.18598]] Anywhere: A Multi-Agent Framework for Reliable and Diverse Foreground-Conditioned Image Inpainting(https://arxiv.org/abs/2404.18598)
• Keywords: diffusion, large language model
• Abstract: Recent advancements in image inpainting, particularly through diffusion modeling, have yielded promising outcomes. However, when tested in scenarios involving the completion of images based on the foreground objects, current methods that aim to inpaint an image in an end-to-end manner encounter challenges such as "over-imagination", inconsistency between foreground and background, and limited diversity. In response, we introduce Anywhere, a pioneering multi-agent framework designed to address these issues. Anywhere utilizes a sophisticated pipeline framework comprising various agents such as Visual Language Model (VLM), Large Language Model (LLM), and image generation models. This framework consists of three principal components: the prompt generation module, the image generation module, and the outcome analyzer. The prompt generation module conducts a semantic analysis of the input foreground image, leveraging VLM to predict relevant language descriptions and LLM to recommend optimal language prompts. In the image generation module, we employ a text-guided canny-to-image generation model to create a template image based on the edge map of the foreground image and language prompts, and an image refiner to produce the outcome by blending the input foreground and the template image. The outcome analyzer employs VLM to evaluate image content rationality, aesthetic score, and foreground-background relevance, triggering prompt and image regeneration as needed. Extensive experiments demonstrate that our Anywhere framework excels in foreground-conditioned image inpainting, mitigating "over-imagination", resolving foreground-background discrepancies, and enhancing diversity. It successfully elevates foreground-conditioned image inpainting to produce more reliable and diverse results.

Title: CSTalk: Correlation Supervised Speech-driven 3D Emotional Facial Animation Generation

• Authors: Xiangyu Liang, Wenlin Zhuang, Tianyong Wang, Guangxing Geng, Guangyue Geng, Haifeng Xia, Siyu Xia
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18604
• Pdf URL: https://arxiv.org/pdf/2404.18604
• Copy Paste: [[2404.18604]] CSTalk: Correlation Supervised Speech-driven 3D Emotional Facial Animation Generation(https://arxiv.org/abs/2404.18604)
• Keywords: generative
• Abstract: Speech-driven 3D facial animation technology has been developed for years, but its practical application still lacks expectations. The main challenges lie in data limitations, lip alignment, and the naturalness of facial expressions. Although lip alignment has seen many related studies, existing methods struggle to synthesize natural and realistic expressions, resulting in a mechanical and stiff appearance of facial animations. Even with some research extracting emotional features from speech, the randomness of facial movements limits the effective expression of emotions. To address this issue, this paper proposes a method called CSTalk (Correlation Supervised) that models the correlations among different regions of facial movements and supervises the training of the generative model to generate realistic expressions that conform to human facial motion patterns. To generate more intricate animations, we employ a rich set of control parameters based on the metahuman character model and capture a dataset for five different emotions. We train a generative network using an autoencoder structure and input an emotion embedding vector to achieve the generation of user-control expressions. Experimental results demonstrate that our method outperforms existing state-of-the-art methods.

Title: FlexiFilm: Long Video Generation with Flexible Conditions

• Authors: Yichen Ouyang, jianhao Yuan, Hao Zhao, Gaoang Wang, Bo zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18620
• Pdf URL: https://arxiv.org/pdf/2404.18620
• Copy Paste: [[2404.18620]] FlexiFilm: Long Video Generation with Flexible Conditions(https://arxiv.org/abs/2404.18620)
• Keywords: diffusion
• Abstract: Generating long and consistent videos has emerged as a significant yet challenging problem. While most existing diffusion-based video generation models, derived from image generation models, demonstrate promising performance in generating short videos, their simple conditioning mechanism and sampling strategy-originally designed for image generation-cause severe performance degradation when adapted to long video generation. This results in prominent temporal inconsistency and overexposure. Thus, in this work, we introduce FlexiFilm, a new diffusion model tailored for long video generation. Our framework incorporates a temporal conditioner to establish a more consistent relationship between generation and multi-modal conditions, and a resampling strategy to tackle overexposure. Empirical results demonstrate FlexiFilm generates long and consistent videos, each over 30 seconds in length, outperforming competitors in qualitative and quantitative analyses. Project page: https://y-ichen.github.io/FlexiFilm-Page/

Title: Uncertainty-boosted Robust Video Activity Anticipation

• Authors: Zhaobo Qi, Shuhui Wang, Weigang Zhang, Qingming Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18648
• Pdf URL: https://arxiv.org/pdf/2404.18648
• Copy Paste: [[2404.18648]] Uncertainty-boosted Robust Video Activity Anticipation(https://arxiv.org/abs/2404.18648)
• Keywords: robust, interpretability
• Abstract: Video activity anticipation aims to predict what will happen in the future, embracing a broad application prospect ranging from robot vision and autonomous driving. Despite the recent progress, the data uncertainty issue, reflected as the content evolution process and dynamic correlation in event labels, has been somehow ignored. This reduces the model generalization ability and deep understanding on video content, leading to serious error accumulation and degraded performance. In this paper, we address the uncertainty learning problem and propose an uncertainty-boosted robust video activity anticipation framework, which generates uncertainty values to indicate the credibility of the anticipation results. The uncertainty value is used to derive a temperature parameter in the softmax function to modulate the predicted target activity distribution. To guarantee the distribution adjustment, we construct a reasonable target activity label representation by incorporating the activity evolution from the temporal class correlation and the semantic relationship. Moreover, we quantify the uncertainty into relative values by comparing the uncertainty among sample pairs and their temporal-lengths. This relative strategy provides a more accessible way in uncertainty modeling than quantifying the absolute uncertainty values on the whole dataset. Experiments on multiple backbones and benchmarks show our framework achieves promising performance and better robustness/interpretability. Source codes are available at https://github.com/qzhb/UbRV2A.

Title: Towards Quantitative Evaluation of Explainable AI Methods for Deepfake Detection

• Authors: Konstantinos Tsigos, Evlampios Apostolidis, Spyridon Baxevanakis, Symeon Papadopoulos, Vasileios Mezaris
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18649
• Pdf URL: https://arxiv.org/pdf/2404.18649
• Copy Paste: [[2404.18649]] Towards Quantitative Evaluation of Explainable AI Methods for Deepfake Detection(https://arxiv.org/abs/2404.18649)
• Keywords: attack
• Abstract: In this paper we propose a new framework for evaluating the performance of explanation methods on the decisions of a deepfake detector. This framework assesses the ability of an explanation method to spot the regions of a fake image with the biggest influence on the decision of the deepfake detector, by examining the extent to which these regions can be modified through a set of adversarial attacks, in order to flip the detector's prediction or reduce its initial prediction; we anticipate a larger drop in deepfake detection accuracy and prediction, for methods that spot these regions more accurately. Based on this framework, we conduct a comparative study using a state-of-the-art model for deepfake detection that has been trained on the FaceForensics++ dataset, and five explanation methods from the literature. The findings of our quantitative and qualitative evaluations document the advanced performance of the LIME explanation method against the other compared ones, and indicate this method as the most appropriate for explaining the decisions of the utilized deepfake detector.

Title: Terrain characterisation for online adaptability of automated sonar processing: Lessons learnt from operationally applying ATR to sidescan sonar in MCM applications

• Authors: Thomas Guerneve, Stephanos Loizou, Andrea Munafo, Pierre-Yves Mignotte
• Subjects: cs.CV, cs.LG, cs.RO, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.18663
• Pdf URL: https://arxiv.org/pdf/2404.18663
• Copy Paste: [[2404.18663]] Terrain characterisation for online adaptability of automated sonar processing: Lessons learnt from operationally applying ATR to sidescan sonar in MCM applications(https://arxiv.org/abs/2404.18663)
• Keywords: explainability
• Abstract: The performance of Automated Recognition (ATR) algorithms on side-scan sonar imagery has shown to degrade rapidly when deployed on non benign environments. Complex seafloors and acoustic artefacts constitute distractors in the form of strong textural patterns, creating false detections or preventing detections of true objects. This paper presents two online seafloor characterisation techniques to improve explainability during Autonomous Underwater Vehicles (AUVs) missions. Importantly and as opposed to previous work in the domain, these techniques are not based on a model and require limited input from human operators, making it suitable for real-time onboard processing. Both techniques rely on an unsupervised machine learning approach to extract terrain features which relate to the human understanding of terrain complexity. The first technnique provides a quantitative, application-driven terrain characterisation metric based on the performance of an ATR algorithm. The second method provides a way to incorporate subject matter expertise and enables contextualisation and explainability in support for scenario-dependent subjective terrain characterisation. The terrain complexity matches the expectation of seasoned users making this tool desirable and trustworthy in comparison to traditional unsupervised approaches. We finally detail an application of these techniques to repair a Mine Countermeasures (MCM) mission carried with SeeByte autonomy framework Neptune.

Title: Reading Order Independent Metrics for Information Extraction in Handwritten Documents

• Authors: David Villanova-Aparisi, Solène Tarride, Carlos-D. Martínez-Hinarejos, Verónica Romero, Christopher Kermorvant, Moisés Pastor-Gadea
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18664
• Pdf URL: https://arxiv.org/pdf/2404.18664
• Copy Paste: [[2404.18664]] Reading Order Independent Metrics for Information Extraction in Handwritten Documents(https://arxiv.org/abs/2404.18664)
• Keywords: extraction
• Abstract: Information Extraction processes in handwritten documents tend to rely on obtaining an automatic transcription and performing Named Entity Recognition (NER) over such transcription. For this reason, in publicly available datasets, the performance of the systems is usually evaluated with metrics particular to each dataset. Moreover, most of the metrics employed are sensitive to reading order errors. Therefore, they do not reflect the expected final application of the system and introduce biases in more complex documents. In this paper, we propose and publicly release a set of reading order independent metrics tailored to Information Extraction evaluation in handwritten documents. In our experimentation, we perform an in-depth analysis of the behavior of the metrics to recommend what we consider to be the minimal set of metrics to evaluate a task correctly.

Title: Leveraging PointNet and PointNet++ for Lyft Point Cloud Classification Challenge

• Authors: Rajat K. Doshi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18665
• Pdf URL: https://arxiv.org/pdf/2404.18665
• Copy Paste: [[2404.18665]] Leveraging PointNet and PointNet++ for Lyft Point Cloud Classification Challenge(https://arxiv.org/abs/2404.18665)
• Keywords: robust
• Abstract: This study investigates the application of PointNet and PointNet++ in the classification of LiDAR-generated point cloud data, a critical component for achieving fully autonomous vehicles. Utilizing a modified dataset from the Lyft 3D Object Detection Challenge, we examine the models' capabilities to handle dynamic and complex environments essential for autonomous navigation. Our analysis shows that PointNet and PointNet++ achieved accuracy rates of 79.53% and 84.24%, respectively. These results underscore the models' robustness in interpreting intricate environmental data, which is pivotal for the safety and efficiency of autonomous vehicles. Moreover, the enhanced detection accuracy, particularly in distinguishing pedestrians from other objects, highlights the potential of these models to contribute substantially to the advancement of autonomous vehicle technology.

Title: FALE: Fairness-Aware ALE Plots for Auditing Bias in Subgroups

• Authors: Giorgos Giannopoulos, Dimitris Sacharidis, Nikolas Theologitis, Loukas Kavouras, Ioannis Emiris
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.18685
• Pdf URL: https://arxiv.org/pdf/2404.18685
• Copy Paste: [[2404.18685]] FALE: Fairness-Aware ALE Plots for Auditing Bias in Subgroups(https://arxiv.org/abs/2404.18685)
• Keywords: fair, explainability
• Abstract: Fairness is steadily becoming a crucial requirement of Machine Learning (ML) systems. A particularly important notion is subgroup fairness, i.e., fairness in subgroups of individuals that are defined by more than one attributes. Identifying bias in subgroups can become both computationally challenging, as well as problematic with respect to comprehensibility and intuitiveness of the finding to end users. In this work we focus on the latter aspects; we propose an explainability method tailored to identifying potential bias in subgroups and visualizing the findings in a user friendly manner to end users. In particular, we extend the ALE plots explainability method, proposing FALE (Fairness aware Accumulated Local Effects) plots, a method for measuring the change in fairness for an affected population corresponding to different values of a feature (attribute). We envision FALE to function as an efficient, user friendly, comprehensible and reliable first-stage tool for identifying subgroups with potential bias issues.

Title: Beyond Gaze Points: Augmenting Eye Movement with Brainwave Data for Multimodal User Authentication in Extended Reality

• Authors: Matin Fallahi, Patricia Arias-Cabarcos, Thorsten Strufe
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.18694
• Pdf URL: https://arxiv.org/pdf/2404.18694
• Copy Paste: [[2404.18694]] Beyond Gaze Points: Augmenting Eye Movement with Brainwave Data for Multimodal User Authentication in Extended Reality(https://arxiv.org/abs/2404.18694)
• Keywords: secure, security, biometric
• Abstract: The increasing adoption of Extended Reality (XR) in various applications underscores the need for secure and user-friendly authentication methods. However, existing methods can disrupt the immersive experience in XR settings, or suffer from higher false acceptance rates. In this paper, we introduce a multimodal biometric authentication system that combines eye movement and brainwave patterns, as captured by consumer-grade low-fidelity sensors. Our multimodal authentication exploits the non-invasive and hands-free properties of eye movement and brainwaves to provide a seamless XR user experience and enhanced security as well. Using synchronized eye and brainwave data collected from 30 participants through consumer-grade devices, we investigated whether twin neural networks can utilize these biometrics for identity verification. Our multimodal authentication system yields an excellent Equal Error Rate (EER) of 0.298\%, which means an 83.6\% reduction in EER compared to the single eye movement modality or a 93.9\% reduction in EER compared to the single brainwave modality.

Title: Convergence Properties of Score-Based Models using Graduated Optimisation for Linear Inverse Problems

• Authors: Pascal Fernsel, Željko Kereta, Alexander Denker
• Subjects: cs.LG, cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.18699
• Pdf URL: https://arxiv.org/pdf/2404.18699
• Copy Paste: [[2404.18699]] Convergence Properties of Score-Based Models using Graduated Optimisation for Linear Inverse Problems(https://arxiv.org/abs/2404.18699)
• Keywords: generative
• Abstract: The incorporation of generative models as regularisers within variational formulations for inverse problems has proven effective across numerous image reconstruction tasks. However, the resulting optimisation problem is often non-convex and challenging to solve. In this work, we show that score-based generative models (SGMs) can be used in a graduated optimisation framework to solve inverse problems. We show that the resulting graduated non-convexity flow converge to stationary points of the original problem and provide a numerical convergence analysis of a 2D toy example. We further provide experiments on computed tomography image reconstruction, where we show that this framework is able to recover high-quality images, independent of the initial value. The experiments highlight the potential of using SGMs in graduated optimisation frameworks.

Title: Why You Should Not Trust Interpretations in Machine Learning: Adversarial Attacks on Partial Dependence Plots

• Authors: Xi Xin, Fei Huang, Giles Hooker
• Subjects: cs.LG, cs.CR, stat.AP, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.18702
• Pdf URL: https://arxiv.org/pdf/2404.18702
• Copy Paste: [[2404.18702]] Why You Should Not Trust Interpretations in Machine Learning: Adversarial Attacks on Partial Dependence Plots(https://arxiv.org/abs/2404.18702)
• Keywords: attack
• Abstract: The adoption of artificial intelligence (AI) across industries has led to the widespread use of complex black-box models and interpretation tools for decision making. This paper proposes an adversarial framework to uncover the vulnerability of permutation-based interpretation methods for machine learning tasks, with a particular focus on partial dependence (PD) plots. This adversarial framework modifies the original black box model to manipulate its predictions for instances in the extrapolation domain. As a result, it produces deceptive PD plots that can conceal discriminatory behaviors while preserving most of the original model's predictions. This framework can produce multiple fooled PD plots via a single model. By using real-world datasets including an auto insurance claims dataset and COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) dataset, our results show that it is possible to intentionally hide the discriminatory behavior of a predictor and make the black-box model appear neutral through interpretation tools like PD plots while retaining almost all the predictions of the original black-box model. Managerial insights for regulators and practitioners are provided based on the findings.

Title: CVTN: Cross Variable and Temporal Integration for Time Series Forecasting

• Authors: Han Zhou, Yuntian Chen
• Subjects: cs.LG, cs.AI, stat.AP
• Abstract URL: https://arxiv.org/abs/2404.18730
• Pdf URL: https://arxiv.org/pdf/2404.18730
• Copy Paste: [[2404.18730]] CVTN: Cross Variable and Temporal Integration for Time Series Forecasting(https://arxiv.org/abs/2404.18730)
• Keywords: transformer
• Abstract: In multivariate time series forecasting, the Transformer architecture encounters two significant challenges: effectively mining features from historical sequences and avoiding overfitting during the learning of temporal dependencies. To tackle these challenges, this paper deconstructs time series forecasting into the learning of historical sequences and prediction sequences, introducing the Cross-Variable and Time Network (CVTN). This unique method divides multivariate time series forecasting into two phases: cross-variable learning for effectively mining fea tures from historical sequences, and cross-time learning to capture the temporal dependencies of prediction sequences. Separating these two phases helps avoid the impact of overfitting in cross-time learning on cross-variable learning. Exten sive experiments on various real-world datasets have confirmed its state-of-the-art (SOTA) performance. CVTN emphasizes three key dimensions in time series fore casting: the short-term and long-term nature of time series (locality and longevity), feature mining from both historical and prediction sequences, and the integration of cross-variable and cross-time learning. This approach not only advances the current state of time series forecasting but also provides a more comprehensive framework for future research in this field.

Title: Real Time Multi Organ Classification on Computed Tomography Images

• Authors: Halid Ziya Yerebakan, Yoshihisa Shinagawa, Gerardo Hermosillo Valadez
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18731
• Pdf URL: https://arxiv.org/pdf/2404.18731
• Copy Paste: [[2404.18731]] Real Time Multi Organ Classification on Computed Tomography Images(https://arxiv.org/abs/2404.18731)
• Keywords: segmentation
• Abstract: Organ segmentation is a fundamental task in medical imaging, and it is useful for many clinical automation pipelines. Typically, the process involves segmenting the entire volume, which can be unnecessary when the points of interest are limited. In those cases, a classifier could be used instead of segmentation. However, there is an inherent trade-off between the context size and the speed of classifiers. To address this issue, we propose a new method that employs a data selection strategy with sparse sampling across a wide field of view without image resampling. This sparse sampling strategy makes it possible to classify voxels into multiple organs in real time without using accelerators. Although our method is an independent classifier, it can generate full segmentation by querying grid locations at any resolution. We have compared our method with existing segmentation techniques, demonstrating its potential for superior runtime in practical applications in medical imaging.

Title: Mapping the Potential of Explainable Artificial Intelligence (XAI) for Fairness Along the AI Lifecycle

• Authors: Luca Deck, Astrid Schoemäcker, Timo Speith, Jakob Schöffer, Lena Kästner, Niklas Kühl
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.18736
• Pdf URL: https://arxiv.org/pdf/2404.18736
• Copy Paste: [[2404.18736]] Mapping the Potential of Explainable Artificial Intelligence (XAI) for Fairness Along the AI Lifecycle(https://arxiv.org/abs/2404.18736)
• Keywords: fair
• Abstract: The widespread use of artificial intelligence (AI) systems across various domains is increasingly highlighting issues related to algorithmic fairness, especially in high-stakes scenarios. Thus, critical considerations of how fairness in AI systems might be improved, and what measures are available to aid this process, are overdue. Many researchers and policymakers see explainable AI (XAI) as a promising way to increase fairness in AI systems. However, there is a wide variety of XAI methods and fairness conceptions expressing different desiderata, and the precise connections between XAI and fairness remain largely nebulous. Besides, different measures to increase algorithmic fairness might be applicable at different points throughout an AI system's lifecycle. Yet, there currently is no coherent mapping of fairness desiderata along the AI lifecycle. In this paper, we set out to bridge both these gaps: We distill eight fairness desiderata, map them along the AI lifecycle, and discuss how XAI could help address each of them. We hope to provide orientation for practical applications and to inspire XAI research specifically focused on these fairness desiderata.

Title: Evaluating the Effectiveness of Video Anomaly Detection in the Wild: Online Learning and Inference for Real-world Deployment

• Authors: Shanle Yao, Ghazal Alinezhad Noghre, Armin Danesh Pazho, Hamed Tabkhi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18747
• Pdf URL: https://arxiv.org/pdf/2404.18747
• Copy Paste: [[2404.18747]] Evaluating the Effectiveness of Video Anomaly Detection in the Wild: Online Learning and Inference for Real-world Deployment(https://arxiv.org/abs/2404.18747)
• Keywords: privacy
• Abstract: Video Anomaly Detection (VAD) identifies unusual activities in video streams, a key technology with broad applications ranging from surveillance to healthcare. Tackling VAD in real-life settings poses significant challenges due to the dynamic nature of human actions, environmental variations, and domain shifts. Many research initiatives neglect these complexities, often concentrating on traditional testing methods that fail to account for performance on unseen datasets, creating a gap between theoretical models and their real-world utility. Online learning is a potential strategy to mitigate this issue by allowing models to adapt to new information continuously. This paper assesses how well current VAD algorithms can adjust to real-life conditions through an online learning framework, particularly those based on pose analysis, for their efficiency and privacy advantages. Our proposed framework enables continuous model updates with streaming data from novel environments, thus mirroring actual world challenges and evaluating the models' ability to adapt in real-time while maintaining accuracy. We investigate three state-of-the-art models in this setting, focusing on their adaptability across different domains. Our findings indicate that, even under the most challenging conditions, our online learning approach allows a model to preserve 89.39% of its original effectiveness compared to its offline-trained counterpart in a specific target domain.

Title: Flow AM: Generating Point Cloud Global Explanations by Latent Alignment

• Authors: Hanxiao Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18760
• Pdf URL: https://arxiv.org/pdf/2404.18760
• Copy Paste: [[2404.18760]] Flow AM: Generating Point Cloud Global Explanations by Latent Alignment(https://arxiv.org/abs/2404.18760)
• Keywords: explainability, generative
• Abstract: Although point cloud models have gained significant improvements in prediction accuracy over recent years, their trustworthiness is still not sufficiently investigated. In terms of global explainability, Activation Maximization (AM) techniques in the image domain are not directly transplantable due to the special structure of the point cloud models. Existing studies exploit generative models to yield global explanations that can be perceived by humans. However, the opacity of the generative models themselves and the introduction of additional priors call into question the plausibility and fidelity of the explanations. In this work, we demonstrate that when the classifier predicts different types of instances, the intermediate layer activations are differently activated, known as activation flows. Based on this property, we propose an activation flow-based AM method that generates global explanations that can be perceived without incorporating any generative model. Furthermore, we reveal that AM based on generative models fails the sanity checks and thus lack of fidelity. Extensive experiments show that our approach dramatically enhances the perceptibility of explanations compared to other AM methods that are not based on generative models. Our code is available at: https://github.com/Explain3D/FlowAM

Title: From Density to Geometry: YOLOv8 Instance Segmentation for Reverse Engineering of Optimized Structures

• Authors: Thomas Rochefort-Beaudoin, Aurelian Vadean, Sofiane Achiche, Niels Aage
• Subjects: cs.CV, cs.CE
• Abstract URL: https://arxiv.org/abs/2404.18763
• Pdf URL: https://arxiv.org/pdf/2404.18763
• Copy Paste: [[2404.18763]] From Density to Geometry: YOLOv8 Instance Segmentation for Reverse Engineering of Optimized Structures(https://arxiv.org/abs/2404.18763)
• Keywords: segmentation
• Abstract: This paper introduces YOLOv8-TO, a novel approach for reverse engineering of topology-optimized structures into interpretable geometric parameters using the YOLOv8 instance segmentation model. Density-based topology optimization methods require post-processing to convert the optimal density distribution into a parametric representation for design exploration and integration with CAD tools. Traditional methods such as skeletonization struggle with complex geometries and require manual intervention. YOLOv8-TO addresses these challenges by training a custom YOLOv8 model to automatically detect and reconstruct structural components from binary density distributions. The model is trained on a diverse dataset of both optimized and random structures generated using the Moving Morphable Components method. A custom reconstruction loss function based on the dice coefficient of the predicted geometry is used to train the new regression head of the model via self-supervised learning. The method is evaluated on test sets generated from different topology optimization methods, including out-of-distribution samples, and compared against a skeletonization approach. Results show that YOLOv8-TO significantly outperforms skeletonization in reconstructing visually and structurally similar designs. The method showcases an average improvement of 13.84% in the Dice coefficient, with peak enhancements reaching 20.78%. The method demonstrates good generalization to complex geometries and fast inference times, making it suitable for integration into design workflows using regular workstations. Limitations include the sensitivity to non-max suppression thresholds. YOLOv8-TO represents a significant advancement in topology optimization post-processing, enabling efficient and accurate reverse engineering of optimized structures for design exploration and manufacturing.

Title: Saliency Suppressed, Semantics Surfaced: Visual Transformations in Neural Networks and the Brain

• Authors: Gustaw Opiełka, Jessica Loke, Steven Scholte
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18772
• Pdf URL: https://arxiv.org/pdf/2404.18772
• Copy Paste: [[2404.18772]] Saliency Suppressed, Semantics Surfaced: Visual Transformations in Neural Networks and the Brain(https://arxiv.org/abs/2404.18772)
• Keywords: robust
• Abstract: Deep learning algorithms lack human-interpretable accounts of how they transform raw visual input into a robust semantic understanding, which impedes comparisons between different architectures, training objectives, and the human brain. In this work, we take inspiration from neuroscience and employ representational approaches to shed light on how neural networks encode information at low (visual saliency) and high (semantic similarity) levels of abstraction. Moreover, we introduce a custom image dataset where we systematically manipulate salient and semantic information. We find that ResNets are more sensitive to saliency information than ViTs, when trained with object classification objectives. We uncover that networks suppress saliency in early layers, a process enhanced by natural language supervision (CLIP) in ResNets. CLIP also enhances semantic encoding in both architectures. Finally, we show that semantic encoding is a key factor in aligning AI with human visual perception, while saliency suppression is a non-brain-like strategy.

Title: A Universal Metric of Dataset Similarity for Cross-silo Federated Learning

• Authors: Ahmed Elhussein, Gamze Gursoy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18773
• Pdf URL: https://arxiv.org/pdf/2404.18773
• Copy Paste: [[2404.18773]] A Universal Metric of Dataset Similarity for Cross-silo Federated Learning(https://arxiv.org/abs/2404.18773)
• Keywords: privacy, robust, federate
• Abstract: Federated Learning is increasingly used in domains such as healthcare to facilitate collaborative model training without data-sharing. However, datasets located in different sites are often non-identically distributed, leading to degradation of model performance in FL. Most existing methods for assessing these distribution shifts are limited by being dataset or task-specific. Moreover, these metrics can only be calculated by exchanging data, a practice restricted in many FL scenarios. To address these challenges, we propose a novel metric for assessing dataset similarity. Our metric exhibits several desirable properties for FL: it is dataset-agnostic, is calculated in a privacy-preserving manner, and is computationally efficient, requiring no model training. In this paper, we first establish a theoretical connection between our metric and training dynamics in FL. Next, we extensively evaluate our metric on a range of datasets including synthetic, benchmark, and medical imaging datasets. We demonstrate that our metric shows a robust and interpretable relationship with model performance and can be calculated in privacy-preserving manner. As the first federated dataset similarity metric, we believe this metric can better facilitate successful collaborations between sites.

Title: Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models

• Authors: Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, Patrick Lewis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18796
• Pdf URL: https://arxiv.org/pdf/2404.18796
• Copy Paste: [[2404.18796]] Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models(https://arxiv.org/abs/2404.18796)
• Keywords: large language model
• Abstract: As Large Language Models (LLMs) have become more advanced, they have outpaced our abilities to accurately evaluate their quality. Not only is finding data to adequately probe particular model properties difficult, but evaluating the correctness of a model's freeform generation alone is a challenge. To address this, many evaluations now rely on using LLMs themselves as judges to score the quality of outputs from other LLMs. Evaluations most commonly use a single large model like GPT4. While this method has grown in popularity, it is costly, has been shown to introduce intramodel bias, and in this work, we find that very large models are often unnecessary. We propose instead to evaluate models using a Panel of LLm evaluators (PoLL). Across three distinct judge settings and spanning six different datasets, we find that using a PoLL composed of a larger number of smaller models outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families, and does so while being over seven times less expensive.

Title: A Partial Replication of MaskFormer in TensorFlow on TPUs for the TensorFlow Model Garden

• Authors: Vishal Purohit, Wenxin Jiang, Akshath R. Ravikiran, James C. Davis
• Subjects: cs.CV, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.18801
• Pdf URL: https://arxiv.org/pdf/2404.18801
• Copy Paste: [[2404.18801]] A Partial Replication of MaskFormer in TensorFlow on TPUs for the TensorFlow Model Garden(https://arxiv.org/abs/2404.18801)
• Keywords: segmentation
• Abstract: This paper undertakes the task of replicating the MaskFormer model a universal image segmentation model originally developed using the PyTorch framework, within the TensorFlow ecosystem, specifically optimized for execution on Tensor Processing Units (TPUs). Our implementation exploits the modular constructs available within the TensorFlow Model Garden (TFMG), encompassing elements such as the data loader, training orchestrator, and various architectural components, tailored and adapted to meet the specifications of the MaskFormer model. We address key challenges encountered during the replication, non-convergence issues, slow training, adaptation of loss functions, and the integration of TPU-specific functionalities. We verify our reproduced implementation and present qualitative results on the COCO dataset. Although our implementation meets some of the objectives for end-to-end reproducibility, we encountered challenges in replicating the PyTorch version of MaskFormer in TensorFlow. This replication process is not straightforward and requires substantial engineering efforts. Specifically, it necessitates the customization of various components within the TFMG, alongside thorough verification and hyper-parameter tuning. The replication is available at: https://github.com/PurdueDualityLab/tf-maskformer/tree/main/official/projects/maskformer

Title: Belt and Brace: When Federated Learning Meets Differential Privacy

• Authors: Xuebin Ren, Shusen Yang, Cong Zhao, Julie McCann, Zongben Xu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.18814
• Pdf URL: https://arxiv.org/pdf/2404.18814
• Copy Paste: [[2404.18814]] Belt and Brace: When Federated Learning Meets Differential Privacy(https://arxiv.org/abs/2404.18814)
• Keywords: privacy, protect, federate
• Abstract: Federated learning (FL) has great potential for large-scale machine learning (ML) without exposing raw data.Differential privacy (DP) is the de facto standard of privacy protection with provable guarantees.Advances in ML suggest that DP would be a perfect fit for FL with comprehensive privacy preservation. Hence, extensive efforts have been devoted to achieving practically usable FL with DP, which however is still challenging.Practitioners often not only are not fully aware of its development and categorization, but also face a hard choice between privacy and utility. Therefore, it calls for a holistic review of current advances and an investigation on the challenges and opportunities for highly usable FL systems with a DP guarantee. In this article, we first introduce the primary concepts of FL and DP, and highlight the benefits of integration. We then review the current developments by categorizing different paradigms and notions. Aiming at usable FL with DP, we present the optimization principles to seek a better tradeoff between model utility and privacy loss. Finally, we discuss future challenges in the emergent areas and relevant research topics.

Title: AppPoet: Large Language Model based Android malware detection via multi-view prompt engineering

• Authors: Wenxiang Zhao, Juntao Wu, Zhaoyi Meng
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.18816
• Pdf URL: https://arxiv.org/pdf/2404.18816
• Copy Paste: [[2404.18816]] AppPoet: Large Language Model based Android malware detection via multi-view prompt engineering(https://arxiv.org/abs/2404.18816)
• Keywords: attack, large language model
• Abstract: Due to the vast array of Android applications, their multifarious functions and intricate behavioral semantics, attackers can adopt various tactics to conceal their genuine attack intentions within legitimate functions. However, numerous feature engineering based methods suffer from a limitation in mining behavioral semantic information, thus impeding the accuracy and efficiency of Android malware detection. Besides, the majority of existing feature engineering based methods are weakly interpretive and fail to furnish researchers with effective and readable detection reports. Inspired by the success of the Large Language Models (LLMs) in natural language understanding, we propose AppPoet, a LLM-assisted multi-view system for Android malware detection. Firstly, AppPoet employs a static method to comprehensively collect application features and formulate various observation views. Subsequently, it steers the LLM to produce function descriptions and behavioral summaries for views via our meticulously devised multi-view prompt engineering technique to realize the deep mining of view semantics. Finally, we collaboratively fuse the multi-view information to efficiently and accurately detect malware through a deep neural network (DNN) classifier and then generate the heuristic diagnostic reports. Experimental results demonstrate that our method achieves a detection accuracy of 97.15% and an F1 score of 97.21%, which is superior to the baseline method Drebin and its variant. Furthermore, the case study evaluates the effectiveness of our generated diagnostic reports.

Title: Hiding from Facebook: An Encryption Protocol resistant to Correlation Attacks

• Authors: Chen-Da Liu, Simone Santini
• Subjects: cs.CR, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.18817
• Pdf URL: https://arxiv.org/pdf/2404.18817
• Copy Paste: [[2404.18817]] Hiding from Facebook: An Encryption Protocol resistant to Correlation Attacks(https://arxiv.org/abs/2404.18817)
• Keywords: privacy, attack
• Abstract: In many social networks, one publishes information that one wants to reveal (e.g., the photograph of some friends) together with information that may lead to privacy breaches (e.g., the name of these people). One might want to hide this sensitive information by encrypting it and sharing the decryption key only with trusted people, but this might not be enough. If the cipher associated to a face is always the same, correlation between the output of a face recognition system and the cipher can give useful clues and help train recognizers to identify untagged instances of the face. We refer to these as "correlation attacks". In this paper we present a coding system that attempts to counter correlation attacks by associating to each instance of a face a different encryption of the same tag in such a way that the correlation between different instances is minimal. In addition, we present a key distribution code that allows only the owner of the images to encode the tags, but allows a group of trusted friends to decode them.

Title: Benchmarking Benchmark Leakage in Large Language Models

• Authors: Ruijie Xu, Zengzhi Wang, Run-Ze Fan, Pengfei Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18824
• Pdf URL: https://arxiv.org/pdf/2404.18824
• Copy Paste: [[2404.18824]] Benchmarking Benchmark Leakage in Large Language Models(https://arxiv.org/abs/2404.18824)
• Keywords: fair, large language model
• Abstract: Amid the expanding use of pre-training data, the phenomenon of benchmark dataset leakage has become increasingly prominent, exacerbated by opaque training processes and the often undisclosed inclusion of supervised data in contemporary Large Language Models (LLMs). This issue skews benchmark effectiveness and fosters potentially unfair comparisons, impeding the field's healthy development. To address this, we introduce a detection pipeline utilizing Perplexity and N-gram accuracy, two simple and scalable metrics that gauge a model's prediction precision on benchmark, to identify potential data leakages. By analyzing 31 LLMs under the context of mathematical reasoning, we reveal substantial instances of training even test set misuse, resulting in potentially unfair comparisons. These findings prompt us to offer several recommendations regarding model documentation, benchmark setup, and future evaluations. Notably, we propose the "Benchmark Transparency Card" to encourage clear documentation of benchmark utilization, promoting transparency and healthy developments of LLMs. we have made our leaderboard, pipeline implementation, and model predictions publicly available, fostering future research.

Title: Harmonic Machine Learning Models are Robust

• Authors: Nicholas S. Kersting, Yi Li, Aman Mohanty, Oyindamola Obisesan, Raphael Okochu
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18825
• Pdf URL: https://arxiv.org/pdf/2404.18825
• Copy Paste: [[2404.18825]] Harmonic Machine Learning Models are Robust(https://arxiv.org/abs/2404.18825)
• Keywords: robust, explainability, transformer
• Abstract: We introduce Harmonic Robustness, a powerful and intuitive method to test the robustness of any machine-learning model either during training or in black-box real-time inference monitoring without ground-truth labels. It is based on functional deviation from the harmonic mean value property, indicating instability and lack of explainability. We show implementation examples in low-dimensional trees and feedforward NNs, where the method reliably identifies overfitting, as well as in more complex high-dimensional models such as ResNet-50 and Vision Transformer where it efficiently measures adversarial vulnerability across image classes.

Title: It's Difficult to be Neutral -- Human and LLM-based Sentiment Annotation of Patient Comments

• Authors: Petter Mæhlum, David Samuel, Rebecka Maria Norman, Elma Jelin, Øyvind Andresen Bjertnæs, Lilja Øvrelid, Erik Velldal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18832
• Pdf URL: https://arxiv.org/pdf/2404.18832
• Copy Paste: [[2404.18832]] It's Difficult to be Neutral -- Human and LLM-based Sentiment Annotation of Patient Comments(https://arxiv.org/abs/2404.18832)
• Keywords: large language model
• Abstract: Sentiment analysis is an important tool for aggregating patient voices, in order to provide targeted improvements in healthcare services. A prerequisite for this is the availability of in-domain data annotated for sentiment. This article documents an effort to add sentiment annotations to free-text comments in patient surveys collected by the Norwegian Institute of Public Health (NIPH). However, annotation can be a time-consuming and resource-intensive process, particularly when it requires domain expertise. We therefore also evaluate a possible alternative to human annotation, using large language models (LLMs) as annotators. We perform an extensive evaluation of the approach for two openly available pretrained LLMs for Norwegian, experimenting with different configurations of prompts and in-context learning, comparing their performance to human annotators. We find that even for zero-shot runs, models perform well above the baseline for binary sentiment, but still cannot compete with human annotators on the full dataset.

Title: VISION: Toward a Standardized Process for Radiology Image Management at the National Level

• Authors: Kathryn Knight, Ioana Danciu, Olga Ovchinnikova, Jacob Hinkle, Mayanka Chandra Shekar, Debangshu Mukherjee, Eileen McAllister, Caitlin Rizy, Kelly Cho, Amy C. Justice, Joseph Erdos, Peter Kuzmak, Lauren Costa, Yuk-Lam Ho, Reddy Madipadga, Suzanne Tamang, Ian Goethert
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18842
• Pdf URL: https://arxiv.org/pdf/2404.18842
• Copy Paste: [[2404.18842]] VISION: Toward a Standardized Process for Radiology Image Management at the National Level(https://arxiv.org/abs/2404.18842)
• Keywords: fair
• Abstract: The compilation and analysis of radiological images poses numerous challenges for researchers. The sheer volume of data as well as the computational needs of algorithms capable of operating on images are extensive. Additionally, the assembly of these images alone is difficult, as these exams may differ widely in terms of clinical context, structured annotation available for model training, modality, and patient identifiers. In this paper, we describe our experiences and challenges in establishing a trusted collection of radiology images linked to the United States Department of Veterans Affairs (VA) electronic health record database. We also discuss implications in making this repository research-ready for medical investigators. Key insights include uncovering the specific procedures required for transferring images from a clinical to a research-ready environment, as well as roadblocks and bottlenecks in this process that may hinder future efforts at automation.

Title: FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition

• Authors: Yuxuan Yan, Shunpu Tang, Zhiguo Shi, Qianqian Yang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18848
• Pdf URL: https://arxiv.org/pdf/2404.18848
• Copy Paste: [[2404.18848]] FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition(https://arxiv.org/abs/2404.18848)
• Keywords: privacy, federate
• Abstract: Pre-trained Language Models (PLMs) have shown excellent performance on various downstream tasks after fine-tuning. Nevertheless, the escalating concerns surrounding user privacy have posed significant challenges to centralized training reliant on extensive data collection. Federated learning(FL), which only requires training on the clients and aggregates weights on the server without sharing data, has emerged as a solution. However, the substantial parameter size of PLMs places a significant burden on the computational resources of client devices, while also leading to costly communication expenses. Introducing Parameter-Efficient Fine-Tuning(PEFT) into FL can effectively address this problem. However, we observe that the non-IID data in federated learning leads to a gap in performance between the PEFT method and full parameter fine-tuning(FT). To overcome this, we propose FeDeRA, an improvement over the LoRA method in FL. FeDeRA uses the same adapter module as LoRA. However, the difference lies in FeDeRA's initialization of the adapter module by performing Singular Value Decomposition (SVD) on the pre-trained matrix and selecting its principal components. We conducted extensive experiments, using RoBERTa and DeBERTaV3, on three tasks and six datasets, comparing the methods including FT and the other three different PEFT methods. FeDeRA outperforms all other PEFT methods and is comparable to or even surpasses the performance of FT methods. We also deployed federated learning on Jetson AGX Orin and compared the time required by different methods to achieve the target accuracy on specific tasks. Compared to FT, FeDeRA reduces the training time by 95.9%, 97.9%, 96.9%, and 97.3%, 96.5%, and 96.5% respectively on three tasks using RoBERTa and DeBERTaV3. The overall experiments indicate that FeDeRA achieves good performance while also maintaining efficiency.

Title: MiPa: Mixed Patch Infrared-Visible Modality Agnostic Object Detection

• Authors: Heitor R. Medeiros, David Latortue, Fidel Guerrero Pena, Eric Granger, Marco Pedersoli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18849
• Pdf URL: https://arxiv.org/pdf/2404.18849
• Copy Paste: [[2404.18849]] MiPa: Mixed Patch Infrared-Visible Modality Agnostic Object Detection(https://arxiv.org/abs/2404.18849)
• Keywords: transformer
• Abstract: In this paper, we present a different way to use two modalities, in which either one modality or the other is seen by a single model. This can be useful when adapting an unimodal model to leverage more information while respecting a limited computational budget. This would mean having a single model that is able to deal with any modalities. To describe this, we coined the term anymodal learning. An example of this, is a use case where, surveillance in a room when the lights are off would be much more valuable using an infrared modality while a visible one would provide more discriminative information when lights are on. This work investigates how to efficiently leverage visible and infrared/thermal modalities for transformer-based object detection backbone to create an anymodal architecture. Our work does not create any inference overhead during the testing while exploring an effective way to exploit the two modalities during the training. To accomplish such a task, we introduce the novel anymodal training technique: Mixed Patches (MiPa), in conjunction with a patch-wise domain agnostic module, which is responsible of learning the best way to find a common representation of both modalities. This approach proves to be able to balance modalities by reaching competitive results on individual modality benchmarks with the alternative of using an unimodal architecture on three different visible-infrared object detection datasets. Finally, our proposed method, when used as a regularization for the strongest modality, can beat the performance of multimodal fusion methods while only requiring a single modality during inference. Notably, MiPa became the state-of-the-art on the LLVIP visible/infrared benchmark. Code: https://github.com/heitorrapela/MiPa

Title: A Comprehensive Rubric for Annotating Pathological Speech

• Authors: Mario Corrales-Astorgano, David Escudero-Mancebo, Lourdes Aguilar, Valle Flores-Lucas, Valentín Cardeñoso-Payo, Carlos Vivaracho-Pascual, César González-Ferreras
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18851
• Pdf URL: https://arxiv.org/pdf/2404.18851
• Copy Paste: [[2404.18851]] A Comprehensive Rubric for Annotating Pathological Speech(https://arxiv.org/abs/2404.18851)
• Keywords: segmentation
• Abstract: Rubrics are a commonly used tool for labeling voice corpora in speech quality assessment, although their application in the context of pathological speech remains relatively limited. In this study, we introduce a comprehensive rubric based on various dimensions of speech quality, including phonetics, fluency, and prosody. The objective is to establish standardized criteria for identifying errors within the speech of individuals with Down syndrome, thereby enabling the development of automated assessment systems. To achieve this objective, we utilized the Prautocal corpus. To assess the quality of annotations using our rubric, two experiments were conducted, focusing on phonetics and fluency. For phonetic evaluation, we employed the Goodness of Pronunciation (GoP) metric, utilizing automatic segmentation systems and correlating the results with evaluations conducted by a specialized speech therapist. While the obtained correlation values were not notably high, a positive trend was observed. In terms of fluency assessment, deep learning models like wav2vec were used to extract audio features, and we employed an SVM classifier trained on a corpus focused on identifying fluency issues to categorize Prautocal corpus samples. The outcomes highlight the complexities of evaluating such phenomena, with variability depending on the specific type of disfluency detected.

Title: A Survey on Vision Mamba: Models, Applications and Challenges

• Authors: Rui Xu, Shu Yang, Yihui Wang, Bo Du, Hao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18861
• Pdf URL: https://arxiv.org/pdf/2404.18861
• Copy Paste: [[2404.18861]] A Survey on Vision Mamba: Models, Applications and Challenges(https://arxiv.org/abs/2404.18861)
• Keywords: transformer
• Abstract: Mamba, a recent selective structured state space model, performs excellently on long sequence modeling tasks. Mamba mitigates the modeling constraints of convolutional neural networks and offers advanced modeling capabilities similar to those of Transformers, through global receptive fields and dynamic weighting. Crucially, it achieves this without incurring the quadratic computational complexity typically associated with Transformers. Due to its advantages over the former two mainstream foundation models, Mamba exhibits great potential to be a visual foundation model. Researchers are actively applying Mamba to various computer vision tasks, leading to numerous emerging works. To help keep pace with the rapid advancements in computer vision, this paper aims to provide a comprehensive review of visual Mamba approaches. This paper begins by delineating the formulation of the original Mamba model. Subsequently, our review of visual Mamba delves into several representative backbone networks to elucidate the core insights of the visual Mamba. We then categorize related works using different modalities, including image, video, point cloud, multi-modal, and others. Specifically, for image applications, we further organize them into distinct tasks to facilitate a more structured discussion. Finally, we discuss the challenges and future research directions for visual Mamba, providing insights for future research in this quickly evolving area. A comprehensive list of visual Mamba models reviewed in this work is available at https://github.com/Ruixxxx/Awesome-Vision-Mamba-Models.

Title: Truth-value judgment in language models: belief directions are context sensitive

• Authors: Stefan F. Schouten, Peter Bloem, Ilia Markov, Piek Vossen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18865
• Pdf URL: https://arxiv.org/pdf/2404.18865
• Copy Paste: [[2404.18865]] Truth-value judgment in language models: belief directions are context sensitive(https://arxiv.org/abs/2404.18865)
• Keywords: large language model
• Abstract: Recent work has demonstrated that the latent spaces of large language models (LLMs) contain directions predictive of the truth of sentences. Multiple methods recover such directions and build probes that are described as getting at a model's "knowledge" or "beliefs". We investigate this phenomenon, looking closely at the impact of context on the probes. Our experiments establish where in the LLM the probe's predictions can be described as being conditional on the preceding (related) sentences. Specifically, we quantify the responsiveness of the probes to the presence of (negated) supporting and contradicting sentences, and score the probes on their consistency. We also perform a causal intervention experiment, investigating whether moving the representation of a premise along these belief directions influences the position of the hypothesis along that same direction. We find that the probes we test are generally context sensitive, but that contexts which should not affect the truth often still impact the probe outputs. Our experiments show that the type of errors depend on the layer, the (type of) model, and the kind of data. Finally, our results suggest that belief directions are (one of the) causal mediators in the inference process that incorporates in-context information.

Title: Learning Mixtures of Gaussians Using Diffusion Models

• Authors: Khashayar Gatmiry, Jonathan Kelner, Holden Lee
• Subjects: cs.LG, cs.DS, math.PR, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.18869
• Pdf URL: https://arxiv.org/pdf/2404.18869
• Copy Paste: [[2404.18869]] Learning Mixtures of Gaussians Using Diffusion Models(https://arxiv.org/abs/2404.18869)
• Keywords: diffusion, generative
• Abstract: We give a new algorithm for learning mixtures of $k$ Gaussians (with identity covariance in $\mathbb{R}^n$) to TV error $\varepsilon$, with quasi-polynomial ($O(n^{\text{poly log}\left(\frac{n+k}{\varepsilon}\right)})$) time and sample complexity, under a minimum weight assumption. Unlike previous approaches, most of which are algebraic in nature, our approach is analytic and relies on the framework of diffusion models. Diffusion models are a modern paradigm for generative modeling, which typically rely on learning the score function (gradient log-pdf) along a process transforming a pure noise distribution, in our case a Gaussian, to the data distribution. Despite their dazzling performance in tasks such as image generation, there are few end-to-end theoretical guarantees that they can efficiently learn nontrivial families of distributions; we give some of the first such guarantees. We proceed by deriving higher-order Gaussian noise sensitivity bounds for the score functions for a Gaussian mixture to show that that they can be inductively learned using piecewise polynomial regression (up to poly-logarithmic degree), and combine this with known convergence results for diffusion models. Our results extend to continuous mixtures of Gaussians where the mixing distribution is supported on a union of $k$ balls of constant radius. In particular, this applies to the case of Gaussian convolutions of distributions on low-dimensional manifolds, or more generally sets with small covering number.

Title: More RLHF, More Trust? On The Impact of Human Preference Alignment On Language Model Trustworthiness

• Authors: Aaron J. Li, Satyapriya Krishna, Himabindu Lakkaraju
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18870
• Pdf URL: https://arxiv.org/pdf/2404.18870
• Copy Paste: [[2404.18870]] More RLHF, More Trust? On The Impact of Human Preference Alignment On Language Model Trustworthiness(https://arxiv.org/abs/2404.18870)
• Keywords: privacy, large language model
• Abstract: The surge in Large Language Models (LLMs) development has led to improved performance on cognitive tasks as well as an urgent need to align these models with human values in order to safely exploit their power. Despite the effectiveness of preference learning algorithms like Reinforcement Learning From Human Feedback (RLHF) in aligning human preferences, their assumed improvements on model trustworthiness haven't been thoroughly testified. Toward this end, this study investigates how models that have been aligned with general-purpose preference data on helpfulness and harmlessness perform across five trustworthiness verticals: toxicity, stereotypical bias, machine ethics, truthfulness, and privacy. For model alignment, we focus on three widely used RLHF variants: Supervised Finetuning (SFT), Proximal Policy Optimization (PPO), and Direct Preference Optimization (DPO). Through extensive empirical investigations, we discover that the improvement in trustworthiness by RLHF is far from guaranteed, and there exists a complex interplay between preference data, alignment algorithms, and specific trustworthiness aspects. Together, our results underscore the need for more nuanced approaches for model alignment. By shedding light on the intricate dynamics of these components within model alignment, we hope this research will guide the community towards developing language models that are both capable and trustworthy.

Title: A Survey on Diffusion Models for Time Series and Spatio-Temporal Data

• Authors: Yiyuan Yang, Ming Jin, Haomin Wen, Chaoli Zhang, Yuxuan Liang, Lintao Ma, Yi Wang, Chenghao Liu, Bin Yang, Zenglin Xu, Jiang Bian, Shirui Pan, Qingsong Wen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18886
• Pdf URL: https://arxiv.org/pdf/2404.18886
• Copy Paste: [[2404.18886]] A Survey on Diffusion Models for Time Series and Spatio-Temporal Data(https://arxiv.org/abs/2404.18886)
• Keywords: diffusion, generative
• Abstract: The study of time series data is crucial for understanding trends and anomalies over time, enabling predictive insights across various sectors. Spatio-temporal data, on the other hand, is vital for analyzing phenomena in both space and time, providing a dynamic perspective on complex system interactions. Recently, diffusion models have seen widespread application in time series and spatio-temporal data mining. Not only do they enhance the generative and inferential capabilities for sequential and temporal data, but they also extend to other downstream tasks. In this survey, we comprehensively and thoroughly review the use of diffusion models in time series and spatio-temporal data, categorizing them by model category, task type, data modality, and practical application domain. In detail, we categorize diffusion models into unconditioned and conditioned types and discuss time series data and spatio-temporal data separately. Unconditioned models, which operate unsupervised, are subdivided into probability-based and score-based models, serving predictive and generative tasks such as forecasting, anomaly detection, classification, and imputation. Conditioned models, on the other hand, utilize extra information to enhance performance and are similarly divided for both predictive and generative tasks. Our survey extensively covers their application in various fields, including healthcare, recommendation, climate, energy, audio, and transportation, providing a foundational understanding of how these models analyze and generate data. Through this structured overview, we aim to provide researchers and practitioners with a comprehensive understanding of diffusion models for time series and spatio-temporal data analysis, aiming to direct future innovations and applications by addressing traditional challenges and exploring innovative solutions within the diffusion model framework.

Title: Hide and Seek: How Does Watermarking Impact Face Recognition?

• Authors: Yuguang Yao, Steven Grosz, Sijia Liu, Anil Jain
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18890
• Pdf URL: https://arxiv.org/pdf/2404.18890
• Copy Paste: [[2404.18890]] Hide and Seek: How Does Watermarking Impact Face Recognition?(https://arxiv.org/abs/2404.18890)
• Keywords: privacy, robust, biometric, watermark, generative
• Abstract: The recent progress in generative models has revolutionized the synthesis of highly realistic images, including face images. This technological development has undoubtedly helped face recognition, such as training data augmentation for higher recognition accuracy and data privacy. However, it has also introduced novel challenges concerning the responsible use and proper attribution of computer generated images. We investigate the impact of digital watermarking, a technique for embedding ownership signatures into images, on the effectiveness of face recognition models. We propose a comprehensive pipeline that integrates face image generation, watermarking, and face recognition to systematically examine this question. The proposed watermarking scheme, based on an encoder-decoder architecture, successfully embeds and recovers signatures from both real and synthetic face images while preserving their visual fidelity. Through extensive experiments, we unveil that while watermarking enables robust image attribution, it results in a slight decline in face recognition accuracy, particularly evident for face images with challenging poses and expressions. Additionally, we find that directly training face recognition models on watermarked images offers only a limited alleviation of this performance decline. Our findings underscore the intricate trade off between watermarking and face recognition accuracy. This work represents a pivotal step towards the responsible utilization of generative models in face recognition and serves to initiate discussions regarding the broader implications of watermarking in biometrics.

Title: IPixMatch: Boost Semi-supervised Semantic Segmentation with Inter-Pixel Relation

• Authors: Kebin Wu, Wenbin Li, Xiaofei Xiao
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18891
• Pdf URL: https://arxiv.org/pdf/2404.18891
• Copy Paste: [[2404.18891]] IPixMatch: Boost Semi-supervised Semantic Segmentation with Inter-Pixel Relation(https://arxiv.org/abs/2404.18891)
• Keywords: segmentation
• Abstract: The scarcity of labeled data in real-world scenarios is a critical bottleneck of deep learning's effectiveness. Semi-supervised semantic segmentation has been a typical solution to achieve a desirable tradeoff between annotation cost and segmentation performance. However, previous approaches, whether based on consistency regularization or self-training, tend to neglect the contextual knowledge embedded within inter-pixel relations. This negligence leads to suboptimal performance and limited generalization. In this paper, we propose a novel approach IPixMatch designed to mine the neglected but valuable Inter-Pixel information for semi-supervised learning. Specifically, IPixMatch is constructed as an extension of the standard teacher-student network, incorporating additional loss terms to capture inter-pixel relations. It shines in low-data regimes by efficiently leveraging the limited labeled data and extracting maximum utility from the available unlabeled data. Furthermore, IPixMatch can be integrated seamlessly into most teacher-student frameworks without the need of model modification or adding additional components. Our straightforward IPixMatch method demonstrates consistent performance improvements across various benchmark datasets under different partitioning protocols.

Title: RSCaMa: Remote Sensing Image Change Captioning with State Space Model

• Authors: Chenyang Liu, Keyan Chen, Bowen Chen, Haotian Zhang, Zhengxia Zou, Zhenwei Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18895
• Pdf URL: https://arxiv.org/pdf/2404.18895
• Copy Paste: [[2404.18895]] RSCaMa: Remote Sensing Image Change Captioning with State Space Model(https://arxiv.org/abs/2404.18895)
• Keywords: transformer
• Abstract: Remote Sensing Image Change Captioning (RSICC) aims to identify surface changes in multi-temporal remote sensing images and describe them in natural language. Current methods typically rely on an encoder-decoder architecture and focus on designing a sophisticated neck to process bi-temporal features extracted by the backbone. Recently, State Space Models (SSMs), especially Mamba, have demonstrated outstanding performance in many fields, owing to their efficient feature-selective modelling capability. However, their potential in the RSICC task remains unexplored. In this paper, we introduce Mamba into RSICC and propose a novel approach called RSCaMa (Remote Sensing Change Captioning Mamba). Specifically, we utilize Siamese backbones to extract bi-temporal features, which are then processed through multiple CaMa layers consisting of Spatial Difference-guided SSM (SD-SSM) and Temporal Traveling SSM (TT-SSM). SD-SSM uses differential features to enhance change perception, while TT-SSM promotes bitemporal interactions in a token-wise cross-scanning manner. Experimental results validate the effectiveness of CaMa layers and demonstrate the superior performance of RSCaMa, as well as the potential of Mamba in the RSICC task. Additionally, we systematically compare the effects of three language decoders, including Mamba, GPT-style decoder with causal attention mechanism, and Transformer decoder with cross-attention mechanism. This provides valuable insights for future RSICC research. The code will be available at https://github.com/Chen-Yang-Liu/RSCaMa

Title: Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty

• Authors: Laixi Shi, Eric Mazumdar, Yuejie Chi, Adam Wierman
• Subjects: cs.LG, cs.MA, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.18909
• Pdf URL: https://arxiv.org/pdf/2404.18909
• Copy Paste: [[2404.18909]] Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty(https://arxiv.org/abs/2404.18909)
• Keywords: robust, generative
• Abstract: To overcome the sim-to-real gap in reinforcement learning (RL), learned policies must maintain robustness against environmental uncertainties. While robust RL has been widely studied in single-agent regimes, in multi-agent environments, the problem remains understudied -- despite the fact that the problems posed by environmental uncertainties are often exacerbated by strategic interactions. This work focuses on learning in distributionally robust Markov games (RMGs), a robust variant of standard Markov games, wherein each agent aims to learn a policy that maximizes its own worst-case performance when the deployed environment deviates within its own prescribed uncertainty set. This results in a set of robust equilibrium strategies for all agents that align with classic notions of game-theoretic equilibria. Assuming a non-adaptive sampling mechanism from a generative model, we propose a sample-efficient model-based algorithm (DRNVI) with finite-sample complexity guarantees for learning robust variants of various notions of game-theoretic equilibria. We also establish an information-theoretic lower bound for solving RMGs, which confirms the near-optimal sample complexity of DRNVI with respect to problem-dependent factors such as the size of the state space, the target accuracy, and the horizon length.

Title: Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting

• Authors: Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, Yunhe Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18911
• Pdf URL: https://arxiv.org/pdf/2404.18911
• Copy Paste: [[2404.18911]] Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting(https://arxiv.org/abs/2404.18911)
• Keywords: large language model
• Abstract: Speculative decoding has demonstrated its effectiveness in accelerating the inference of large language models while maintaining a consistent sampling distribution. However, the conventional approach of training a separate draft model to achieve a satisfactory token acceptance rate can be costly. Drawing inspiration from early exiting, we propose a novel self-speculative decoding framework \emph{Kangaroo}, which uses a fixed shallow sub-network as a self-draft model, with the remaining layers serving as the larger target model. We train a lightweight and efficient adapter module on top of the sub-network to bridge the gap between the sub-network and the full model's representation ability. It is noteworthy that the inference latency of the self-draft model may no longer be negligible compared to the large model, necessitating strategies to increase the token acceptance rate while minimizing the drafting steps of the small model. To address this challenge, we introduce an additional early exiting mechanism for generating draft tokens. Specifically, we halt the small model's subsequent prediction during the drafting phase once the confidence level for the current token falls below a certain threshold. Extensive experiments on the Spec-Bench demonstrate the effectiveness of Kangaroo. Under single-sequence verification, Kangaroo achieves speedups up to $1.68\times$ on Spec-Bench, outperforming Medusa-1 with 88.7\% fewer additional parameters (67M compared to 591M). The code for Kangaroo is available at https://github.com/Equationliu/Kangaroo.

Title: TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation

• Authors: Junhao Cheng, Baiqiao Yin, Kaixin Cai, Minbin Huang, Hanhui Li, Yuxin He, Xi Lu, Yue Li, Yifei Li, Yuhao Cheng, Yiqiang Yan, Xiaodan Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18919
• Pdf URL: https://arxiv.org/pdf/2404.18919
• Copy Paste: [[2404.18919]] TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation(https://arxiv.org/abs/2404.18919)
• Keywords: diffusion, large language model
• Abstract: Recent advances in diffusion models can generate high-quality and stunning images from text. However, multi-turn image generation, which is of high demand in real-world scenarios, still faces challenges in maintaining semantic consistency between images and texts, as well as contextual consistency of the same subject across multiple interactive turns. To address this issue, we introduce TheaterGen, a training-free framework that integrates large language models (LLMs) and text-to-image (T2I) models to provide the capability of multi-turn image generation. Within this framework, LLMs, acting as a "Screenwriter", engage in multi-turn interaction, generating and managing a standardized prompt book that encompasses prompts and layout designs for each character in the target image. Based on these, Theatergen generate a list of character images and extract guidance information, akin to the "Rehearsal". Subsequently, through incorporating the prompt book and guidance information into the reverse denoising process of T2I diffusion models, Theatergen generate the final image, as conducting the "Final Performance". With the effective management of prompt books and character images, TheaterGen significantly improves semantic and contextual consistency in synthesized images. Furthermore, we introduce a dedicated benchmark, CMIGBench (Consistent Multi-turn Image Generation Benchmark) with 8000 multi-turn instructions. Different from previous multi-turn benchmarks, CMIGBench does not define characters in advance. Both the tasks of story generation and multi-turn editing are included on CMIGBench for comprehensive evaluation. Extensive experimental results show that TheaterGen outperforms state-of-the-art methods significantly. It raises the performance bar of the cutting-edge Mini DALLE 3 model by 21% in average character-character similarity and 19% in average text-image similarity.

Title: DPO Meets PPO: Reinforced Token Optimization for RLHF

• Authors: Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, Liwei Wang
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.18922
• Pdf URL: https://arxiv.org/pdf/2404.18922
• Copy Paste: [[2404.18922]] DPO Meets PPO: Reinforced Token Optimization for RLHF(https://arxiv.org/abs/2404.18922)
• Keywords: large language model
• Abstract: In the classical Reinforcement Learning from Human Feedback (RLHF) framework, Proximal Policy Optimization (PPO) is employed to learn from sparse, sentence-level rewards -- a challenging scenario in traditional deep reinforcement learning. Despite the great successes of PPO in the alignment of state-of-the-art closed-source large language models (LLMs), its open-source implementation is still largely sub-optimal, as widely reported by numerous research studies. To address these issues, we introduce a framework that models RLHF problems as a Markov decision process (MDP), enabling the capture of fine-grained token-wise information. Furthermore, we provide theoretical insights that demonstrate the superiority of our MDP framework over the previous sentence-level bandit formulation. Under this framework, we introduce an algorithm, dubbed as Reinforced Token Optimization (\texttt{RTO}), which learns the token-wise reward function from preference data and performs policy optimization based on this learned token-wise reward signal. Theoretically, \texttt{RTO} is proven to have the capability of finding the near-optimal policy sample-efficiently. For its practical implementation, \texttt{RTO} innovatively integrates Direct Preference Optimization (DPO) and PPO. DPO, originally derived from sparse sentence rewards, surprisingly provides us with a token-wise characterization of response quality, which is seamlessly incorporated into our subsequent PPO training stage. Extensive real-world alignment experiments verify the effectiveness of the proposed approach.

Title: Swin2-MoSE: A New Single Image Super-Resolution Model for Remote Sensing

• Authors: Leonardo Rossi, Vittorio Bernuzzi, Tomaso Fontanini, Massimo Bertozzi, Andrea Prati
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.18924
• Pdf URL: https://arxiv.org/pdf/2404.18924
• Copy Paste: [[2404.18924]] Swin2-MoSE: A New Single Image Super-Resolution Model for Remote Sensing(https://arxiv.org/abs/2404.18924)
• Keywords: transformer, segmentation
• Abstract: Due to the limitations of current optical and sensor technologies and the high cost of updating them, the spectral and spatial resolution of satellites may not always meet desired requirements. For these reasons, Remote-Sensing Single-Image Super-Resolution (RS-SISR) techniques have gained significant interest. In this paper, we propose Swin2-MoSE model, an enhanced version of Swin2SR. Our model introduces MoE-SM, an enhanced Mixture-of-Experts (MoE) to replace the Feed-Forward inside all Transformer block. MoE-SM is designed with Smart-Merger, and new layer for merging the output of individual experts, and with a new way to split the work between experts, defining a new per-example strategy instead of the commonly used per-token one. Furthermore, we analyze how positional encodings interact with each other, demonstrating that per-channel bias and per-head bias can positively cooperate. Finally, we propose to use a combination of Normalized-Cross-Correlation (NCC) and Structural Similarity Index Measure (SSIM) losses, to avoid typical MSE loss limitations. Experimental results demonstrate that Swin2-MoSE outperforms SOTA by up to 0.377 ~ 0.958 dB (PSNR) on task of 2x, 3x and 4x resolution-upscaling (Sen2Venus and OLI2MSI datasets). We show the efficacy of Swin2-MoSE, applying it to a semantic segmentation task (SeasoNet dataset). Code and pretrained are available on https://github.com/IMPLabUniPr/swin2-mose/tree/official_code

Title: Stylus: Automatic Adapter Selection for Diffusion Models

• Authors: Michael Luo, Justin Wong, Brandon Trabucco, Yanping Huang, Joseph E. Gonzalez, Zhifeng Chen, Ruslan Salakhutdinov, Ion Stoica
• Subjects: cs.CV, cs.AI, cs.CL, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18928
• Pdf URL: https://arxiv.org/pdf/2404.18928
• Copy Paste: [[2404.18928]] Stylus: Automatic Adapter Selection for Diffusion Models(https://arxiv.org/abs/2404.18928)
• Keywords: diffusion
• Abstract: Beyond scaling base models with more data or parameters, fine-tuned adapters provide an alternative way to generate high fidelity, custom images at reduced costs. As such, adapters have been widely adopted by open-source communities, accumulating a database of over 100K adapters-most of which are highly customized with insufficient descriptions. This paper explores the problem of matching the prompt to a set of relevant adapters, built on recent work that highlight the performance gains of composing adapters. We introduce Stylus, which efficiently selects and automatically composes task-specific adapters based on a prompt's keywords. Stylus outlines a three-stage approach that first summarizes adapters with improved descriptions and embeddings, retrieves relevant adapters, and then further assembles adapters based on prompts' keywords by checking how well they fit the prompt. To evaluate Stylus, we developed StylusDocs, a curated dataset featuring 75K adapters with pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion checkpoints, Stylus achieves greater CLIP-FID Pareto efficiency and is twice as preferred, with humans and multimodal models as evaluators, over the base model. See stylus-diffusion.github.io for more.

Title: Hallucination of Multimodal Large Language Models: A Survey

• Authors: Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, Mike Zheng Shou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.18930
• Pdf URL: https://arxiv.org/pdf/2404.18930
• Copy Paste: [[2404.18930]] Hallucination of Multimodal Large Language Models: A Survey(https://arxiv.org/abs/2404.18930)
• Keywords: robust, large language model
• Abstract: This survey presents a comprehensive analysis of the phenomenon of hallucination in multimodal large language models (MLLMs), also known as Large Vision-Language Models (LVLMs), which have demonstrated significant advancements and remarkable abilities in multimodal tasks. Despite these promising developments, MLLMs often generate outputs that are inconsistent with the visual content, a challenge known as hallucination, which poses substantial obstacles to their practical deployment and raises concerns regarding their reliability in real-world applications. This problem has attracted increasing attention, prompting efforts to detect and mitigate such inaccuracies. We review recent advances in identifying, evaluating, and mitigating these hallucinations, offering a detailed overview of the underlying causes, evaluation benchmarks, metrics, and strategies developed to address this issue. Additionally, we analyze the current challenges and limitations, formulating open questions that delineate potential pathways for future research. By drawing the granular classification and landscapes of hallucination causes, evaluation benchmarks, and mitigation methods, this survey aims to deepen the understanding of hallucinations in MLLMs and inspire further advancements in the field. Through our thorough and in-depth review, we contribute to the ongoing dialogue on enhancing the robustness and reliability of MLLMs, providing valuable insights and resources for researchers and practitioners alike. Resources are available at: https://github.com/showlab/Awesome-MLLM-Hallucination.