2024-03-14

Title: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection

• Authors: Wei Ye, Chaoya Jiang, Haiyang Xu, Chenhao Ye, Chenliang Li, Ming Yan, Shikun Zhang, Songhang Huang, Fei Huang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.07883
• Pdf URL: https://arxiv.org/pdf/2403.07883
• Copy Paste: [[2403.07883]] Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection(https://arxiv.org/abs/2403.07883)
• Keywords: transformer, generative
• Abstract: Vision Transformers (ViTs) have become increasingly popular in large-scale Vision and Language Pre-training (VLP) models. Although previous VLP research has demonstrated the efficacy of ViTs, these efforts still struggle with computational inefficiencies caused by lengthy visual sequences. To address this challenge, we introduce an efficient VLP approach called TRIPS, which stands for Text-Relevant Image Patch Selection. TRIPS progressively reduces the visual sequence using a text-guided patch-selection layer in the visual backbone, thereby accelerating both training and inference processes. This patch-selection layer dynamically computes text-dependent visual attention, enabling it to identify attentive image tokens with text guidance and fuse inattentive ones in an end-to-end fashion. Importantly, TRIPS does not add any extra parameters and generalizes to most ViT-based VLP models. We incorporate TRIPS into three representative VLP models covering single-stream, dual-stream, and generative paradigms, and conduct extensive experiments on five widely-used multi-modal benchmark datasets. Our experimental results reveal that TRIPS delivers a 40% speedup, while maintaining competitive or superior performance on downstream tasks.

Title: Seg-metrics: a Python package to compute segmentation metrics

• Authors: Jingnan Jia, Marius Staring, Berend C. Stoel
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.07884
• Pdf URL: https://arxiv.org/pdf/2403.07884
• Copy Paste: [[2403.07884]] Seg-metrics: a Python package to compute segmentation metrics(https://arxiv.org/abs/2403.07884)
• Keywords: segmentation
• Abstract: In response to a concerning trend of selectively emphasizing metrics in medical image segmentation (MIS) studies, we introduce \texttt{seg-metrics}, an open-source Python package for standardized MIS model evaluation. Unlike existing packages, \texttt{seg-metrics} offers user-friendly interfaces for various overlap-based and distance-based metrics, providing a comprehensive solution. \texttt{seg-metrics} supports multiple file formats and is easily installable through the Python Package Index (PyPI). With a focus on speed and convenience, \texttt{seg-metrics} stands as a valuable tool for efficient MIS model assessment.

Title: Cross-modality debiasing: using language to mitigate sub-population shifts in imaging

• Authors: Yijiang Pang, Hoang Bao, Jiayu Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.07888
• Pdf URL: https://arxiv.org/pdf/2403.07888
• Copy Paste: [[2403.07888]] Cross-modality debiasing: using language to mitigate sub-population shifts in imaging(https://arxiv.org/abs/2403.07888)
• Keywords: robust
• Abstract: Sub-population shift is a specific type of domain shift that highlights changes in data distribution within specific sub-groups or populations between training and testing. Sub-population shift accounts for a significant source of algorithmic bias and calls for distributional robustness. Recent studies found inherent distributional robustness in multi-modality foundation models, such as the vision-language model CLIP, yet this robustness is vulnerable through parameter fine-tuning. In this paper, we propose leveraging the connection of robustness among different modalities and reshaping the distributional robustness of one modality with another. Specifically, in the context of the distributional robustness of CLIP, we propose to leverage natural language inputs to debias the image feature representations, to improve worst-case performance on sub-populations. Our extensive empirical studies show that image representations debiased by natural language can achieve significant performance improvement and reduction of performance instability under sub-population shifts.

Title: MIP: CLIP-based Image Reconstruction from PEFT Gradients

• Authors: Peiheng Zhou, Ming Hu, Xiaofei Xie, Yihao Huang, Kangjie Chen, Mingsong Chen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.07901
• Pdf URL: https://arxiv.org/pdf/2403.07901
• Copy Paste: [[2403.07901]] MIP: CLIP-based Image Reconstruction from PEFT Gradients(https://arxiv.org/abs/2403.07901)
• Keywords: attack, federate
• Abstract: Contrastive Language-Image Pre-training (CLIP) model, as an effective pre-trained multimodal neural network, has been widely used in distributed machine learning tasks, especially Federated Learning (FL). Typically, CLIP-based FL adopts Parameter-Efficient Fine-Tuning (PEFT) for model training, which only fine-tunes adapter parameters or soft prompts rather than the full parameters. Although PEFT is different from the traditional training mode, in this paper, we theoretically analyze that the gradients of adapters or soft prompts can still be used to perform image reconstruction attacks. Based on our theoretical analysis, we propose Multm-In-Parvo (MIP), a proprietary reconstruction attack method targeting CLIP-based distributed machine learning architecture. Specifically, MIP can reconstruct CLIP training images according to the gradients of soft prompts or an adapter. In addition, MIP includes a label prediction strategy to accelerate convergence and an inverse gradient estimation mechanism to avoid the vanishing gradient problem on the text encoder. Experimental results show that MIP can effectively reconstruct training images according to the gradients of soft prompts or adapters of CLIP models.

Title: HandGCAT: Occlusion-Robust 3D Hand Mesh Reconstruction from Monocular Images

• Authors: Shuaibing Wang, Shunli Wang, Dingkang Yang, Mingcheng Li, Ziyun Qian, Liuzhen Su, Lihua Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.07912
• Pdf URL: https://arxiv.org/pdf/2403.07912
• Copy Paste: [[2403.07912]] HandGCAT: Occlusion-Robust 3D Hand Mesh Reconstruction from Monocular Images(https://arxiv.org/abs/2403.07912)
• Keywords: robust, transformer
• Abstract: We propose a robust and accurate method for reconstructing 3D hand mesh from monocular images. This is a very challenging problem, as hands are often severely occluded by objects. Previous works often have disregarded 2D hand pose information, which contains hand prior knowledge that is strongly correlated with occluded regions. Thus, in this work, we propose a novel 3D hand mesh reconstruction network HandGCAT, that can fully exploit hand prior as compensation information to enhance occluded region features. Specifically, we designed the Knowledge-Guided Graph Convolution (KGC) module and the Cross-Attention Transformer (CAT) module. KGC extracts hand prior information from 2D hand pose by graph convolution. CAT fuses hand prior into occluded regions by considering their high correlation. Extensive experiments on popular datasets with challenging hand-object occlusions, such as HO3D v2, HO3D v3, and DexYCB demonstrate that our HandGCAT reaches state-of-the-art performance. The code is available at https://github.com/heartStrive/HandGCAT.

Title: ACTrack: Adding Spatio-Temporal Condition for Visual Object Tracking

• Authors: Yushan Han, Kaer Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.07914
• Pdf URL: https://arxiv.org/pdf/2403.07914
• Copy Paste: [[2403.07914]] ACTrack: Adding Spatio-Temporal Condition for Visual Object Tracking(https://arxiv.org/abs/2403.07914)
• Keywords: transformer
• Abstract: Efficiently modeling spatio-temporal relations of objects is a key challenge in visual object tracking (VOT). Existing methods track by appearance-based similarity or long-term relation modeling, resulting in rich temporal contexts between consecutive frames being easily overlooked. Moreover, training trackers from scratch or fine-tuning large pre-trained models needs more time and memory consumption. In this paper, we present ACTrack, a new tracking framework with additive spatio-temporal conditions. It preserves the quality and capabilities of the pre-trained Transformer backbone by freezing its parameters, and makes a trainable lightweight additive net to model spatio-temporal relations in tracking. We design an additive siamese convolutional network to ensure the integrity of spatial features and perform temporal sequence modeling to simplify the tracking pipeline. Experimental results on several benchmarks prove that ACTrack could balance training efficiency and tracking performance.

Title: Merino: Entropy-driven Design for Generative Language Models on IoT Devices

• Authors: Youpeng Zhao, Ming Lin, Huadong Tang, Qiang Wu, Jun Wang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.07921
• Pdf URL: https://arxiv.org/pdf/2403.07921
• Copy Paste: [[2403.07921]] Merino: Entropy-driven Design for Generative Language Models on IoT Devices(https://arxiv.org/abs/2403.07921)
• Keywords: transformer, generative, large language model
• Abstract: Generative Large Language Models (LLMs) stand as a revolutionary advancement in the modern era of artificial intelligence (AI). However, directly deploying LLMs in resource-constrained hardware, such as Internet-of-Things (IoT) devices, is difficult due to their high computational cost. In this paper, we propose a novel information-entropy framework for designing mobile-friendly generative language models. Our key design paradigm is to maximize the entropy of transformer decoders within the given computational budgets. The whole design procedure involves solving a mathematical programming (MP) problem, which can be done on the CPU within minutes, making it nearly zero-cost. We evaluate our designed models, termed MeRino, across nine NLP downstream tasks, showing their competitive performance against the state-of-the-art autoregressive transformer models under the mobile setting. Notably, MeRino achieves similar or better zero performance compared to the 350M parameter OPT while being 4.9x faster on NVIDIA Jetson Nano with 5.5x reduction in model size. Code will be made available soon.

Title: Sketching the Heat Kernel: Using Gaussian Processes to Embed Data

• Authors: Anna C. Gilbert, Kevin O'Neill
• Subjects: cs.LG, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.07929
• Pdf URL: https://arxiv.org/pdf/2403.07929
• Copy Paste: [[2403.07929]] Sketching the Heat Kernel: Using Gaussian Processes to Embed Data(https://arxiv.org/abs/2403.07929)
• Keywords: robust, diffusion
• Abstract: This paper introduces a novel, non-deterministic method for embedding data in low-dimensional Euclidean space based on computing realizations of a Gaussian process depending on the geometry of the data. This type of embedding first appeared in (Adler et al, 2018) as a theoretical model for a generic manifold in high dimensions. In particular, we take the covariance function of the Gaussian process to be the heat kernel, and computing the embedding amounts to sketching a matrix representing the heat kernel. The Karhunen-Lo\`eve expansion reveals that the straight-line distances in the embedding approximate the diffusion distance in a probabilistic sense, avoiding the need for sharp cutoffs and maintaining some of the smaller-scale structure. Our method demonstrates further advantage in its robustness to outliers. We justify the approach with both theory and experiments.

Title: Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification

• Authors: Tingting Zheng, Kui Jiang, Hongxun Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.07939
• Pdf URL: https://arxiv.org/pdf/2403.07939
• Copy Paste: [[2403.07939]] Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification(https://arxiv.org/abs/2403.07939)
• Keywords: robust
• Abstract: Multi-Instance Learning (MIL) has shown impressive performance for histopathology whole slide image (WSI) analysis using bags or pseudo-bags. It involves instance sampling, feature representation, and decision-making. However, existing MIL-based technologies at least suffer from one or more of the following problems: 1) requiring high storage and intensive pre-processing for numerous instances (sampling); 2) potential over-fitting with limited knowledge to predict bag labels (feature representation); 3) pseudo-bag counts and prior biases affect model robustness and generalizability (decision-making). Inspired by clinical diagnostics, using the past sampling instances can facilitate the final WSI analysis, but it is barely explored in prior technologies. To break free these limitations, we integrate the dynamic instance sampling and reinforcement learning into a unified framework to improve the instance selection and feature aggregation, forming a novel Dynamic Policy Instance Selection (DPIS) scheme for better and more credible decision-making. Specifically, the measurement of feature distance and reward function are employed to boost continuous instance sampling. To alleviate the over-fitting, we explore the latent global relations among instances for more robust and discriminative feature representation while establishing reward and punishment mechanisms to correct biases in pseudo-bags using contrastive learning. These strategies form the final Dynamic Policy-Driven Adaptive Multi-Instance Learning (PAMIL) method for WSI tasks. Extensive experiments reveal that our PAMIL method outperforms the state-of-the-art by 3.8\% on CAMELYON16 and 4.4\% on TCGA lung cancer datasets.

Title: Attacking Transformers with Feature Diversity Adversarial Perturbation

• Authors: Chenxing Gao, Hang Zhou, Junqing Yu, YuTeng Ye, Jiale Cai, Junle Wang, Wei Yang
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.07942
• Pdf URL: https://arxiv.org/pdf/2403.07942
• Copy Paste: [[2403.07942]] Attacking Transformers with Feature Diversity Adversarial Perturbation(https://arxiv.org/abs/2403.07942)
• Keywords: attack, transformer
• Abstract: Understanding the mechanisms behind Vision Transformer (ViT), particularly its vulnerability to adversarial perturba tions, is crucial for addressing challenges in its real-world applications. Existing ViT adversarial attackers rely on la bels to calculate the gradient for perturbation, and exhibit low transferability to other structures and tasks. In this paper, we present a label-free white-box attack approach for ViT-based models that exhibits strong transferability to various black box models, including most ViT variants, CNNs, and MLPs, even for models developed for other modalities. Our inspira tion comes from the feature collapse phenomenon in ViTs, where the critical attention mechanism overly depends on the low-frequency component of features, causing the features in middle-to-end layers to become increasingly similar and eventually collapse. We propose the feature diversity attacker to naturally accelerate this process and achieve remarkable performance and transferability.

Title: Revisiting Edge Perturbation for Graph Neural Network in Graph Data Augmentation and Attack

• Authors: Xin Liu, Yuxiang Zhang, Meng Wu, Mingyu Yan, Kun He, Wei Yan, Shirui Pan, Xiaochun Ye, Dongrui Fan
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.07943
• Pdf URL: https://arxiv.org/pdf/2403.07943
• Copy Paste: [[2403.07943]] Revisiting Edge Perturbation for Graph Neural Network in Graph Data Augmentation and Attack(https://arxiv.org/abs/2403.07943)
• Keywords: attack
• Abstract: Edge perturbation is a basic method to modify graph structures. It can be categorized into two veins based on their effects on the performance of graph neural networks (GNNs), i.e., graph data augmentation and attack. Surprisingly, both veins of edge perturbation methods employ the same operations, yet yield opposite effects on GNNs' accuracy. A distinct boundary between these methods in using edge perturbation has never been clearly defined. Consequently, inappropriate perturbations may lead to undesirable outcomes, necessitating precise adjustments to achieve desired effects. Therefore, questions of ``why edge perturbation has a two-faced effect?'' and ``what makes edge perturbation flexible and effective?'' still remain unanswered. In this paper, we will answer these questions by proposing a unified formulation and establishing a clear boundary between two categories of edge perturbation methods. Specifically, we conduct experiments to elucidate the differences and similarities between these methods and theoretically unify the workflow of these methods by casting it to one optimization problem. Then, we devise Edge Priority Detector (EPD) to generate a novel priority metric, bridging these methods up in the workflow. Experiments show that EPD can make augmentation or attack flexibly and achieve comparable or superior performance to other counterparts with less time overhead.

Title: WorldGPT: A Sora-Inspired Video AI Agent as Rich World Models from Text and Image Inputs

• Authors: Deshun Yang, Luhui Hu, Yu Tian, Zihao Li, Chris Kelly, Bang Yang, Cindy Yang, Yuexian Zou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.07944
• Pdf URL: https://arxiv.org/pdf/2403.07944
• Copy Paste: [[2403.07944]] WorldGPT: A Sora-Inspired Video AI Agent as Rich World Models from Text and Image Inputs(https://arxiv.org/abs/2403.07944)
• Keywords: diffusion
• Abstract: Several text-to-video diffusion models have demonstrated commendable capabilities in synthesizing high-quality video content. However, it remains a formidable challenge pertaining to maintaining temporal consistency and ensuring action smoothness throughout the generated sequences. In this paper, we present an innovative video generation AI agent that harnesses the power of Sora-inspired multimodal learning to build skilled world models framework based on textual prompts and accompanying images. The framework includes two parts: prompt enhancer and full video translation. The first part employs the capabilities of ChatGPT to meticulously distill and proactively construct precise prompts for each subsequent step, thereby guaranteeing the utmost accuracy in prompt communication and accurate execution in following model operations. The second part employ compatible with existing advanced diffusion techniques to expansively generate and refine the key frame at the conclusion of a video. Then we can expertly harness the power of leading and trailing key frames to craft videos with enhanced temporal consistency and action smoothness. The experimental results confirm that our method has strong effectiveness and novelty in constructing world models from text and image inputs over the other methods.

Title: A Mathematical Framework for the Problem of Security for Cognition in Neurotechnology

• Authors: Bryce Allen Bagley
• Subjects: cs.CR, cs.CY, cs.ET, cs.LG, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2403.07945
• Pdf URL: https://arxiv.org/pdf/2403.07945
• Copy Paste: [[2403.07945]] A Mathematical Framework for the Problem of Security for Cognition in Neurotechnology(https://arxiv.org/abs/2403.07945)
• Keywords: security, privacy, attack
• Abstract: The rapid advancement in neurotechnology in recent years has created an emerging critical intersection between neurotechnology and security. Implantable devices, non-invasive monitoring, and non-invasive therapies all carry with them the prospect of violating the privacy and autonomy of individuals' cognition. A growing number of scientists and physicians have made calls to address this issue -- which we term Cognitive Security -- but applied efforts have been limited. A major barrier hampering scientific and engineering efforts to address Cognitive Security is the lack of a clear means of describing and analyzing relevant problems. In this paper we develop Cognitive Security, a mathematical framework which enables such description and analysis by drawing on methods and results from multiple fields. We demonstrate certain statistical properties which have significant implications for Cognitive Security, and then present descriptions of the algorithmic problems faced by attackers attempting to violate privacy and autonomy, and defenders attempting to obstruct such attempts.

Title: AesopAgent: Agent-driven Evolutionary System on Story-to-Video Production

• Authors: Jiuniu Wang, Zehua Du, Yuyuan Zhao, Bo Yuan, Kexiang Wang, Jian Liang, Yaxi Zhao, Yihen Lu, Gengliang Li, Junlong Gao, Xin Tu, Zhenyu Guo
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.07952
• Pdf URL: https://arxiv.org/pdf/2403.07952
• Copy Paste: [[2403.07952]] AesopAgent: Agent-driven Evolutionary System on Story-to-Video Production(https://arxiv.org/abs/2403.07952)
• Keywords: generative
• Abstract: The Agent and AIGC (Artificial Intelligence Generated Content) technologies have recently made significant progress. We propose AesopAgent, an Agent-driven Evolutionary System on Story-to-Video Production. AesopAgent is a practical application of agent technology for multimodal content generation. The system integrates multiple generative capabilities within a unified framework, so that individual users can leverage these modules easily. This innovative system would convert user story proposals into scripts, images, and audio, and then integrate these multimodal contents into videos. Additionally, the animating units (e.g., Gen-2 and Sora) could make the videos more infectious. The AesopAgent system could orchestrate task workflow for video generation, ensuring that the generated video is both rich in content and coherent. This system mainly contains two layers, i.e., the Horizontal Layer and the Utility Layer. In the Horizontal Layer, we introduce a novel RAG-based evolutionary system that optimizes the whole video generation workflow and the steps within the workflow. It continuously evolves and iteratively optimizes workflow by accumulating expert experience and professional knowledge, including optimizing the LLM prompts and utilities usage. The Utility Layer provides multiple utilities, leading to consistent image generation that is visually coherent in terms of composition, characters, and style. Meanwhile, it provides audio and special effects, integrating them into expressive and logically arranged videos. Overall, our AesopAgent achieves state-of-the-art performance compared with many previous works in visual storytelling. Our AesopAgent is designed for convenient service for individual users, which is available on the following page: https://aesopai.github.io/.

Title: DeepCDCL: An CDCL-based Neural Network Verification Framework

• Authors: Zongxin Liu, Pengfei Yang, Lijun Zhang, Xiaowei Huang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.07956
• Pdf URL: https://arxiv.org/pdf/2403.07956
• Copy Paste: [[2403.07956]] DeepCDCL: An CDCL-based Neural Network Verification Framework(https://arxiv.org/abs/2403.07956)
• Keywords: security
• Abstract: Neural networks in safety-critical applications face increasing safety and security concerns due to their susceptibility to little disturbance. In this paper, we propose DeepCDCL, a novel neural network verification framework based on the Conflict-Driven Clause Learning (CDCL) algorithm. We introduce an asynchronous clause learning and management structure, reducing redundant time consumption compared to the direct application of the CDCL framework. Furthermore, we also provide a detailed evaluation of the performance of our approach on the ACAS Xu and MNIST datasets, showing that a significant speed-up is achieved in most cases.

Title: An Interpretable Generalization Mechanism for Accurately Detecting Anomaly and Identifying Networking Intrusion Techniques

• Authors: Hao-Ting Pai, Yu-Hsuan Kang, Wen-Cheng Chung
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.07959
• Pdf URL: https://arxiv.org/pdf/2403.07959
• Copy Paste: [[2403.07959]] An Interpretable Generalization Mechanism for Accurately Detecting Anomaly and Identifying Networking Intrusion Techniques(https://arxiv.org/abs/2403.07959)
• Keywords: security, attack, interpretability
• Abstract: Recent advancements in Intrusion Detection Systems (IDS), integrating Explainable AI (XAI) methodologies, have led to notable improvements in system performance via precise feature selection. However, a thorough understanding of cyber-attacks requires inherently explainable decision-making processes within IDS. In this paper, we present the Interpretable Generalization Mechanism (IG), poised to revolutionize IDS capabilities. IG discerns coherent patterns, making it interpretable in distinguishing between normal and anomalous network traffic. Further, the synthesis of coherent patterns sheds light on intricate intrusion pathways, providing essential insights for cybersecurity forensics. By experiments with real-world datasets NSL-KDD, UNSW-NB15, and UKM-IDS20, IG is accurate even at a low ratio of training-to-test. With 10%-to-90%, IG achieves Precision (PRE)=0.93, Recall (REC)=0.94, and Area Under Curve (AUC)=0.94 in NSL-KDD; PRE=0.98, REC=0.99, and AUC=0.99 in UNSW-NB15; and PRE=0.98, REC=0.98, and AUC=0.99 in UKM-IDS20. Notably, in UNSW-NB15, IG achieves REC=1.0 and at least PRE=0.98 since 40%-to-60%; in UKM-IDS20, IG achieves REC=1.0 and at least PRE=0.88 since 20%-to-80%. Importantly, in UKM-IDS20, IG successfully identifies all three anomalous instances without prior exposure, demonstrating its generalization capabilities. These results and inferences are reproducible. In sum, IG showcases superior generalization by consistently performing well across diverse datasets and training-to-test ratios (from 10%-to-90% to 90%-to-10%), and excels in identifying novel anomalies without prior exposure. Its interpretability is enhanced by coherent evidence that accurately distinguishes both normal and anomalous activities, significantly improving detection accuracy and reducing false alarms, thereby strengthening IDS reliability and trustworthiness.

Title: Conditional computation in neural networks: principles and research trends

• Authors: Simone Scardapane, Alessandro Baiocchi, Alessio Devoto, Valerio Marsocci, Pasquale Minervini, Jary Pomponi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.07965
• Pdf URL: https://arxiv.org/pdf/2403.07965
• Copy Paste: [[2403.07965]] Conditional computation in neural networks: principles and research trends(https://arxiv.org/abs/2403.07965)
• Keywords: explainability
• Abstract: This article summarizes principles and ideas from the emerging area of applying \textit{conditional computation} methods to the design of neural networks. In particular, we focus on neural networks that can dynamically activate or de-activate parts of their computational graph conditionally on their input. Examples include the dynamic selection of, e.g., input tokens, layers (or sets of layers), and sub-modules inside each layer (e.g., channels in a convolutional filter). We first provide a general formalism to describe these techniques in an uniform way. Then, we introduce three notable implementations of these principles: mixture-of-experts (MoEs) networks, token selection mechanisms, and early-exit neural networks. The paper aims to provide a tutorial-like introduction to this growing field. To this end, we analyze the benefits of these modular designs in terms of efficiency, explainability, and transfer learning, with a focus on emerging applicative areas ranging from automated scientific discovery to semantic communication.

Title: Applying ranking techniques for estimating influence of Earth variables on temperature forecast error

• Authors: M. Julia Flores, Melissa Ruiz-Vásquez, Ana Bastos, René Orth
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2403.07966
• Pdf URL: https://arxiv.org/pdf/2403.07966
• Copy Paste: [[2403.07966]] Applying ranking techniques for estimating influence of Earth variables on temperature forecast error(https://arxiv.org/abs/2403.07966)
• Keywords: robust
• Abstract: This paper describes how to analyze the influence of Earth system variables on the errors when providing temperature forecasts. The initial framework to get the data has been based on previous research work, which resulted in a very interesting discovery. However, the aforementioned study only worked on individual correlations of the variables with respect to the error. This research work is going to re-use the main ideas but introduce three main novelties: (1) applying a data science approach by a few representative locations; (2) taking advantage of the rankings created by Spearman correlation but enriching them with other metrics looking for a more robust ranking of the variables; (3) evaluation of the methodology by learning random forest models for regression with the distinct experimental variations. The main contribution is the framework that shows how to convert correlations into rankings and combine them into an aggregate ranking. We have carried out experiments on five chosen locations to analyze the behavior of this ranking-based methodology. The results show that the specific performance is dependent on the location and season, which is expected, and that this selection technique works properly with Random Forest models but can also improve simpler regression models such as Bayesian Ridge. This work also contributes with an extensive analysis of the results. We can conclude that this selection based on the top-k ranked variables seems promising for this real problem, and it could also be applied in other domains.

Title: KnowCoder: Coding Structured Knowledge into LLMs for Universal Information Extraction

• Authors: Zixuan Li, Yutao Zeng, Yuxin Zuo, Weicheng Ren, Wenxuan Liu, Miao Su, Yucan Guo, Yantao Liu, Xiang Li, Zhilei Hu, Long Bai, Wei Li, Yidan Liu, Pan Yang, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.07969
• Pdf URL: https://arxiv.org/pdf/2403.07969
• Copy Paste: [[2403.07969]] KnowCoder: Coding Structured Knowledge into LLMs for Universal Information Extraction(https://arxiv.org/abs/2403.07969)
• Keywords: extraction, large language model
• Abstract: In this paper, we propose KnowCoder, a Large Language Model (LLM) to conduct Universal Information Extraction (UIE) via code generation. KnowCoder aims to develop a kind of unified schema representation that LLMs can easily understand and an effective learning framework that encourages LLMs to follow schemas and extract structured knowledge accurately. To achieve these, KnowCoder introduces a code-style schema representation method to uniformly transform different schemas into Python classes, with which complex schema information, such as constraints among tasks in UIE, can be captured in an LLM-friendly manner. We further construct a code-style schema library covering over $\textbf{30,000}$ types of knowledge, which is the largest one for UIE, to the best of our knowledge. To ease the learning process of LLMs, KnowCoder contains a two-phase learning framework that enhances its schema understanding ability via code pretraining and its schema following ability via instruction tuning. After code pretraining on around $1.5$B automatically constructed data, KnowCoder already attains remarkable generalization ability and achieves relative improvements by $\textbf{49.8\%}$ F1, compared to LLaMA2, under the few-shot setting. After instruction tuning, KnowCoder further exhibits strong generalization ability on unseen schemas and achieves up to $\textbf{12.5\%}$ and $\textbf{21.9\%}$, compared to sota baselines, under the zero-shot setting and the low resource setting, respectively. Additionally, based on our unified schema representations, various human-annotated datasets can simultaneously be utilized to refine KnowCoder, which achieves significant improvements up to $\textbf{7.5\%}$ under the supervised setting.

Title: Do Agents Dream of Electric Sheep?: Improving Generalization in Reinforcement Learning through Generative Learning

• Authors: Giorgio Franceschelli, Mirco Musolesi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.07979
• Pdf URL: https://arxiv.org/pdf/2403.07979
• Copy Paste: [[2403.07979]] Do Agents Dream of Electric Sheep?: Improving Generalization in Reinforcement Learning through Generative Learning(https://arxiv.org/abs/2403.07979)
• Keywords: generative
• Abstract: The Overfitted Brain hypothesis suggests dreams happen to allow generalization in the human brain. Here, we ask if the same is true for reinforcement learning agents as well. Given limited experience in a real environment, we use imagination-based reinforcement learning to train a policy on dream-like episodes, where non-imaginative, predicted trajectories are modified through generative augmentations. Experiments on four ProcGen environments show that, compared to classic imagination and offline training on collected experience, our method can reach a higher level of generalization when dealing with sparsely rewarded environments.

Title: Real-time Surgical Instrument Segmentation in Video Using Point Tracking and Segment Anything

• Authors: Zijian Wu, Adam Schmidt, Peter Kazanzides, Septimiu E. Salcudean
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08003
• Pdf URL: https://arxiv.org/pdf/2403.08003
• Copy Paste: [[2403.08003]] Real-time Surgical Instrument Segmentation in Video Using Point Tracking and Segment Anything(https://arxiv.org/abs/2403.08003)
• Keywords: segmentation
• Abstract: The Segment Anything Model (SAM) is a powerful vision foundation model that is revolutionizing the traditional paradigm of segmentation. Despite this, a reliance on prompting each frame and large computational cost limit its usage in robotically assisted surgery. Applications, such as augmented reality guidance, require little user intervention along with efficient inference to be usable clinically. In this study, we address these limitations by adopting lightweight SAM variants to meet the speed requirement and employing fine-tuning techniques to enhance their generalization in surgical scenes. Recent advancements in Tracking Any Point (TAP) have shown promising results in both accuracy and efficiency, particularly when points are occluded or leave the field of view. Inspired by this progress, we present a novel framework that combines an online point tracker with a lightweight SAM model that is fine-tuned for surgical instrument segmentation. Sparse points within the region of interest are tracked and used to prompt SAM throughout the video sequence, providing temporal consistency. The quantitative results surpass the state-of-the-art semi-supervised video object segmentation method on the EndoVis 2015 dataset, with an over 25 FPS inference speed running on a single GeForce RTX 4060 GPU.

Title: IndicSTR12: A Dataset for Indic Scene Text Recognition

• Authors: Harsh Lunia, Ajoy Mondal, C V Jawahar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08007
• Pdf URL: https://arxiv.org/pdf/2403.08007
• Copy Paste: [[2403.08007]] IndicSTR12: A Dataset for Indic Scene Text Recognition(https://arxiv.org/abs/2403.08007)
• Keywords: robust
• Abstract: The importance of Scene Text Recognition (STR) in today's increasingly digital world cannot be overstated. Given the significance of STR, data intensive deep learning approaches that auto-learn feature mappings have primarily driven the development of STR solutions. Several benchmark datasets and substantial work on deep learning models are available for Latin languages to meet this need. On more complex, syntactically and semantically, Indian languages spoken and read by 1.3 billion people, there is less work and datasets available. This paper aims to address the Indian space's lack of a comprehensive dataset by proposing the largest and most comprehensive real dataset - IndicSTR12 - and benchmarking STR performance on 12 major Indian languages. A few works have addressed the same issue, but to the best of our knowledge, they focused on a small number of Indian languages. The size and complexity of the proposed dataset are comparable to those of existing Latin contemporaries, while its multilingualism will catalyse the development of robust text detection and recognition models. It was created specifically for a group of related languages with different scripts. The dataset contains over 27000 word-images gathered from various natural scenes, with over 1000 word-images for each language. Unlike previous datasets, the images cover a broader range of realistic conditions, including blur, illumination changes, occlusion, non-iconic texts, low resolution, perspective text etc. Along with the new dataset, we provide a high-performing baseline on three models - PARSeq, CRNN, and STARNet.

Title: Debatrix: Multi-dimensinal Debate Judge with Iterative Chronological Analysis Based on LLM

• Authors: Jingcong Liang, Rong Ye, Meng Han, Ruofei Lai, Xinyu Zhang, Xuanjing Huang, Zhongyu Wei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08010
• Pdf URL: https://arxiv.org/pdf/2403.08010
• Copy Paste: [[2403.08010]] Debatrix: Multi-dimensinal Debate Judge with Iterative Chronological Analysis Based on LLM(https://arxiv.org/abs/2403.08010)
• Keywords: large language model
• Abstract: How can we construct an automated debate judge to evaluate an extensive, vibrant, multi-turn debate? This task is challenging, as judging a debate involves grappling with lengthy texts, intricate argument relationships, and multi-dimensional assessments. At the same time, current research mainly focuses on short dialogues, rarely touching upon the evaluation of an entire debate. In this paper, by leveraging Large Language Models (LLMs), we propose Debatrix, which makes the analysis and assessment of multi-turn debates more aligned with majority preferences. Specifically, Debatrix features a vertical, iterative chronological analysis and a horizontal, multi-dimensional evaluation collaboration. To align with real-world debate scenarios, we introduced the PanelBench benchmark, comparing our system's performance to actual debate outcomes. The findings indicate a notable enhancement over directly using LLMs for debate evaluation. Source code and benchmark data are available online at https://github.com/ljcleo/Debatrix .

Title: Gujarati-English Code-Switching Speech Recognition using ensemble prediction of spoken language

• Authors: Yash Sharma, Basil Abraham, Preethi Jyothi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08011
• Pdf URL: https://arxiv.org/pdf/2403.08011
• Copy Paste: [[2403.08011]] Gujarati-English Code-Switching Speech Recognition using ensemble prediction of spoken language(https://arxiv.org/abs/2403.08011)
• Keywords: explainability, transformer
• Abstract: An important and difficult task in code-switched speech recognition is to recognize the language, as lots of words in two languages can sound similar, especially in some accents. We focus on improving performance of end-to-end Automatic Speech Recognition models by conditioning transformer layers on language ID of words and character in the output in an per layer supervised manner. To this end, we propose two methods of introducing language specific parameters and explainability in the multi-head attention mechanism, and implement a Temporal Loss that helps maintain continuity in input alignment. Despite being unable to reduce WER significantly, our method shows promise in predicting the correct language from just spoken data. We introduce regularization in the language prediction by dropping LID in the sequence, which helps align long repeated output sequences.

Title: Red Teaming Models for Hyperspectral Image Analysis Using Explainable AI

• Authors: Vladimir Zaigrajew, Hubert Baniecki, Lukasz Tulczyjew, Agata M. Wijata, Jakub Nalepa, Nicolas Longépé, Przemyslaw Biecek
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08017
• Pdf URL: https://arxiv.org/pdf/2403.08017
• Copy Paste: [[2403.08017]] Red Teaming Models for Hyperspectral Image Analysis Using Explainable AI(https://arxiv.org/abs/2403.08017)
• Keywords: robust
• Abstract: Remote sensing (RS) applications in the space domain demand machine learning (ML) models that are reliable, robust, and quality-assured, making red teaming a vital approach for identifying and exposing potential flaws and biases. Since both fields advance independently, there is a notable gap in integrating red teaming strategies into RS. This paper introduces a methodology for examining ML models operating on hyperspectral images within the HYPERVIEW challenge, focusing on soil parameters' estimation. We use post-hoc explanation methods from the Explainable AI (XAI) domain to critically assess the best performing model that won the HYPERVIEW challenge and served as an inspiration for the model deployed on board the INTUITION-1 hyperspectral mission. Our approach effectively red teams the model by pinpointing and validating key shortcomings, constructing a model that achieves comparable performance using just 1% of the input features and a mere up to 5% performance loss. Additionally, we propose a novel way of visualizing explanations that integrate domain-specific information about hyperspectral bands (wavelengths) and data transformations to better suit interpreting models for hyperspectral image analysis.

Title: Learning Data Association for Multi-Object Tracking using Only Coordinates

• Authors: Mehdi Miah, Guillaume-Alexandre Bilodeau, Nicolas Saunier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08018
• Pdf URL: https://arxiv.org/pdf/2403.08018
• Copy Paste: [[2403.08018]] Learning Data Association for Multi-Object Tracking using Only Coordinates(https://arxiv.org/abs/2403.08018)
• Keywords: transformer
• Abstract: We propose a novel Transformer-based module to address the data association problem for multi-object tracking. From detections obtained by a pretrained detector, this module uses only coordinates from bounding boxes to estimate an affinity score between pairs of tracks extracted from two distinct temporal windows. This module, named TWiX, is trained on sets of tracks with the objective of discriminating pairs of tracks coming from the same object from those which are not. Our module does not use the intersection over union measure, nor does it requires any motion priors or any camera motion compensation technique. By inserting TWiX within an online cascade matching pipeline, our tracker C-TWiX achieves state-of-the-art performance on the DanceTrack and KITTIMOT datasets, and gets competitive results on the MOT17 dataset. The code will be made available upon publication.

Title: McCatch: Scalable Microcluster Detection in Dimensional and Nondimensional Datasets

• Authors: Braulio V. Sánchez Vinces, Robson L. F. Cordeiro, Christos Faloutsos
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08027
• Pdf URL: https://arxiv.org/pdf/2403.08027
• Copy Paste: [[2403.08027]] McCatch: Scalable Microcluster Detection in Dimensional and Nondimensional Datasets(https://arxiv.org/abs/2403.08027)
• Keywords: attack
• Abstract: How could we have an outlier detector that works even with nondimensional data, and ranks together both singleton microclusters ('one-off' outliers) and nonsingleton microclusters by their anomaly scores? How to obtain scores that are principled in one scalable and 'hands-off' manner? Microclusters of outliers indicate coalition or repetition in fraud activities, etc.; their identification is thus highly desirable. This paper presents McCatch: a new algorithm that detects microclusters by leveraging our proposed 'Oracle' plot (1NN Distance versus Group 1NN Distance). We study 31 real and synthetic datasets with up to 1M data elements to show that McCatch is the only method that answers both of the questions above; and, it outperforms 11 other methods, especially when the data has nonsingleton microclusters or is nondimensional. We also showcase McCatch's ability to detect meaningful microclusters in graphs, fingerprints, logs of network connections, text data, and satellite imagery. For example, it found a 30-elements microcluster of confirmed 'Denial of Service' attacks in the network logs, taking only ~3 minutes for 222K data elements on a stock desktop.

Title: LG-Traj: LLM Guided Pedestrian Trajectory Prediction

• Authors: Pranav Singh Chib, Pravendra Singh
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08032
• Pdf URL: https://arxiv.org/pdf/2403.08032
• Copy Paste: [[2403.08032]] LG-Traj: LLM Guided Pedestrian Trajectory Prediction(https://arxiv.org/abs/2403.08032)
• Keywords: transformer, large language model
• Abstract: Accurate pedestrian trajectory prediction is crucial for various applications, and it requires a deep understanding of pedestrian motion patterns in dynamic environments. However, existing pedestrian trajectory prediction methods still need more exploration to fully leverage these motion patterns. This paper investigates the possibilities of using Large Language Models (LLMs) to improve pedestrian trajectory prediction tasks by inducing motion cues. We introduce LG-Traj, a novel approach incorporating LLMs to generate motion cues present in pedestrian past/observed trajectories. Our approach also incorporates motion cues present in pedestrian future trajectories by clustering future trajectories of training data using a mixture of Gaussians. These motion cues, along with pedestrian coordinates, facilitate a better understanding of the underlying representation. Furthermore, we utilize singular value decomposition to augment the observed trajectories, incorporating them into the model learning process to further enhance representation learning. Our method employs a transformer-based architecture comprising a motion encoder to model motion patterns and a social decoder to capture social interactions among pedestrians. We demonstrate the effectiveness of our approach on popular pedestrian trajectory prediction benchmarks, namely ETH-UCY and SDD, and present various ablation experiments to validate our approach.

Title: Harnessing Artificial Intelligence to Combat Online Hate: Exploring the Challenges and Opportunities of Large Language Models in Hate Speech Detection

• Authors: Tharindu Kumarage, Amrita Bhattacharjee, Joshua Garland
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08035
• Pdf URL: https://arxiv.org/pdf/2403.08035
• Copy Paste: [[2403.08035]] Harnessing Artificial Intelligence to Combat Online Hate: Exploring the Challenges and Opportunities of Large Language Models in Hate Speech Detection(https://arxiv.org/abs/2403.08035)
• Keywords: large language model
• Abstract: Large language models (LLMs) excel in many diverse applications beyond language generation, e.g., translation, summarization, and sentiment analysis. One intriguing application is in text classification. This becomes pertinent in the realm of identifying hateful or toxic speech -- a domain fraught with challenges and ethical dilemmas. In our study, we have two objectives: firstly, to offer a literature review revolving around LLMs as classifiers, emphasizing their role in detecting and classifying hateful or toxic content. Subsequently, we explore the efficacy of several LLMs in classifying hate speech: identifying which LLMs excel in this task as well as their underlying attributes and training. Providing insight into the factors that contribute to an LLM proficiency (or lack thereof) in discerning hateful content. By combining a comprehensive literature review with an empirical analysis, our paper strives to shed light on the capabilities and constraints of LLMs in the crucial domain of hate speech detection.

Title: A Review of Cybersecurity Incidents in the Food and Agriculture Sector

• Authors: Ajay Kulkarni, Yingjie Wang, Munisamy Gopinath, Dan Sobien, Abdul Rahman, Feras A. Batarseh
• Subjects: cs.CR, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.08036
• Pdf URL: https://arxiv.org/pdf/2403.08036
• Copy Paste: [[2403.08036]] A Review of Cybersecurity Incidents in the Food and Agriculture Sector(https://arxiv.org/abs/2403.08036)
• Keywords: security
• Abstract: The increasing utilization of emerging technologies in the Food & Agriculture (FA) sector has heightened the need for security to minimize cyber risks. Considering this aspect, this manuscript reviews disclosed and documented cybersecurity incidents in the FA sector. For this purpose, thirty cybersecurity incidents were identified, which took place between July 2011 and April 2023. The details of these incidents are reported from multiple sources such as: the private industry and flash notifications generated by the Federal Bureau of Investigation (FBI), internal reports from the affected organizations, and available media sources. Considering the available information, a brief description of the security threat, ransom amount, and impact on the organization are discussed for each incident. This review reports an increased frequency of cybersecurity threats to the FA sector. To minimize these cyber risks, popular cybersecurity frameworks and recent agriculture-specific cybersecurity solutions are also discussed. Further, the need for AI assurance in the FA sector is explained, and the Farmer-Centered AI (FCAI) framework is proposed. The main aim of the FCAI framework is to support farmers in decision-making for agricultural production, by incorporating AI assurance. Lastly, the effects of the reported cyber incidents on other critical infrastructures, food security, and the economy are noted, along with specifying the open issues for future development.

Title: Big City Bias: Evaluating the Impact of Metropolitan Size on Computational Job Market Abilities of Language Models

• Authors: Charlie Campanella, Rob van der Goot
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08046
• Pdf URL: https://arxiv.org/pdf/2403.08046
• Copy Paste: [[2403.08046]] Big City Bias: Evaluating the Impact of Metropolitan Size on Computational Job Market Abilities of Language Models(https://arxiv.org/abs/2403.08046)
• Keywords: large language model
• Abstract: Large language models (LLMs) have emerged as a useful technology for job matching, for both candidates and employers. Job matching is often based on a particular geographic location, such as a city or region. However, LLMs have known biases, commonly derived from their training data. In this work, we aim to quantify the metropolitan size bias encoded within large language models, evaluating zero-shot salary, employer presence, and commute duration predictions in 384 of the United States' metropolitan regions. Across all benchmarks, we observe negative correlations between the metropolitan size and the performance of the LLMS, indicating that smaller regions are indeed underrepresented. More concretely, the smallest 10 metropolitan regions show upwards of 300% worse benchmark performance than the largest 10.

Title: CHAI: Clustered Head Attention for Efficient LLM Inference

• Authors: Saurabh Agarwal, Bilge Acun, Basil Homer, Mostafa Elhoushi, Yejin Lee, Shivaram Venkataraman, Dimitris Papailiopoulos, Carole-Jean Wu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08058
• Pdf URL: https://arxiv.org/pdf/2403.08058
• Copy Paste: [[2403.08058]] CHAI: Clustered Head Attention for Efficient LLM Inference(https://arxiv.org/abs/2403.08058)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) with hundreds of billions of parameters have transformed the field of machine learning. However, serving these models at inference time is both compute and memory intensive, where a single request can require multiple GPUs and tens of Gigabytes of memory. Multi-Head Attention is one of the key components of LLMs, which can account for over 50% of LLMs memory and compute requirement. We observe that there is a high amount of redundancy across heads on which tokens they pay attention to. Based on this insight, we propose Clustered Head Attention (CHAI). CHAI combines heads with a high amount of correlation for self-attention at runtime, thus reducing both memory and compute. In our experiments, we show that CHAI is able to reduce the memory requirements for storing K,V cache by up to 21.4% and inference time latency by up to 1.73x without any fine-tuning required. CHAI achieves this with a maximum 3.2% deviation in accuracy across 3 different models (i.e. OPT-66B, LLAMA-7B, LLAMA-33B) and 5 different evaluation datasets.

Title: FluoroSAM: A Language-aligned Foundation Model for X-ray Image Segmentation

• Authors: Benjamin D. Killeen, Liam J. Wang, Han Zhang, Mehran Armand, Russell H. Taylor, Greg Osgood, Mathias Unberath
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08059
• Pdf URL: https://arxiv.org/pdf/2403.08059
• Copy Paste: [[2403.08059]] FluoroSAM: A Language-aligned Foundation Model for X-ray Image Segmentation(https://arxiv.org/abs/2403.08059)
• Keywords: segmentation
• Abstract: Automated X-ray image segmentation would accelerate research and development in diagnostic and interventional precision medicine. Prior efforts have contributed task-specific models capable of solving specific image analysis problems, but the utility of these models is restricted to their particular task domain, and expanding to broader use requires additional data, labels, and retraining efforts. Recently, foundation models (FMs) -- machine learning models trained on large amounts of highly variable data thus enabling broad applicability -- have emerged as promising tools for automated image analysis. Existing FMs for medical image analysis focus on scenarios and modalities where objects are clearly defined by visually apparent boundaries, such as surgical tool segmentation in endoscopy. X-ray imaging, by contrast, does not generally offer such clearly delineated boundaries or structure priors. During X-ray image formation, complex 3D structures are projected in transmission onto the imaging plane, resulting in overlapping features of varying opacity and shape. To pave the way toward an FM for comprehensive and automated analysis of arbitrary medical X-ray images, we develop FluoroSAM, a language-aligned variant of the Segment-Anything Model, trained from scratch on 1.6M synthetic X-ray images. FluoroSAM is trained on data including masks for 128 organ types and 464 non-anatomical objects, such as tools and implants. In real X-ray images of cadaveric specimens, FluoroSAM is able to segment bony anatomical structures based on text-only prompting with 0.51 and 0.79 DICE with point-based refinement, outperforming competing SAM variants for all structures. FluoroSAM is also capable of zero-shot generalization to segmenting classes beyond the training set thanks to its language alignment, which we demonstrate for full lung segmentation on real chest X-rays.

Title: SCALHEALTH: Scalable Blockchain Integration for Secure IoT Healthcare Systems

• Authors: Mehrzad Mohammadi, Reza Javan, Mohammad Beheshti-Atashgah, Mohammad Reza Aref
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.08068
• Pdf URL: https://arxiv.org/pdf/2403.08068
• Copy Paste: [[2403.08068]] SCALHEALTH: Scalable Blockchain Integration for Secure IoT Healthcare Systems(https://arxiv.org/abs/2403.08068)
• Keywords: secure, privacy
• Abstract: Internet of Things (IoT) devices are capable of allowing for far-reaching access to and evaluation of patient data to monitor health and diagnose from a distance. An electronic healthcare system that checks patient data, prepares medicines and provides financial assistance is necessary. Providing safe data transmission, monitoring, decentralization, preserving patient privacy, and maintaining confidentiality are essential to an electronic healthcare system. In this study, we introduce (SCALHEALTH) which is a blockchain-based scheme of the Hyperledger Fabric consortium. In this study, we use authentication to agree on a common key for data encryption to send data confidentially. Also, sending data through IPFS is decentralized. Non-fungible token (NFT) is used to send patient prescriptions to pharmacies and insurance companies to ensure the authenticity of patient prescriptions. As the system's main body, blockchain creates authorization and validation for all devices and institutions. Also, all metadata in the system is recorded on the blockchain to maintain integrity, transparency, and timely data monitoring. The proposed study uses two types of blockchain: a health blockchain and a financial blockchain. The financial blockchain is for financial transactions and is based on Ethereum. The health blockchain also introduces a mechanism that allows several blockchains to be active in parallel, instead of only one blockchain. The prototype of this mechanism is simulated in two scenarios. In comparison to the normal state, the proposed plan has superior results.

Title: A Multimodal Intermediate Fusion Network with Manifold Learning for Stress Detection

• Authors: Morteza Bodaghi, Majid Hosseini, Raju Gottumukkala
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08077
• Pdf URL: https://arxiv.org/pdf/2403.08077
• Copy Paste: [[2403.08077]] A Multimodal Intermediate Fusion Network with Manifold Learning for Stress Detection(https://arxiv.org/abs/2403.08077)
• Keywords: biometric
• Abstract: Multimodal deep learning methods capture synergistic features from multiple modalities and have the potential to improve accuracy for stress detection compared to unimodal methods. However, this accuracy gain typically comes from high computational cost due to the high-dimensional feature spaces, especially for intermediate fusion. Dimensionality reduction is one way to optimize multimodal learning by simplifying data and making the features more amenable to processing and analysis, thereby reducing computational complexity. This paper introduces an intermediate multimodal fusion network with manifold learning-based dimensionality reduction. The multimodal network generates independent representations from biometric signals and facial landmarks through 1D-CNN and 2D-CNN. Finally, these features are fused and fed to another 1D-CNN layer, followed by a fully connected dense layer. We compared various dimensionality reduction techniques for different variations of unimodal and multimodal networks. We observe that the intermediate-level fusion with the Multi-Dimensional Scaling (MDS) manifold method showed promising results with an accuracy of 96.00\% in a Leave-One-Subject-Out Cross-Validation (LOSO-CV) paradigm over other dimensional reduction methods. MDS had the highest computational cost among manifold learning methods. However, while outperforming other networks, it managed to reduce the computational cost of the proposed networks by 25\% when compared to six well-known conventional feature selection methods used in the preprocessing step.

Title: Mechanics of Next Token Prediction with Self-Attention

• Authors: Yingcong Li, Yixiao Huang, M. Emrullah Ildiz, Ankit Singh Rawat, Samet Oymak
• Subjects: cs.LG, cs.AI, cs.CL, math.OC
• Abstract URL: https://arxiv.org/abs/2403.08081
• Pdf URL: https://arxiv.org/pdf/2403.08081
• Copy Paste: [[2403.08081]] Mechanics of Next Token Prediction with Self-Attention(https://arxiv.org/abs/2403.08081)
• Keywords: transformer
• Abstract: Transformer-based language models are trained on large datasets to predict the next token given an input sequence. Despite this simple training objective, they have led to revolutionary advances in natural language processing. Underlying this success is the self-attention mechanism. In this work, we ask: $\textit{What}$ $\textit{does}$ $\textit{a}$ $\textit{single}$ $\textit{self-attention}$ $\textit{layer}$ $\textit{learn}$ $\textit{from}$ $\textit{next-token}$ $\textit{prediction?}$ We show that training self-attention with gradient descent learns an automaton which generates the next token in two distinct steps: $\textbf{(1)}$ $\textbf{Hard}$ $\textbf{retrieval:}$ Given input sequence, self-attention precisely selects the $\textit{high-priority}$ $\textit{input}$ $\textit{tokens}$ associated with the last input token. $\textbf{(2)}$ $\textbf{Soft}$ $\textbf{composition:}$ It then creates a convex combination of the high-priority tokens from which the next token can be sampled. Under suitable conditions, we rigorously characterize these mechanics through a directed graph over tokens extracted from the training data. We prove that gradient descent implicitly discovers the strongly-connected components (SCC) of this graph and self-attention learns to retrieve the tokens that belong to the highest-priority SCC available in the context window. Our theory relies on decomposing the model weights into a directional component and a finite component that correspond to hard retrieval and soft composition steps respectively. This also formalizes a related implicit bias formula conjectured in [Tarzanagh et al. 2023]. We hope that these findings shed light on how self-attention processes sequential data and pave the path toward demystifying more complex architectures.

Title: Mitigating the Impact of Attribute Editing on Face Recognition

• Authors: Sudipta Banerjee, Sai Pranaswi Mullangi, Shruti Wagle, Chinmay Hegde, Nasir Memon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08092
• Pdf URL: https://arxiv.org/pdf/2403.08092
• Copy Paste: [[2403.08092]] Mitigating the Impact of Attribute Editing on Face Recognition(https://arxiv.org/abs/2403.08092)
• Keywords: generative, segmentation
• Abstract: Facial attribute editing using generative models can impair automated face recognition. This degradation persists even with recent identity-preserving models such as InstantID. To mitigate this issue, we propose two techniques that perform local and global attribute editing. Local editing operates on the finer details via a regularization-free method based on ControlNet conditioned on depth maps and auxiliary semantic segmentation masks. Global editing operates on coarser details via a regularization-based method guided by custom loss and regularization set. In this work, we empirically ablate twenty-six facial semantic, demographic and expression-based attributes altered using state-of-the-art generative models and evaluate them using ArcFace and AdaFace matchers on CelebA, CelebAMaskHQ and LFW datasets. Finally, we use LLaVA, a vision-language framework for attribute prediction to validate our editing techniques. Our methods outperform SoTA (BLIP, InstantID) at facial editing while retaining identity.

Title: Efficient Language Model Architectures for Differentially Private Federated Learning

• Authors: Jae Hun Ro, Srinadh Bhojanapalli, Zheng Xu, Yanxiang Zhang, Ananda Theertha Suresh
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.08100
• Pdf URL: https://arxiv.org/pdf/2403.08100
• Copy Paste: [[2403.08100]] Efficient Language Model Architectures for Differentially Private Federated Learning(https://arxiv.org/abs/2403.08100)
• Keywords: privacy, federate, transformer
• Abstract: Cross-device federated learning (FL) is a technique that trains a model on data distributed across typically millions of edge devices without data leaving the devices. SGD is the standard client optimizer for on device training in cross-device FL, favored for its memory and computational efficiency. However, in centralized training of neural language models, adaptive optimizers are preferred as they offer improved stability and performance. In light of this, we ask if language models can be modified such that they can be efficiently trained with SGD client optimizers and answer this affirmatively. We propose a scale-invariant Coupled Input Forget Gate (SI CIFG) recurrent network by modifying the sigmoid and tanh activations in the recurrent cell and show that this new model converges faster and achieves better utility than the standard CIFG recurrent model in cross-device FL in large scale experiments. We further show that the proposed scale invariant modification also helps in federated learning of larger transformer models. Finally, we demonstrate the scale invariant modification is also compatible with other non-adaptive algorithms. Particularly, our results suggest an improved privacy utility trade-off in federated learning with differential privacy.

Title: Contextual Clarity: Generating Sentences with Transformer Models using Context-Reverso Data

• Authors: Ruslan Musaev
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08103
• Pdf URL: https://arxiv.org/pdf/2403.08103
• Copy Paste: [[2403.08103]] Contextual Clarity: Generating Sentences with Transformer Models using Context-Reverso Data(https://arxiv.org/abs/2403.08103)
• Keywords: transformer
• Abstract: In the age of information abundance, the ability to provide users with contextually relevant and concise information is crucial. Keyword in Context (KIC) generation is a task that plays a vital role in and generation applications, such as search engines, personal assistants, and content summarization. In this paper, we present a novel approach to generating unambiguous and brief sentence-contexts for given keywords using the T5 transformer model, leveraging data obtained from the Context-Reverso API. The code is available at https://github.com/Rusamus/word2context/tree/main .

Title: TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection

• Authors: Hanning Chen, Wenjun Huang, Yang Ni, Sanggeon Yun, Fei Wen, Hugo Latapie, Mohsen Imani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08108
• Pdf URL: https://arxiv.org/pdf/2403.08108
• Copy Paste: [[2403.08108]] TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection(https://arxiv.org/abs/2403.08108)
• Keywords: transformer
• Abstract: Task-oriented object detection aims to find objects suitable for accomplishing specific tasks. As a challenging task, it requires simultaneous visual data processing and reasoning under ambiguous semantics. Recent solutions are mainly all-in-one models. However, the object detection backbones are pre-trained without text supervision. Thus, to incorporate task requirements, their intricate models undergo extensive learning on a highly imbalanced and scarce dataset, resulting in capped performance, laborious training, and poor generalizability. In contrast, we propose TaskCLIP, a more natural two-stage design composed of general object detection and task-guided object selection. Particularly for the latter, we resort to the recently successful large Vision-Language Models (VLMs) as our backbone, which provides rich semantic knowledge and a uniform embedding space for images and texts. Nevertheless, the naive application of VLMs leads to sub-optimal quality, due to the misalignment between embeddings of object images and their visual attributes, which are mainly adjective phrases. To this end, we design a transformer-based aligner after the pre-trained VLMs to re-calibrate both embeddings. Finally, we employ a trainable score function to post-process the VLM matching results for object selection. Experimental results demonstrate that our TaskCLIP outperforms the state-of-the-art DETR-based model TOIST by 3.5% and only requires a single NVIDIA RTX 4090 for both training and inference.

Title: Towards Independence Criterion in Machine Unlearning of Features and Labels

• Authors: Ling Han, Nanqing Luo, Hao Huang, Jing Chen, Mary-Anne Hartley
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.08124
• Pdf URL: https://arxiv.org/pdf/2403.08124
• Copy Paste: [[2403.08124]] Towards Independence Criterion in Machine Unlearning of Features and Labels(https://arxiv.org/abs/2403.08124)
• Keywords: privacy, protect, robust
• Abstract: This work delves into the complexities of machine unlearning in the face of distributional shifts, particularly focusing on the challenges posed by non-uniform feature and label removal. With the advent of regulations like the GDPR emphasizing data privacy and the right to be forgotten, machine learning models face the daunting task of unlearning sensitive information without compromising their integrity or performance. Our research introduces a novel approach that leverages influence functions and principles of distributional independence to address these challenges. By proposing a comprehensive framework for machine unlearning, we aim to ensure privacy protection while maintaining model performance and adaptability across varying distributions. Our method not only facilitates efficient data removal but also dynamically adjusts the model to preserve its generalization capabilities. Through extensive experimentation, we demonstrate the efficacy of our approach in scenarios characterized by significant distributional shifts, making substantial contributions to the field of machine unlearning. This research paves the way for developing more resilient and adaptable unlearning techniques, ensuring models remain robust and accurate in the dynamic landscape of data privacy and machine learning.

Title: Q-SLAM: Quadric Representations for Monocular SLAM

• Authors: Chensheng Peng, Chenfeng Xu, Yue Wang, Mingyu Ding, Heng Yang, Masayoshi Tomizuka, Kurt Keutzer, Marco Pavone, Wei Zhan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08125
• Pdf URL: https://arxiv.org/pdf/2403.08125
• Copy Paste: [[2403.08125]] Q-SLAM: Quadric Representations for Monocular SLAM(https://arxiv.org/abs/2403.08125)
• Keywords: transformer
• Abstract: Monocular SLAM has long grappled with the challenge of accurately modeling 3D geometries. Recent advances in Neural Radiance Fields (NeRF)-based monocular SLAM have shown promise, yet these methods typically focus on novel view synthesis rather than precise 3D geometry modeling. This focus results in a significant disconnect between NeRF applications, i.e., novel-view synthesis and the requirements of SLAM. We identify that the gap results from the volumetric representations used in NeRF, which are often dense and noisy. In this study, we propose a novel approach that reimagines volumetric representations through the lens of quadric forms. We posit that most scene components can be effectively represented as quadric planes. Leveraging this assumption, we reshape the volumetric representations with million of cubes by several quadric planes, which leads to more accurate and efficient modeling of 3D scenes in SLAM contexts. Our method involves two key steps: First, we use the quadric assumption to enhance coarse depth estimations obtained from tracking modules, e.g., Droid-SLAM. This step alone significantly improves depth estimation accuracy. Second, in the subsequent mapping phase, we diverge from previous NeRF-based SLAM methods that distribute sampling points across the entire volume space. Instead, we concentrate sampling points around quadric planes and aggregate them using a novel quadric-decomposed Transformer. Additionally, we introduce an end-to-end joint optimization strategy that synchronizes pose estimation with 3D reconstruction.

Title: Information Leakage through Physical Layer Supply Voltage Coupling Vulnerability

• Authors: Sahan Sanjaya, Aruna Jayasena, Prabhat Mishra
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.08132
• Pdf URL: https://arxiv.org/pdf/2403.08132
• Copy Paste: [[2403.08132]] Information Leakage through Physical Layer Supply Voltage Coupling Vulnerability(https://arxiv.org/abs/2403.08132)
• Keywords: security, attack
• Abstract: Side-channel attacks exploit variations in non-functional behaviors to expose sensitive information across security boundaries. Existing methods leverage side-channels based on power consumption, electromagnetic radiation, silicon substrate coupling, and channels created by malicious implants. Power-based side-channel attacks are widely known for extracting information from data processed within a device while assuming that an attacker has physical access or the ability to modify the device. In this paper, we introduce a novel side-channel vulnerability that leaks data-dependent power variations through physical layer supply voltage coupling (PSVC). Unlike traditional power side-channel attacks, the proposed vulnerability allows an adversary to mount an attack and extract information without modifying the device. We assess the effectiveness of PSVC vulnerability through three case studies, demonstrating several end-to-end attacks on general-purpose microcontrollers with varying adversary capabilities. These case studies provide evidence for the existence of PSVC vulnerability, its applicability for on-chip as well as on-board side-channel attacks, and how it can eliminate the need for physical access to the target device, making it applicable to any off-the-shelf hardware. Our experiments also reveal that designing devices to operate at the lowest operational voltage significantly reduces the risk of PSVC side-channel vulnerability.

Title: ShadowRemovalNet: Efficient Real-Time Shadow Removal

• Authors: Alzayat Saleh, Alex Olsen, Jake Wood, Bronson Philippa, Mostafa Rahimi Azghadi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08142
• Pdf URL: https://arxiv.org/pdf/2403.08142
• Copy Paste: [[2403.08142]] ShadowRemovalNet: Efficient Real-Time Shadow Removal(https://arxiv.org/abs/2403.08142)
• Keywords: robust, generative
• Abstract: Shadows significantly impact computer vision tasks, particularly in outdoor environments. State-of-the-art shadow removal methods are typically too computationally intensive for real-time image processing on edge hardware. We propose ShadowRemovalNet, a novel method designed for real-time image processing on resource-constrained hardware. ShadowRemovalNet achieves significantly higher frame rates compared to existing methods, making it suitable for real-time computer vision pipelines like those used in field robotics. Beyond speed, ShadowRemovalNet offers advantages in efficiency and simplicity, as it does not require a separate shadow mask during inference. ShadowRemovalNet also addresses challenges associated with Generative Adversarial Networks (GANs) for shadow removal, including artefacts, inaccurate mask estimations, and inconsistent supervision between shadow and boundary pixels. To address these limitations, we introduce a novel loss function that substantially reduces shadow removal errors. ShadowRemovalNet's efficiency and straightforwardness make it a robust and effective solution for real-time shadow removal in outdoor robotics and edge computing applications.

Title: Representing Molecules as Random Walks Over Interpretable Grammars

• Authors: Michael Sun, Minghao Guo, Weize Yuan, Veronika Thost, Crystal Elaine Owens, Aristotle Franklin Grosz, Sharvaa Selvan, Katelyn Zhou, Hassan Mohiuddin, Benjamin J Pedretti, Zachary P Smith, Jie Chen, Wojciech Matusik
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2403.08147
• Pdf URL: https://arxiv.org/pdf/2403.08147
• Copy Paste: [[2403.08147]] Representing Molecules as Random Walks Over Interpretable Grammars(https://arxiv.org/abs/2403.08147)
• Keywords: interpretability
• Abstract: Recent research in molecular discovery has primarily been devoted to small, drug-like molecules, leaving many similarly important applications in material design without adequate technology. These applications often rely on more complex molecular structures with fewer examples that are carefully designed using known substructures. We propose a data-efficient and interpretable model for representing and reasoning over such molecules in terms of graph grammars that explicitly describe the hierarchical design space featuring motifs to be the design basis. We present a novel representation in the form of random walks over the design space, which facilitates both molecule generation and property prediction. We demonstrate clear advantages over existing methods in terms of performance, efficiency, and synthesizability of predicted molecules, and we provide detailed insights into the method's chemical interpretability.

Title: Multiscale Low-Frequency Memory Network for Improved Feature Extraction in Convolutional Neural Networks

• Authors: Fuzhi Wu, Jiasong Wu, Youyong Kong, Chunfeng Yang, Guanyu Yang, Huazhong Shu, Guy Carrault, Lotfi Senhadji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08157
• Pdf URL: https://arxiv.org/pdf/2403.08157
• Copy Paste: [[2403.08157]] Multiscale Low-Frequency Memory Network for Improved Feature Extraction in Convolutional Neural Networks(https://arxiv.org/abs/2403.08157)
• Keywords: extraction, transformer, segmentation
• Abstract: Deep learning and Convolutional Neural Networks (CNNs) have driven major transformations in diverse research areas. However, their limitations in handling low-frequency information present obstacles in certain tasks like interpreting global structures or managing smooth transition images. Despite the promising performance of transformer structures in numerous tasks, their intricate optimization complexities highlight the persistent need for refined CNN enhancements using limited resources. Responding to these complexities, we introduce a novel framework, the Multiscale Low-Frequency Memory (MLFM) Network, with the goal to harness the full potential of CNNs while keeping their complexity unchanged. The MLFM efficiently preserves low-frequency information, enhancing performance in targeted computer vision tasks. Central to our MLFM is the Low-Frequency Memory Unit (LFMU), which stores various low-frequency data and forms a parallel channel to the core network. A key advantage of MLFM is its seamless compatibility with various prevalent networks, requiring no alterations to their original core structure. Testing on ImageNet demonstrated substantial accuracy improvements in multiple 2D CNNs, including ResNet, MobileNet, EfficientNet, and ConvNeXt. Furthermore, we showcase MLFM's versatility beyond traditional image classification by successfully integrating it into image-to-image translation tasks, specifically in semantic segmentation networks like FCN and U-Net. In conclusion, our work signifies a pivotal stride in the journey of optimizing the efficacy and efficiency of CNNs with limited resources. This research builds upon the existing CNN foundations and paves the way for future advancements in computer vision. Our codes are available at https://github.com/AlphaWuSeu/ MLFM.

Title: Versatile Defense Against Adversarial Attacks on Image Recognition

• Authors: Haibo Zhang, Zhihua Yao, Kouichi Sakurai
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.08170
• Pdf URL: https://arxiv.org/pdf/2403.08170
• Copy Paste: [[2403.08170]] Versatile Defense Against Adversarial Attacks on Image Recognition(https://arxiv.org/abs/2403.08170)
• Keywords: security, defense, attack, robust
• Abstract: Adversarial attacks present a significant security risk to image recognition tasks. Defending against these attacks in a real-life setting can be compared to the way antivirus software works, with a key consideration being how well the defense can adapt to new and evolving attacks. Another important factor is the resources involved in terms of time and cost for training defense models and updating the model database. Training many models that are specific to each type of attack can be time-consuming and expensive. Ideally, we should be able to train one single model that can handle a wide range of attacks. It appears that a defense method based on image-to-image translation may be capable of this. The proposed versatile defense approach in this paper only requires training one model to effectively resist various unknown adversarial attacks. The trained model has successfully improved the classification accuracy from nearly zero to an average of 86%, performing better than other defense methods proposed in prior studies. When facing the PGD attack and the MI-FGSM attack, versatile defense model even outperforms the attack-specific models trained based on these two attacks. The robustness check also shows that our versatile defense model performs stably regardless with the attack strength.

Title: Embedded Translations for Low-resource Automated Glossing

• Authors: Changbing Yang, Garrett Nicolai, Miikka Silfverberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08189
• Pdf URL: https://arxiv.org/pdf/2403.08189
• Copy Paste: [[2403.08189]] Embedded Translations for Low-resource Automated Glossing(https://arxiv.org/abs/2403.08189)
• Keywords: large language model
• Abstract: We investigate automatic interlinear glossing in low-resource settings. We augment a hard-attentional neural model with embedded translation information extracted from interlinear glossed text. After encoding these translations using large language models, specifically BERT and T5, we introduce a character-level decoder for generating glossed output. Aided by these enhancements, our model demonstrates an average improvement of 3.97\%-points over the previous state of the art on datasets from the SIGMORPHON 2023 Shared Task on Interlinear Glossing. In a simulated ultra low-resource setting, trained on as few as 100 sentences, our system achieves an average 9.78\%-point improvement over the plain hard-attentional baseline. These results highlight the critical role of translation information in boosting the system's performance, especially in processing and interpreting modest data sources. Our findings suggest a promising avenue for the documentation and preservation of languages, with our experiments on shared task datasets indicating significant advancements over the existing state of the art.

Title: MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension

• Authors: Xingyu Lu, He Cao, Zijing Liu, Shengyuan Bai, Leqing Chen, Yuan Yao, Hai-Tao Zheng, Yu Li
• Subjects: cs.CL, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2403.08192
• Pdf URL: https://arxiv.org/pdf/2403.08192
• Copy Paste: [[2403.08192]] MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension(https://arxiv.org/abs/2403.08192)
• Keywords: large language model
• Abstract: Large language models are playing an increasingly significant role in molecular research, yet existing models often generate erroneous information, posing challenges to accurate molecular comprehension. Traditional evaluation metrics for generated content fail to assess a model's accuracy in molecular understanding. To rectify the absence of factual evaluation, we present MoleculeQA, a novel question answering (QA) dataset which possesses 62K QA pairs over 23K molecules. Each QA pair, composed of a manual question, a positive option and three negative options, has consistent semantics with a molecular description from authoritative molecular corpus. MoleculeQA is not only the first benchmark for molecular factual bias evaluation but also the largest QA dataset for molecular research. A comprehensive evaluation on MoleculeQA for existing molecular LLMs exposes their deficiencies in specific areas and pinpoints several particularly crucial factors for molecular understanding.

Title: SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation

• Authors: Jiayu Du, Jinpeng Li, Guoguo Chen, Wei-Qiang Zhang
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.08196
• Pdf URL: https://arxiv.org/pdf/2403.08196
• Copy Paste: [[2403.08196]] SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation(https://arxiv.org/abs/2403.08196)
• Keywords: robust
• Abstract: In the wake of the surging tide of deep learning over the past decade, Automatic Speech Recognition (ASR) has garnered substantial attention, leading to the emergence of numerous publicly accessible ASR systems that are actively being integrated into our daily lives. Nonetheless, the impartial and replicable evaluation of these ASR systems encounters challenges due to various crucial subtleties. In this paper we introduce the SpeechColab Leaderboard, a general-purpose, open-source platform designed for ASR evaluation. With this platform: (i) We report a comprehensive benchmark, unveiling the current state-of-the-art panorama for ASR systems, covering both open-source models and industrial commercial services. (ii) We quantize how distinct nuances in the scoring pipeline influence the final benchmark outcomes. These include nuances related to capitalization, punctuation, interjection, contraction, synonym usage, compound words, etc. These issues have gained prominence in the context of the transition towards an End-to-End future. (iii) We propose a practical modification to the conventional Token-Error-Rate (TER) evaluation metric, with inspirations from Kolmogorov complexity and Normalized Information Distance (NID). This adaptation, called modified-TER (mTER), achieves proper normalization and symmetrical treatment of reference and hypothesis. By leveraging this platform as a large-scale testing ground, this study demonstrates the robustness and backward compatibility of mTER when compared to TER. The SpeechColab Leaderboard is accessible at https://github.com/SpeechColab/Leaderboard

Title: PAGE: Domain-Incremental Adaptation with Past-Agnostic Generative Replay for Smart Healthcare

• Authors: Chia-Hao Li, Niraj K. Jha
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08197
• Pdf URL: https://arxiv.org/pdf/2403.08197
• Copy Paste: [[2403.08197]] PAGE: Domain-Incremental Adaptation with Past-Agnostic Generative Replay for Smart Healthcare(https://arxiv.org/abs/2403.08197)
• Keywords: privacy, generative
• Abstract: We propose PAGE, a domain-incremental adaptation strategy with past-agnostic generative replay for smart healthcare. PAGE enables generative replay without the aid of any preserved data or information from prior domains. When adapting to a new domain, it exploits real data from the new distribution and the current model to generate synthetic data that retain the learned knowledge of previous domains. By replaying the synthetic data with the new real data during training, PAGE achieves a good balance between domain adaptation and knowledge retention. In addition, we incorporate an extended inductive conformal prediction (EICP) method into PAGE to produce a confidence score and a credibility value for each detection result. This makes the predictions interpretable and provides statistical guarantees for disease detection in smart healthcare applications. We demonstrate PAGE's effectiveness in domain-incremental disease detection with three distinct disease datasets collected from commercially available WMSs. PAGE achieves highly competitive performance against state-of-the-art with superior scalability, data privacy, and feasibility. Furthermore, PAGE can enable up to 75% reduction in clinical workload with the help of EICP.

Title: AutoDFP: Automatic Data-Free Pruning via Channel Similarity Reconstruction

• Authors: Siqi Li, Jun Chen, Jingyang Xiang, Chengrui Zhu, Yong Liu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08204
• Pdf URL: https://arxiv.org/pdf/2403.08204
• Copy Paste: [[2403.08204]] AutoDFP: Automatic Data-Free Pruning via Channel Similarity Reconstruction(https://arxiv.org/abs/2403.08204)
• Keywords: security, privacy, data-free
• Abstract: Structured pruning methods are developed to bridge the gap between the massive scale of neural networks and the limited hardware resources. Most current structured pruning methods rely on training datasets to fine-tune the compressed model, resulting in high computational burdens and being inapplicable for scenarios with stringent requirements on privacy and security. As an alternative, some data-free methods have been proposed, however, these methods often require handcraft parameter tuning and can only achieve inflexible reconstruction. In this paper, we propose the Automatic Data-Free Pruning (AutoDFP) method that achieves automatic pruning and reconstruction without fine-tuning. Our approach is based on the assumption that the loss of information can be partially compensated by retaining focused information from similar channels. Specifically, We formulate data-free pruning as an optimization problem, which can be effectively addressed through reinforcement learning. AutoDFP assesses the similarity of channels for each layer and provides this information to the reinforcement learning agent, guiding the pruning and reconstruction process of the network. We evaluate AutoDFP with multiple networks on multiple datasets, achieving impressive compression results. For instance, on the CIFAR-10 dataset, AutoDFP demonstrates a 2.87\% reduction in accuracy loss compared to the recently proposed data-free pruning method DFPC with fewer FLOPs on VGG-16. Furthermore, on the ImageNet dataset, AutoDFP achieves 43.17\% higher accuracy than the SOTA method with the same 80\% preserved ratio on MobileNet-V1.

Title: Advancing Security in AI Systems: A Novel Approach to Detecting Backdoors in Deep Neural Networks

• Authors: Khondoker Murad Hossain, Tim Oates
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08208
• Pdf URL: https://arxiv.org/pdf/2403.08208
• Copy Paste: [[2403.08208]] Advancing Security in AI Systems: A Novel Approach to Detecting Backdoors in Deep Neural Networks(https://arxiv.org/abs/2403.08208)
• Keywords: security
• Abstract: In the rapidly evolving landscape of communication and network security, the increasing reliance on deep neural networks (DNNs) and cloud services for data processing presents a significant vulnerability: the potential for backdoors that can be exploited by malicious actors. Our approach leverages advanced tensor decomposition algorithms Independent Vector Analysis (IVA), Multiset Canonical Correlation Analysis (MCCA), and Parallel Factor Analysis (PARAFAC2) to meticulously analyze the weights of pre-trained DNNs and distinguish between backdoored and clean models effectively. The key strengths of our method lie in its domain independence, adaptability to various network architectures, and ability to operate without access to the training data of the scrutinized models. This not only ensures versatility across different application scenarios but also addresses the challenge of identifying backdoors without prior knowledge of the specific triggers employed to alter network behavior. We have applied our detection pipeline to three distinct computer vision datasets, encompassing both image classification and object detection tasks. The results demonstrate a marked improvement in both accuracy and efficiency over existing backdoor detection methods. This advancement enhances the security of deep learning and AI in networked systems, providing essential cybersecurity against evolving threats in emerging technologies.

Title: Large Language Models are Contrastive Reasoners

• Authors: Liang Yao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08211
• Pdf URL: https://arxiv.org/pdf/2403.08211
• Copy Paste: [[2403.08211]] Large Language Models are Contrastive Reasoners(https://arxiv.org/abs/2403.08211)
• Keywords: large language model
• Abstract: Prompting methods play a crucial role in enhancing the capabilities of pre-trained large language models (LLMs). We explore how contrastive prompting (CP) significantly improves the ability of large language models to perform complex reasoning. We demonstrate that LLMs are decent contrastive reasoners by simply adding "Let's give a correct and a wrong answer." before LLMs provide answers. Experiments on two large language models show that zero-shot contrastive prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks without any hand-crafted few-shot examples, such as increasing the accuracy on GSM8K from 35.9% to 88.8% and AQUA-RAT from 41.3% to 62.2% with the state-of-the-art GPT-4 model. Our method not only surpasses zero-shot CoT and few-shot CoT in most arithmetic and commonsense reasoning tasks but also can seamlessly integrate with existing prompting methods, resulting in improved or comparable results when compared to state-of-the-art methods. Our code is available at https://github.com/yao8839836/cp

Title: Can Large Language Models Identify Authorship?

• Authors: Baixiang Huang, Canyu Chen, Kai Shu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08213
• Pdf URL: https://arxiv.org/pdf/2403.08213
• Copy Paste: [[2403.08213]] Can Large Language Models Identify Authorship?(https://arxiv.org/abs/2403.08213)
• Keywords: explainability, large language model
• Abstract: The ability to accurately identify authorship is crucial for verifying content authenticity and mitigating misinformation. Large Language Models (LLMs) have demonstrated exceptional capacity for reasoning and problem-solving. However, their potential in authorship analysis, encompassing authorship verification and attribution, remains underexplored. This paper conducts a comprehensive evaluation of LLMs in these critical tasks. Traditional studies have depended on hand-crafted stylistic features, whereas state-of-the-art approaches leverage text embeddings from pre-trained language models. These methods, which typically require fine-tuning on labeled data, often suffer from performance degradation in cross-domain applications and provide limited explainability. This work seeks to address three research questions: (1) Can LLMs perform zero-shot, end-to-end authorship verification effectively? (2) Are LLMs capable of accurately attributing authorship among multiple candidates authors (e.g., 10 and 20)? (3) How can LLMs provide explainability in authorship analysis, particularly through the role of linguistic features? Moreover, we investigate the integration of explicit linguistic features to guide LLMs in their reasoning processes. Our extensive assessment demonstrates LLMs' proficiency in both tasks without the need for domain-specific fine-tuning, providing insights into their decision-making via a detailed analysis of linguistic features. This establishes a new benchmark for future research on LLM-based authorship analysis. The code and data are available at https://github.com/baixianghuang/authorship-llm.

Title: P2LHAP:Wearable sensor-based human activity recognition, segmentation and forecast through Patch-to-Label Seq2Seq Transformer

• Authors: Shuangjian Li, Tao Zhu, Mingxing Nie, Huansheng Ning, Zhenyu Liu, Liming Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08214
• Pdf URL: https://arxiv.org/pdf/2403.08214
• Copy Paste: [[2403.08214]] P2LHAP:Wearable sensor-based human activity recognition, segmentation and forecast through Patch-to-Label Seq2Seq Transformer(https://arxiv.org/abs/2403.08214)
• Keywords: transformer, segmentation
• Abstract: Traditional deep learning methods struggle to simultaneously segment, recognize, and forecast human activities from sensor data. This limits their usefulness in many fields such as healthcare and assisted living, where real-time understanding of ongoing and upcoming activities is crucial. This paper introduces P2LHAP, a novel Patch-to-Label Seq2Seq framework that tackles all three tasks in a efficient single-task model. P2LHAP divides sensor data streams into a sequence of "patches", served as input tokens, and outputs a sequence of patch-level activity labels including the predicted future activities. A unique smoothing technique based on surrounding patch labels, is proposed to identify activity boundaries accurately. Additionally, P2LHAP learns patch-level representation by sensor signal channel-independent Transformer encoders and decoders. All channels share embedding and Transformer weights across all sequences. Evaluated on three public datasets, P2LHAP significantly outperforms the state-of-the-art in all three tasks, demonstrating its effectiveness and potential for real-world applications.

Title: LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Driving

• Authors: Sicen Guo, Zhiyuan Wu, Qijun Chen, Ioannis Pitas, Rui Fan
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.08215
• Pdf URL: https://arxiv.org/pdf/2403.08215
• Copy Paste: [[2403.08215]] LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Driving(https://arxiv.org/abs/2403.08215)
• Keywords: segmentation
• Abstract: Despite the impressive performance achieved by data-fusion networks with duplex encoders for visual semantic segmentation, they become ineffective when spatial geometric data are not available. Implicitly infusing the spatial geometric prior knowledge acquired by a duplex-encoder teacher model into a single-encoder student model is a practical, albeit less explored research avenue. This paper delves into this topic and resorts to knowledge distillation approaches to address this problem. We introduce the Learning to Infuse "X" (LIX) framework, with novel contributions in both logit distillation and feature distillation aspects. We present a mathematical proof that underscores the limitation of using a single fixed weight in decoupled knowledge distillation and introduce a logit-wise dynamic weight controller as a solution to this issue. Furthermore, we develop an adaptively-recalibrated feature distillation algorithm, including two technical novelties: feature recalibration via kernel regression and in-depth feature consistency quantification via centered kernel alignment. Extensive experiments conducted with intermediate-fusion and late-fusion networks across various public datasets provide both quantitative and qualitative evaluations, demonstrating the superior performance of our LIX framework when compared to other state-of-the-art approaches.

Title: PaddingFlow: Improving Normalizing Flows with Padding-Dimensional Noise

• Authors: Qinglong Meng, Chongkun Xia, Xueqian Wang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08216
• Pdf URL: https://arxiv.org/pdf/2403.08216
• Copy Paste: [[2403.08216]] PaddingFlow: Improving Normalizing Flows with Padding-Dimensional Noise(https://arxiv.org/abs/2403.08216)
• Keywords: generative
• Abstract: Normalizing flow is a generative modeling approach with efficient sampling. However, Flow-based models suffer two issues, which are manifold and discrete data. If the target distribution is a manifold, which means the dimension of the latent target distribution and the dimension of the data distribution are unmatched, flow-based models might perform badly. Discrete data makes flow-based models collapse into a degenerate mixture of point masses. In this paper, to sidestep such two issues we propose PaddingFlow, a novel dequantization method, which improves normalizing flows with padding-dimensional noise. PaddingFlow is easy to implement, computationally cheap, widely suitable for various tasks, and generates samples that are unbiased estimations of the data. Especially, our method can overcome the limitation of existing dequantization methods that have to change the data distribution, which might degrade performance. We validate our method on the main benchmarks of unconditional density estimation, including five tabular datasets and four image datasets for VAE models, and the IK experiments which are conditional density estimation. The results show that PaddingFlow can provide improvement on all tasks in this paper.

Title: Research on the Application of Deep Learning-based BERT Model in Sentiment Analysis

• Authors: Yichao Wu, Zhengyu Jin, Chenxi Shi, Penghao Liang, Tong Zhan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08217
• Pdf URL: https://arxiv.org/pdf/2403.08217
• Copy Paste: [[2403.08217]] Research on the Application of Deep Learning-based BERT Model in Sentiment Analysis(https://arxiv.org/abs/2403.08217)
• Keywords: robust
• Abstract: This paper explores the application of deep learning techniques, particularly focusing on BERT models, in sentiment analysis. It begins by introducing the fundamental concept of sentiment analysis and how deep learning methods are utilized in this domain. Subsequently, it delves into the architecture and characteristics of BERT models. Through detailed explanation, it elucidates the application effects and optimization strategies of BERT models in sentiment analysis, supported by experimental validation. The experimental findings indicate that BERT models exhibit robust performance in sentiment analysis tasks, with notable enhancements post fine-tuning. Lastly, the paper concludes by summarizing the potential applications of BERT models in sentiment analysis and suggests directions for future research and practical implementations.

Title: Robust Decision Aggregation with Adversarial Experts

• Authors: Yongkang Guo, Yuqing Kong
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08222
• Pdf URL: https://arxiv.org/pdf/2403.08222
• Copy Paste: [[2403.08222]] Robust Decision Aggregation with Adversarial Experts(https://arxiv.org/abs/2403.08222)
• Keywords: robust
• Abstract: We consider a binary decision aggregation problem in the presence of both truthful and adversarial experts. The truthful experts will report their private signals truthfully with proper incentive, while the adversarial experts can report arbitrarily. The decision maker needs to design a robust aggregator to forecast the true state of the world based on the reports of experts. The decision maker does not know the specific information structure, which is a joint distribution of signals, states, and strategies of adversarial experts. We want to find the optimal aggregator minimizing regret under the worst information structure. The regret is defined by the difference in expected loss between the aggregator and a benchmark who makes the optimal decision given the joint distribution and reports of truthful experts. We prove that when the truthful experts are symmetric and adversarial experts are not too numerous, the truncated mean is optimal, which means that we remove some lowest reports and highest reports and take averaging among the left reports. Moreover, for many settings, the optimal aggregators are in the family of piecewise linear functions. The regret is independent of the total number of experts but only depends on the ratio of adversaries. We evaluate our aggregators by numerical experiment in an ensemble learning task. We also obtain some negative results for the aggregation problem with adversarial experts under some more general information structures and experts' report space.

Title: REPAIR: Rank Correlation and Noisy Pair Half-replacing with Memory for Noisy Correspondence

• Authors: Ruochen Zheng, Jiahao Hong, Changxin Gao, Nong Sang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08224
• Pdf URL: https://arxiv.org/pdf/2403.08224
• Copy Paste: [[2403.08224]] REPAIR: Rank Correlation and Noisy Pair Half-replacing with Memory for Noisy Correspondence(https://arxiv.org/abs/2403.08224)
• Keywords: robust
• Abstract: The presence of noise in acquired data invariably leads to performance degradation in cross-modal matching. Unfortunately, obtaining precise annotations in the multimodal field is expensive, which has prompted some methods to tackle the mismatched data pair issue in cross-modal matching contexts, termed as noisy correspondence. However, most of these existing noisy correspondence methods exhibit the following limitations: a) the problem of self-reinforcing error accumulation, and b) improper handling of noisy data pair. To tackle the two problems, we propose a generalized framework termed as Rank corrElation and noisy Pair hAlf-replacing wIth memoRy (REPAIR), which benefits from maintaining a memory bank for features of matched pairs. Specifically, we calculate the distances between the features in the memory bank and those of the target pair for each respective modality, and use the rank correlation of these two sets of distances to estimate the soft correspondence label of the target pair. Estimating soft correspondence based on memory bank features rather than using a similarity network can avoid the accumulation of errors due to incorrect network identifications. For pairs that are completely mismatched, REPAIR searches the memory bank for the most matching feature to replace one feature of one modality, instead of using the original pair directly or merely discarding the mismatched pair. We conduct experiments on three cross-modal datasets, i.e., Flickr30K, MSCOCO, and CC152K, proving the effectiveness and robustness of our REPAIR on synthetic and real-world noise.

Title: Matching Non-Identical Objects

• Authors: Yusuke Marumo, Kazuhiko Kawamoto, Hiroshi Kera
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08227
• Pdf URL: https://arxiv.org/pdf/2403.08227
• Copy Paste: [[2403.08227]] Matching Non-Identical Objects(https://arxiv.org/abs/2403.08227)
• Keywords: robust
• Abstract: Not identical but similar objects are everywhere in the world. Examples include four-legged animals such as dogs and cats, cars of different models, akin flowers in various colors, and countless others. In this study, we address a novel task of matching such non-identical objects. We propose a simple weighting scheme of descriptors that enhance various sparse image matching methods, which are originally designed for matching identical objects captured from different perspectives, and achieve semantically robust matching. The experiments show successful matching between non-identical objects in various cases including domain shift. Further, we present a first evaluation of the robustness of the image matching methods under common corruptions, which is a sort of domain shift, and the proposed method improves the matching in this case as well.

Title: Boosting Disfluency Detection with Large Language Model as Disfluency Generator

• Authors: Zhenrong Cheng, Jiayan Guo, Hao Sun, Yan Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08229
• Pdf URL: https://arxiv.org/pdf/2403.08229
• Copy Paste: [[2403.08229]] Boosting Disfluency Detection with Large Language Model as Disfluency Generator(https://arxiv.org/abs/2403.08229)
• Keywords: generative, large language model
• Abstract: Current disfluency detection methods heavily rely on costly and scarce human-annotated data. To tackle this issue, some approaches employ heuristic or statistical features to generate disfluent sentences, partially improving detection performance. However, these sentences often deviate from real-life scenarios, constraining overall model enhancement. In this study, we propose a lightweight data augmentation approach for disfluency detection, utilizing the superior generative and semantic understanding capabilities of large language model (LLM) to generate disfluent sentences as augmentation data. We leverage LLM to generate diverse and more realistic sentences guided by specific prompts, without the need for fine-tuning the LLM. Subsequently, we apply an uncertainty-aware data filtering approach to improve the quality of the generated sentences, utilized in training a small detection model for improved performance. Experiments using enhanced data yielded state-of-the-art results. The results showed that using a small amount of LLM-generated enhanced data can significantly improve performance, thereby further enhancing cost-effectiveness.

Title: Point Cloud Compression via Constrained Optimal Transport

• Authors: Zezeng Li, Weimin Wang, Ziliang Wang, Na Lei
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.08236
• Pdf URL: https://arxiv.org/pdf/2403.08236
• Copy Paste: [[2403.08236]] Point Cloud Compression via Constrained Optimal Transport(https://arxiv.org/abs/2403.08236)
• Keywords: generative
• Abstract: This paper presents a novel point cloud compression method COT-PCC by formulating the task as a constrained optimal transport (COT) problem. COT-PCC takes the bitrate of compressed features as an extra constraint of optimal transport (OT) which learns the distribution transformation between original and reconstructed points. Specifically, the formulated COT is implemented with a generative adversarial network (GAN) and a bitrate loss for training. The discriminator measures the Wasserstein distance between input and reconstructed points, and a generator calculates the optimal mapping between distributions of input and reconstructed point cloud. Moreover, we introduce a learnable sampling module for downsampling in the compression procedure. Extensive results on both sparse and dense point cloud datasets demonstrate that COT-PCC outperforms state-of-the-art methods in terms of both CD and PSNR metrics. Source codes are available at \url{https://github.com/cognaclee/PCC-COT}.

Title: Machine Unlearning: Taxonomy, Metrics, Applications, Challenges, and Prospects

• Authors: Na Li, Chunyi Zhou, Yansong Gao, Hui Chen, Anmin Fu, Zhi Zhang, Yu Shui
• Subjects: cs.LG, cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.08254
• Pdf URL: https://arxiv.org/pdf/2403.08254
• Copy Paste: [[2403.08254]] Machine Unlearning: Taxonomy, Metrics, Applications, Challenges, and Prospects(https://arxiv.org/abs/2403.08254)
• Keywords: privacy, protect, attack
• Abstract: Personal digital data is a critical asset, and governments worldwide have enforced laws and regulations to protect data privacy. Data users have been endowed with the right to be forgotten of their data. In the course of machine learning (ML), the forgotten right requires a model provider to delete user data and its subsequent impact on ML models upon user requests. Machine unlearning emerges to address this, which has garnered ever-increasing attention from both industry and academia. While the area has developed rapidly, there is a lack of comprehensive surveys to capture the latest advancements. Recognizing this shortage, we conduct an extensive exploration to map the landscape of machine unlearning including the (fine-grained) taxonomy of unlearning algorithms under centralized and distributed settings, debate on approximate unlearning, verification and evaluation metrics, challenges and solutions for unlearning under different applications, as well as attacks targeting machine unlearning. The survey concludes by outlining potential directions for future research, hoping to serve as a guide for interested scholars.

Title: Make Me Happier: Evoking Emotions Through Image Diffusion Models

• Authors: Qing Lin, Jingfeng Zhang, Yew Soon Ong, Mengmi Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08255
• Pdf URL: https://arxiv.org/pdf/2403.08255
• Copy Paste: [[2403.08255]] Make Me Happier: Evoking Emotions Through Image Diffusion Models(https://arxiv.org/abs/2403.08255)
• Keywords: diffusion
• Abstract: Despite the rapid progress in image generation, emotional image editing remains under-explored. The semantics, context, and structure of an image can evoke emotional responses, making emotional image editing techniques valuable for various real-world applications, including treatment of psychological disorders, commercialization of products, and artistic design. For the first time, we present a novel challenge of emotion-evoked image generation, aiming to synthesize images that evoke target emotions while retaining the semantics and structures of the original scenes. To address this challenge, we propose a diffusion model capable of effectively understanding and editing source images to convey desired emotions and sentiments. Moreover, due to the lack of emotion editing datasets, we provide a unique dataset consisting of 340,000 pairs of images and their emotion annotations. Furthermore, we conduct human psychophysics experiments and introduce four new evaluation metrics to systematically benchmark all the methods. Experimental results demonstrate that our method surpasses all competitive baselines. Our diffusion model is capable of identifying emotional cues from original images, editing images that elicit desired emotions, and meanwhile, preserving the semantic structure of the original images. All code, model, and data will be made public.

Title: IG-FIQA: Improving Face Image Quality Assessment through Intra-class Variance Guidance robust to Inaccurate Pseudo-Labels

• Authors: Minsoo Kim, Gi Pyo Nam, Haksub Kim, Haesol Park, Ig-Jae Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08256
• Pdf URL: https://arxiv.org/pdf/2403.08256
• Copy Paste: [[2403.08256]] IG-FIQA: Improving Face Image Quality Assessment through Intra-class Variance Guidance robust to Inaccurate Pseudo-Labels(https://arxiv.org/abs/2403.08256)
• Keywords: robust
• Abstract: In the realm of face image quality assesment (FIQA), method based on sample relative classification have shown impressive performance. However, the quality scores used as pseudo-labels assigned from images of classes with low intra-class variance could be unrelated to the actual quality in this method. To address this issue, we present IG-FIQA, a novel approach to guide FIQA training, introducing a weight parameter to alleviate the adverse impact of these classes. This method involves estimating sample intra-class variance at each iteration during training, ensuring minimal computational overhead and straightforward implementation. Furthermore, this paper proposes an on-the-fly data augmentation methodology for improved generalization performance in FIQA. On various benchmark datasets, our proposed method, IG-FIQA, achieved novel state-of-the-art (SOTA) performance.

Title: CoroNetGAN: Controlled Pruning of GANs via Hypernetworks

• Authors: Aman Kumar, Khushboo Anand, Shubham Mandloi, Ashutosh Mishra, Avinash Thakur, Neeraj Kasera, Prathosh A P
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.08261
• Pdf URL: https://arxiv.org/pdf/2403.08261
• Copy Paste: [[2403.08261]] CoroNetGAN: Controlled Pruning of GANs via Hypernetworks(https://arxiv.org/abs/2403.08261)
• Keywords: generative
• Abstract: Generative Adversarial Networks (GANs) have proven to exhibit remarkable performance and are widely used across many generative computer vision applications. However, the unprecedented demand for the deployment of GANs on resource-constrained edge devices still poses a challenge due to huge number of parameters involved in the generation process. This has led to focused attention on the area of compressing GANs. Most of the existing works use knowledge distillation with the overhead of teacher dependency. Moreover, there is no ability to control the degree of compression in these methods. Hence, we propose CoroNet-GAN for compressing GAN using the combined strength of differentiable pruning method via hypernetworks. The proposed method provides the advantage of performing controllable compression while training along with reducing training time by a substantial factor. Experiments have been done on various conditional GAN architectures (Pix2Pix and CycleGAN) to signify the effectiveness of our approach on multiple benchmark datasets such as Edges-to-Shoes, Horse-to-Zebra and Summer-to-Winter. The results obtained illustrate that our approach succeeds to outperform the baselines on Zebra-to-Horse and Summer-to-Winter achieving the best FID score of 32.3 and 72.3 respectively, yielding high-fidelity images across all the datasets. Additionally, our approach also outperforms the state-of-the-art methods in achieving better inference time on various smart-phone chipsets and data-types making it a feasible solution for deployment on edge devices.

Title: Sketch2Manga: Shaded Manga Screening from Sketch with Diffusion Models

• Authors: Jian Lin, Xueting Liu, Chengze Li, Minshan Xie, Tien-Tsin Wong
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2403.08266
• Pdf URL: https://arxiv.org/pdf/2403.08266
• Copy Paste: [[2403.08266]] Sketch2Manga: Shaded Manga Screening from Sketch with Diffusion Models(https://arxiv.org/abs/2403.08266)
• Keywords: diffusion
• Abstract: While manga is a popular entertainment form, creating manga is tedious, especially adding screentones to the created sketch, namely manga screening. Unfortunately, there is no existing method that tailors for automatic manga screening, probably due to the difficulty of generating high-quality shaded high-frequency screentones. The classic manga screening approaches generally require user input to provide screentone exemplars or a reference manga image. The recent deep learning models enables the automatic generation by learning from a large-scale dataset. However, the state-of-the-art models still fail to generate high-quality shaded screentones due to the lack of a tailored model and high-quality manga training data. In this paper, we propose a novel sketch-to-manga framework that first generates a color illustration from the sketch and then generates a screentoned manga based on the intensity guidance. Our method significantly outperforms existing methods in generating high-quality manga with shaded high-frequency screentones.

Title: SNOW-SCA: ML-assisted Side-Channel Attack on SNOW-V

• Authors: Harshit Saurabh, Anupam Golder, Samarth Shivakumar Titti, Suparna Kundu, Chaoyun Li, Angshuman Karmakar, Debayan Das
• Subjects: cs.CR, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.08267
• Pdf URL: https://arxiv.org/pdf/2403.08267
• Copy Paste: [[2403.08267]] SNOW-SCA: ML-assisted Side-Channel Attack on SNOW-V(https://arxiv.org/abs/2403.08267)
• Keywords: security, attack
• Abstract: This paper presents SNOW-SCA, the first power side-channel analysis (SCA) attack of a 5G mobile communication security standard candidate, SNOW-V, running on a 32-bit ARM Cortex-M4 microcontroller. First, we perform a generic known-key correlation (KKC) analysis to identify the leakage points. Next, a correlation power analysis (CPA) attack is performed, which reduces the attack complexity to two key guesses for each key byte. The correct secret key is then uniquely identified utilizing linear discriminant analysis (LDA). The profiled SCA attack with LDA achieves 100% accuracy after training with $<200$ traces, which means the attack succeeds with just a single trace. Overall, using the \textit{combined CPA and LDA attack} model, the correct secret key byte is recovered with <50 traces collected using the ChipWhisperer platform. The entire 256-bit secret key of SNOW-V can be recovered incrementally using the proposed SCA attack. Finally, we suggest low-overhead countermeasures that can be used to prevent these SCA attacks.

Title: Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classification

• Authors: Long Lan, Fengxiang Wang, Shuyan Li, Xiangtao Zheng, Zengmao Wang, Xinwang Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08271
• Pdf URL: https://arxiv.org/pdf/2403.08271
• Copy Paste: [[2403.08271]] Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classification(https://arxiv.org/abs/2403.08271)
• Keywords: privacy
• Abstract: Fine-grained ship classification in remote sensing (RS-FGSC) poses a significant challenge due to the high similarity between classes and the limited availability of labeled data, limiting the effectiveness of traditional supervised classification methods. Recent advancements in large pre-trained Vision-Language Models (VLMs) have demonstrated impressive capabilities in few-shot or zero-shot learning, particularly in understanding image content. This study delves into harnessing the potential of VLMs to enhance classification accuracy for unseen ship categories, which holds considerable significance in scenarios with restricted data due to cost or privacy constraints. Directly fine-tuning VLMs for RS-FGSC often encounters the challenge of overfitting the seen classes, resulting in suboptimal generalization to unseen classes, which highlights the difficulty in differentiating complex backgrounds and capturing distinct ship features. To address these issues, we introduce a novel prompt tuning technique that employs a hierarchical, multi-granularity prompt design. Our approach integrates remote sensing ship priors through bias terms, learned from a small trainable network. This strategy enhances the model's generalization capabilities while improving its ability to discern intricate backgrounds and learn discriminative ship features. Furthermore, we contribute to the field by introducing a comprehensive dataset, FGSCM-52, significantly expanding existing datasets with more extensive data and detailed annotations for less common ship classes. Extensive experimental evaluations demonstrate the superiority of our proposed method over current state-of-the-art techniques. The source code will be made publicly available.

Title: RECIPE4U: Student-ChatGPT Interaction Dataset in EFL Writing Education

• Authors: Jieun Han, Haneul Yoo, Junho Myung, Minsun Kim, Tak Yeon Lee, So-Yeon Ahn, Alice Oh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08272
• Pdf URL: https://arxiv.org/pdf/2403.08272
• Copy Paste: [[2403.08272]] RECIPE4U: Student-ChatGPT Interaction Dataset in EFL Writing Education(https://arxiv.org/abs/2403.08272)
• Keywords: generative
• Abstract: The integration of generative AI in education is expanding, yet empirical analyses of large-scale and real-world interactions between students and AI systems still remain limited. Addressing this gap, we present RECIPE4U (RECIPE for University), a dataset sourced from a semester-long experiment with 212 college students in English as Foreign Language (EFL) writing courses. During the study, students engaged in dialogues with ChatGPT to revise their essays. RECIPE4U includes comprehensive records of these interactions, including conversation logs, students' intent, students' self-rated satisfaction, and students' essay edit histories. In particular, we annotate the students' utterances in RECIPE4U with 13 intention labels based on our coding schemes. We establish baseline results for two subtasks in task-oriented dialogue systems within educational contexts: intent detection and satisfaction estimation. As a foundational step, we explore student-ChatGPT interaction patterns through RECIPE4U and analyze them by focusing on students' dialogue, essay data statistics, and students' essay edits. We further illustrate potential applications of RECIPE4U dataset for enhancing the incorporation of LLMs in educational frameworks. RECIPE4U is publicly available at https://zeunie.github.io/RECIPE4U/.

Title: LiqD: A Dynamic Liquid Level Detection Model under Tricky Small Containers

• Authors: Yukun Ma, Zikun Mao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08273
• Pdf URL: https://arxiv.org/pdf/2403.08273
• Copy Paste: [[2403.08273]] LiqD: A Dynamic Liquid Level Detection Model under Tricky Small Containers(https://arxiv.org/abs/2403.08273)
• Keywords: robust
• Abstract: In daily life and industrial production, it is crucial to accurately detect changes in liquid level in containers. Traditional contact measurement methods have some limitations, while emerging non-contact image processing technology shows good application prospects. This paper proposes a container dynamic liquid level detection model based on U^2-Net. This model uses the SAM model to generate an initial data set, and then evaluates and filters out high-quality pseudo-label images through the SemiReward framework to build an exclusive data set. The model uses U^2-Net to extract mask images of containers from the data set, and uses morphological processing to compensate for mask defects. Subsequently, the model calculates the grayscale difference between adjacent video frame images at the same position, segments the liquid level change area by setting a difference threshold, and finally uses a lightweight neural network to classify the liquid level state. This approach not only mitigates the impact of intricate surroundings, but also reduces the demand for training data, showing strong robustness and versatility. A large number of experimental results show that the proposed model can effectively detect the dynamic liquid level changes of the liquid in the container, providing a novel and efficient solution for related fields.

Title: VIGFace: Virtual Identity Generation Model for Face Image Synthesis

• Authors: Minsoo Kim, Min-Cheol Sagong, Gi Pyo Nam, Junghyun Cho, Ig-Jae Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08277
• Pdf URL: https://arxiv.org/pdf/2403.08277
• Copy Paste: [[2403.08277]] VIGFace: Virtual Identity Generation Model for Face Image Synthesis(https://arxiv.org/abs/2403.08277)
• Keywords: privacy, diffusion
• Abstract: Deep learning-based face recognition continues to face challenges due to its reliance on huge datasets obtained from web crawling, which can be costly to gather and raise significant real-world privacy concerns. To address this issue, we propose VIGFace, a novel framework capable of generating synthetic facial images. Initially, we train the face recognition model using a real face dataset and create a feature space for both real and virtual IDs where virtual prototypes are orthogonal to other prototypes. Subsequently, we generate synthetic images by using the diffusion model based on the feature space. Our proposed framework provides two significant benefits. Firstly, it allows for creating virtual facial images without concerns about portrait rights, guaranteeing that the generated virtual face images are clearly differentiated from existing individuals. Secondly, it serves as an effective augmentation method by incorporating real existing images. Further experiments demonstrate the efficacy of our framework, achieving state-of-the-art results from both perspectives without any external data.

Title: Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models

• Authors: Ning Ding, Yulin Chen, Ganqu Cui, Xingtai Lv, Ruobing Xie, Bowen Zhou, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08281
• Pdf URL: https://arxiv.org/pdf/2403.08281
• Copy Paste: [[2403.08281]] Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models(https://arxiv.org/abs/2403.08281)
• Keywords: large language model
• Abstract: Underlying data distributions of natural language, programming code, and mathematical symbols vary vastly, presenting a complex challenge for large language models (LLMs) that strive to achieve high performance across all three domains simultaneously. Achieving a very high level of proficiency for an LLM within a specific domain often requires extensive training with relevant corpora, which is typically accompanied by a sacrifice in performance in other domains. In this paper, we propose to fuse models that are already highly-specialized directly. The proposed fusing framework, UltraFuser, consists of three distinct specialists that are already sufficiently trained on language, coding, and mathematics. A token-level gating mechanism is introduced to blend the specialists' outputs. A two-stage training strategy accompanied by balanced sampling is designed to ensure stability. To effectively train the fused model, we further construct a high-quality supervised instruction tuning dataset, UltraChat 2, which includes text, code, and mathematical content. This dataset comprises approximately 300,000 instructions and covers a wide range of topics in each domain. Experiments show that our model could simultaneously achieve mastery of the three crucial domains.

Title: MGIC: A Multi-Label Gradient Inversion Attack based on Canny Edge Detection on Federated Learning

• Authors: Can Liu, Jin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08284
• Pdf URL: https://arxiv.org/pdf/2403.08284
• Copy Paste: [[2403.08284]] MGIC: A Multi-Label Gradient Inversion Attack based on Canny Edge Detection on Federated Learning(https://arxiv.org/abs/2403.08284)
• Keywords: privacy, protect, attack, federate
• Abstract: As a new distributed computing framework that can protect data privacy, federated learning (FL) has attracted more and more attention in recent years. It receives gradients from users to train the global model and releases the trained global model to working users. Nonetheless, the gradient inversion (GI) attack reflects the risk of privacy leakage in federated learning. Attackers only need to use gradients through hundreds of thousands of simple iterations to obtain relatively accurate private data stored on users' local devices. For this, some works propose simple but effective strategies to obtain user data under a single-label dataset. However, these strategies induce a satisfactory visual effect of the inversion image at the expense of higher time costs. Due to the semantic limitation of a single label, the image obtained by gradient inversion may have semantic errors. We present a novel gradient inversion strategy based on canny edge detection (MGIC) in both the multi-label and single-label datasets. To reduce semantic errors caused by a single label, we add new convolution layers' blocks in the trained model to obtain the image's multi-label. Through multi-label representation, serious semantic errors in inversion images are reduced. Then, we analyze the impact of parameters on the difficulty of input image reconstruction and discuss how image multi-subjects affect the inversion performance. Our proposed strategy has better visual inversion image results than the most widely used ones, saving more than 78% of time costs in the ImageNet dataset.

Title: CleanAgent: Automating Data Standardization with LLM-based Agents

• Authors: Danrui Qi, Jiannan Wang
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2403.08291
• Pdf URL: https://arxiv.org/pdf/2403.08291
• Copy Paste: [[2403.08291]] CleanAgent: Automating Data Standardization with LLM-based Agents(https://arxiv.org/abs/2403.08291)
• Keywords: robust, large language model
• Abstract: Data standardization is a crucial part in data science life cycle. While tools like Pandas offer robust functionalities, their complexity and the manual effort required for customizing code to diverse column types pose significant challenges. Although large language models (LLMs) like ChatGPT have shown promise in automating this process through natural language understanding and code generation, it still demands expert-level programming knowledge and continuous interaction for prompt refinement. To solve these challenges, our key idea is to propose a Python library with declarative, unified APIs for standardizing column types, simplifying the code generation of LLM with concise API calls. We first propose Dataprep.Clean which is written as a component of the Dataprep Library, offers a significant reduction in complexity by enabling the standardization of specific column types with a single line of code. Then we introduce the CleanAgent framework integrating Dataprep.Clean and LLM-based agents to automate the data standardization process. With CleanAgent, data scientists need only provide their requirements once, allowing for a hands-free, automatic standardization process.

Title: Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale

• Authors: Xiang Hu, Pengyu Ji, Qingyang Zhu, Wei Wu, Kewei Tu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08293
• Pdf URL: https://arxiv.org/pdf/2403.08293
• Copy Paste: [[2403.08293]] Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale(https://arxiv.org/abs/2403.08293)
• Keywords: transformer, generative
• Abstract: A syntactic language model (SLM) incrementally generates a sentence with its syntactic tree in a left-to-right manner. We present Generative Pretrained Structured Transformers (GPST), an unsupervised SLM at scale capable of being pre-trained from scratch on raw texts with high parallelism. GPST circumvents the limitations of previous SLMs such as relying on gold trees and sequential training. It consists of two components, a usual SLM supervised by a uni-directional language modeling loss, and an additional composition model, which induces syntactic parse trees and computes constituent representations, supervised by a bi-directional language modeling loss. We propose a representation surrogate to enable joint parallel training of the two models in a hard-EM fashion. We pre-train GPST on OpenWebText, a corpus with $9$ billion tokens, and demonstrate the superiority of GPST over GPT-2 with a comparable size in numerous tasks covering both language understanding and language generation. Meanwhile, GPST also significantly outperforms existing unsupervised SLMs on left-to-right grammar induction, while holding a substantial acceleration on training.

Title: Attack Deterministic Conditional Image Generative Models for Diverse and Controllable Generation

• Authors: Tianyi Chu, Wei Xing, Jiafu Chen, Zhizhong Wang, Jiakai Sun, Lei Zhao, Haibo Chen, Huaizhong Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08294
• Pdf URL: https://arxiv.org/pdf/2403.08294
• Copy Paste: [[2403.08294]] Attack Deterministic Conditional Image Generative Models for Diverse and Controllable Generation(https://arxiv.org/abs/2403.08294)
• Keywords: attack, generative
• Abstract: Existing generative adversarial network (GAN) based conditional image generative models typically produce fixed output for the same conditional input, which is unreasonable for highly subjective tasks, such as large-mask image inpainting or style transfer. On the other hand, GAN-based diverse image generative methods require retraining/fine-tuning the network or designing complex noise injection functions, which is computationally expensive, task-specific, or struggle to generate high-quality results. Given that many deterministic conditional image generative models have been able to produce high-quality yet fixed results, we raise an intriguing question: is it possible for pre-trained deterministic conditional image generative models to generate diverse results without changing network structures or parameters? To answer this question, we re-examine the conditional image generation tasks from the perspective of adversarial attack and propose a simple and efficient plug-in projected gradient descent (PGD) like method for diverse and controllable image generation. The key idea is attacking the pre-trained deterministic generative models by adding a micro perturbation to the input condition. In this way, diverse results can be generated without any adjustment of network structures or fine-tuning of the pre-trained models. In addition, we can also control the diverse results to be generated by specifying the attack direction according to a reference text or image. Our work opens the door to applying adversarial attack to low-level vision tasks, and experiments on various conditional image generation tasks demonstrate the effectiveness and superiority of the proposed method.

Title: Towards Personalized Evaluation of Large Language Models with An Anonymous Crowd-Sourcing Platform

• Authors: Mingyue Cheng, Hao Zhang, Jiqian Yang, Qi Liu, Li Li, Xin Huang, Liwei Song, Zhi Li, Zhenya Huang, Enhong Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08305
• Pdf URL: https://arxiv.org/pdf/2403.08305
• Copy Paste: [[2403.08305]] Towards Personalized Evaluation of Large Language Models with An Anonymous Crowd-Sourcing Platform(https://arxiv.org/abs/2403.08305)
• Keywords: large language model
• Abstract: Large language model evaluation plays a pivotal role in the enhancement of its capacity. Previously, numerous methods for evaluating large language models have been proposed in this area. Despite their effectiveness, these existing works mainly focus on assessing objective questions, overlooking the capability to evaluate subjective questions which is extremely common for large language models. Additionally, these methods predominantly utilize centralized datasets for evaluation, with question banks concentrated within the evaluation platforms themselves. Moreover, the evaluation processes employed by these platforms often overlook personalized factors, neglecting to consider the individual characteristics of both the evaluators and the models being evaluated. To address these limitations, we propose a novel anonymous crowd-sourcing evaluation platform, BingJian, for large language models that employs a competitive scoring mechanism where users participate in ranking models based on their performance. This platform stands out not only for its support of centralized evaluations to assess the general capabilities of models but also for offering an open evaluation gateway. Through this gateway, users have the opportunity to submit their questions, testing the models on a personalized and potentially broader range of capabilities. Furthermore, our platform introduces personalized evaluation scenarios, leveraging various forms of human-computer interaction to assess large language models in a manner that accounts for individual user preferences and contexts. The demonstration of BingJian can be accessed at https://github.com/Mingyue-Cheng/Bingjian.

Title: HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback

• Authors: Ang Li, Qiugen Xiao, Peng Cao, Jian Tang, Yi Yuan, Zijie Zhao, Xiaoyuan Chen, Liang Zhang, Xiangyang Li, Kaitong Yang, Weidong Guo, Yukang Gan, Daniell Wang, Ying Shan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08309
• Pdf URL: https://arxiv.org/pdf/2403.08309
• Copy Paste: [[2403.08309]] HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback(https://arxiv.org/abs/2403.08309)
• Keywords: robust, large language model
• Abstract: Reinforcement Learning from AI Feedback (RLAIF) has the advantages of shorter annotation cycles and lower costs over Reinforcement Learning from Human Feedback (RLHF), making it highly efficient during the rapid strategy iteration periods of large language model (LLM) training. Using ChatGPT as a labeler to provide feedback on open-domain prompts in RLAIF training, we observe an increase in human evaluators' preference win ratio for model responses, but a decrease in evaluators' satisfaction rate. Analysis suggests that the decrease in satisfaction rate is mainly due to some responses becoming less helpful, particularly in terms of correctness and truthfulness, highlighting practical limitations of basic RLAIF. In this paper, we propose Hybrid Reinforcement Learning from AI Feedback (HRLAIF). This method enhances the accuracy of AI annotations for responses, making the model's helpfulness more robust in training process. Additionally, it employs AI for Red Teaming, further improving the model's harmlessness. Human evaluation results show that HRLAIF inherits the ability of RLAIF to enhance human preference for outcomes at a low cost while also improving the satisfaction rate of responses. Compared to the policy model before Reinforcement Learning (RL), it achieves an increase of 2.08\% in satisfaction rate, effectively addressing the issue of a decrease of 4.58\% in satisfaction rate after basic RLAIF.

Title: StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses

• Authors: Jia-Nan Li, Quan Tu, Cunli Mao, Zhengtao Yu, Ji-Rong Wen, Rui Yan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08312
• Pdf URL: https://arxiv.org/pdf/2403.08312
• Copy Paste: [[2403.08312]] StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses(https://arxiv.org/abs/2403.08312)
• Keywords: large language model
• Abstract: Standard Large Language Models (LLMs) struggle with handling dialogues with long contexts due to efficiency and consistency issues. According to our observation, dialogue contexts are highly structured, and the special token of \textit{End-of-Utterance} (EoU) in dialogues has the potential to aggregate information. We refer to the EoU tokens as ``conversational attention sinks'' (conv-attn sinks). Accordingly, we introduce StreamingDialogue, which compresses long dialogue history into conv-attn sinks with minimal losses, and thus reduces computational complexity quadratically with the number of sinks (i.e., the number of utterances). Current LLMs already demonstrate the ability to handle long context window, e.g., a window size of 200k or more. To this end, by compressing utterances into EoUs, our method has the potential to handle more than 200k of utterances, resulting in a prolonged dialogue learning. In order to minimize information losses from reconstruction after compression, we design two learning strategies of short-memory reconstruction (SMR) and long-memory reactivation (LMR). Our method outperforms strong baselines in dialogue tasks and achieves a 4 $\times$ speedup while reducing memory usage by 18 $\times$ compared to dense attention recomputation.

Title: Is Context Helpful for Chat Translation Evaluation?

• Authors: Sweta Agrawal, Amin Farajian, Patrick Fernandes, Ricardo Rei, André F.T. Martins
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08314
• Pdf URL: https://arxiv.org/pdf/2403.08314
• Copy Paste: [[2403.08314]] Is Context Helpful for Chat Translation Evaluation?(https://arxiv.org/abs/2403.08314)
• Keywords: large language model
• Abstract: Despite the recent success of automatic metrics for assessing translation quality, their application in evaluating the quality of machine-translated chats has been limited. Unlike more structured texts like news, chat conversations are often unstructured, short, and heavily reliant on contextual information. This poses questions about the reliability of existing sentence-level metrics in this domain as well as the role of context in assessing the translation quality. Motivated by this, we conduct a meta-evaluation of existing sentence-level automatic metrics, primarily designed for structured domains such as news, to assess the quality of machine-translated chats. We find that reference-free metrics lag behind reference-based ones, especially when evaluating translation quality in out-of-English settings. We then investigate how incorporating conversational contextual information in these metrics affects their performance. Our findings show that augmenting neural learned metrics with contextual information helps improve correlation with human judgments in the reference-free scenario and when evaluating translations in out-of-English settings. Finally, we propose a new evaluation metric, Context-MQM, that utilizes bilingual context with a large language model (LLM) and further validate that adding context helps even for LLM-based evaluation metrics.

Title: DrFER: Learning Disentangled Representations for 3D Facial Expression Recognition

• Authors: Hebeizi Li, Hongyu Yang, Di Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08318
• Pdf URL: https://arxiv.org/pdf/2403.08318
• Copy Paste: [[2403.08318]] DrFER: Learning Disentangled Representations for 3D Facial Expression Recognition(https://arxiv.org/abs/2403.08318)
• Keywords: extraction
• Abstract: Facial Expression Recognition (FER) has consistently been a focal point in the field of facial analysis. In the context of existing methodologies for 3D FER or 2D+3D FER, the extraction of expression features often gets entangled with identity information, compromising the distinctiveness of these features. To tackle this challenge, we introduce the innovative DrFER method, which brings the concept of disentangled representation learning to the field of 3D FER. DrFER employs a dual-branch framework to effectively disentangle expression information from identity information. Diverging from prior disentanglement endeavors in the 3D facial domain, we have carefully reconfigured both the loss functions and network structure to make the overall framework adaptable to point cloud data. This adaptation enhances the capability of the framework in recognizing facial expressions, even in cases involving varying head poses. Extensive evaluations conducted on the BU-3DFE and Bosphorus datasets substantiate that DrFER surpasses the performance of other 3D FER methods.

Title: Knowledge Conflicts for LLMs: A Survey

• Authors: Rongwu Xu, Zehan Qi, Cunxiang Wang, Hongru Wang, Yue Zhang, Wei Xu
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08319
• Pdf URL: https://arxiv.org/pdf/2403.08319
• Copy Paste: [[2403.08319]] Knowledge Conflicts for LLMs: A Survey(https://arxiv.org/abs/2403.08319)
• Keywords: robust, large language model
• Abstract: This survey provides an in-depth analysis of knowledge conflicts for large language models (LLMs), highlighting the complex challenges they encounter when blending contextual and parametric knowledge. Our focus is on three categories of knowledge conflicts: context-memory, inter-context, and intra-memory conflict. These conflicts can significantly impact the trustworthiness and performance of LLMs, especially in real-world applications where noise and misinformation are common. By categorizing these conflicts, exploring the causes, examining the behaviors of LLMs under such conflicts, and reviewing available solutions, this survey aims to shed light on strategies for improving the robustness of LLMs, thereby serving as a valuable resource for advancing research in this evolving area.

Title: Activating Wider Areas in Image Super-Resolution

• Authors: Cheng Cheng, Hang Wang, Hongbin Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08330
• Pdf URL: https://arxiv.org/pdf/2403.08330
• Copy Paste: [[2403.08330]] Activating Wider Areas in Image Super-Resolution(https://arxiv.org/abs/2403.08330)
• Keywords: transformer
• Abstract: The prevalence of convolution neural networks (CNNs) and vision transformers (ViTs) has markedly revolutionized the area of single-image super-resolution (SISR). To further boost the SR performances, several techniques, such as residual learning and attention mechanism, are introduced, which can be largely attributed to a wider range of activated area, that is, the input pixels that strongly influence the SR results. However, the possibility of further improving SR performance through another versatile vision backbone remains an unresolved challenge. To address this issue, in this paper, we unleash the representation potential of the modern state space model, i.e., Vision Mamba (Vim), in the context of SISR. Specifically, we present three recipes for better utilization of Vim-based models: 1) Integration into a MetaFormer-style block; 2) Pre-training on a larger and broader dataset; 3) Employing complementary attention mechanism, upon which we introduce the MMA. The resulting network MMA is capable of finding the most relevant and representative input pixels to reconstruct the corresponding high-resolution images. Comprehensive experimental analysis reveals that MMA not only achieves competitive or even superior performance compared to state-of-the-art SISR methods but also maintains relatively low memory and computational overheads (e.g., +0.5 dB PSNR elevation on Manga109 dataset with 19.8 M parameters at the scale of 2). Furthermore, MMA proves its versatility in lightweight SR applications. Through this work, we aim to illuminate the potential applications of state space models in the broader realm of image processing rather than SISR, encouraging further exploration in this innovative direction.

Title: Fast Inference of Removal-Based Node Influence

• Authors: Weikai Li, Zhiping Xiao, Xiao Luo, Yizhou Sun
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08333
• Pdf URL: https://arxiv.org/pdf/2403.08333
• Copy Paste: [[2403.08333]] Fast Inference of Removal-Based Node Influence(https://arxiv.org/abs/2403.08333)
• Keywords: attack
• Abstract: Graph neural networks (GNNs) are widely utilized to capture the information spreading patterns in graphs. While remarkable performance has been achieved, there is a new trending topic of evaluating node influence. We propose a new method of evaluating node influence, which measures the prediction change of a trained GNN model caused by removing a node. A real-world application is, "In the task of predicting Twitter accounts' polarity, had a particular account been removed, how would others' polarity change?". We use the GNN as a surrogate model whose prediction could simulate the change of nodes or edges caused by node removal. To obtain the influence for every node, a straightforward way is to alternately remove every node and apply the trained GNN on the modified graph. It is reliable but time-consuming, so we need an efficient method. The related lines of work, such as graph adversarial attack and counterfactual explanation, cannot directly satisfy our needs, since they do not focus on the global influence score for every node. We propose an efficient and intuitive method, NOde-Removal-based fAst GNN inference (NORA), which uses the gradient to approximate the node-removal influence. It only costs one forward propagation and one backpropagation to approximate the influence score for all nodes. Extensive experiments on six datasets and six GNN models verify the effectiveness of NORA. Our code is available at https://github.com/weikai-li/NORA.git.

Title: DONAPI: Malicious NPM Packages Detector using Behavior Sequence Knowledge Mapping

• Authors: Cheng Huang (1), Nannan Wang (1), Ziyan Wang (1), Siqi Sun (1), Lingzi Li (1), Junren Chen (1), Qianchong Zhao (1), Jiaxuan Han (1), Zhen Yang (1), Lei Shi (2) ((1) Sichuan University, (2) Huawei Technologies)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.08334
• Pdf URL: https://arxiv.org/pdf/2403.08334
• Copy Paste: [[2403.08334]] DONAPI: Malicious NPM Packages Detector using Behavior Sequence Knowledge Mapping(https://arxiv.org/abs/2403.08334)
• Keywords: security
• Abstract: With the growing popularity of modularity in software development comes the rise of package managers and language ecosystems. Among them, npm stands out as the most extensive package manager, hosting more than 2 million third-party open-source packages that greatly simplify the process of building code. However, this openness also brings security risks, as evidenced by numerous package poisoning incidents. In this paper, we synchronize a local package cache containing more than 3.4 million packages in near real-time to give us access to more package code details. Further, we perform manual inspection and API call sequence analysis on packages collected from public datasets and security reports to build a hierarchical classification framework and behavioral knowledge base covering different sensitive behaviors. In addition, we propose the DONAPI, an automatic malicious npm packages detector that combines static and dynamic analysis. It makes preliminary judgments on the degree of maliciousness of packages by code reconstruction techniques and static analysis, extracts dynamic API call sequences to confirm and identify obfuscated content that static analysis can not handle alone, and finally tags malicious software packages based on the constructed behavior knowledge base. To date, we have identified and manually confirmed 325 malicious samples and discovered 2 unusual API calls and 246 API call sequences that have not appeared in known samples.

Title: From human experts to machines: An LLM supported approach to ontology and knowledge graph construction

• Authors: Vamsi Krishna Kommineni, Birgitta König-Ries, Sheeba Samuel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08345
• Pdf URL: https://arxiv.org/pdf/2403.08345
• Copy Paste: [[2403.08345]] From human experts to machines: An LLM supported approach to ontology and knowledge graph construction(https://arxiv.org/abs/2403.08345)
• Keywords: large language model
• Abstract: The conventional process of building Ontologies and Knowledge Graphs (KGs) heavily relies on human domain experts to define entities and relationship types, establish hierarchies, maintain relevance to the domain, fill the ABox (or populate with instances), and ensure data quality (including amongst others accuracy and completeness). On the other hand, Large Language Models (LLMs) have recently gained popularity for their ability to understand and generate human-like natural language, offering promising ways to automate aspects of this process. This work explores the (semi-)automatic construction of KGs facilitated by open-source LLMs. Our pipeline involves formulating competency questions (CQs), developing an ontology (TBox) based on these CQs, constructing KGs using the developed ontology, and evaluating the resultant KG with minimal to no involvement of human experts. We showcase the feasibility of our semi-automated pipeline by creating a KG on deep learning methodologies by exploiting scholarly publications. To evaluate the answers generated via Retrieval-Augmented-Generation (RAG) as well as the KG concepts automatically extracted using LLMs, we design a judge LLM, which rates the generated content based on ground truth. Our findings suggest that employing LLMs could potentially reduce the human effort involved in the construction of KGs, although a human-in-the-loop approach is recommended to evaluate automatically generated KGs.

Title: CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model

• Authors: Cheng Chen, Junchen Zhu, Xu Luo, Hengtao Shen, Lianli Gao, Jingkuan Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08350
• Pdf URL: https://arxiv.org/pdf/2403.08350
• Copy Paste: [[2403.08350]] CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model(https://arxiv.org/abs/2403.08350)
• Keywords: large language model
• Abstract: Instruction tuning represents a prevalent strategy employed by Multimodal Large Language Models (MLLMs) to align with human instructions and adapt to new tasks. Nevertheless, MLLMs encounter the challenge of adapting to users' evolving knowledge and demands. Therefore, how to retain existing skills while acquiring new knowledge needs to be investigated. In this paper, we present a comprehensive benchmark, namely Continual Instruction tuNing (CoIN), to assess existing MLLMs in the sequential instruction tuning paradigm. CoIN comprises 10 commonly used datasets spanning 8 task categories, ensuring a diverse range of instructions and tasks. Besides, the trained model is evaluated from two aspects: Instruction Following and General Knowledge, which assess the alignment with human intention and knowledge preserved for reasoning, respectively. Experiments on CoIN demonstrate that current powerful MLLMs still suffer catastrophic forgetting, and the failure in intention alignment assumes the main responsibility, instead of the knowledge forgetting. To this end, we introduce MoELoRA to MLLMs which is effective to retain the previous instruction alignment. Experimental results consistently illustrate the forgetting decreased from this method on CoIN.

Title: Decoupled Federated Learning on Long-Tailed and Non-IID data with Feature Statistics

• Authors: Zhuoxin Chen, Zhenyu Wu, Yang Ji
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08364
• Pdf URL: https://arxiv.org/pdf/2403.08364
• Copy Paste: [[2403.08364]] Decoupled Federated Learning on Long-Tailed and Non-IID data with Feature Statistics(https://arxiv.org/abs/2403.08364)
• Keywords: security, privacy, federate
• Abstract: Federated learning is designed to enhance data security and privacy, but faces challenges when dealing with heterogeneous data in long-tailed and non-IID distributions. This paper explores an overlooked scenario where tail classes are sparsely distributed over a few clients, causing the models trained with these classes to have a lower probability of being selected during client aggregation, leading to slower convergence rates and poorer model performance. To address this issue, we propose a two-stage Decoupled Federated learning framework using Feature Statistics (DFL-FS). In the first stage, the server estimates the client's class coverage distributions through masked local feature statistics clustering to select models for aggregation to accelerate convergence and enhance feature learning without privacy leakage. In the second stage, DFL-FS employs federated feature regeneration based on global feature statistics and utilizes resampling and weighted covariance to calibrate the global classifier to enhance the model's adaptability to long-tailed data distributions. We conducted experiments on CIFAR10-LT and CIFAR100-LT datasets with various long-tailed rates. The results demonstrate that our method outperforms state-of-the-art methods in both accuracy and convergence rate.

Title: METER: a mobile vision transformer architecture for monocular depth estimation

• Authors: L. Papa, P. Russo, I. Amerini
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08368
• Pdf URL: https://arxiv.org/pdf/2403.08368
• Copy Paste: [[2403.08368]] METER: a mobile vision transformer architecture for monocular depth estimation(https://arxiv.org/abs/2403.08368)
• Keywords: transformer
• Abstract: Depth estimation is a fundamental knowledge for autonomous systems that need to assess their own state and perceive the surrounding environment. Deep learning algorithms for depth estimation have gained significant interest in recent years, owing to the potential benefits of this methodology in overcoming the limitations of active depth sensing systems. Moreover, due to the low cost and size of monocular cameras, researchers have focused their attention on monocular depth estimation (MDE), which consists in estimating a dense depth map from a single RGB video frame. State of the art MDE models typically rely on vision transformers (ViT) architectures that are highly deep and complex, making them unsuitable for fast inference on devices with hardware constraints. Purposely, in this paper, we address the problem of exploiting ViT in MDE on embedded devices. Those systems are usually characterized by limited memory capabilities and low-power CPU/GPU. We propose METER, a novel lightweight vision transformer architecture capable of achieving state of the art estimations and low latency inference performances on the considered embedded hardwares: NVIDIA Jetson TX1 and NVIDIA Jetson Nano. We provide a solution consisting of three alternative configurations of METER, a novel loss function to balance pixel estimation and reconstruction of image details, and a new data augmentation strategy to improve the overall final predictions. The proposed method outperforms previous lightweight works over the two benchmark datasets: the indoor NYU Depth v2 and the outdoor KITTI.

Title: Nonlinear Manifold Learning Determines Microgel Size from Raman Spectroscopy

• Authors: Eleni D. Koronaki, Luise F. Kaven, Johannes M. M. Faust, Ioannis G. Kevrekidis, Alexander Mitsos
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.08376
• Pdf URL: https://arxiv.org/pdf/2403.08376
• Copy Paste: [[2403.08376]] Nonlinear Manifold Learning Determines Microgel Size from Raman Spectroscopy(https://arxiv.org/abs/2403.08376)
• Keywords: diffusion
• Abstract: Polymer particle size constitutes a crucial characteristic of product quality in polymerization. Raman spectroscopy is an established and reliable process analytical technology for in-line concentration monitoring. Recent approaches and some theoretical considerations show a correlation between Raman signals and particle sizes but do not determine polymer size from Raman spectroscopic measurements accurately and reliably. With this in mind, we propose three alternative machine learning workflows to perform this task, all involving diffusion maps, a nonlinear manifold learning technique for dimensionality reduction: (i) directly from diffusion maps, (ii) alternating diffusion maps, and (iii) conformal autoencoder neural networks. We apply the workflows to a data set of Raman spectra with associated size measured via dynamic light scattering of 47 microgel (cross-linked polymer) samples in a diameter range of 208nm to 483 nm. The conformal autoencoders substantially outperform state-of-the-art methods and results for the first time in a promising prediction of polymer size from Raman spectra.

Title: Mitigate Target-level Insensitivity of Infrared Small Target Detection via Posterior Distribution Modeling

• Authors: Haoqing Li, Jinfu Yang, Yifei Xu, Runshi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08380
• Pdf URL: https://arxiv.org/pdf/2403.08380
• Copy Paste: [[2403.08380]] Mitigate Target-level Insensitivity of Infrared Small Target Detection via Posterior Distribution Modeling(https://arxiv.org/abs/2403.08380)
• Keywords: diffusion, generative, segmentation
• Abstract: Infrared Small Target Detection (IRSTD) aims to segment small targets from infrared clutter background. Existing methods mainly focus on discriminative approaches, i.e., a pixel-level front-background binary segmentation. Since infrared small targets are small and low signal-to-clutter ratio, empirical risk has few disturbances when a certain false alarm and missed detection exist, which seriously affect the further improvement of such methods. Motivated by the dense prediction generative methods, in this paper, we propose a diffusion model framework for Infrared Small Target Detection which compensates pixel-level discriminant with mask posterior distribution modeling. Furthermore, we design a Low-frequency Isolation in the wavelet domain to suppress the interference of intrinsic infrared noise on the diffusion noise estimation. This transition from the discriminative paradigm to generative one enables us to bypass the target-level insensitivity. Experiments show that the proposed method achieves competitive performance gains over state-of-the-art methods on NUAA-SIRST, IRSTD-1k, and NUDT-SIRST datasets. Code are available at https://github.com/Li-Haoqing/IRSTD-Diff.

Title: Tackling the Singularities at the Endpoints of Time Intervals in Diffusion Models

• Authors: Pengze Zhang, Hubery Yin, Chen Li, Xiaohua Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08381
• Pdf URL: https://arxiv.org/pdf/2403.08381
• Copy Paste: [[2403.08381]] Tackling the Singularities at the Endpoints of Time Intervals in Diffusion Models(https://arxiv.org/abs/2403.08381)
• Keywords: diffusion
• Abstract: Most diffusion models assume that the reverse process adheres to a Gaussian distribution. However, this approximation has not been rigorously validated, especially at singularities, where t=0 and t=1. Improperly dealing with such singularities leads to an average brightness issue in applications, and limits the generation of images with extreme brightness or darkness. We primarily focus on tackling singularities from both theoretical and practical perspectives. Initially, we establish the error bounds for the reverse process approximation, and showcase its Gaussian characteristics at singularity time steps. Based on this theoretical insight, we confirm the singularity at t=1 is conditionally removable while it at t=0 is an inherent property. Upon these significant conclusions, we propose a novel plug-and-play method SingDiffusion to address the initial singular time step sampling, which not only effectively resolves the average brightness issue for a wide range of diffusion models without extra training efforts, but also enhances their generation capability in achieving notable lower FID scores. Code and models are released at https://github.com/PangzeCheung/SingDiffusion.

Title: RAF-GI: Towards Robust, Accurate and Fast-Convergent Gradient Inversion Attack in Federated Learning

• Authors: Can Liu, Jin Wang, Dongyang Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08383
• Pdf URL: https://arxiv.org/pdf/2403.08383
• Copy Paste: [[2403.08383]] RAF-GI: Towards Robust, Accurate and Fast-Convergent Gradient Inversion Attack in Federated Learning(https://arxiv.org/abs/2403.08383)
• Keywords: privacy, attack, robust, federate
• Abstract: Federated learning (FL) empowers privacy-preservation in model training by only exposing users' model gradients. Yet, FL users are susceptible to the gradient inversion (GI) attack which can reconstruct ground-truth training data such as images based on model gradients. However, reconstructing high-resolution images by existing GI attack works faces two challenges: inferior accuracy and slow-convergence, especially when the context is complicated, e.g., the training batch size is much greater than 1 on each FL user. To address these challenges, we present a Robust, Accurate and Fast-convergent GI attack algorithm, called RAF-GI, with two components: 1) Additional Convolution Block (ACB) which can restore labels with up to 20% improvement compared with existing works; 2) Total variance, three-channel mEan and cAnny edge detection regularization term (TEA), which is a white-box attack strategy to reconstruct images based on labels inferred by ACB. Moreover, RAF-GI is robust that can still accurately reconstruct ground-truth data when the users' training batch size is no more than 48. Our experimental results manifest that RAF-GI can diminish 94% time costs while achieving superb inversion quality in ImageNet dataset. Notably, with a batch size of 1, RAF-GI exhibits a 7.89 higher Peak Signal-to-Noise Ratio (PSNR) compared to the state-of-the-art baselines.

Title: Iterative Online Image Synthesis via Diffusion Model for Imbalanced Classification

• Authors: Shuhan Li, Yi Lin, Hao Chen, Kwang-Ting Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08407
• Pdf URL: https://arxiv.org/pdf/2403.08407
• Copy Paste: [[2403.08407]] Iterative Online Image Synthesis via Diffusion Model for Imbalanced Classification(https://arxiv.org/abs/2403.08407)
• Keywords: robust, diffusion
• Abstract: Accurate and robust classification of diseases is important for proper diagnosis and treatment. However, medical datasets often face challenges related to limited sample sizes and inherent imbalanced distributions, due to difficulties in data collection and variations in disease prevalence across different types. In this paper, we introduce an Iterative Online Image Synthesis (IOIS) framework to address the class imbalance problem in medical image classification. Our framework incorporates two key modules, namely Online Image Synthesis (OIS) and Accuracy Adaptive Sampling (AAS), which collectively target the imbalance classification issue at both the instance level and the class level. The OIS module alleviates the data insufficiency problem by generating representative samples tailored for online training of the classifier. On the other hand, the AAS module dynamically balances the synthesized samples among various classes, targeting those with low training accuracy. To evaluate the effectiveness of our proposed method in addressing imbalanced classification, we conduct experiments on the HAM10000 and APTOS datasets. The results obtained demonstrate the superiority of our approach over state-of-the-art methods as well as the effectiveness of each component. The source code will be released upon acceptance.

Title: Causal Graph Neural Networks for Wildfire Danger Prediction

• Authors: Shan Zhao, Ioannis Prapas, Ilektra Karasante, Zhitong Xiong, Ioannis Papoutsis, Gustau Camps-Valls, Xiao Xiang Zhu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08414
• Pdf URL: https://arxiv.org/pdf/2403.08414
• Copy Paste: [[2403.08414]] Causal Graph Neural Networks for Wildfire Danger Prediction(https://arxiv.org/abs/2403.08414)
• Keywords: robust
• Abstract: Wildfire forecasting is notoriously hard due to the complex interplay of different factors such as weather conditions, vegetation types and human activities. Deep learning models show promise in dealing with this complexity by learning directly from data. However, to inform critical decision making, we argue that we need models that are right for the right reasons; that is, the implicit rules learned should be grounded by the underlying processes driving wildfires. In that direction, we propose integrating causality with Graph Neural Networks (GNNs) that explicitly model the causal mechanism among complex variables via graph learning. The causal adjacency matrix considers the synergistic effect among variables and removes the spurious links from highly correlated impacts. Our methodology's effectiveness is demonstrated through superior performance forecasting wildfire patterns in the European boreal and mediterranean biome. The gain is especially prominent in a highly imbalanced dataset, showcasing an enhanced robustness of the model to adapt to regime shifts in functional relationships. Furthermore, SHAP values from our trained model further enhance our understanding of the model's inner workings.

Title: Tastle: Distract Large Language Models for Automatic Jailbreak Attack

• Authors: Zeguan Xiao, Yan Yang, Guanhua Chen, Yun Chen
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08424
• Pdf URL: https://arxiv.org/pdf/2403.08424
• Copy Paste: [[2403.08424]] Tastle: Distract Large Language Models for Automatic Jailbreak Attack(https://arxiv.org/abs/2403.08424)
• Keywords: security, defense, attack, large language model
• Abstract: Large language models (LLMs) have achieved significant advances in recent days. Extensive efforts have been made before the public release of LLMs to align their behaviors with human values. The primary goal of alignment is to ensure their helpfulness, honesty and harmlessness. However, even meticulously aligned LLMs remain vulnerable to malicious manipulations such as jailbreaking, leading to unintended behaviors. The jailbreak is to intentionally develop a malicious prompt that escapes from the LLM security restrictions to produce uncensored detrimental contents. Previous works explore different jailbreak methods for red teaming LLMs, yet they encounter challenges regarding to effectiveness and scalability. In this work, we propose Tastle, a novel black-box jailbreak framework for automated red teaming of LLMs. We designed malicious content concealing and memory reframing with an iterative optimization algorithm to jailbreak LLMs, motivated by the research about the distractibility and over-confidence phenomenon of LLMs. Extensive experiments of jailbreaking both open-source and proprietary LLMs demonstrate the superiority of our framework in terms of effectiveness, scalability and transferability. We also evaluate the effectiveness of existing jailbreak defense methods against our attack and highlight the crucial need to develop more effective and practical defense strategies.

Title: Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation

• Authors: Zicheng Zhang, Tong Zhang, Yi Zhu, Jianzhuang Liu, Xiaodan Liang, QiXiang Ye, Wei Ke
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08426
• Pdf URL: https://arxiv.org/pdf/2403.08426
• Copy Paste: [[2403.08426]] Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation(https://arxiv.org/abs/2403.08426)
• Keywords: transformer, segmentation
• Abstract: The pre-trained vision-language model, exemplified by CLIP, advances zero-shot semantic segmentation by aligning visual features with class embeddings through a transformer decoder to generate semantic masks. Despite its effectiveness, prevailing methods within this paradigm encounter challenges, including overfitting on seen classes and small fragmentation in masks. To mitigate these issues, we propose a Language-Driven Visual Consensus (LDVC) approach, fostering improved alignment of semantic and visual information.Specifically, we leverage class embeddings as anchors due to their discrete and abstract nature, steering vision features toward class embeddings. Moreover, to circumvent noisy alignments from the vision part due to its redundant nature, we introduce route attention into self-attention for finding visual consensus, thereby enhancing semantic consistency within the same object. Equipped with a vision-language prompting strategy, our approach significantly boosts the generalization capacity of segmentation models for unseen classes. Experimental results underscore the effectiveness of our approach, showcasing mIoU gains of 4.5 on the PASCAL VOC 2012 and 3.6 on the COCO-Stuff 164k for unseen classes compared with the state-of-the-art methods.

Title: PFStorer: Personalized Face Restoration and Super-Resolution

• Authors: Tuomas Varanka, Tapani Toivonen, Soumya Tripathy, Guoying Zhao, Erman Acar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08436
• Pdf URL: https://arxiv.org/pdf/2403.08436
• Copy Paste: [[2403.08436]] PFStorer: Personalized Face Restoration and Super-Resolution(https://arxiv.org/abs/2403.08436)
• Keywords: robust, diffusion, generative
• Abstract: Recent developments in face restoration have achieved remarkable results in producing high-quality and lifelike outputs. The stunning results however often fail to be faithful with respect to the identity of the person as the models lack necessary context. In this paper, we explore the potential of personalized face restoration with diffusion models. In our approach a restoration model is personalized using a few images of the identity, leading to tailored restoration with respect to the identity while retaining fine-grained details. By using independent trainable blocks for personalization, the rich prior of a base restoration model can be exploited to its fullest. To avoid the model relying on parts of identity left in the conditioning low-quality images, a generative regularizer is employed. With a learnable parameter, the model learns to balance between the details generated based on the input image and the degree of personalization. Moreover, we improve the training pipeline of face restoration models to enable an alignment-free approach. We showcase the robust capabilities of our approach in several real-world scenarios with multiple identities, demonstrating our method's ability to generate fine-grained details with faithful restoration. In the user study we evaluate the perceptual quality and faithfulness of the genereated details, with our method being voted best 61% of the time compared to the second best with 25% of the votes.

Title: Reproducibility and Geometric Intrinsic Dimensionality: An Investigation on Graph Neural Network Research

• Authors: Tobias Hille, Maximilian Stubbemann, Tom Hanika
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08438
• Pdf URL: https://arxiv.org/pdf/2403.08438
• Copy Paste: [[2403.08438]] Reproducibility and Geometric Intrinsic Dimensionality: An Investigation on Graph Neural Network Research(https://arxiv.org/abs/2403.08438)
• Keywords: robust
• Abstract: Difficulties in replication and reproducibility of empirical evidences in machine learning research have become a prominent topic in recent years. Ensuring that machine learning research results are sound and reliable requires reproducibility, which verifies the reliability of research findings using the same code and data. This promotes open and accessible research, robust experimental workflows, and the rapid integration of new findings. Evaluating the degree to which research publications support these different aspects of reproducibility is one goal of the present work. For this we introduce an ontology of reproducibility in machine learning and apply it to methods for graph neural networks. Building on these efforts we turn towards another critical challenge in machine learning, namely the curse of dimensionality, which poses challenges in data collection, representation, and analysis, making it harder to find representative data and impeding the training and inference processes. Using the closely linked concept of geometric intrinsic dimension we investigate to which extend the used machine learning models are influenced by the intrinsic dimension of the data sets they are trained on.

Title: Actor-Critic Physics-informed Neural Lyapunov Control

• Authors: Jiarui Wang, Mahyar Fazlyab
• Subjects: cs.LG, cs.RO, eess.SY
• Abstract URL: https://arxiv.org/abs/2403.08448
• Pdf URL: https://arxiv.org/pdf/2403.08448
• Copy Paste: [[2403.08448]] Actor-Critic Physics-informed Neural Lyapunov Control(https://arxiv.org/abs/2403.08448)
• Keywords: robust
• Abstract: Designing control policies for stabilization tasks with provable guarantees is a long-standing problem in nonlinear control. A crucial performance metric is the size of the resulting region of attraction, which essentially serves as a robustness "margin" of the closed-loop system against uncertainties. In this paper, we propose a new method to train a stabilizing neural network controller along with its corresponding Lyapunov certificate, aiming to maximize the resulting region of attraction while respecting the actuation constraints. Crucial to our approach is the use of Zubov's Partial Differential Equation (PDE), which precisely characterizes the true region of attraction of a given control policy. Our framework follows an actor-critic pattern where we alternate between improving the control policy (actor) and learning a Zubov function (critic). Finally, we compute the largest certifiable region of attraction by invoking an SMT solver after the training procedure. Our numerical experiments on several design problems show consistent and significant improvements in the size of the resulting region of attraction.

Title: Towards Dense and Accurate Radar Perception Via Efficient Cross-Modal Diffusion Model

• Authors: Ruibin Zhang, Donglai Xue, Yuhan Wang, Ruixu Geng, Fei Gao
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.08460
• Pdf URL: https://arxiv.org/pdf/2403.08460
• Copy Paste: [[2403.08460]] Towards Dense and Accurate Radar Perception Via Efficient Cross-Modal Diffusion Model(https://arxiv.org/abs/2403.08460)
• Keywords: diffusion, generative
• Abstract: Millimeter wave (mmWave) radars have attracted significant attention from both academia and industry due to their capability to operate in extreme weather conditions. However, they face challenges in terms of sparsity and noise interference, which hinder their application in the field of micro aerial vehicle (MAV) autonomous navigation. To this end, this paper proposes a novel approach to dense and accurate mmWave radar point cloud construction via cross-modal learning. Specifically, we introduce diffusion models, which possess state-of-the-art performance in generative modeling, to predict LiDAR-like point clouds from paired raw radar data. We also incorporate the most recent diffusion model inference accelerating techniques to ensure that the proposed method can be implemented on MAVs with limited computing resources.We validate the proposed method through extensive benchmark comparisons and real-world experiments, demonstrating its superior performance and generalization ability. Code and pretrained models will be available at https://github.com/ZJU-FAST-Lab/Radar-Diffusion.

Title: Authorship Verification based on the Likelihood Ratio of Grammar Models

• Authors: Andrea Nini, Oren Halvani, Lukas Graner, Valerio Gherardi, Shunichi Ishihara
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08462
• Pdf URL: https://arxiv.org/pdf/2403.08462
• Copy Paste: [[2403.08462]] Authorship Verification based on the Likelihood Ratio of Grammar Models(https://arxiv.org/abs/2403.08462)
• Keywords: robust, transformer
• Abstract: Authorship Verification (AV) is the process of analyzing a set of documents to determine whether they were written by a specific author. This problem often arises in forensic scenarios, e.g., in cases where the documents in question constitute evidence for a crime. Existing state-of-the-art AV methods use computational solutions that are not supported by a plausible scientific explanation for their functioning and that are often difficult for analysts to interpret. To address this, we propose a method relying on calculating a quantity we call $\lambda_G$ (LambdaG): the ratio between the likelihood of a document given a model of the Grammar for the candidate author and the likelihood of the same document given a model of the Grammar for a reference population. These Grammar Models are estimated using $n$-gram language models that are trained solely on grammatical features. Despite not needing large amounts of data for training, LambdaG still outperforms other established AV methods with higher computational complexity, including a fine-tuned Siamese Transformer network. Our empirical evaluation based on four baseline methods applied to twelve datasets shows that LambdaG leads to better results in terms of both accuracy and AUC in eleven cases and in all twelve cases if considering only topic-agnostic methods. The algorithm is also highly robust to important variations in the genre of the reference population in many cross-genre comparisons. In addition to these properties, we demonstrate how LambdaG is easier to interpret than the current state-of-the-art. We argue that the advantage of LambdaG over other methods is due to fact that it is compatible with Cognitive Linguistic theories of language processing.

Title: An Analysis of Human Alignment of Latent Diffusion Models

• Authors: Lorenz Linhardt, Marco Morik, Sidney Bender, Naima Elosegui Borras
• Subjects: cs.LG, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.08469
• Pdf URL: https://arxiv.org/pdf/2403.08469
• Copy Paste: [[2403.08469]] An Analysis of Human Alignment of Latent Diffusion Models(https://arxiv.org/abs/2403.08469)
• Keywords: diffusion
• Abstract: Diffusion models, trained on large amounts of data, showed remarkable performance for image synthesis. They have high error consistency with humans and low texture bias when used for classification. Furthermore, prior work demonstrated the decomposability of their bottleneck layer representations into semantic directions. In this work, we analyze how well such representations are aligned to human responses on a triplet odd-one-out task. We find that despite the aforementioned observations: I) The representational alignment with humans is comparable to that of models trained only on ImageNet-1k. II) The most aligned layers of the denoiser U-Net are intermediate layers and not the bottleneck. III) Text conditioning greatly improves alignment at high noise levels, hinting at the importance of abstract textual information, especially in the early stage of generation.

Title: SoK: Reducing the Vulnerability of Fine-tuned Language Models to Membership Inference Attacks

• Authors: Guy Amit, Abigail Goldsteen, Ariel Farkash
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.08481
• Pdf URL: https://arxiv.org/pdf/2403.08481
• Copy Paste: [[2403.08481]] SoK: Reducing the Vulnerability of Fine-tuned Language Models to Membership Inference Attacks(https://arxiv.org/abs/2403.08481)
• Keywords: privacy, protect, defense, attack, membership infer, large language model
• Abstract: Natural language processing models have experienced a significant upsurge in recent years, with numerous applications being built upon them. Many of these applications require fine-tuning generic base models on customized, proprietary datasets. This fine-tuning data is especially likely to contain personal or sensitive information about individuals, resulting in increased privacy risk. Membership inference attacks are the most commonly employed attack to assess the privacy leakage of a machine learning model. However, limited research is available on the factors that affect the vulnerability of language models to this kind of attack, or on the applicability of different defense strategies in the language domain. We provide the first systematic review of the vulnerability of fine-tuned large language models to membership inference attacks, the various factors that come into play, and the effectiveness of different defense strategies. We find that some training methods provide significantly reduced privacy risk, with the combination of differential privacy and low-rank adaptors achieving the best privacy protection against these attacks.

Title: Data-oriented Dynamic Fine-tuning Parameter Selection Strategy for FISH Mask based Efficient Fine-tuning

• Authors: Ming Dong, Kang Xue, Bolong Zheng, Tingting He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08484
• Pdf URL: https://arxiv.org/pdf/2403.08484
• Copy Paste: [[2403.08484]] Data-oriented Dynamic Fine-tuning Parameter Selection Strategy for FISH Mask based Efficient Fine-tuning(https://arxiv.org/abs/2403.08484)
• Keywords: large language model
• Abstract: In view of the huge number of parameters of Large language models (LLMs) , tuning all parameters is very costly, and accordingly fine-tuning specific parameters is more sensible. Most of parameter efficient fine-tuning (PEFT) concentrate on parameter selection strategies, such as additive method, selective method and reparametrization-based method. However, there are few methods that consider the impact of data samples on parameter selecting, such as Fish Mask based method. Fish Mask randomly choose a part of data samples and treat them equally during parameter selection, which is unable to dynamically select optimal parameters for inconstant data distributions. In this work, we adopt a data-oriented perspective, then proposing an IRD ($\mathrm{\underline I}$terative sample-parameter $\mathrm{\underline R}$ange $\mathrm{\underline D}$ecreasing) algorithm to search the best setting of sample-parameter pair for FISH Mask. In each iteration, by searching the set of samples and parameters with larger Fish information, IRD can find better sample-parameter pair in most scale. We demonstrate the effectiveness and rationality of proposed strategy by conducting experiments on GLUE benchmark. Experimental results show our strategy optimizes the parameter selection and achieves preferable performance.

Title: Model Will Tell: Training Membership Inference for Diffusion Models

• Authors: Xiaomeng Fu, Xi Wang, Qiao Li, Jin Liu, Jiao Dai, Jizhong Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08487
• Pdf URL: https://arxiv.org/pdf/2403.08487
• Copy Paste: [[2403.08487]] Model Will Tell: Training Membership Inference for Diffusion Models(https://arxiv.org/abs/2403.08487)
• Keywords: privacy, membership infer, diffusion, generative
• Abstract: Diffusion models pose risks of privacy breaches and copyright disputes, primarily stemming from the potential utilization of unauthorized data during the training phase. The Training Membership Inference (TMI) task aims to determine whether a specific sample has been used in the training process of a target model, representing a critical tool for privacy violation verification. However, the increased stochasticity inherent in diffusion renders traditional shadow-model-based or metric-based methods ineffective when applied to diffusion models. Moreover, existing methods only yield binary classification labels which lack necessary comprehensibility in practical applications. In this paper, we explore a novel perspective for the TMI task by leveraging the intrinsic generative priors within the diffusion model. Compared with unseen samples, training samples exhibit stronger generative priors within the diffusion model, enabling the successful reconstruction of substantially degraded training images. Consequently, we propose the Degrade Restore Compare (DRC) framework. In this framework, an image undergoes sequential degradation and restoration, and its membership is determined by comparing it with the restored counterpart. Experimental results verify that our approach not only significantly outperforms existing methods in terms of accuracy but also provides comprehensible decision criteria, offering evidence for potential privacy violations.

Title: Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking

• Authors: Ming Dong, Yujing Chen, Miao Zhang, Hao Sun, Tingting He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08492
• Pdf URL: https://arxiv.org/pdf/2403.08492
• Copy Paste: [[2403.08492]] Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking(https://arxiv.org/abs/2403.08492)
• Keywords: large language model
• Abstract: Chinese Spell Checking (CSC) is a widely used technology, which plays a vital role in speech to text (STT) and optical character recognition (OCR). Most of the existing CSC approaches relying on BERT architecture achieve excellent performance. However, limited by the scale of the foundation model, BERT-based method does not work well in few-shot scenarios, showing certain limitations in practical applications. In this paper, we explore using an in-context learning method named RS-LLM (Rich Semantic based LLMs) to introduce large language models (LLMs) as the foundation model. Besides, we study the impact of introducing various Chinese rich semantic information in our framework. We found that by introducing a small number of specific Chinese rich semantic structures, LLMs achieve better performance than the BERT-based model on few-shot CSC task. Furthermore, we conduct experiments on multiple datasets, and the experimental results verified the superiority of our proposed framework.

Title: Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator

• Authors: Yusheng Liao, Yutong Meng, Yuhao Wang, Hongcheng Liu, Yanfeng Wang, Yu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08495
• Pdf URL: https://arxiv.org/pdf/2403.08495
• Copy Paste: [[2403.08495]] Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator(https://arxiv.org/abs/2403.08495)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable proficiency in human interactions, yet their application within the medical field remains insufficiently explored. Previous works mainly focus on the performance of medical knowledge with examinations, which is far from the realistic scenarios, falling short in assessing the abilities of LLMs on clinical tasks. In the quest to enhance the application of Large Language Models (LLMs) in healthcare, this paper introduces the Automated Interactive Evaluation (AIE) framework and the State-Aware Patient Simulator (SAPS), targeting the gap between traditional LLM evaluations and the nuanced demands of clinical practice. Unlike prior methods that rely on static medical knowledge assessments, AIE and SAPS provide a dynamic, realistic platform for assessing LLMs through multi-turn doctor-patient simulations. This approach offers a closer approximation to real clinical scenarios and allows for a detailed analysis of LLM behaviors in response to complex patient interactions. Our extensive experimental validation demonstrates the effectiveness of the AIE framework, with outcomes that align well with human evaluations, underscoring its potential to revolutionize medical LLM testing for improved healthcare delivery.

Title: Masked Generative Story Transformer with Character Guidance and Caption Augmentation

• Authors: Christos Papadimitriou, Giorgos Filandrianos, Maria Lymperaiou, Giorgos Stamou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08502
• Pdf URL: https://arxiv.org/pdf/2403.08502
• Copy Paste: [[2403.08502]] Masked Generative Story Transformer with Character Guidance and Caption Augmentation(https://arxiv.org/abs/2403.08502)
• Keywords: robust, transformer, generative, large language model
• Abstract: Story Visualization (SV) is a challenging generative vision task, that requires both visual quality and consistency between different frames in generated image sequences. Previous approaches either employ some kind of memory mechanism to maintain context throughout an auto-regressive generation of the image sequence, or model the generation of the characters and their background separately, to improve the rendering of characters. On the contrary, we embrace a completely parallel transformer-based approach, exclusively relying on Cross-Attention with past and future captions to achieve consistency. Additionally, we propose a Character Guidance technique to focus on the generation of characters in an implicit manner, by forming a combination of text-conditional and character-conditional logits in the logit space. We also employ a caption-augmentation technique, carried out by a Large Language Model (LLM), to enhance the robustness of our approach. The combination of these methods culminates into state-of-the-art (SOTA) results over various metrics in the most prominent SV benchmark (Pororo-SV), attained with constraint resources while achieving superior computational complexity compared to previous arts. The validity of our quantitative results is supported by a human survey.

Title: DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning

• Authors: Sikai Bai, Jie Zhang, Shuaicheng Li, Song Guo, Jingcai Guo, Jun Hou, Tao Han, Xiaocheng Lu
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08506
• Pdf URL: https://arxiv.org/pdf/2403.08506
• Copy Paste: [[2403.08506]] DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning(https://arxiv.org/abs/2403.08506)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) has emerged as a powerful paradigm for learning from decentralized data, and federated domain generalization further considers the test dataset (target domain) is absent from the decentralized training data (source domains). However, most existing FL methods assume that domain labels are provided during training, and their evaluation imposes explicit constraints on the number of domains, which must strictly match the number of clients. Because of the underutilization of numerous edge devices and additional cross-client domain annotations in the real world, such restrictions may be impractical and involve potential privacy leaks. In this paper, we propose an efficient and novel approach, called Disentangled Prompt Tuning (DiPrompT), a method that tackles the above restrictions by learning adaptive prompts for domain generalization in a distributed manner. Specifically, we first design two types of prompts, i.e., global prompt to capture general knowledge across all clients and domain prompts to capture domain-specific knowledge. They eliminate the restriction on the one-to-one mapping between source domains and local clients. Furthermore, a dynamic query metric is introduced to automatically search the suitable domain label for each sample, which includes two-substep text-image alignments based on prompt tuning without labor-intensive annotation. Extensive experiments on multiple datasets demonstrate that our DiPrompT achieves superior domain generalization performance over state-of-the-art FL methods when domain labels are not provided, and even outperforms many centralized learning methods using domain labels.

Title: A Multimodal Fusion Network For Student Emotion Recognition Based on Transformer and Tensor Product

• Authors: Ao Xiang, Zongqing Qi, Han Wang, Qin Yang, Danqing Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08511
• Pdf URL: https://arxiv.org/pdf/2403.08511
• Copy Paste: [[2403.08511]] A Multimodal Fusion Network For Student Emotion Recognition Based on Transformer and Tensor Product(https://arxiv.org/abs/2403.08511)
• Keywords: transformer
• Abstract: In recent years, there have been frequent incidents of foreign objects intruding into railway and Airport runways. These objects can include pedestrians, vehicles, animals, and debris. This paper introduces an improved YOLOv5 architecture incorporating FasterNet and attention mechanisms to enhance the detection of foreign objects on railways and Airport runways. This study proposes a new dataset, AARFOD (Aero and Rail Foreign Object Detection), which combines two public datasets for detecting foreign objects in aviation and railway systems. The dataset aims to improve the recognition capabilities of foreign object targets. Experimental results on this large dataset have demonstrated significant performance improvements of the proposed model over the baseline YOLOv5 model, reducing computational requirements. improved YOLO model shows a significant improvement in precision by 1.2%, recall rate by 1.0%, and mAP@.5 by 0.6%, while mAP@.5-.95 remained unchanged. The parameters were reduced by approximately 25.12%, and GFLOPs were reduced by about 10.63%. In the ablation experiment, it is found that the FasterNet module can significantly reduce the number of parameters of the model, and the reference of the attention mechanism can slow down the performance loss caused by lightweight.

Title: Pig aggression classification using CNN, Transformers and Recurrent Networks

• Authors: Junior Silva Souza, Eduardo Bedin, Gabriel Toshio Hirokawa Higa, Newton Loebens, Hemerson Pistori
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08528
• Pdf URL: https://arxiv.org/pdf/2403.08528
• Copy Paste: [[2403.08528]] Pig aggression classification using CNN, Transformers and Recurrent Networks(https://arxiv.org/abs/2403.08528)
• Keywords: transformer
• Abstract: The development of techniques that can be used to analyze and detect animal behavior is a crucial activity for the livestock sector, as it is possible to monitor the stress and animal welfare and contributes to decision making in the farm. Thus, the development of applications can assist breeders in making decisions to improve production performance and reduce costs, once the animal behavior is analyzed by humans and this can lead to susceptible errors and time consumption. Aggressiveness in pigs is an example of behavior that is studied to reduce its impact through animal classification and identification. However, this process is laborious and susceptible to errors, which can be reduced through automation by visually classifying videos captured in controlled environment. The captured videos can be used for training and, as a result, for classification through computer vision and artificial intelligence, employing neural network techniques. The main techniques utilized in this study are variants of transformers: STAM, TimeSformer, and ViViT, as well as techniques using convolutions, such as ResNet3D2, Resnet(2+1)D, and CnnLstm. These techniques were employed for pig video classification with the objective of identifying aggressive and non-aggressive behaviors. In this work, various techniques were compared to analyze the contribution of using transformers, in addition to the effectiveness of the convolution technique in video classification. The performance was evaluated using accuracy, precision, and recall. The TimerSformer technique showed the best results in video classification, with median accuracy of 0.729.

Title: HOLMES: HOLonym-MEronym based Semantic inspection for Convolutional Image Classifiers

• Authors: Francesco Dibitonto, Fabio Garcea, André Panisson, Alan Perotti, Lia Morra
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08536
• Pdf URL: https://arxiv.org/pdf/2403.08536
• Copy Paste: [[2403.08536]] HOLMES: HOLonym-MEronym based Semantic inspection for Convolutional Image Classifiers(https://arxiv.org/abs/2403.08536)
• Keywords: extraction
• Abstract: Convolutional Neural Networks (CNNs) are nowadays the model of choice in Computer Vision, thanks to their ability to automatize the feature extraction process in visual tasks. However, the knowledge acquired during training is fully subsymbolic, and hence difficult to understand and explain to end users. In this paper, we propose a new technique called HOLMES (HOLonym-MEronym based Semantic inspection) that decomposes a label into a set of related concepts, and provides component-level explanations for an image classification model. Specifically, HOLMES leverages ontologies, web scraping and transfer learning to automatically construct meronym (parts)-based detectors for a given holonym (class). Then, it produces heatmaps at the meronym level and finally, by probing the holonym CNN with occluded images, it highlights the importance of each part on the classification output. Compared to state-of-the-art saliency methods, HOLMES takes a step further and provides information about both where and what the holonym CNN is looking at, without relying on densely annotated datasets and without forcing concepts to be associated to single computational units. Extensive experimental evaluation on different categories of objects (animals, tools and vehicles) shows the feasibility of our approach. On average, HOLMES explanations include at least two meronyms, and the ablation of a single meronym roughly halves the holonym model confidence. The resulting heatmaps were quantitatively evaluated using the deletion/insertion/preservation curves. All metrics were comparable to those achieved by GradCAM, while offering the advantage of further decomposing the heatmap in human-understandable concepts, thus highlighting both the relevance of meronyms to object classification, as well as HOLMES ability to capture it. The code is available at https://github.com/FrancesC0de/HOLMES.

Title: CINA: Conditional Implicit Neural Atlas for Spatio-Temporal Representation of Fetal Brains

• Authors: Maik Dannecker, Vanessa Kyriakopoulou, Lucilio Cordero-Grande, Anthony N. Price, Joseph V. Hajnal, Daniel Rueckert
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08550
• Pdf URL: https://arxiv.org/pdf/2403.08550
• Copy Paste: [[2403.08550]] CINA: Conditional Implicit Neural Atlas for Spatio-Temporal Representation of Fetal Brains(https://arxiv.org/abs/2403.08550)
• Keywords: segmentation
• Abstract: We introduce a conditional implicit neural atlas (CINA) for spatio-temporal atlas generation from Magnetic Resonance Images (MRI) of the neurotypical and pathological fetal brain, that is fully independent of affine or non-rigid registration. During training, CINA learns a general representation of the fetal brain and encodes subject specific information into latent code. After training, CINA can construct a faithful atlas with tissue probability maps of the fetal brain for any gestational age (GA) and anatomical variation covered within the training domain. Thus, CINA is competent to represent both, neurotypical and pathological brains. Furthermore, a trained CINA model can be fit to brain MRI of unseen subjects via test-time optimization of the latent code. CINA can then produce probabilistic tissue maps tailored to a particular subject. We evaluate our method on a total of 198 T2 weighted MRI of normal and abnormal fetal brains from the dHCP and FeTA datasets. We demonstrate CINA's capability to represent a fetal brain atlas that can be flexibly conditioned on GA and on anatomical variations like ventricular volume or degree of cortical folding, making it a suitable tool for modeling both neurotypical and pathological brains. We quantify the fidelity of our atlas by means of tissue segmentation and age prediction and compare it to an established baseline. CINA demonstrates superior accuracy for neurotypical brains and pathological brains with ventriculomegaly. Moreover, CINA scores a mean absolute error of 0.23 weeks in fetal brain age prediction, further confirming an accurate representation of fetal brain development.

Title: Federated Knowledge Graph Unlearning via Diffusion Model

• Authors: Bingchen Liu, Yuanyuan Fang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08554
• Pdf URL: https://arxiv.org/pdf/2403.08554
• Copy Paste: [[2403.08554]] Federated Knowledge Graph Unlearning via Diffusion Model(https://arxiv.org/abs/2403.08554)
• Keywords: privacy, protect, federate, diffusion
• Abstract: Federated learning (FL) promotes the development and application of artificial intelligence technologies by enabling model sharing and collaboration while safeguarding data privacy. Knowledge graph (KG) embedding representation provides a foundation for knowledge reasoning and applications by mapping entities and relations into vector space. Federated KG embedding enables the utilization of knowledge from diverse client sources while safeguarding the privacy of local data. However, due to demands such as privacy protection and the need to adapt to dynamic data changes, investigations into machine unlearning (MU) have been sparked. However, it is challenging to maintain the performance of KG embedding models while forgetting the influence of specific forgotten data on the model. In this paper, we propose FedDM, a novel framework tailored for machine unlearning in federated knowledge graphs. Leveraging diffusion models, we generate noisy data to sensibly mitigate the influence of specific knowledge on FL models while preserving the overall performance concerning the remaining data. We conduct experimental evaluations on benchmark datasets to assess the efficacy of the proposed model. Extensive experiments demonstrate that FedDM yields promising results in knowledge forgetting.

Title: Non-discrimination Criteria for Generative Language Models

• Authors: Sara Sterlie, Nina Weng, Aasa Feragen
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.08564
• Pdf URL: https://arxiv.org/pdf/2403.08564
• Copy Paste: [[2403.08564]] Non-discrimination Criteria for Generative Language Models(https://arxiv.org/abs/2403.08564)
• Keywords: generative, large language model
• Abstract: Within recent years, generative AI, such as large language models, has undergone rapid development. As these models become increasingly available to the public, concerns arise about perpetuating and amplifying harmful biases in applications. Gender stereotypes can be harmful and limiting for the individuals they target, whether they consist of misrepresentation or discrimination. Recognizing gender bias as a pervasive societal construct, this paper studies how to uncover and quantify the presence of gender biases in generative language models. In particular, we derive generative AI analogues of three well-known non-discrimination criteria from classification, namely independence, separation and sufficiency. To demonstrate these criteria in action, we design prompts for each of the criteria with a focus on occupational gender stereotype, specifically utilizing the medical test to introduce the ground truth in the generative AI context. Our results address the presence of occupational gender bias within such conversational language models.

Title: Consistent Prompting for Rehearsal-Free Continual Learning

• Authors: Zhanxin Gao, Jun Cen, Xiaobin Chang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08568
• Pdf URL: https://arxiv.org/pdf/2403.08568
• Copy Paste: [[2403.08568]] Consistent Prompting for Rehearsal-Free Continual Learning(https://arxiv.org/abs/2403.08568)
• Keywords: robust
• Abstract: Continual learning empowers models to adapt autonomously to the ever-changing environment or data streams without forgetting old knowledge. Prompt-based approaches are built on frozen pre-trained models to learn the task-specific prompts and classifiers efficiently. Existing prompt-based methods are inconsistent between training and testing, limiting their effectiveness. Two types of inconsistency are revealed. Test predictions are made from all classifiers while training only focuses on the current task classifier without holistic alignment, leading to Classifier inconsistency. Prompt inconsistency indicates that the prompt selected during testing may not correspond to the one associated with this task during training. In this paper, we propose a novel prompt-based method, Consistent Prompting (CPrompt), for more aligned training and testing. Specifically, all existing classifiers are exposed to prompt training, resulting in classifier consistency learning. In addition, prompt consistency learning is proposed to enhance prediction robustness and boost prompt selection accuracy. Our Consistent Prompting surpasses its prompt-based counterparts and achieves state-of-the-art performance on multiple continual learning benchmarks. Detailed analysis shows that improvements come from more consistent training and testing.

Title: A Physics-driven GraphSAGE Method for Physical Process Simulations Described by Partial Differential Equations

• Authors: Hang Hu, Sidi Wu, Guoxiong Cai, Na Liu
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2403.08569
• Pdf URL: https://arxiv.org/pdf/2403.08569
• Copy Paste: [[2403.08569]] A Physics-driven GraphSAGE Method for Physical Process Simulations Described by Partial Differential Equations(https://arxiv.org/abs/2403.08569)
• Keywords: robust
• Abstract: Physics-informed neural networks (PINNs) have successfully addressed various computational physics problems based on partial differential equations (PDEs). However, while tackling issues related to irregularities like singularities and oscillations, trained solutions usually suffer low accuracy. In addition, most current works only offer the trained solution for predetermined input parameters. If any change occurs in input parameters, transfer learning or retraining is required, and traditional numerical techniques also need an independent simulation. In this work, a physics-driven GraphSAGE approach (PD-GraphSAGE) based on the Galerkin method and piecewise polynomial nodal basis functions is presented to solve computational problems governed by irregular PDEs and to develop parametric PDE surrogate models. This approach employs graph representations of physical domains, thereby reducing the demands for evaluated points due to local refinement. A distance-related edge feature and a feature mapping strategy are devised to help training and convergence for singularity and oscillation situations, respectively. The merits of the proposed method are demonstrated through a couple of cases. Moreover, the robust PDE surrogate model for heat conduction problems parameterized by the Gaussian random field source is successfully established, which not only provides the solution accurately but is several times faster than the finite element method in our experiments.

Title: Caformer: Rethinking Time Series Analysis from Causal Perspective

• Authors: Kexuan Zhang, Xiaobei Zou, Yang Tang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08572
• Pdf URL: https://arxiv.org/pdf/2403.08572
• Copy Paste: [[2403.08572]] Caformer: Rethinking Time Series Analysis from Causal Perspective(https://arxiv.org/abs/2403.08572)
• Keywords: robust, interpretability
• Abstract: Time series analysis is a vital task with broad applications in various domains. However, effectively capturing cross-dimension and cross-time dependencies in non-stationary time series poses significant challenges, particularly in the context of environmental factors. The spurious correlation induced by the environment confounds the causal relationships between cross-dimension and cross-time dependencies. In this paper, we introduce a novel framework called Caformer (\underline{\textbf{Ca}}usal Trans\underline{\textbf{former}}) for time series analysis from a causal perspective. Specifically, our framework comprises three components: Dynamic Learner, Environment Learner, and Dependency Learner. The Dynamic Learner unveils dynamic interactions among dimensions, the Environment Learner mitigates spurious correlations caused by environment with a back-door adjustment, and the Dependency Learner aims to infer robust interactions across both time and dimensions. Our Caformer demonstrates consistent state-of-the-art performance across five mainstream time series analysis tasks, including long- and short-term forecasting, imputation, classification, and anomaly detection, with proper interpretability.

Title: Improving Implicit Regularization of SGD with Preconditioning for Least Square Problems

• Authors: Junwei Su, Difan Zou, Chuan Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08585
• Pdf URL: https://arxiv.org/pdf/2403.08585
• Copy Paste: [[2403.08585]] Improving Implicit Regularization of SGD with Preconditioning for Least Square Problems(https://arxiv.org/abs/2403.08585)
• Keywords: robust
• Abstract: Stochastic gradient descent (SGD) exhibits strong algorithmic regularization effects in practice and plays an important role in the generalization of modern machine learning. However, prior research has revealed instances where the generalization performance of SGD is worse than ridge regression due to uneven optimization along different dimensions. Preconditioning offers a natural solution to this issue by rebalancing optimization across different directions. Yet, the extent to which preconditioning can enhance the generalization performance of SGD and whether it can bridge the existing gap with ridge regression remains uncertain. In this paper, we study the generalization performance of SGD with preconditioning for the least squared problem. We make a comprehensive comparison between preconditioned SGD and (standard \& preconditioned) ridge regression. Our study makes several key contributions toward understanding and improving SGD with preconditioning. First, we establish excess risk bounds (generalization performance) for preconditioned SGD and ridge regression under an arbitrary preconditions matrix. Second, leveraging the excessive risk characterization of preconditioned SGD and ridge regression, we show that (through construction) there exists a simple preconditioned matrix that can outperform (standard \& preconditioned) ridge regression. Finally, we show that our proposed preconditioning matrix is straightforward enough to allow robust estimation from finite samples while maintaining a theoretical advantage over ridge regression. Our empirical results align with our theoretical findings, collectively showcasing the enhanced regularization effect of preconditioned SGD.

Title: PRAGO: Differentiable Multi-View Pose Optimization From Objectness Detections

• Authors: Matteo Taiana, Matteo Toso, Stuart James, Alessio Del Bue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08586
• Pdf URL: https://arxiv.org/pdf/2403.08586
• Copy Paste: [[2403.08586]] PRAGO: Differentiable Multi-View Pose Optimization From Objectness Detections(https://arxiv.org/abs/2403.08586)
• Keywords: robust
• Abstract: Robustly estimating camera poses from a set of images is a fundamental task which remains challenging for differentiable methods, especially in the case of small and sparse camera pose graphs. To overcome this challenge, we propose Pose-refined Rotation Averaging Graph Optimization (PRAGO). From a set of objectness detections on unordered images, our method reconstructs the rotational pose, and in turn, the absolute pose, in a differentiable manner benefiting from the optimization of a sequence of geometrical tasks. We show how our objectness pose-refinement module in PRAGO is able to refine the inherent ambiguities in pairwise relative pose estimation without removing edges and avoiding making early decisions on the viability of graph edges. PRAGO then refines the absolute rotations through iterative graph construction, reweighting the graph edges to compute the final rotational pose, which can be converted into absolute poses using translation averaging. We show that PRAGO is able to outperform non-differentiable solvers on small and sparse scenes extracted from 7-Scenes achieving a relative improvement of 21% for rotations while achieving similar translation estimates.

Title: ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videos

• Authors: Lei Shi, Paul Bürkner, Andreas Bulling
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08591
• Pdf URL: https://arxiv.org/pdf/2403.08591
• Copy Paste: [[2403.08591]] ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videos(https://arxiv.org/abs/2403.08591)
• Keywords: diffusion
• Abstract: We present ActionDiffusion -- a novel diffusion model for procedure planning in instructional videos that is the first to take temporal inter-dependencies between actions into account in a diffusion model for procedure planning. This approach is in stark contrast to existing methods that fail to exploit the rich information content available in the particular order in which actions are performed. Our method unifies the learning of temporal dependencies between actions and denoising of the action plan in the diffusion process by projecting the action information into the noise space. This is achieved 1) by adding action embeddings in the noise masks in the noise-adding phase and 2) by introducing an attention mechanism in the noise prediction network to learn the correlations between different action steps. We report extensive experiments on three instructional video benchmark datasets (CrossTask, Coin, and NIV) and show that our method outperforms previous state-of-the-art methods on all metrics on CrossTask and NIV and all metrics except accuracy on Coin dataset. We show that by adding action embeddings into the noise mask the diffusion model can better learn action temporal dependencies and increase the performances on procedure planning.

Title: Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments

• Authors: Sitao Cheng, Ziyuan Zhuang, Yong Xu, Fangkai Yang, Chaoyun Zhang, Xiaoting Qin, Xiang Huang, Ling Chen, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Qi Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08593
• Pdf URL: https://arxiv.org/pdf/2403.08593
• Copy Paste: [[2403.08593]] Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments(https://arxiv.org/abs/2403.08593)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown potential in reasoning over structured environments, e.g., knowledge graph and table. Such tasks typically require multi-hop reasoning, i.e., match natural language utterance with instances in the environment. Previous methods leverage LLMs to incrementally build a reasoning path, where the LLMs either invoke tools or pick up schemas by step-by-step interacting with the environment. We propose Reasoning-Path-Editing (Readi), a novel framework where LLMs can efficiently and faithfully reason over structured environments. In Readi, LLMs initially generate a reasoning path given a query, and edit the path only when necessary. We instantiate the path on structured environments and provide feedback to edit the path if anything goes wrong. Experimental results on three KGQA datasets and two TableQA datasets show the effectiveness of Readi, significantly surpassing all LLM-based methods (by 9.1% on WebQSP, 12.4% on MQA-3H and 10.9% on WTQ), comparable with state-of-the-art fine-tuned methods (67% on CWQ and 74.7% on WebQSP) and substantially boosting the vanilla LLMs (by 14.9% on CWQ). Our code will be available upon publication.

Title: DevBench: A Comprehensive Benchmark for Software Development

• Authors: Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, Kai Chen
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.08604
• Pdf URL: https://arxiv.org/pdf/2403.08604
• Copy Paste: [[2403.08604]] DevBench: A Comprehensive Benchmark for Software Development(https://arxiv.org/abs/2403.08604)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have significantly enhanced their coding capabilities. However, existing benchmarks predominantly focused on simplified or isolated aspects of programming, such as single-file code generation or repository issue debugging, falling short of measuring the full spectrum of challenges raised by real-world programming activities. To this end, we propose DevBench, a comprehensive benchmark that evaluates LLMs across various stages of the software development lifecycle, including software design, environment setup, implementation, acceptance testing, and unit testing. DevBench features a wide range of programming languages and domains, high-quality data collection, and carefully designed and verified metrics for each task. Empirical studies show that current LLMs, including GPT-4-Turbo, fail to solve the challenges presented within DevBench. Analyses reveal that models struggle with understanding the complex structures in the repository, managing the compilation process, and grasping advanced programming concepts. Our findings offer actionable insights for the future development of LLMs toward real-world programming applications. Our benchmark is available at https://github.com/open-compass/DevBench

Title: MedInsight: A Multi-Source Context Augmentation Framework for Generating Patient-Centric Medical Responses using Large Language Models

• Authors: Subash Neupane, Shaswata Mitra, Sudip Mittal, Noorbakhsh Amiri Golilarz, Shahram Rahimi, Amin Amirlatifi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08607
• Pdf URL: https://arxiv.org/pdf/2403.08607
• Copy Paste: [[2403.08607]] MedInsight: A Multi-Source Context Augmentation Framework for Generating Patient-Centric Medical Responses using Large Language Models(https://arxiv.org/abs/2403.08607)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown impressive capabilities in generating human-like responses. However, their lack of domain-specific knowledge limits their applicability in healthcare settings, where contextual and comprehensive responses are vital. To address this challenge and enable the generation of patient-centric responses that are contextually relevant and comprehensive, we propose MedInsight:a novel retrieval augmented framework that augments LLM inputs (prompts) with relevant background information from multiple sources. MedInsight extracts pertinent details from the patient's medical record or consultation transcript. It then integrates information from authoritative medical textbooks and curated web resources based on the patient's health history and condition. By constructing an augmented context combining the patient's record with relevant medical knowledge, MedInsight generates enriched, patient-specific responses tailored for healthcare applications such as diagnosis, treatment recommendations, or patient education. Experiments on the MTSamples dataset validate MedInsight's effectiveness in generating contextually appropriate medical responses. Quantitative evaluation using the Ragas metric and TruLens for answer similarity and answer correctness demonstrates the model's efficacy. Furthermore, human evaluation studies involving Subject Matter Expert (SMEs) confirm MedInsight's utility, with moderate inter-rater agreement on the relevance and correctness of the generated responses.

Title: On the Convergence of Locally Adaptive and Scalable Diffusion-Based Sampling Methods for Deep Bayesian Neural Network Posteriors

• Authors: Tim Rensmeyer, Oliver Niggemann
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.08609
• Pdf URL: https://arxiv.org/pdf/2403.08609
• Copy Paste: [[2403.08609]] On the Convergence of Locally Adaptive and Scalable Diffusion-Based Sampling Methods for Deep Bayesian Neural Network Posteriors(https://arxiv.org/abs/2403.08609)
• Keywords: robust, diffusion
• Abstract: Achieving robust uncertainty quantification for deep neural networks represents an important requirement in many real-world applications of deep learning such as medical imaging where it is necessary to assess the reliability of a neural network's prediction. Bayesian neural networks are a promising approach for modeling uncertainties in deep neural networks. Unfortunately, generating samples from the posterior distribution of neural networks is a major challenge. One significant advance in that direction would be the incorporation of adaptive step sizes, similar to modern neural network optimizers, into Monte Carlo Markov chain sampling algorithms without significantly increasing computational demand. Over the past years, several papers have introduced sampling algorithms with claims that they achieve this property. However, do they indeed converge to the correct distribution? In this paper, we demonstrate that these methods can have a substantial bias in the distribution they sample, even in the limit of vanishing step sizes and at full batch size.

Title: Verifix: Post-Training Correction to Improve Label Noise Robustness with Verified Samples

• Authors: Sangamesh Kodge, Deepak Ravikumar, Gobinda Saha, Kaushik Roy
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.08618
• Pdf URL: https://arxiv.org/pdf/2403.08618
• Copy Paste: [[2403.08618]] Verifix: Post-Training Correction to Improve Label Noise Robustness with Verified Samples(https://arxiv.org/abs/2403.08618)
• Keywords: attack, robust
• Abstract: Label corruption, where training samples have incorrect labels, can significantly degrade the performance of machine learning models. This corruption often arises from non-expert labeling or adversarial attacks. Acquiring large, perfectly labeled datasets is costly, and retraining large models from scratch when a clean dataset becomes available is computationally expensive. To address this challenge, we propose Post-Training Correction, a new paradigm that adjusts model parameters after initial training to mitigate label noise, eliminating the need for retraining. We introduce Verifix, a novel Singular Value Decomposition (SVD) based algorithm that leverages a small, verified dataset to correct the model weights using a single update. Verifix uses SVD to estimate a Clean Activation Space and then projects the model's weights onto this space to suppress activations corresponding to corrupted data. We demonstrate Verifix's effectiveness on both synthetic and real-world label noise. Experiments on the CIFAR dataset with 25% synthetic corruption show 7.36% generalization improvements on average. Additionally, we observe generalization improvements of up to 2.63% on naturally corrupted datasets like WebVision1.0 and Clothing1M.

Title: Scaling Up Dynamic Human-Scene Interaction Modeling

• Authors: Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Siyuan Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08629
• Pdf URL: https://arxiv.org/pdf/2403.08629
• Copy Paste: [[2403.08629]] Scaling Up Dynamic Human-Scene Interaction Modeling(https://arxiv.org/abs/2403.08629)
• Keywords: diffusion
• Abstract: Confronting the challenges of data scarcity and advanced motion synthesis in human-scene interaction modeling, we introduce the TRUMANS dataset alongside a novel HSI motion synthesis method. TRUMANS stands as the most comprehensive motion-captured HSI dataset currently available, encompassing over 15 hours of human interactions across 100 indoor scenes. It intricately captures whole-body human motions and part-level object dynamics, focusing on the realism of contact. This dataset is further scaled up by transforming physical environments into exact virtual models and applying extensive augmentations to appearance and motion for both humans and objects while maintaining interaction fidelity. Utilizing TRUMANS, we devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length, taking into account both scene context and intended actions. In experiments, our approach shows remarkable zero-shot generalizability on a range of 3D scene datasets (e.g., PROX, Replica, ScanNet, ScanNet++), producing motions that closely mimic original motion-captured sequences, as confirmed by quantitative experiments and human studies.

Title: Human Alignment of Large Language Models through Online Preference Optimisation

• Authors: Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, Bilal Piot
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.08635
• Pdf URL: https://arxiv.org/pdf/2403.08635
• Copy Paste: [[2403.08635]] Human Alignment of Large Language Models through Online Preference Optimisation(https://arxiv.org/abs/2403.08635)
• Keywords: large language model
• Abstract: Ensuring alignment of language models' outputs with human preferences is critical to guarantee a useful, safe, and pleasant user experience. Thus, human alignment has been extensively studied recently and several methods such as Reinforcement Learning from Human Feedback (RLHF), Direct Policy Optimisation (DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In this paper, our contribution is two-fold. First, we show the equivalence between two recent alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror Descent (Nash-MD). Second, we introduce a generalisation of IPO, named IPO-MD, that leverages the regularised sampling approach proposed by Nash-MD. This equivalence may seem surprising at first sight, since IPO is an offline method whereas Nash-MD is an online method using a preference model. However, this equivalence can be proven when we consider the online version of IPO, that is when both generations are sampled by the online policy and annotated by a trained preference model. Optimising the IPO loss with such a stream of data becomes then equivalent to finding the Nash equilibrium of the preference model through self-play. Building on this equivalence, we introduce the IPO-MD algorithm that generates data with a mixture policy (between the online and reference policy) similarly as the general Nash-MD algorithm. We compare online-IPO and IPO-MD to different online versions of existing losses on preference data such as DPO and SLiC on a summarisation task.

Title: Refractive COLMAP: Refractive Structure-from-Motion Revisited

• Authors: Mengkun She, Felix Seegräber, David Nakath, Kevin Köser
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08640
• Pdf URL: https://arxiv.org/pdf/2403.08640
• Copy Paste: [[2403.08640]] Refractive COLMAP: Refractive Structure-from-Motion Revisited(https://arxiv.org/abs/2403.08640)
• Keywords: robust
• Abstract: In this paper, we present a complete refractive Structure-from-Motion (RSfM) framework for underwater 3D reconstruction using refractive camera setups (for both, flat- and dome-port underwater housings). Despite notable achievements in refractive multi-view geometry over the past decade, a robust, complete and publicly available solution for such tasks is not available at present, and often practical applications have to resort to approximating refraction effects by the intrinsic (distortion) parameters of a pinhole camera model. To fill this gap, we have integrated refraction considerations throughout the entire SfM process within the state-of-the-art, open-source SfM framework COLMAP. Numerical simulations and reconstruction results on synthetically generated but photo-realistic images with ground truth validate that enabling refraction does not compromise accuracy or robustness as compared to in-air reconstructions. Finally, we demonstrate the capability of our approach for large-scale refractive scenarios using a dataset consisting of nearly 6000 images. The implementation is released as open-source at: https://cau-git.rz.uni-kiel.de/inf-ag-koeser/colmap_underwater.

Title: Data Augmentation in Human-Centric Vision

• Authors: Wentao Jiang, Yige Zhang, Shaozhong Zheng, Si Liu, Shuicheng Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08650
• Pdf URL: https://arxiv.org/pdf/2403.08650
• Copy Paste: [[2403.08650]] Data Augmentation in Human-Centric Vision(https://arxiv.org/abs/2403.08650)
• Keywords: robust, diffusion, generative
• Abstract: This survey presents a comprehensive analysis of data augmentation techniques in human-centric vision tasks, a first of its kind in the field. It delves into a wide range of research areas including person ReID, human parsing, human pose estimation, and pedestrian detection, addressing the significant challenges posed by overfitting and limited training data in these domains. Our work categorizes data augmentation methods into two main types: data generation and data perturbation. Data generation covers techniques like graphic engine-based generation, generative model-based generation, and data recombination, while data perturbation is divided into image-level and human-level perturbations. Each method is tailored to the unique requirements of human-centric tasks, with some applicable across multiple areas. Our contributions include an extensive literature review, providing deep insights into the influence of these augmentation techniques in human-centric vision and highlighting the nuances of each method. We also discuss open issues and future directions, such as the integration of advanced generative models like Latent Diffusion Models, for creating more realistic and diverse training data. This survey not only encapsulates the current state of data augmentation in human-centric vision but also charts a course for future research, aiming to develop more robust, accurate, and efficient human-centric vision systems.

Title: Extracting Explanations, Justification, and Uncertainty from Black-Box Deep Neural Networks

• Authors: Paul Ardis, Arjuna Flenner
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.08652
• Pdf URL: https://arxiv.org/pdf/2403.08652
• Copy Paste: [[2403.08652]] Extracting Explanations, Justification, and Uncertainty from Black-Box Deep Neural Networks(https://arxiv.org/abs/2403.08652)
• Keywords: interpretability
• Abstract: Deep Neural Networks (DNNs) do not inherently compute or exhibit empirically-justified task confidence. In mission critical applications, it is important to both understand associated DNN reasoning and its supporting evidence. In this paper, we propose a novel Bayesian approach to extract explanations, justifications, and uncertainty estimates from DNNs. Our approach is efficient both in terms of memory and computation, and can be applied to any black box DNN without any retraining, including applications to anomaly detection and out-of-distribution detection tasks. We validate our approach on the CIFAR-10 dataset, and show that it can significantly improve the interpretability and reliability of DNNs.

Title: Physical Memory Attacks and a Memory Safe Management System for Memory Defense

• Authors: Alon Hillel-Tuch, Aspen Olmstead
• Subjects: cs.CR, cs.OS
• Abstract URL: https://arxiv.org/abs/2403.08656
• Pdf URL: https://arxiv.org/pdf/2403.08656
• Copy Paste: [[2403.08656]] Physical Memory Attacks and a Memory Safe Management System for Memory Defense(https://arxiv.org/abs/2403.08656)
• Keywords: protect, defense, attack
• Abstract: Programming errors, defective hardware components (such as hard disk spindle defects), and environmental hazards can lead to invalid memory operations. In addition, less predictable forms of environmental stress, such as radiation, thermal influence, and energy fluctuations, can induce hardware faults. Sometimes, a soft error can occur instead of a complete failure, such as a bit-flip. The 'natural' factors that can cause bit-flips are replicable through targeted attacks that result in significant compromises, including full privileged system access. Existing physical defense solutions have consistently been circumvented shortly after deployment. We will explore the concept of a novel software-based low-level layer that can protect vulnerable memory targeted by physical attack vectors related to bit-flip vulnerabilities.

Title: Zero-shot and Few-shot Generation Strategies for Artificial Clinical Records

• Authors: Erlend Frayling, Jake Lever, Graham McDonald
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08664
• Pdf URL: https://arxiv.org/pdf/2403.08664
• Copy Paste: [[2403.08664]] Zero-shot and Few-shot Generation Strategies for Artificial Clinical Records(https://arxiv.org/abs/2403.08664)
• Keywords: privacy, large language model
• Abstract: The challenge of accessing historical patient data for clinical research, while adhering to privacy regulations, is a significant obstacle in medical science. An innovative approach to circumvent this issue involves utilising synthetic medical records that mirror real patient data without compromising individual privacy. The creation of these synthetic datasets, particularly without using actual patient data to train Large Language Models (LLMs), presents a novel solution as gaining access to sensitive patient information to train models is also a challenge. This study assesses the capability of the Llama 2 LLM to create synthetic medical records that accurately reflect real patient information, employing zero-shot and few-shot prompting strategies for comparison against fine-tuned methodologies that do require sensitive patient data during training. We focus on generating synthetic narratives for the History of Present Illness section, utilising data from the MIMIC-IV dataset for comparison. In this work introduce a novel prompting technique that leverages a chain-of-thought approach, enhancing the model's ability to generate more accurate and contextually relevant medical narratives without prior fine-tuning. Our findings suggest that this chain-of-thought prompted approach allows the zero-shot model to achieve results on par with those of fine-tuned models, based on Rouge metrics evaluation.

Title: OneVOS: Unifying Video Object Segmentation with All-in-One Transformer Framework

• Authors: Wanyun Li, Pinxue Guo, Xinyu Zhou, Lingyi Hong, Yangji He, Xiangyu Zheng, Wei Zhang, Wenqiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08682
• Pdf URL: https://arxiv.org/pdf/2403.08682
• Copy Paste: [[2403.08682]] OneVOS: Unifying Video Object Segmentation with All-in-One Transformer Framework(https://arxiv.org/abs/2403.08682)
• Keywords: extraction, transformer, segmentation
• Abstract: Contemporary Video Object Segmentation (VOS) approaches typically consist stages of feature extraction, matching, memory management, and multiple objects aggregation. Recent advanced models either employ a discrete modeling for these components in a sequential manner, or optimize a combined pipeline through substructure aggregation. However, these existing explicit staged approaches prevent the VOS framework from being optimized as a unified whole, leading to the limited capacity and suboptimal performance in tackling complex videos. In this paper, we propose OneVOS, a novel framework that unifies the core components of VOS with All-in-One Transformer. Specifically, to unify all aforementioned modules into a vision transformer, we model all the features of frames, masks and memory for multiple objects as transformer tokens, and integrally accomplish feature extraction, matching and memory management of multiple objects through the flexible attention mechanism. Furthermore, a Unidirectional Hybrid Attention is proposed through a double decoupling of the original attention operation, to rectify semantic errors and ambiguities of stored tokens in OneVOS framework. Finally, to alleviate the storage burden and expedite inference, we propose the Dynamic Token Selector, which unveils the working mechanism of OneVOS and naturally leads to a more efficient version of OneVOS. Extensive experiments demonstrate the superiority of OneVOS, achieving state-of-the-art performance across 7 datasets, particularly excelling in complex LVOS and MOSE datasets with 70.1% and 66.4% $J \& F$ scores, surpassing previous state-of-the-art methods by 4.2% and 7.0%, respectively. And our code will be available for reproducibility and further research.

Title: Token Alignment via Character Matching for Subword Completion

• Authors: Ben Athiwaratkun, Shiqi Wang, Mingyue Shang, Yuchen Tian, Zijian Wang, Sujan Kumar Gonugondla, Sanjay Krishna Gouda, Rob Kwiatowski, Ramesh Nallapati, Bing Xiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08688
• Pdf URL: https://arxiv.org/pdf/2403.08688
• Copy Paste: [[2403.08688]] Token Alignment via Character Matching for Subword Completion(https://arxiv.org/abs/2403.08688)
• Keywords: generative
• Abstract: Generative models, widely utilized in various applications, can often struggle with prompts corresponding to partial tokens. This struggle stems from tokenization, where partial tokens fall out of distribution during inference, leading to incorrect or nonsensical outputs. This paper examines a technique to alleviate the tokenization artifact on text completion in generative models, maintaining performance even in regular non-subword cases. The method, termed token alignment, involves backtracking to the last complete tokens and ensuring the model's generation aligns with the prompt. This approach showcases marked improvement across many partial token scenarios, including nuanced cases like space-prefix and partial indentation, with only a minor time increase. The technique and analysis detailed in this paper contribute to the continuous advancement of generative models in handling partial inputs, bearing relevance for applications like code completion and text autocompletion.

Title: TeaMs-RL: Teaching LLMs to Teach Themselves Better Instructions via Reinforcement Learning

• Authors: Shangding Gu, Alois Knoll, Ming Jin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08694
• Pdf URL: https://arxiv.org/pdf/2403.08694
• Copy Paste: [[2403.08694]] TeaMs-RL: Teaching LLMs to Teach Themselves Better Instructions via Reinforcement Learning(https://arxiv.org/abs/2403.08694)
• Keywords: privacy, protect, large language model
• Abstract: The development of Large Language Models (LLMs) often confronts challenges stemming from the heavy reliance on human annotators in the reinforcement learning with human feedback (RLHF) framework, or the frequent and costly external queries tied to the self-instruct paradigm. In this work, we pivot to Reinforcement Learning (RL) -- but with a twist. Diverging from the typical RLHF, which refines LLMs following instruction data training, we use RL to directly generate the foundational instruction dataset that alone suffices for fine-tuning. Our method, TeaMs-RL, uses a suite of textual operations and rules, prioritizing the diversification of training datasets. It facilitates the generation of high-quality data without excessive reliance on external advanced models, paving the way for a single fine-tuning step and negating the need for subsequent RLHF stages. Our findings highlight key advantages of our approach: reduced need for human involvement and fewer model queries (only $5.73\%$ of WizardLM's total), along with enhanced capabilities of LLMs in crafting and comprehending complex instructions compared to strong baselines, and substantially improved model privacy protection.

Title: Deep Learning for In-Orbit Cloud Segmentation and Classification in Hyperspectral Satellite Data

• Authors: Daniel Kovac, Jan Mucha, Jon Alvarez Justo, Jiri Mekyska, Zoltan Galaz, Krystof Novotny, Radoslav Pitonak, Jan Knezik, Jonas Herec, Tor Arne Johansen
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.08695
• Pdf URL: https://arxiv.org/pdf/2403.08695
• Copy Paste: [[2403.08695]] Deep Learning for In-Orbit Cloud Segmentation and Classification in Hyperspectral Satellite Data(https://arxiv.org/abs/2403.08695)
• Keywords: segmentation
• Abstract: This article explores the latest Convolutional Neural Networks (CNNs) for cloud detection aboard hyperspectral satellites. The performance of the latest 1D CNN (1D-Justo-LiuNet) and two recent 2D CNNs (nnU-net and 2D-Justo-UNet-Simple) for cloud segmentation and classification is assessed. Evaluation criteria include precision and computational efficiency for in-orbit deployment. Experiments utilize NASA's EO-1 Hyperion data, with varying spectral channel numbers after Principal Component Analysis. Results indicate that 1D-Justo-LiuNet achieves the highest accuracy, outperforming 2D CNNs, while maintaining compactness with larger spectral channel sets, albeit with increased inference times. However, the performance of 1D CNN degrades with significant channel reduction. In this context, the 2D-Justo-UNet-Simple offers the best balance for in-orbit deployment, considering precision, memory, and time costs. While nnU-net is suitable for on-ground processing, deployment of lightweight 1D-Justo-LiuNet is recommended for high-precision applications. Alternatively, lightweight 2D-Justo-UNet-Simple is recommended for balanced costs between timing and precision in orbit.

Title: Review of Generative AI Methods in Cybersecurity

• Authors: Yagmur Yigit, William J Buchanan, Madjid G Tehrani, Leandros Maglaras
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.08701
• Pdf URL: https://arxiv.org/pdf/2403.08701
• Copy Paste: [[2403.08701]] Review of Generative AI Methods in Cybersecurity(https://arxiv.org/abs/2403.08701)
• Keywords: security, generative, large language model
• Abstract: Large language models (LLMs) and generative artificial intelligence (GenAI) constitute paradigm shifts in cybersecurity that present hitherto unseen challenges as well as opportunities. In examining the state-of-the-art application of GenAI in cybersecurity, this work highlights how models like Google's Gemini and ChatGPT-4 potentially enhance security protocols, vulnerability assessment, and threat identification. Our research highlights the significance of a novel approach that employs LLMs to identify and eliminate sophisticated cyber threats. This paper presents a thorough assessment of LLMs' ability to produce important security insights, hence broadening the potential applications of AI-driven cybersecurity solutions. Our findings demonstrate the significance of GenAI in improving digital security. It offers recommendations for further investigations into the intricate relationship between cybersecurity requirements and artificial intelligence's potential.

Title: SOTOPIA-$π$: Interactive Learning of Socially Intelligent Language Agents

• Authors: Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Graham Neubig, Yonatan Bisk, Hao Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08715
• Pdf URL: https://arxiv.org/pdf/2403.08715
• Copy Paste: [[2403.08715]] SOTOPIA-$π$: Interactive Learning of Socially Intelligent Language Agents(https://arxiv.org/abs/2403.08715)
• Keywords: large language model
• Abstract: Humans learn social skills through both imitation and social interaction. This social learning process is largely understudied by existing research on building language agents. Motivated by this gap, we propose an interactive learning method, SOTOPIA-$\pi$, improving the social intelligence of language agents. This method leverages behavior cloning and self-reinforcement training on filtered social interaction data according to large language model (LLM) ratings. We show that our training method allows a 7B LLM to reach the social goal completion ability of an expert model (GPT-4-based agent), while improving the safety of language agents and maintaining general QA ability on the MMLU benchmark. We also find that this training paradigm uncovers some difficulties in LLM-based evaluation of social intelligence: LLM-based evaluators overestimate the abilities of the language agents trained specifically for social interaction.

Title: Historical Astronomical Diagrams Decomposition in Geometric Primitives

• Authors: Syrine Kalleli, Scott Trigg, Ségolène Albouy, Mathieu Husson, Mathieu Aubry
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08721
• Pdf URL: https://arxiv.org/pdf/2403.08721
• Copy Paste: [[2403.08721]] Historical Astronomical Diagrams Decomposition in Geometric Primitives(https://arxiv.org/abs/2403.08721)
• Keywords: diffusion
• Abstract: Automatically extracting the geometric content from the hundreds of thousands of diagrams drawn in historical manuscripts would enable historians to study the diffusion of astronomical knowledge on a global scale. However, state-of-the-art vectorization methods, often designed to tackle modern data, are not adapted to the complexity and diversity of historical astronomical diagrams. Our contribution is thus twofold. First, we introduce a unique dataset of 303 astronomical diagrams from diverse traditions, ranging from the XIIth to the XVIIIth century, annotated with more than 3000 line segments, circles and arcs. Second, we develop a model that builds on DINO-DETR to enable the prediction of multiple geometric primitives. We show that it can be trained solely on synthetic data and accurately predict primitives on our challenging dataset. Our approach widely improves over the LETR baseline, which is restricted to lines, by introducing a meaningful parametrization for multiple primitives, jointly training for detection and parameter refinement, using deformable attention and training on rich synthetic data. Our dataset and code are available on our webpage.

Title: Ambient Diffusion Posterior Sampling: Solving Inverse Problems with Diffusion Models trained on Corrupted Data

• Authors: Asad Aali, Giannis Daras, Brett Levac, Sidharth Kumar, Alexandros G. Dimakis, Jonathan I. Tamir
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08728
• Pdf URL: https://arxiv.org/pdf/2403.08728
• Copy Paste: [[2403.08728]] Ambient Diffusion Posterior Sampling: Solving Inverse Problems with Diffusion Models trained on Corrupted Data(https://arxiv.org/abs/2403.08728)
• Keywords: diffusion, generative
• Abstract: We provide a framework for solving inverse problems with diffusion models learned from linearly corrupted data. Our method, Ambient Diffusion Posterior Sampling (A-DPS), leverages a generative model pre-trained on one type of corruption (e.g. image inpainting) to perform posterior sampling conditioned on measurements from a potentially different forward process (e.g. image blurring). We test the efficacy of our approach on standard natural image datasets (CelebA, FFHQ, and AFHQ) and we show that A-DPS can sometimes outperform models trained on clean data for several image restoration tasks in both speed and performance. We further extend the Ambient Diffusion framework to train MRI models with access only to Fourier subsampled multi-coil MRI measurements at various acceleration factors (R=2, 4, 6, 8). We again observe that models trained on highly subsampled data are better priors for solving inverse problems in the high acceleration regime than models trained on fully sampled data. We open-source our code and the trained Ambient Diffusion MRI models: https://github.com/utcsilab/ambient-diffusion-mri .

Title: Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization

• Authors: Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, Tong Zhang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08730
• Pdf URL: https://arxiv.org/pdf/2403.08730
• Copy Paste: [[2403.08730]] Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization(https://arxiv.org/abs/2403.08730)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) excel in generating responses based on visual inputs. However, they often suffer from a bias towards generating responses similar to their pretraining corpus, overshadowing the importance of visual information. We treat this bias as a "preference" for pretraining statistics, which hinders the model's grounding in visual input. To mitigate this issue, we propose Bootstrapped Preference Optimization (BPO), which conducts preference learning with datasets containing negative responses bootstrapped from the model itself. Specifically, we propose the following two strategies: 1) using distorted image inputs to the MLLM for eliciting responses that contain signified pretraining bias; 2) leveraging text-based LLM to explicitly inject erroneous but common elements into the original response. Those undesirable responses are paired with original annotated responses from the datasets to construct the preference dataset, which is subsequently utilized to perform preference learning. Our approach effectively suppresses pretrained LLM bias, enabling enhanced grounding in visual inputs. Extensive experimentation demonstrates significant performance improvements across multiple benchmarks, advancing the state-of-the-art in multimodal conversational systems.

Title: GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editing

• Authors: Jing Wu, Jia-Wang Bian, Xinghui Li, Guangrun Wang, Ian Reid, Philip Torr, Victor Adrian Prisacariu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08733
• Pdf URL: https://arxiv.org/pdf/2403.08733
• Copy Paste: [[2403.08733]] GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editing(https://arxiv.org/abs/2403.08733)
• Keywords: diffusion
• Abstract: We propose GaussCtrl, a text-driven method to edit a 3D scene reconstructed by the 3D Gaussian Splatting (3DGS). Our method first renders a collection of images by using the 3DGS and edits them by using a pre-trained 2D diffusion model (ControlNet) based on the input prompt, which is then used to optimise the 3D model. Our key contribution is multi-view consistent editing, which enables editing all images together instead of iteratively editing one image while updating the 3D model as in previous works. It leads to faster editing as well as higher visual quality. This is achieved by the two terms: (a) depth-conditioned editing that enforces geometric consistency across multi-view images by leveraging naturally consistent depth maps. (b) attention-based latent code alignment that unifies the appearance of edited images by conditioning their editing to several reference views through self and cross-view attention between images' latent representations. Experiments demonstrate that our method achieves faster editing and better visual results than previous state-of-the-art methods.

Title: The Garden of Forking Paths: Observing Dynamic Parameters Distribution in Large Language Models

• Authors: Carlo Nicolini, Jacopo Staiano, Bruno Lepri, Raffaele Marino
• Subjects: cs.CL, cond-mat.dis-nn, cond-mat.stat-mech, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08739
• Pdf URL: https://arxiv.org/pdf/2403.08739
• Copy Paste: [[2403.08739]] The Garden of Forking Paths: Observing Dynamic Parameters Distribution in Large Language Models(https://arxiv.org/abs/2403.08739)
• Keywords: transformer, large language model
• Abstract: A substantial gap persists in understanding the reasons behind the exceptional performance of the Transformer architecture in NLP. A particularly unexplored area involves the mechanistic description of how the distribution of parameters evolves over time during training. In this work we suggest that looking at the time evolution of the statistic distribution of model parameters, and specifically at bifurcation effects, can help understanding the model quality, potentially reducing training costs and evaluation efforts and empirically showing the reasons behind the effectiveness of weights sparsification.

Title: Acoustic Side Channel Attack on Keyboards Based on Typing Patterns

• Authors: Alireza Taheritajar, Reza Rahaeimehr
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.08740
• Pdf URL: https://arxiv.org/pdf/2403.08740
• Copy Paste: [[2403.08740]] Acoustic Side Channel Attack on Keyboards Based on Typing Patterns(https://arxiv.org/abs/2403.08740)
• Keywords: security, attack
• Abstract: Acoustic side-channel attacks on keyboards can bypass security measures in many systems that use keyboards as one of the input devices. These attacks aim to reveal users' sensitive information by targeting the sounds made by their keyboards as they type. Most existing approaches in this field ignore the negative impacts of typing patterns and environmental noise in their results. This paper seeks to address these shortcomings by proposing an applicable method that takes into account the user's typing pattern in a realistic environment. Our method achieved an average success rate of 43% across all our case studies when considering real-world scenarios.

Title: Steering LLMs Towards Unbiased Responses: A Causality-Guided Debiasing Framework

• Authors: Jingling Li, Zeyu Tang, Xiaoyu Liu, Peter Spirtes, Kun Zhang, Liu Leqi, Yang Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08743
• Pdf URL: https://arxiv.org/pdf/2403.08743
• Copy Paste: [[2403.08743]] Steering LLMs Towards Unbiased Responses: A Causality-Guided Debiasing Framework(https://arxiv.org/abs/2403.08743)
• Keywords: large language model
• Abstract: Large language models (LLMs) can easily generate biased and discriminative responses. As LLMs tap into consequential decision-making (e.g., hiring and healthcare), it is of crucial importance to develop strategies to mitigate these biases. This paper focuses on social bias, tackling the association between demographic information and LLM outputs. We propose a causality-guided debiasing framework that utilizes causal understandings of (1) the data-generating process of the training corpus fed to LLMs, and (2) the internal reasoning process of LLM inference, to guide the design of prompts for debiasing LLM outputs through selection mechanisms. Our framework unifies existing de-biasing prompting approaches such as inhibitive instructions and in-context contrastive examples, and sheds light on new ways of debiasing by encouraging bias-free reasoning. Our strong empirical performance on real-world datasets demonstrates that our framework provides principled guidelines on debiasing LLM outputs even with only the black-box access.

Title: MIM4D: Masked Modeling with Multi-View Video for Autonomous Driving Representation Learning

• Authors: Jialv Zou, Bencheng Liao, Qian Zhang, Wenyu Liu, Xinggang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08760
• Pdf URL: https://arxiv.org/pdf/2403.08760
• Copy Paste: [[2403.08760]] MIM4D: Masked Modeling with Multi-View Video for Autonomous Driving Representation Learning(https://arxiv.org/abs/2403.08760)
• Keywords: robust, segmentation
• Abstract: Learning robust and scalable visual representations from massive multi-view video data remains a challenge in computer vision and autonomous driving. Existing pre-training methods either rely on expensive supervised learning with 3D annotations, limiting the scalability, or focus on single-frame or monocular inputs, neglecting the temporal information. We propose MIM4D, a novel pre-training paradigm based on dual masked image modeling (MIM). MIM4D leverages both spatial and temporal relations by training on masked multi-view video inputs. It constructs pseudo-3D features using continuous scene flow and projects them onto 2D plane for supervision. To address the lack of dense 3D supervision, MIM4D reconstruct pixels by employing 3D volumetric differentiable rendering to learn geometric representations. We demonstrate that MIM4D achieves state-of-the-art performance on the nuScenes dataset for visual representation learning in autonomous driving. It significantly improves existing methods on multiple downstream tasks, including BEV segmentation (8.7% IoU), 3D object detection (3.5% mAP), and HD map construction (1.4% mAP). Our work offers a new choice for learning representation at scale in autonomous driving. Code and models are released at https://github.com/hustvl/MIM4D

Title: Simple and Scalable Strategies to Continually Pre-train Large Language Models

• Authors: Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, Irina Rish
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08763
• Pdf URL: https://arxiv.org/pdf/2403.08763
• Copy Paste: [[2403.08763]] Simple and Scalable Strategies to Continually Pre-train Large Language Models(https://arxiv.org/abs/2403.08763)
• Keywords: large language model
• Abstract: Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by final loss and language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English$\rightarrow$English) and a stronger distribution shift (English$\rightarrow$German) at the $405$M parameter model scale with large dataset sizes (hundreds of billions of tokens). Selecting the weak but realistic shift for larger-scale experiments, we also find that our continual learning strategies match the re-training baseline for a 10B parameter LLM. Our results demonstrate that LLMs can be successfully updated via simple and scalable continual learning strategies, matching the re-training baseline using only a fraction of the compute. Finally, inspired by previous work, we propose alternatives to the cosine learning rate schedule that help circumvent forgetting induced by LR re-warming and that are not bound to a fixed token budget.

Title: VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis

• Authors: Enric Corona, Andrei Zanfir, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, Cristian Sminchisescu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08764
• Pdf URL: https://arxiv.org/pdf/2403.08764
• Copy Paste: [[2403.08764]] VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis(https://arxiv.org/abs/2403.08764)
• Keywords: fair, diffusion, generative
• Abstract: We propose VLOGGER, a method for audio-driven human video generation from a single input image of a person, which builds on the success of recent generative diffusion models. Our method consists of 1) a stochastic human-to-3d-motion diffusion model, and 2) a novel diffusion-based architecture that augments text-to-image models with both spatial and temporal controls. This supports the generation of high quality video of variable length, easily controllable through high-level representations of human faces and bodies. In contrast to previous work, our method does not require training for each person, does not rely on face detection and cropping, generates the complete image (not just the face or the lips), and considers a broad spectrum of scenarios (e.g. visible torso or diverse subject identities) that are critical to correctly synthesize humans who communicate. We also curate MENTOR, a new and diverse dataset with 3d pose and expression annotations, one order of magnitude larger than previous ones (800,000 identities) and with dynamic gestures, on which we train and ablate our main technical contributions. VLOGGER outperforms state-of-the-art methods in three public benchmarks, considering image quality, identity preservation and temporal consistency while also generating upper-body gestures. We analyze the performance of VLOGGER with respect to multiple diversity metrics, showing that our architectural choices and the use of MENTOR benefit training a fair and unbiased model at scale. Finally we show applications in video editing and personalization.