2024-01-18

Title: Curve-based Neural Style Transfer

• Authors: Yu-hsuan Chen, Levent Burak Kara, Jonathan Cagan
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.08579
• Pdf URL: https://arxiv.org/pdf/2401.08579
• Copy Paste: [[2401.08579]] Curve-based Neural Style Transfer(https://arxiv.org/abs/2401.08579)
• Keywords: extraction
• Abstract: This research presents a new parametric style transfer framework specifically designed for curve-based design sketches. In this research, traditional challenges faced by neural style transfer methods in handling binary sketch transformations are effectively addressed through the utilization of parametric shape-editing rules, efficient curve-to-pixel conversion techniques, and the fine-tuning of VGG19 on ImageNet-Sketch, enhancing its role as a feature pyramid network for precise style extraction. By harmonizing intuitive curve-based imagery with rule-based editing, this study holds the potential to significantly enhance design articulation and elevate the practice of style transfer within the realm of product design.

Title: Temporal Embeddings: Scalable Self-Supervised Temporal Representation Learning from Spatiotemporal Data for Multimodal Computer Vision

• Authors: Yi Cao, Swetava Ganguli, Vipul Pandey
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08581
• Pdf URL: https://arxiv.org/pdf/2401.08581
• Copy Paste: [[2401.08581]] Temporal Embeddings: Scalable Self-Supervised Temporal Representation Learning from Spatiotemporal Data for Multimodal Computer Vision(https://arxiv.org/abs/2401.08581)
• Keywords: segmentation
• Abstract: There exists a correlation between geospatial activity temporal patterns and type of land use. A novel self-supervised approach is proposed to stratify landscape based on mobility activity time series. First, the time series signal is transformed to the frequency domain and then compressed into task-agnostic temporal embeddings by a contractive autoencoder, which preserves cyclic temporal patterns observed in time series. The pixel-wise embeddings are converted to image-like channels that can be used for task-based, multimodal modeling of downstream geospatial tasks using deep semantic segmentation. Experiments show that temporal embeddings are semantically meaningful representations of time series data and are effective across different tasks such as classifying residential area and commercial areas. Temporal embeddings transform sequential, spatiotemporal motion trajectory data into semantically meaningful image-like tensor representations that can be combined (multimodal fusion) with other data modalities that are or can be transformed into image-like tensor representations (for e.g., RBG imagery, graph embeddings of road networks, passively collected imagery like SAR, etc.) to facilitate multimodal learning in geospatial computer vision. Multimodal computer vision is critical for training machine learning models for geospatial feature detection to keep a geospatial mapping service up-to-date in real-time and can significantly improve user experience and above all, user safety.

Title: Automatic extraction and 3D reconstruction of split wire from point cloud data based on improved DPC algorithm

• Authors: Jia Cheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.08587
• Pdf URL: https://arxiv.org/pdf/2401.08587
• Copy Paste: [[2401.08587]] Automatic extraction and 3D reconstruction of split wire from point cloud data based on improved DPC algorithm(https://arxiv.org/abs/2401.08587)
• Keywords: extraction
• Abstract: In order to solve the problem of point cloud data splitting improved by DPC algorithm, a research on automatic separation and 3D reconstruction of point cloud data split lines is proposed. First, the relative coordinates of each point in the cloud point are calculated. Second, it is planned to develop a relative ensemble-based DPC swarm algorithm for analyzing the number of separation lines to determine all parts in the cloud content. Finally, fit each separator using the least squares method. iron. The cloud point of the resulting split subconductors has a clear demarcation line, and the distance between adjacent split subconductors is 0.45 m, divided by the four vertices of the square.

Title: Improved Pothole Detection Using YOLOv7 and ESRGAN

• Authors: Nirmal Kumar Rout, Gyanateet Dutta, Varun Sinha, Arghadeep Dey, Subhrangshu Mukherjee, Gopal Gupta
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08588
• Pdf URL: https://arxiv.org/pdf/2401.08588
• Copy Paste: [[2401.08588]] Improved Pothole Detection Using YOLOv7 and ESRGAN(https://arxiv.org/abs/2401.08588)
• Keywords: generative
• Abstract: Potholes are common road hazards that is causing damage to vehicles and posing a safety risk to drivers. The introduction of Convolutional Neural Networks (CNNs) is widely used in the industry for object detection based on Deep Learning methods and has achieved significant progress in hardware improvement and software implementations. In this paper, a unique better algorithm is proposed to warrant the use of low-resolution cameras or low-resolution images and video feed for automatic pothole detection using Super Resolution (SR) through Super Resolution Generative Adversarial Networks (SRGANs). Then we have proceeded to establish a baseline pothole detection performance on low quality and high quality dashcam images using a You Only Look Once (YOLO) network, namely the YOLOv7 network. We then have illustrated and examined the speed and accuracy gained above the benchmark after having upscaling implementation on the low quality images.

Title: Automatic measurement of coverage area of water-based pesticides-surfactant formulation on plant leaves using deep learning tools

• Authors: Fabio Grazioso, Anzhelika A. Atsapina, Gardoon L. O. Obaeed, Natalia A. Ivanova
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08593
• Pdf URL: https://arxiv.org/pdf/2401.08593
• Copy Paste: [[2401.08593]] Automatic measurement of coverage area of water-based pesticides-surfactant formulation on plant leaves using deep learning tools(https://arxiv.org/abs/2401.08593)
• Keywords: segmentation
• Abstract: A method to efficiently and quantitatively study the delivery of a pesticide-surfactant formulation in water solution over plants leaves is presented. Instead of measuring the contact angle, the surface of the leaves wet area is used as key parameter. To this goal, a deep learning model has been trained and tested, to automatically measure the surface of area wet with water solution over cucumber leaves, processing the frames of video footage. We have individuated an existing deep learning model, reported in literature for other applications, and we have applied it to this different task. We present the measurement technique, some details of the deep learning model, its training procedure and its image segmentation performance. Finally, we report the results of the wet areas surface measurement as a function of the concentration of a surfactant in the pesticide solution.

Title: NutritionVerse-Real: An Open Access Manually Collected 2D Food Scene Dataset for Dietary Intake Estimation

• Authors: Chi-en Amy Tai, Saeejith Nair, Olivia Markham, Matthew Keller, Yifan Wu, Yuhao Chen, Alexander Wong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08598
• Pdf URL: https://arxiv.org/pdf/2401.08598
• Copy Paste: [[2401.08598]] NutritionVerse-Real: An Open Access Manually Collected 2D Food Scene Dataset for Dietary Intake Estimation(https://arxiv.org/abs/2401.08598)
• Keywords: segmentation
• Abstract: Dietary intake estimation plays a crucial role in understanding the nutritional habits of individuals and populations, aiding in the prevention and management of diet-related health issues. Accurate estimation requires comprehensive datasets of food scenes, including images, segmentation masks, and accompanying dietary intake metadata. In this paper, we introduce NutritionVerse-Real, an open access manually collected 2D food scene dataset for dietary intake estimation with 889 images of 251 distinct dishes and 45 unique food types. The NutritionVerse-Real dataset was created by manually collecting images of food scenes in real life, measuring the weight of every ingredient and computing the associated dietary content of each dish using the ingredient weights and nutritional information from the food packaging or the Canada Nutrient File. Segmentation masks were then generated through human labelling of the images. We provide further analysis on the data diversity to highlight potential biases when using this data to develop models for dietary intake estimation. NutritionVerse-Real is publicly available at https://www.kaggle.com/datasets/nutritionverse/nutritionverse-real as part of an open initiative to accelerate machine learning for dietary sensing.

Title: SAM4UDASS: When SAM Meets Unsupervised Domain Adaptive Semantic Segmentation in Intelligent Vehicles

• Authors: Weihao Yan, Yeqiang Qian, Xingyuan Chen, Hanyang Zhuang, Chunxiang Wang, Ming Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.08604
• Pdf URL: https://arxiv.org/pdf/2401.08604
• Copy Paste: [[2401.08604]] SAM4UDASS: When SAM Meets Unsupervised Domain Adaptive Semantic Segmentation in Intelligent Vehicles(https://arxiv.org/abs/2401.08604)
• Keywords: segmentation
• Abstract: Semantic segmentation plays a critical role in enabling intelligent vehicles to comprehend their surrounding environments. However, deep learning-based methods usually perform poorly in domain shift scenarios due to the lack of labeled data for training. Unsupervised domain adaptation (UDA) techniques have emerged to bridge the gap across different driving scenes and enhance model performance on unlabeled target environments. Although self-training UDA methods have achieved state-of-the-art results, the challenge of generating precise pseudo-labels persists. These pseudo-labels tend to favor majority classes, consequently sacrificing the performance of rare classes or small objects like traffic lights and signs. To address this challenge, we introduce SAM4UDASS, a novel approach that incorporates the Segment Anything Model (SAM) into self-training UDA methods for refining pseudo-labels. It involves Semantic-Guided Mask Labeling, which assigns semantic labels to unlabeled SAM masks using UDA pseudo-labels. Furthermore, we devise fusion strategies aimed at mitigating semantic granularity inconsistency between SAM masks and the target domain. SAM4UDASS innovatively integrate SAM with UDA for semantic segmentation in driving scenes and seamlessly complements existing self-training UDA methodologies. Extensive experiments on synthetic-to-real and normal-to-adverse driving datasets demonstrate its effectiveness. It brings more than 3% mIoU gains on GTA5-to-Cityscapes, SYNTHIA-to-Cityscapes, and Cityscapes-to-ACDC when using DAFormer and achieves SOTA when using MIC. The code will be available at https://github.com/ywher/SAM4UDASS.

Title: Immature Green Apple Detection and Sizing in Commercial Orchards using YOLOv8 and Shape Fitting Techniques

• Authors: Ranjan Sapkota, Dawood Ahmed, Martin Churuvija, Manoj Karkee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08629
• Pdf URL: https://arxiv.org/pdf/2401.08629
• Copy Paste: [[2401.08629]] Immature Green Apple Detection and Sizing in Commercial Orchards using YOLOv8 and Shape Fitting Techniques(https://arxiv.org/abs/2401.08629)
• Keywords: segmentation
• Abstract: Detecting and estimating size of apples during the early stages of growth is crucial for predicting yield, pest management, and making informed decisions related to crop-load management, harvest and post-harvest logistics, and marketing. Traditional fruit size measurement methods are laborious and time-consuming. This study employs the state-of-the-art YOLOv8 object detection and instance segmentation algorithm in conjunction with geometric shape fitting techniques on 3D point cloud data to accurately determine the size of immature green apples (or fruitlet) in a commercial orchard environment. The methodology utilized two RGB-D sensors: the Intel RealSense D435i and the Microsoft Azure Kinect DK. Notably, the YOLOv8 instance segmentation models exhibited proficiency in immature green apple detection, with the YOLOv8m-seg model clinching the highest AP@0.5 and AP@0.75 scores of 0.94 and 0.91, respectively. Leveraging the ellipsoid fitting technique on images from the Azure Kinect, we observed remarkable metrics, including an RMSE of 2.35, MAE of 1.66, MAPE of 6.15, and an R-squared value of 0.9. Challenges such as partial occlusion, where YOLOv8 sometimes misinterpreted immature green apple clusters, were recognized. In a comparison of 102 outdoor samples, the Microsoft Azure Kinect showed better performance than the Intel Realsense D435i, as supported by the MAE data. This study emphasizes the combined effectiveness of shape-fitting methods and 3D sensors in improving fruitlet sizing for agriculture.

Title: One-Step Diffusion Distillation via Deep Equilibrium Models

• Authors: Zhengyang Geng, Ashwini Pokle, J. Zico Kolter
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08639
• Pdf URL: https://arxiv.org/pdf/2401.08639
• Copy Paste: [[2401.08639]] One-Step Diffusion Distillation via Deep Equilibrium Models(https://arxiv.org/abs/2401.08639)
• Keywords: diffusion, transformer, generative
• Abstract: Diffusion models excel at producing high-quality samples but naively require hundreds of iterations, prompting multiple attempts to distill the generation process into a faster network. However, many existing approaches suffer from a variety of challenges: the process for distillation training can be complex, often requiring multiple training stages, and the resulting models perform poorly when utilized in single-step generative applications. In this paper, we introduce a simple yet effective means of distilling diffusion models directly from initial noise to the resulting image. Of particular importance to our approach is to leverage a new Deep Equilibrium (DEQ) model as the distilled architecture: the Generative Equilibrium Transformer (GET). Our method enables fully offline training with just noise/image pairs from the diffusion model while achieving superior performance compared to existing one-step methods on comparable training budgets. We demonstrate that the DEQ architecture is crucial to this capability, as GET matches a $5\times$ larger ViT in terms of FID scores while striking a critical balance of computational cost and image quality. Code, checkpoints, and datasets are available.

Title: SAiD: Speech-driven Blendshape Facial Animation with Diffusion

• Authors: Inkyu Park, Jaewoong Cho
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2401.08655
• Pdf URL: https://arxiv.org/pdf/2401.08655
• Copy Paste: [[2401.08655]] SAiD: Speech-driven Blendshape Facial Animation with Diffusion(https://arxiv.org/abs/2401.08655)
• Keywords: diffusion, transformer
• Abstract: Speech-driven 3D facial animation is challenging due to the scarcity of large-scale visual-audio datasets despite extensive research. Most prior works, typically focused on learning regression models on a small dataset using the method of least squares, encounter difficulties generating diverse lip movements from speech and require substantial effort in refining the generated outputs. To address these issues, we propose a speech-driven 3D facial animation with a diffusion model (SAiD), a lightweight Transformer-based U-Net with a cross-modality alignment bias between audio and visual to enhance lip synchronization. Moreover, we introduce BlendVOCA, a benchmark dataset of pairs of speech audio and parameters of a blendshape facial model, to address the scarcity of public resources. Our experimental results demonstrate that the proposed approach achieves comparable or superior performance in lip synchronization to baselines, ensures more diverse lip movements, and streamlines the animation editing process.

Title: DA-BEV: Unsupervised Domain Adaptation for Bird's Eye View Perception

• Authors: Kai Jiang, Jiaxing Huang, Weiying Xie, Yunsong Li, Ling Shao, Shijian Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08687
• Pdf URL: https://arxiv.org/pdf/2401.08687
• Copy Paste: [[2401.08687]] DA-BEV: Unsupervised Domain Adaptation for Bird's Eye View Perception(https://arxiv.org/abs/2401.08687)
• Keywords: segmentation
• Abstract: Camera-only Bird's Eye View (BEV) has demonstrated great potential in environment perception in a 3D space. However, most existing studies were conducted under a supervised setup which cannot scale well while handling various new data. Unsupervised domain adaptive BEV, which effective learning from various unlabelled target data, is far under-explored. In this work, we design DA-BEV, the first domain adaptive camera-only BEV framework that addresses domain adaptive BEV challenges by exploiting the complementary nature of image-view features and BEV features. DA-BEV introduces the idea of query into the domain adaptation framework to derive useful information from image-view and BEV features. It consists of two query-based designs, namely, query-based adversarial learning (QAL) and query-based self-training (QST), which exploits image-view features or BEV features to regularize the adaptation of the other. Extensive experiments show that DA-BEV achieves superior domain adaptive BEV perception performance consistently across multiple datasets and tasks such as 3D object detection and 3D scene segmentation.

Title: NODI: Out-Of-Distribution Detection with Noise from Diffusion

• Authors: Jingqiu Zhou, Aojun Zou, Hongshen Li
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08689
• Pdf URL: https://arxiv.org/pdf/2401.08689
• Copy Paste: [[2401.08689]] NODI: Out-Of-Distribution Detection with Noise from Diffusion(https://arxiv.org/abs/2401.08689)
• Keywords: robust, diffusion, transformer
• Abstract: Out-of-distribution (OOD) detection is a crucial part of deploying machine learning models safely. It has been extensively studied with a plethora of methods developed in the literature. This problem is tackled with an OOD score computation, however, previous methods compute the OOD scores with limited usage of the in-distribution dataset. For instance, the OOD scores are computed with information from a small portion of the in-distribution data. Furthermore, these methods encode images with a neural image encoder. The robustness of these methods is rarely checked with respect to image encoders of different training methods and architectures. In this work, we introduce the diffusion process into the OOD task. The diffusion model integrates information on the whole training set into the predicted noise vectors. What's more, we deduce a closed-form solution for the noise vector (stable point). Then the noise vector is converted into our OOD score, we test both the deep model predicted noise vector and the closed-form noise vector on the OOD benchmarks \cite{openood}. Our method outperforms previous OOD methods across all types of image encoders (Table. \ref{main}). A $3.5\%$ performance gain is achieved with the MAE-based image encoder. Moreover, we studied the robustness of OOD methods by applying different types of image encoders. Some OOD methods failed to generalize well when switching image encoders from ResNet to Vision Transformers, our method performs exhibits good robustness with all the image encoders.

Title: Combining Confidence Elicitation and Sample-based Methods for Uncertainty Quantification in Misinformation Mitigation

• Authors: Mauricio Rivera, Jean-François Godbout, Reihaneh Rabbany, Kellin Pelrine
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.08694
• Pdf URL: https://arxiv.org/pdf/2401.08694
• Copy Paste: [[2401.08694]] Combining Confidence Elicitation and Sample-based Methods for Uncertainty Quantification in Misinformation Mitigation(https://arxiv.org/abs/2401.08694)
• Keywords: robust, large language model
• Abstract: Large Language Models have emerged as prime candidates to tackle misinformation mitigation. However, existing approaches struggle with hallucinations and overconfident predictions. We propose an uncertainty quantification framework that leverages both direct confidence elicitation and sampled-based consistency methods to provide better calibration for NLP misinformation mitigation solutions. We first investigate the calibration of sample-based consistency methods that exploit distinct features of consistency across sample sizes and stochastic levels. Next, we evaluate the performance and distributional shift of a robust numeric verbalization prompt across single vs. two-step confidence elicitation procedure. We also compare the performance of the same prompt with different versions of GPT and different numerical scales. Finally, we combine the sample-based consistency and verbalized methods to propose a hybrid framework that yields a better uncertainty estimation for GPT models. Overall, our work proposes novel uncertainty quantification methods that will improve the reliability of Large Language Models in misinformation mitigation applications.

Title: Decoupled Prototype Learning for Reliable Test-Time Adaptation

• Authors: Guowei Wang, Changxing Ding, Wentao Tan, Mingkui Tan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08703
• Pdf URL: https://arxiv.org/pdf/2401.08703
• Copy Paste: [[2401.08703]] Decoupled Prototype Learning for Reliable Test-Time Adaptation(https://arxiv.org/abs/2401.08703)
• Keywords: robust
• Abstract: Test-time adaptation (TTA) is a task that continually adapts a pre-trained source model to the target domain during inference. One popular approach involves fine-tuning model with cross-entropy loss according to estimated pseudo-labels. However, its performance is significantly affected by noisy pseudo-labels. This study reveals that minimizing the classification error of each sample causes the cross-entropy loss's vulnerability to label noise. To address this issue, we propose a novel Decoupled Prototype Learning (DPL) method that features prototype-centric loss computation. First, we decouple the optimization of class prototypes. For each class prototype, we reduce its distance with positive samples and enlarge its distance with negative samples in a contrastive manner. This strategy prevents the model from overfitting to noisy pseudo-labels. Second, we propose a memory-based strategy to enhance DPL's robustness for the small batch sizes often encountered in TTA. We update each class's pseudo-feature from a memory in a momentum manner and insert an additional DPL loss. Finally, we introduce a consistency regularization-based approach to leverage samples with unconfident pseudo-labels. This approach transfers feature styles of samples with unconfident pseudo-labels to those with confident pseudo-labels. Thus, more reliable samples for TTA are created. The experimental results demonstrate that our methods achieve state-of-the-art performance on domain generalization benchmarks, and reliably improve the performance of self-training-based methods on image corruption benchmarks. The code will be released.

Title: Unsupervised Pre-Training for 3D Leaf Instance Segmentation

• Authors: Gianmarco Roggiolani, Federico Magistri, Tiziano Guadagnino, Jens Behley, Cyrill Stachniss
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08720
• Pdf URL: https://arxiv.org/pdf/2401.08720
• Copy Paste: [[2401.08720]] Unsupervised Pre-Training for 3D Leaf Instance Segmentation(https://arxiv.org/abs/2401.08720)
• Keywords: segmentation
• Abstract: Crops for food, feed, fiber, and fuel are key natural resources for our society. Monitoring plants and measuring their traits is an important task in agriculture often referred to as plant phenotyping. Traditionally, this task is done manually, which is time- and labor-intensive. Robots can automate phenotyping providing reproducible and high-frequency measurements. Today's perception systems use deep learning to interpret these measurements, but require a substantial amount of annotated data to work well. Obtaining such labels is challenging as it often requires background knowledge on the side of the labelers. This paper addresses the problem of reducing the labeling effort required to perform leaf instance segmentation on 3D point clouds, which is a first step toward phenotyping in 3D. Separating all leaves allows us to count them and compute relevant traits as their areas, lengths, and widths. We propose a novel self-supervised task-specific pre-training approach to initialize the backbone of a network for leaf instance segmentation. We also introduce a novel automatic postprocessing that considers the difficulty of correctly segmenting the points close to the stem, where all the leaves petiole overlap. The experiments presented in this paper suggest that our approach boosts the performance over all the investigated scenarios. We also evaluate the embeddings to assess the quality of the fully unsupervised approach and see a higher performance of our domain-specific postprocessing.

Title: HierSFL: Local Differential Privacy-aided Split Federated Learning in Mobile Edge Computing

• Authors: Minh K. Quan, Dinh C. Nguyen, Van-Dinh Nguyen, Mayuri Wijayasundara, Sujeeva Setunge, Pubudu N. Pathirana
• Subjects: cs.CR, cs.CV, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08723
• Pdf URL: https://arxiv.org/pdf/2401.08723
• Copy Paste: [[2401.08723]] HierSFL: Local Differential Privacy-aided Split Federated Learning in Mobile Edge Computing(https://arxiv.org/abs/2401.08723)
• Keywords: privacy, federate
• Abstract: Federated Learning is a promising approach for learning from user data while preserving data privacy. However, the high requirements of the model training process make it difficult for clients with limited memory or bandwidth to participate. To tackle this problem, Split Federated Learning is utilized, where clients upload their intermediate model training outcomes to a cloud server for collaborative server-client model training. This methodology facilitates resource-constrained clients' participation in model training but also increases the training time and communication overhead. To overcome these limitations, we propose a novel algorithm, called Hierarchical Split Federated Learning (HierSFL), that amalgamates models at the edge and cloud phases, presenting qualitative directives for determining the best aggregation timeframes to reduce computation and communication expenses. By implementing local differential privacy at the client and edge server levels, we enhance privacy during local model parameter updates. Our experiments using CIFAR-10 and MNIST datasets show that HierSFL outperforms standard FL approaches with better training accuracy, training time, and communication-computing trade-offs. HierSFL offers a promising solution to mobile edge computing's challenges, ultimately leading to faster content delivery and improved mobile service quality.

Title: Revealing Vulnerabilities in Stable Diffusion via Targeted Attacks

• Authors: Chenyu Zhang, Lanjun Wang, Anan Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08725
• Pdf URL: https://arxiv.org/pdf/2401.08725
• Copy Paste: [[2401.08725]] Revealing Vulnerabilities in Stable Diffusion via Targeted Attacks(https://arxiv.org/abs/2401.08725)
• Keywords: attack, diffusion
• Abstract: Recent developments in text-to-image models, particularly Stable Diffusion, have marked significant achievements in various applications. With these advancements, there are growing safety concerns about the vulnerability of the model that malicious entities exploit to generate targeted harmful images. However, the existing methods in the vulnerability of the model mainly evaluate the alignment between the prompt and generated images, but fall short in revealing the vulnerability associated with targeted image generation. In this study, we formulate the problem of targeted adversarial attack on Stable Diffusion and propose a framework to generate adversarial prompts. Specifically, we design a gradient-based embedding optimization method to craft reliable adversarial prompts that guide stable diffusion to generate specific images. Furthermore, after obtaining successful adversarial prompts, we reveal the mechanisms that cause the vulnerability of the model. Extensive experiments on two targeted attack tasks demonstrate the effectiveness of our method in targeted attacks. The code can be obtained in https://github.com/datar001/Revealing-Vulnerabilities-in-Stable-Diffusion-via-Targeted-Attacks.

Title: Bag of Tricks to Boost Adversarial Transferability

• Authors: Zeliang Zhang, Rongyi Zhu, Wei Yao, Xiaosen Wang, Chenliang Xu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08734
• Pdf URL: https://arxiv.org/pdf/2401.08734
• Copy Paste: [[2401.08734]] Bag of Tricks to Boost Adversarial Transferability(https://arxiv.org/abs/2401.08734)
• Keywords: attack
• Abstract: Deep neural networks are widely known to be vulnerable to adversarial examples. However, vanilla adversarial examples generated under the white-box setting often exhibit low transferability across different models. Since adversarial transferability poses more severe threats to practical applications, various approaches have been proposed for better transferability, including gradient-based, input transformation-based, and model-related attacks, \etc. In this work, we find that several tiny changes in the existing adversarial attacks can significantly affect the attack performance, \eg, the number of iterations and step size. Based on careful studies of existing adversarial attacks, we propose a bag of tricks to enhance adversarial transferability, including momentum initialization, scheduled step size, dual example, spectral-based input transformation, and several ensemble strategies. Extensive experiments on the ImageNet dataset validate the high effectiveness of our proposed tricks and show that combining them can further boost adversarial transferability. Our work provides practical insights and techniques to enhance adversarial transferability, and offers guidance to improve the attack performance on the real-world application through simple adjustments.

Title: SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers

• Authors: Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, Saining Xie
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08740
• Pdf URL: https://arxiv.org/pdf/2401.08740
• Copy Paste: [[2401.08740]] SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers(https://arxiv.org/abs/2401.08740)
• Keywords: diffusion, transformer, generative
• Abstract: We present Scalable Interpolant Transformers (SiT), a family of generative models built on the backbone of Diffusion Transformers (DiT). The interpolant framework, which allows for connecting two distributions in a more flexible way than standard diffusion models, makes possible a modular study of various design choices impacting generative models built on dynamical transport: using discrete vs. continuous time learning, deciding the objective for the model to learn, choosing the interpolant connecting the distributions, and deploying a deterministic or stochastic sampler. By carefully introducing the above ingredients, SiT surpasses DiT uniformly across model sizes on the conditional ImageNet 256x256 benchmark using the exact same backbone, number of parameters, and GFLOPs. By exploring various diffusion coefficients, which can be tuned separately from learning, SiT achieves an FID-50K score of 2.06.

Title: Fixed Point Diffusion Models

• Authors: Xingjian Bai, Luke Melas-Kyriazi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08741
• Pdf URL: https://arxiv.org/pdf/2401.08741
• Copy Paste: [[2401.08741]] Fixed Point Diffusion Models(https://arxiv.org/abs/2401.08741)
• Keywords: diffusion, generative
• Abstract: We introduce the Fixed Point Diffusion Model (FPDM), a novel approach to image generation that integrates the concept of fixed point solving into the framework of diffusion-based generative modeling. Our approach embeds an implicit fixed point solving layer into the denoising network of a diffusion model, transforming the diffusion process into a sequence of closely-related fixed point problems. Combined with a new stochastic training method, this approach significantly reduces model size, reduces memory usage, and accelerates training. Moreover, it enables the development of two new techniques to improve sampling efficiency: reallocating computation across timesteps and reusing fixed point solutions between timesteps. We conduct extensive experiments with state-of-the-art models on ImageNet, FFHQ, CelebA-HQ, and LSUN-Church, demonstrating substantial improvements in performance and efficiency. Compared to the state-of-the-art DiT model, FPDM contains 87% fewer parameters, consumes 60% less memory during training, and improves image generation quality in situations where sampling computation or time is limited. Our code and pretrained models are available at https://lukemelas.github.io/fixed-point-diffusion-models.

Title: HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance

• Authors: Huanjun Kong, Songyang Zhang, Kai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.08772
• Pdf URL: https://arxiv.org/pdf/2401.08772
• Copy Paste: [[2401.08772]] HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance(https://arxiv.org/abs/2401.08772)
• Keywords: large language model
• Abstract: In this work, we present HuixiangDou, a technical assistant powered by Large Language Models (LLM). This system is designed to assist algorithm developers by providing insightful responses to questions related to open-source algorithm projects, such as computer vision and deep learning projects from OpenMMLab. We further explore the integration of this assistant into the group chats of instant messaging (IM) tools such as WeChat and Lark. Through several iterative improvements and trials, we have developed a sophisticated technical chat assistant capable of effectively answering users' technical questions without causing message flooding. This paper's contributions include: 1) Designing an algorithm pipeline specifically for group chat scenarios; 2) Verifying the reliable performance of text2vec in task rejection; 3) Identifying three critical requirements for LLMs in technical-assistant-like products, namely scoring ability, In-Context Learning (ICL), and Long Context. We have made the software and source code available at https://github.com/internlm/huixiangdou to aid in future research and application. HuixiangDou is applicable to any group chat within IM tools.

Title: Segment Anything Model Can Not Segment Anything: Assessing AI Foundation Model's Generalizability in Permafrost Mapping

• Authors: Wenwen Li, Chia-Yu Hsu, Sizhe Wang, Yezhou Yang, Hyunho Lee, Anna Liljedahl, Chandi Witharana, Yili Yang, Brendan M. Rogers, Samantha T. Arundel, Matthew B. Jones, Kenton McHenry, Patricia Solis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08787
• Pdf URL: https://arxiv.org/pdf/2401.08787
• Copy Paste: [[2401.08787]] Segment Anything Model Can Not Segment Anything: Assessing AI Foundation Model's Generalizability in Permafrost Mapping(https://arxiv.org/abs/2401.08787)
• Keywords: large language model, segmentation
• Abstract: This paper assesses trending AI foundation models, especially emerging computer vision foundation models and their performance in natural landscape feature segmentation. While the term foundation model has quickly garnered interest from the geospatial domain, its definition remains vague. Hence, this paper will first introduce AI foundation models and their defining characteristics. Built upon the tremendous success achieved by Large Language Models (LLMs) as the foundation models for language tasks, this paper discusses the challenges of building foundation models for geospatial artificial intelligence (GeoAI) vision tasks. To evaluate the performance of large AI vision models, especially Meta's Segment Anything Model (SAM), we implemented different instance segmentation pipelines that minimize the changes to SAM to leverage its power as a foundation model. A series of prompt strategies was developed to test SAM's performance regarding its theoretical upper bound of predictive accuracy, zero-shot performance, and domain adaptability through fine-tuning. The analysis used two permafrost feature datasets, ice-wedge polygons and retrogressive thaw slumps because (1) these landform features are more challenging to segment than manmade features due to their complicated formation mechanisms, diverse forms, and vague boundaries; (2) their presence and changes are important indicators for Arctic warming and climate change. The results show that although promising, SAM still has room for improvement to support AI-augmented terrain mapping. The spatial and domain generalizability of this finding is further validated using a more general dataset EuroCrop for agricultural field mapping. Finally, we discuss future research directions that strengthen SAM's applicability in challenging geospatial domains.

Title: The Impact of Differential Feature Under-reporting on Algorithmic Fairness

• Authors: Nil-Jana Akpinar, Zachary C. Lipton, Alexandra Chouldechova
• Subjects: cs.LG, cs.CY, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.08788
• Pdf URL: https://arxiv.org/pdf/2401.08788
• Copy Paste: [[2401.08788]] The Impact of Differential Feature Under-reporting on Algorithmic Fairness(https://arxiv.org/abs/2401.08788)
• Keywords: fair
• Abstract: Predictive risk models in the public sector are commonly developed using administrative data that is more complete for subpopulations that more greatly rely on public services. In the United States, for instance, information on health care utilization is routinely available to government agencies for individuals supported by Medicaid and Medicare, but not for the privately insured. Critiques of public sector algorithms have identified such differential feature under-reporting as a driver of disparities in algorithmic decision-making. Yet this form of data bias remains understudied from a technical viewpoint. While prior work has examined the fairness impacts of additive feature noise and features that are clearly marked as missing, the setting of data missingness absent indicators (i.e. differential feature under-reporting) has been lacking in research attention. In this work, we present an analytically tractable model of differential feature under-reporting which we then use to characterize the impact of this kind of data bias on algorithmic fairness. We demonstrate how standard missing data methods typically fail to mitigate bias in this setting, and propose a new set of methods specifically tailored to differential feature under-reporting. Our results show that, in real world data settings, under-reporting typically leads to increasing disparities. The proposed solution methods show success in mitigating increases in unfairness.

Title: Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive

• Authors: Yumeng Li, Margret Keuper, Dan Zhang, Anna Khoreva
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08815
• Pdf URL: https://arxiv.org/pdf/2401.08815
• Copy Paste: [[2401.08815]] Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive(https://arxiv.org/abs/2401.08815)
• Keywords: diffusion, segmentation
• Abstract: Despite the recent advances in large-scale diffusion models, little progress has been made on the layout-to-image (L2I) synthesis task. Current L2I models either suffer from poor editability via text or weak alignment between the generated image and the input layout. This limits their usability in practice. To mitigate this, we propose to integrate adversarial supervision into the conventional training pipeline of L2I diffusion models (ALDM). Specifically, we employ a segmentation-based discriminator which provides explicit feedback to the diffusion generator on the pixel-level alignment between the denoised image and the input layout. To encourage consistent adherence to the input layout over the sampling steps, we further introduce the multistep unrolling strategy. Instead of looking at a single timestep, we unroll a few steps recursively to imitate the inference process, and ask the discriminator to assess the alignment of denoised images with the layout over a certain time window. Our experiments show that ALDM enables layout faithfulness of the generated images, while allowing broad editability via text prompts. Moreover, we showcase its usefulness for practical applications: by synthesizing target distribution samples via text control, we improve domain generalization of semantic segmentation models by a large margin (~12 mIoU points).

Title: AiGen-FoodReview: A Multimodal Dataset of Machine-Generated Restaurant Reviews and Images on Social Media

• Authors: Alessandro Gambetti, Qiwei Han
• Subjects: cs.LG, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08825
• Pdf URL: https://arxiv.org/pdf/2401.08825
• Copy Paste: [[2401.08825]] AiGen-FoodReview: A Multimodal Dataset of Machine-Generated Restaurant Reviews and Images on Social Media(https://arxiv.org/abs/2401.08825)
• Keywords: large language model
• Abstract: Online reviews in the form of user-generated content (UGC) significantly impact consumer decision-making. However, the pervasive issue of not only human fake content but also machine-generated content challenges UGC's reliability. Recent advances in Large Language Models (LLMs) may pave the way to fabricate indistinguishable fake generated content at a much lower cost. Leveraging OpenAI's GPT-4-Turbo and DALL-E-2 models, we craft AiGen-FoodReview, a multi-modal dataset of 20,144 restaurant review-image pairs divided into authentic and machine-generated. We explore unimodal and multimodal detection models, achieving 99.80% multimodal accuracy with FLAVA. We use attributes from readability and photographic theories to score reviews and images, respectively, demonstrating their utility as hand-crafted features in scalable and interpretable detection models, with comparable performance. The paper contributes by open-sourcing the dataset and releasing fake review detectors, recommending its use in unimodal and multimodal fake review detection tasks, and evaluating linguistic and visual features in synthetic versus authentic data.

Title: Stochastic Subnetwork Annealing: A Regularization Technique for Fine Tuning Pruned Subnetworks

• Authors: Tim Whitaker, Darrell Whitley
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08830
• Pdf URL: https://arxiv.org/pdf/2401.08830
• Copy Paste: [[2401.08830]] Stochastic Subnetwork Annealing: A Regularization Technique for Fine Tuning Pruned Subnetworks(https://arxiv.org/abs/2401.08830)
• Keywords: robust
• Abstract: Pruning methods have recently grown in popularity as an effective way to reduce the size and computational complexity of deep neural networks. Large numbers of parameters can be removed from trained models with little discernible loss in accuracy after a small number of continued training epochs. However, pruning too many parameters at once often causes an initial steep drop in accuracy which can undermine convergence quality. Iterative pruning approaches mitigate this by gradually removing a small number of parameters over multiple epochs. However, this can still lead to subnetworks that overfit local regions of the loss landscape. We introduce a novel and effective approach to tuning subnetworks through a regularization technique we call Stochastic Subnetwork Annealing. Instead of removing parameters in a discrete manner, we instead represent subnetworks with stochastic masks where each parameter has a probabilistic chance of being included or excluded on any given forward pass. We anneal these probabilities over time such that subnetwork structure slowly evolves as mask values become more deterministic, allowing for a smoother and more robust optimization of subnetworks at high levels of sparsity.

Title: Improving ASR Contextual Biasing with Guided Attention

• Authors: Jiyang Tang, Kwangyoun Kim, Suwon Shon, Felix Wu, Prashant Sridhar, Shinji Watanabe
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.08835
• Pdf URL: https://arxiv.org/pdf/2401.08835
• Copy Paste: [[2401.08835]] Improving ASR Contextual Biasing with Guided Attention(https://arxiv.org/abs/2401.08835)
• Keywords: robust
• Abstract: In this paper, we propose a Guided Attention (GA) auxiliary training loss, which improves the effectiveness and robustness of automatic speech recognition (ASR) contextual biasing without introducing additional parameters. A common challenge in previous literature is that the word error rate (WER) reduction brought by contextual biasing diminishes as the number of bias phrases increases. To address this challenge, we employ a GA loss as an additional training objective besides the Transducer loss. The proposed GA loss aims to teach the cross attention how to align bias phrases with text tokens or audio frames. Compared to studies with similar motivations, the proposed loss operates directly on the cross attention weights and is easier to implement. Through extensive experiments based on Conformer Transducer with Contextual Adapter, we demonstrate that the proposed method not only leads to a lower WER but also retains its effectiveness as the number of bias phrases increases. Specifically, the GA loss decreases the WER of rare vocabularies by up to 19.2% on LibriSpeech compared to the contextual biasing baseline, and up to 49.3% compared to a vanilla Transducer.

Title: Robust Localization of Key Fob Using Channel Impulse Response of Ultra Wide Band Sensors for Keyless Entry Systems

• Authors: Abhiram Kolli, Filippo Casamassima, Horst Possegger, Horst Bischof
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.08863
• Pdf URL: https://arxiv.org/pdf/2401.08863
• Copy Paste: [[2401.08863]] Robust Localization of Key Fob Using Channel Impulse Response of Ultra Wide Band Sensors for Keyless Entry Systems(https://arxiv.org/abs/2401.08863)
• Keywords: security, robust
• Abstract: Using neural networks for localization of key fob within and surrounding a car as a security feature for keyless entry is fast emerging. In this paper we study: 1) the performance of pre-computed features of neural networks based UWB (ultra wide band) localization classification forming the baseline of our experiments. 2) Investigate the inherent robustness of various neural networks; therefore, we include the study of robustness of the adversarial examples without any adversarial training in this work. 3) Propose a multi-head self-supervised neural network architecture which outperforms the baseline neural networks without any adversarial training. The model's performance improved by 67% at certain ranges of adversarial magnitude for fast gradient sign method and 37% each for basic iterative method and projected gradient descent method.

Title: The Effect of Intrinsic Dataset Properties on Generalization: Unraveling Learning Differences Between Natural and Medical Images

• Authors: Nicholas Konz, Maciej A. Mazurowski
• Subjects: cs.CV, cs.LG, eess.IV, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.08865
• Pdf URL: https://arxiv.org/pdf/2401.08865
• Copy Paste: [[2401.08865]] The Effect of Intrinsic Dataset Properties on Generalization: Unraveling Learning Differences Between Natural and Medical Images(https://arxiv.org/abs/2401.08865)
• Keywords: attack, robust
• Abstract: This paper investigates discrepancies in how neural networks learn from different imaging domains, which are commonly overlooked when adopting computer vision techniques from the domain of natural images to other specialized domains such as medical images. Recent works have found that the generalization error of a trained network typically increases with the intrinsic dimension ($d_{data}$) of its training set. Yet, the steepness of this relationship varies significantly between medical (radiological) and natural imaging domains, with no existing theoretical explanation. We address this gap in knowledge by establishing and empirically validating a generalization scaling law with respect to $d_{data}$, and propose that the substantial scaling discrepancy between the two considered domains may be at least partially attributed to the higher intrinsic "label sharpness" ($K_F$) of medical imaging datasets, a metric which we propose. Next, we demonstrate an additional benefit of measuring the label sharpness of a training set: it is negatively correlated with the trained model's adversarial robustness, which notably leads to models for medical images having a substantially higher vulnerability to adversarial attack. Finally, we extend our $d_{data}$ formalism to the related metric of learned representation intrinsic dimension ($d_{repr}$), derive a generalization scaling law with respect to $d_{repr}$, and show that $d_{data}$ serves as an upper bound for $d_{repr}$. Our theoretical results are supported by thorough experiments with six models and eleven natural and medical imaging datasets over a range of training set sizes. Our findings offer insights into the influence of intrinsic dataset properties on generalization, representation learning, and robustness in deep neural networks.

Title: MambaTab: A Simple Yet Effective Approach for Handling Tabular Data

• Authors: Md Atik Ahamed, Qiang Cheng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08867
• Pdf URL: https://arxiv.org/pdf/2401.08867
• Copy Paste: [[2401.08867]] MambaTab: A Simple Yet Effective Approach for Handling Tabular Data(https://arxiv.org/abs/2401.08867)
• Keywords: transformer
• Abstract: Tabular data remains ubiquitous across domains despite growing use of images and texts for machine learning. While deep learning models like convolutional neural networks and transformers achieve strong performance on tabular data, they require extensive data preprocessing, tuning, and resources, limiting accessibility and scalability. This work develops an innovative approach based on a structured state-space model (SSM), MambaTab, for tabular data. SSMs have strong capabilities for efficiently extracting effective representations from data with long-range dependencies. MambaTab leverages Mamba, an emerging SSM variant, for end-to-end supervised learning on tables. Compared to state-of-the-art baselines, MambaTab delivers superior performance while requiring significantly fewer parameters and minimal preprocessing, as empirically validated on diverse benchmark datasets. MambaTab's efficiency, scalability, generalizability, and predictive gains signify it as a lightweight, "out-of-the-box" solution for diverse tabular data with promise for enabling wider practical applications.

Title: B-Cos Aligned Transformers Learn Human-Interpretable Features

• Authors: Manuel Tran, Amal Lahiani, Yashin Dicente Cid, Melanie Boxberg, Peter Lienemann, Christian Matek, Sophia J. Wagner, Fabian J. Theis, Eldad Klaiman, Tingying Peng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08868
• Pdf URL: https://arxiv.org/pdf/2401.08868
• Copy Paste: [[2401.08868]] B-Cos Aligned Transformers Learn Human-Interpretable Features(https://arxiv.org/abs/2401.08868)
• Keywords: interpretability, transformer
• Abstract: Vision Transformers (ViTs) and Swin Transformers (Swin) are currently state-of-the-art in computational pathology. However, domain experts are still reluctant to use these models due to their lack of interpretability. This is not surprising, as critical decisions need to be transparent and understandable. The most common approach to understanding transformers is to visualize their attention. However, attention maps of ViTs are often fragmented, leading to unsatisfactory explanations. Here, we introduce a novel architecture called the B-cos Vision Transformer (BvT) that is designed to be more interpretable. It replaces all linear transformations with the B-cos transform to promote weight-input alignment. In a blinded study, medical experts clearly ranked BvTs above ViTs, suggesting that our network is better at capturing biomedically relevant structures. This is also true for the B-cos Swin Transformer (Bwin). Compared to the Swin Transformer, it even improves the F1-score by up to 4.7% on two public datasets.

Title: DCRMTA: Unbiased Causal Representation for Multi-touch Attribution

• Authors: Jiaming Tang
• Subjects: cs.LG, cs.AI, stat.ME
• Abstract URL: https://arxiv.org/abs/2401.08875
• Pdf URL: https://arxiv.org/pdf/2401.08875
• Copy Paste: [[2401.08875]] DCRMTA: Unbiased Causal Representation for Multi-touch Attribution(https://arxiv.org/abs/2401.08875)
• Keywords: fair
• Abstract: Multi-touch attribution (MTA) currently plays a pivotal role in achieving a fair estimation of the contributions of each advertising touchpoint to-wards conversion behavior, deeply influencing budget allocation and advertising recommenda-tion. Traditional multi-touch attribution methods initially build a conversion prediction model, an-ticipating learning the inherent relationship be-tween touchpoint sequences and user purchasing behavior through historical data. Based on this, counterfactual touchpoint sequences are con-structed from the original sequence subset, and conversions are estimated using the prediction model, thus calculating advertising contributions. A covert assumption of these methods is the un-biased nature of conversion prediction models. However, due to confounding variables factors arising from user preferences and internet recom-mendation mechanisms such as homogenization of ad recommendations resulting from past shop-ping records, bias can easily occur in conversion prediction models trained on observational data. This paper redefines the causal effect of user fea-tures on conversions and proposes a novel end-to-end approach, Deep Causal Representation for MTA (DCRMTA). Our model while eliminating confounding variables, extracts features with causal relations to conversions from users. Fur-thermore, Extensive experiments on both synthet-ic and real-world Criteo data demonstrate DCRMTA's superior performance in converting prediction across varying data distributions, while also effectively attributing value across dif-ferent advertising channels

Title: Whispering Pixels: Exploiting Uninitialized Register Accesses in Modern GPUs

• Authors: Frederik Dermot Pustelnik, Xhani Marvin Saß, Jean-Pierre Seifert
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.08881
• Pdf URL: https://arxiv.org/pdf/2401.08881
• Copy Paste: [[2401.08881]] Whispering Pixels: Exploiting Uninitialized Register Accesses in Modern GPUs(https://arxiv.org/abs/2401.08881)
• Keywords: security, attack, large language model
• Abstract: Graphic Processing Units (GPUs) have transcended their traditional use-case of rendering graphics and nowadays also serve as a powerful platform for accelerating ubiquitous, non-graphical rendering tasks. One prominent task is inference of neural networks, which process vast amounts of personal data, such as audio, text or images. Thus, GPUs became integral components for handling vast amounts of potentially confidential data, which has awakened the interest of security researchers. This lead to the discovery of various vulnerabilities in GPUs in recent years. In this paper, we uncover yet another vulnerability class in GPUs: We found that some GPU implementations lack proper register initialization routines before shader execution, leading to unintended register content leakage of previously executed shader kernels. We showcase the existence of the aforementioned vulnerability on products of 3 major vendors - Apple, NVIDIA and Qualcomm. The vulnerability poses unique challenges to an adversary due to opaque scheduling and register remapping algorithms present in the GPU firmware, complicating the reconstruction of leaked data. In order to illustrate the real-world impact of this flaw, we showcase how these challenges can be solved for attacking various workloads on the GPU. First, we showcase how uninitialized registers leak arbitrary pixel data processed by fragment shaders. We further implement information leakage attacks on intermediate data of Convolutional Neural Networks (CNNs) and present the attack's capability to leak and reconstruct the output of Large Language Models (LLMs).

Title: RiemannONets: Interpretable Neural Operators for Riemann Problems

• Authors: Ahmad Peyvan, Vivek Oommen, Ameya D. Jagtap, George Em Karniadakis
• Subjects: cs.LG, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2401.08886
• Pdf URL: https://arxiv.org/pdf/2401.08886
• Copy Paste: [[2401.08886]] RiemannONets: Interpretable Neural Operators for Riemann Problems(https://arxiv.org/abs/2401.08886)
• Keywords: robust
• Abstract: Developing the proper representations for simulating high-speed flows with strong shock waves, rarefactions, and contact discontinuities has been a long-standing question in numerical analysis. Herein, we employ neural operators to solve Riemann problems encountered in compressible flows for extreme pressure jumps (up to $10^{10}$ pressure ratio). In particular, we first consider the DeepONet that we train in a two-stage process, following the recent work of Lee and Shin, wherein the first stage, a basis is extracted from the trunk net, which is orthonormalized and subsequently is used in the second stage in training the branch net. This simple modification of DeepONet has a profound effect on its accuracy, efficiency, and robustness and leads to very accurate solutions to Riemann problems compared to the vanilla version. It also enables us to interpret the results physically as the hierarchical data-driven produced basis reflects all the flow features that would otherwise be introduced using ad hoc feature expansion layers. We also compare the results with another neural operator based on the U-Net for low, intermediate, and very high-pressure ratios that are very accurate for Riemann problems, especially for large pressure ratios, due to their multiscale nature but computationally more expensive. Overall, our study demonstrates that simple neural network architectures, if properly pre-trained, can achieve very accurate solutions of Riemann problems for real-time forecasting.

Title: MADA: Meta-Adaptive Optimizers through hyper-gradient Descent

• Authors: Kaan Ozkara, Can Karakus, Parameswaran Raman, Mingyi Hong, Shoham Sabach, Branislav Kveton, Volkan Cevher
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2401.08893
• Pdf URL: https://arxiv.org/pdf/2401.08893
• Copy Paste: [[2401.08893]] MADA: Meta-Adaptive Optimizers through hyper-gradient Descent(https://arxiv.org/abs/2401.08893)
• Keywords: robust
• Abstract: Since Adam was introduced, several novel adaptive optimizers for deep learning have been proposed. These optimizers typically excel in some tasks but may not outperform Adam uniformly across all tasks. In this work, we introduce Meta-Adaptive Optimizers (MADA), a unified optimizer framework that can generalize several known optimizers and dynamically learn the most suitable one during training. The key idea in MADA is to parameterize the space of optimizers and search through it using hyper-gradient descent. Numerical results suggest that MADA is robust against sub-optimally tuned hyper-parameters, and outperforms Adam, Lion, and Adan with their default hyper-parameters, often even with optimized hyper-parameters. We also propose AVGrad, a variant of AMSGrad where the maximum operator is replaced with averaging, and observe that it performs better within MADA. Finally, we provide a convergence analysis to show that interpolation of optimizers (specifically, AVGrad and Adam) can improve their error bounds (up to constants), hinting at an advantage for meta-optimizers.

Title: HasTEE+ : Confidential Cloud Computing and Analytics with Haskell

• Authors: Abhiroop Sarkar, Alejandro Russo
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2401.08901
• Pdf URL: https://arxiv.org/pdf/2401.08901
• Copy Paste: [[2401.08901]] HasTEE+ : Confidential Cloud Computing and Analytics with Haskell(https://arxiv.org/abs/2401.08901)
• Keywords: security, protect
• Abstract: Confidential computing is a security paradigm that enables the protection of confidential code and data in a co-tenanted cloud deployment using specialized hardware isolation units called Trusted Execution Environments (TEEs). By integrating TEEs with a Remote Attestation protocol, confidential computing allows a third party to establish the integrity of an \textit{enclave} hosted within an untrusted cloud. However, TEE solutions, such as Intel SGX and ARM TrustZone, offer low-level C/C++-based toolchains that are susceptible to inherent memory safety vulnerabilities and lack language constructs to monitor explicit and implicit information-flow leaks. Moreover, the toolchains involve complex multi-project hierarchies and the deployment of hand-written attestation protocols for verifying \textit{enclave} integrity. We address the above with HasTEE+, a domain-specific language (DSL) embedded in Haskell that enables programming TEEs in a high-level language with strong type-safety. HasTEE+ assists in multi-tier cloud application development by (1) introducing a \textit{tierless} programming model for expressing distributed client-server interactions as a single program, (2) integrating a general remote-attestation architecture that removes the necessity to write application-specific cross-cutting attestation code, and (3) employing a dynamic information flow control mechanism to prevent explicit as well as implicit data leaks. We demonstrate the practicality of HasTEE+ through a case study on confidential data analytics, presenting a data-sharing pattern applicable to mutually distrustful participants and providing overall performance metrics.

Title: PPR: Enhancing Dodging Attacks while Maintaining Impersonation Attacks on Face Recognition Systems

• Authors: Fengfan Zhou, Heifei Ling
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08903
• Pdf URL: https://arxiv.org/pdf/2401.08903
• Copy Paste: [[2401.08903]] PPR: Enhancing Dodging Attacks while Maintaining Impersonation Attacks on Face Recognition Systems(https://arxiv.org/abs/2401.08903)
• Keywords: attack
• Abstract: Adversarial Attacks on Face Recognition (FR) encompass two types: impersonation attacks and evasion attacks. We observe that achieving a successful impersonation attack on FR does not necessarily ensure a successful dodging attack on FR in the black-box setting. Introducing a novel attack method named Pre-training Pruning Restoration Attack (PPR), we aim to enhance the performance of dodging attacks whilst avoiding the degradation of impersonation attacks. Our method employs adversarial example pruning, enabling a portion of adversarial perturbations to be set to zero, while tending to maintain the attack performance. By utilizing adversarial example pruning, we can prune the pre-trained adversarial examples and selectively free up certain adversarial perturbations. Thereafter, we embed adversarial perturbations in the pruned area, which enhances the dodging performance of the adversarial face examples. The effectiveness of our proposed attack method is demonstrated through our experimental results, showcasing its superior performance.

Title: Partial Diacritization: A Context-Contrastive Inference Approach

• Authors: Muhammad ElNokrashy, Badr AlKhamissi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08919
• Pdf URL: https://arxiv.org/pdf/2401.08919
• Copy Paste: [[2401.08919]] Partial Diacritization: A Context-Contrastive Inference Approach(https://arxiv.org/abs/2401.08919)
• Keywords: transformer
• Abstract: Diacritization plays a pivotal role in improving readability and disambiguating the meaning of Arabic texts. Efforts have so far focused on marking every eligible character (Full Diacritization). Comparatively overlooked, Partial Diacritzation (PD) is the selection of a subset of characters to be marked to aid comprehension where needed. Research has indicated that excessive diacritic marks can hinder skilled readers--reducing reading speed and accuracy. We conduct a behavioral experiment and show that partially marked text is often easier to read than fully marked text, and sometimes easier than plain text. In this light, we introduce Context-Contrastive Partial Diacritization (CCPD)--a novel approach to PD which integrates seamlessly with existing Arabic diacritization systems. CCPD processes each word twice, once with context and once without, and diacritizes only the characters with disparities between the two inferences. Further, we introduce novel indicators for measuring partial diacritization quality (SR, PDER, HDER, ERE), essential for establishing this as a machine learning task. Lastly, we introduce TD2, a Transformer-variant of an established model which offers a markedly different per formance profile on our proposed indicators compared to all other known systems.

Title: RandOhm: Mitigating Impedance Side-channel Attacks using Randomized Circuit Configurations

• Authors: Saleh Khalaj Monfared, Domenic Forte, Shahin Tajik
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.08925
• Pdf URL: https://arxiv.org/pdf/2401.08925
• Copy Paste: [[2401.08925]] RandOhm: Mitigating Impedance Side-channel Attacks using Randomized Circuit Configurations(https://arxiv.org/abs/2401.08925)
• Keywords: security, defense, attack
• Abstract: Physical side-channel attacks can compromise the security of integrated circuits. Most of the physical side-channel attacks (e.g., power or electromagnetic) exploit the dynamic behavior of a chip, typically manifesting as changes in current consumption or voltage fluctuations where algorithmic countermeasures, such as masking, can effectively mitigate the attacks. However, as demonstrated recently, these mitigation techniques are not entirely effective against backscattered side-channel attacks such as impedance analysis. In the case of an impedance attack, an adversary exploits the data-dependent impedance variations of chip power delivery network (PDN) to extract secret information. In this work, we introduce RandOhm, which exploits moving target defense (MTD) strategy based on partial reconfiguration of mainstream FPGAs, to defend against impedance side-channel attacks. We demonstrate that the information leakage through the PDN impedance could be reduced via run-time reconfiguration of the secret-sensitive parts of the circuitry. Hence, by constantly randomizing the placement and routing of the circuit, one can decorrelate the data-dependent computation from the impedance value. To validate our claims, we present a systematic approach equipped with two different partial reconfiguration strategies on implementations of the AES cipher realized on 28-nm FPGAs. We investigate the overhead of our mitigation in terms of delay and performance and provide security analysis by performing non-profiled and profiled impedance analysis attacks against these implementations to demonstrate the resiliency of our approach.

Title: Uncertainty-aware No-Reference Point Cloud Quality Assessment

• Authors: Songlin Fan, Zixuan Guo, Wei Gao, Ge Li
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.08926
• Pdf URL: https://arxiv.org/pdf/2401.08926
• Copy Paste: [[2401.08926]] Uncertainty-aware No-Reference Point Cloud Quality Assessment(https://arxiv.org/abs/2401.08926)
• Keywords: robust
• Abstract: The evolution of compression and enhancement algorithms necessitates an accurate quality assessment for point clouds. Previous works consistently regard point cloud quality assessment (PCQA) as a MOS regression problem and devise a deterministic mapping, ignoring the stochasticity in generating MOS from subjective tests. Besides, the viewpoint switching of 3D point clouds in subjective tests reinforces the judging stochasticity of different subjects compared with traditional images. This work presents the first probabilistic architecture for no-reference PCQA, motivated by the labeling process of existing datasets. The proposed method can model the quality judging stochasticity of subjects through a tailored conditional variational autoencoder (CVAE) and produces multiple intermediate quality ratings. These intermediate ratings simulate the judgments from different subjects and are then integrated into an accurate quality prediction, mimicking the generation process of a ground truth MOS. Specifically, our method incorporates a Prior Module, a Posterior Module, and a Quality Rating Generator, where the former two modules are introduced to model the judging stochasticity in subjective tests, while the latter is developed to generate diverse quality ratings. Extensive experiments indicate that our approach outperforms previous cutting-edge methods by a large margin and exhibits gratifying cross-dataset robustness.

Title: 3D Human Pose Analysis via Diffusion Synthesis

• Authors: Haorui Ji, Hongdong Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.08930
• Pdf URL: https://arxiv.org/pdf/2401.08930
• Copy Paste: [[2401.08930]] 3D Human Pose Analysis via Diffusion Synthesis(https://arxiv.org/abs/2401.08930)
• Keywords: robust, diffusion, generative
• Abstract: Diffusion models have demonstrated remarkable success in generative modeling. In this paper, we propose PADS (Pose Analysis by Diffusion Synthesis), a novel framework designed to address various challenges in 3D human pose analysis through a unified pipeline. Central to PADS are two distinctive strategies: i) learning a task-agnostic pose prior using a diffusion synthesis process to effectively capture the kinematic constraints in human pose data, and ii) unifying multiple pose analysis tasks like estimation, completion, denoising, etc, as instances of inverse problems. The learned pose prior will be treated as a regularization imposing on task-specific constraints, guiding the optimization process through a series of conditional denoising steps. PADS represents the first diffusion-based framework for tackling general 3D human pose analysis within the inverse problem framework. Its performance has been validated on different benchmarks, signaling the adaptability and robustness of this pipeline.

Title: Learning to detect cloud and snow in remote sensing images from noisy labels

• Authors: Zili Liu, Hao Chen, Wenyuan Li, Keyan Chen, Zipeng Qi, Chenyang Liu, Zhengxia Zou, Zhenwei Shi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.08932
• Pdf URL: https://arxiv.org/pdf/2401.08932
• Copy Paste: [[2401.08932]] Learning to detect cloud and snow in remote sensing images from noisy labels(https://arxiv.org/abs/2401.08932)
• Keywords: segmentation
• Abstract: Detecting clouds and snow in remote sensing images is an essential preprocessing task for remote sensing imagery. Previous works draw inspiration from semantic segmentation models in computer vision, with most research focusing on improving model architectures to enhance detection performance. However, unlike natural images, the complexity of scenes and the diversity of cloud types in remote sensing images result in many inaccurate labels in cloud and snow detection datasets, introducing unnecessary noises into the training and testing processes. By constructing a new dataset and proposing a novel training strategy with the curriculum learning paradigm, we guide the model in reducing overfitting to noisy labels. Additionally, we design a more appropriate model performance evaluation method, that alleviates the performance assessment bias caused by noisy labels. By conducting experiments on models with UNet and Segformer, we have validated the effectiveness of our proposed method. This paper is the first to consider the impact of label noise on the detection of clouds and snow in remote sensing images.

Title: CEL: A Continual Learning Model for Disease Outbreak Prediction by Leveraging Domain Adaptation via Elastic Weight Consolidation

• Authors: Saba Aslam, Abdur Rasool, Hongyan Wu, Xiaoli Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.08940
• Pdf URL: https://arxiv.org/pdf/2401.08940
• Copy Paste: [[2401.08940]] CEL: A Continual Learning Model for Disease Outbreak Prediction by Leveraging Domain Adaptation via Elastic Weight Consolidation(https://arxiv.org/abs/2401.08940)
• Keywords: robust
• Abstract: Continual learning, the ability of a model to learn over time without forgetting previous knowledge and, therefore, be adaptive to new data, is paramount in dynamic fields such as disease outbreak prediction. Deep neural networks, i.e., LSTM, are prone to error due to catastrophic forgetting. This study introduces a novel CEL model for continual learning by leveraging domain adaptation via Elastic Weight Consolidation (EWC). This model aims to mitigate the catastrophic forgetting phenomenon in a domain incremental setting. The Fisher Information Matrix (FIM) is constructed with EWC to develop a regularization term that penalizes changes to important parameters, namely, the important previous knowledge. CEL's performance is evaluated on three distinct diseases, Influenza, Mpox, and Measles, with different metrics. The high R-squared values during evaluation and reevaluation outperform the other state-of-the-art models in several contexts, indicating that CEL adapts to incremental data well. CEL's robustness and reliability are underscored by its minimal 65% forgetting rate and 18% higher memory stability compared to existing benchmark studies. This study highlights CEL's versatility in disease outbreak prediction, addressing evolving data with temporal patterns. It offers a valuable model for proactive disease control with accurate, timely predictions.

Title: AntiPhishStack: LSTM-based Stacked Generalization Model for Optimized Phishing URLs Detection

• Authors: Saba Aslam, Hafsa Aslam, Arslan Manzoor, Chen Hui, Abdur Rasool
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08947
• Pdf URL: https://arxiv.org/pdf/2401.08947
• Copy Paste: [[2401.08947]] AntiPhishStack: LSTM-based Stacked Generalization Model for Optimized Phishing URLs Detection(https://arxiv.org/abs/2401.08947)
• Keywords: security, robust
• Abstract: The escalating reliance on revolutionary online web services has introduced heightened security risks, with persistent challenges posed by phishing despite extensive security measures. Traditional phishing systems, reliant on machine learning and manual features, struggle with evolving tactics. Recent advances in deep learning offer promising avenues for tackling novel phishing challenges and malicious URLs. This paper introduces a two-phase stack generalized model named AntiPhishStack, designed to detect phishing sites. The model leverages the learning of URLs and character-level TF-IDF features symmetrically, enhancing its ability to combat emerging phishing threats. In Phase I, features are trained on a base machine learning classifier, employing K-fold cross-validation for robust mean prediction. Phase II employs a two-layered stacked-based LSTM network with five adaptive optimizers for dynamic compilation, ensuring premier prediction on these features. Additionally, the symmetrical predictions from both phases are optimized and integrated to train a meta-XGBoost classifier, contributing to a final robust prediction. The significance of this work lies in advancing phishing detection with AntiPhishStack, operating without prior phishing-specific feature knowledge. Experimental validation on two benchmark datasets, comprising benign and phishing or malicious URLs, demonstrates the model's exceptional performance, achieving a notable 96.04% accuracy compared to existing studies. This research adds value to the ongoing discourse on symmetry and asymmetry in information security and provides a forward-thinking solution for enhancing network security in the face of evolving cyber threats.

Title: An Efficient and Scalable Auditing Scheme for Cloud Data Storage using an Enhanced B-tree

• Authors: Tariqul Islam, Faisal Haque Bappy, Md Nafis Ul Haque Shifat, Farhan Ahmad, Kamrul Hasan, Tarannum Shaila Zaman
• Subjects: cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2401.08953
• Pdf URL: https://arxiv.org/pdf/2401.08953
• Copy Paste: [[2401.08953]] An Efficient and Scalable Auditing Scheme for Cloud Data Storage using an Enhanced B-tree(https://arxiv.org/abs/2401.08953)
• Keywords: secure
• Abstract: An efficient, scalable, and provably secure dynamic auditing scheme is highly desirable in the cloud storage environment for verifying the integrity of the outsourced data. Most of the existing work on remote integrity checking focuses on static archival data and therefore cannot be applied to cases where dynamic data updates are more common. Additionally, existing auditing schemes suffer from performance bottlenecks and scalability issues. To address these issues, in this paper, we present a novel dynamic auditing scheme for centralized cloud environments leveraging an enhanced version of the B-tree. Our proposed scheme achieves the immutable characteristic of a decentralized system (i.e., blockchain technology) while effectively addressing the synchronization and performance challenges of such systems. Unlike other static auditing schemes, our scheme supports dynamic insert, update, and delete operations. Also, by leveraging an enhanced B-tree, our scheme maintains a balanced tree after any alteration to a certain file, improving performance significantly. Experimental results show that our scheme outperforms both traditional Merkle Hash Tree-based centralized auditing and decentralized blockchain-based auditing schemes in terms of block modifications (e.g., insert, delete, update), block retrieval, and data verification time.

Title: Dynamic DNNs and Runtime Management for Efficient Inference on Mobile/Embedded Devices

• Authors: Lei Xun, Jonathon Hare, Geoff V. Merrett
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08965
• Pdf URL: https://arxiv.org/pdf/2401.08965
• Copy Paste: [[2401.08965]] Dynamic DNNs and Runtime Management for Efficient Inference on Mobile/Embedded Devices(https://arxiv.org/abs/2401.08965)
• Keywords: privacy
• Abstract: Deep neural network (DNN) inference is increasingly being executed on mobile and embedded platforms due to several key advantages in latency, privacy and always-on availability. However, due to limited computing resources, efficient DNN deployment on mobile and embedded platforms is challenging. Although many hardware accelerators and static model compression methods were proposed by previous works, at system runtime, multiple applications are typically executed concurrently and compete for hardware resources. This raises two main challenges: Runtime Hardware Availability and Runtime Application Variability. Previous works have addressed these challenges through either dynamic neural networks that contain sub-networks with different performance trade-offs or runtime hardware resource management. In this thesis, we proposed a combined method, a system was developed for DNN performance trade-off management, combining the runtime trade-off opportunities in both algorithms and hardware to meet dynamically changing application performance targets and hardware constraints in real time. We co-designed novel Dynamic Super-Networks to maximise runtime system-level performance and energy efficiency on heterogeneous hardware platforms. Compared with SOTA, our experimental results using ImageNet on the GPU of Jetson Xavier NX show our model is 2.4x faster for similar ImageNet Top-1 accuracy, or 5.1% higher accuracy at similar latency. We also designed a hierarchical runtime resource manager that tunes both dynamic neural networks and DVFS at runtime. Compared with the Linux DVFS governor schedutil, our runtime approach achieves up to a 19% energy reduction and a 9% latency reduction in single model deployment scenario, and an 89% energy reduction and a 23% latency reduction in a two concurrent model deployment scenario.

Title: ReFT: Reasoning with Reinforced Fine-Tuning

• Authors: Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, Hang Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.08967
• Pdf URL: https://arxiv.org/pdf/2401.08967
• Copy Paste: [[2401.08967]] ReFT: Reasoning with Reinforced Fine-Tuning(https://arxiv.org/abs/2401.08967)
• Keywords: large language model
• Abstract: One way to enhance the reasoning capability of Large Language Models (LLMs) is to conduct Supervised Fine-Tuning (SFT) using Chain-of-Thought (CoT) annotations. This approach does not show sufficiently strong generalization ability, however, because the training only relies on the given CoT data. In math problem-solving, for example, there is usually only one annotated reasoning path for each question in the training data. Intuitively, it would be better for the algorithm to learn from multiple annotated reasoning paths given a question. To address this issue, we propose a simple yet effective approach called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of learning LLMs for reasoning, with math problem-solving as an example. ReFT first warmups the model with SFT, and then employs on-line reinforcement learning, specifically the PPO algorithm in this paper, to further fine-tune the model, where an abundance of reasoning paths are automatically sampled given the question and the rewards are naturally derived from the ground-truth answers. Extensive experiments on GSM8K, MathQA, and SVAMP datasets show that ReFT significantly outperforms SFT, and the performance can be potentially further boosted by combining inference-time strategies such as majority voting and re-ranking. Note that ReFT obtains the improvement by learning from the same training questions as SFT, without relying on extra or augmented training questions. This indicates a superior generalization ability for ReFT.

Title: COCO is "ALL'' You Need for Visual Instruction Fine-tuning

• Authors: Xiaotian Han, Yiqi Wang, Bohan Zhai, Quanzeng You, Hongxia Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08968
• Pdf URL: https://arxiv.org/pdf/2401.08968
• Copy Paste: [[2401.08968]] COCO is "ALL'' You Need for Visual Instruction Fine-tuning(https://arxiv.org/abs/2401.08968)
• Keywords: generative, large language model
• Abstract: Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. Visual instruction fine-tuning (IFT) is a vital process for aligning MLLMs' output with user's intentions. High-quality and diversified instruction following data is the key to this fine-tuning process. Recent studies propose to construct visual IFT datasets through a multifaceted approach: transforming existing datasets with rule-based templates, employing GPT-4 for rewriting annotations, and utilizing GPT-4V for visual dataset pseudo-labeling. LLaVA-1.5 adopted similar approach and construct LLaVA-mix-665k, which is one of the simplest, most widely used, yet most effective IFT datasets today. Notably, when properly fine-tuned with this dataset, MLLMs can achieve state-of-the-art performance on several benchmarks. However, we noticed that models trained with this dataset often struggle to follow user instructions properly in multi-round dialog. In addition, tradition caption and VQA evaluation benchmarks, with their closed-form evaluation structure, are not fully equipped to assess the capabilities of modern open-ended generative MLLMs. This problem is not unique to the LLaVA-mix-665k dataset, but may be a potential issue in all IFT datasets constructed from image captioning or VQA sources, though the extent of this issue may vary. We argue that datasets with diverse and high-quality detailed instruction following annotations are essential and adequate for MLLMs IFT. In this work, we establish a new IFT dataset, with images sourced from the COCO dataset along with more diverse instructions. Our experiments show that when fine-tuned with out proposed dataset, MLLMs achieve better performance on open-ended evaluation benchmarks in both single-round and multi-round dialog setting.

Title: ACT-GAN: Radio map construction based on generative adversarial networks with ACT blocks

• Authors: Chen Qi, Yang Jingjing, Huang Ming, Zhou Qiang
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.08976
• Pdf URL: https://arxiv.org/pdf/2401.08976
• Copy Paste: [[2401.08976]] ACT-GAN: Radio map construction based on generative adversarial networks with ACT blocks(https://arxiv.org/abs/2401.08976)
• Keywords: robust, generative
• Abstract: The radio map, serving as a visual representation of electromagnetic spatial characteristics, plays a pivotal role in assessment of wireless communication networks and radio monitoring coverage. Addressing the issue of low accuracy existing in the current radio map construction, this paper presents a novel radio map construction method based on generative adversarial network (GAN) in which the Aggregated Contextual-Transformation (AOT) block, Convolutional Block Attention Module (CBAM), and Transposed Convolution (T-Conv) block are applied to the generator, and we name it as ACT-GAN. It significantly improves the reconstruction accuracy and local texture of the radio maps. The performance of ACT-GAN across three different scenarios is demonstrated. Experiment results reveal that in the scenario without sparse discrete observations, the proposed method reduces the root mean square error (RMSE) by 14.6% in comparison to the state-of-the-art models. In the scenario with sparse discrete observations, the RMSE is diminished by 13.2%. Furthermore, the predictive results of the proposed model show a more lucid representation of electromagnetic spatial field distribution. To verify the universality of this model in radio map construction tasks, the scenario of unknown radio emission source is investigated. The results indicate that the proposed model is robust radio map construction and accurate in predicting the location of the emission source.

Title: FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data

• Authors: Zikai Xiao, Zihan Chen, Liyinglan Liu, Yang Feng, Jian Wu, Wanlu Liu, Joey Tianyi Zhou, Howard Hao Yang, Zuozhu Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.08977
• Pdf URL: https://arxiv.org/pdf/2401.08977
• Copy Paste: [[2401.08977]] FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data(https://arxiv.org/abs/2401.08977)
• Keywords: federate
• Abstract: Federated Long-Tailed Learning (Fed-LT), a paradigm wherein data collected from decentralized local clients manifests a globally prevalent long-tailed distribution, has garnered considerable attention in recent times. In the context of Fed-LT, existing works have predominantly centered on addressing the data imbalance issue to enhance the efficacy of the generic global model while neglecting the performance at the local level. In contrast, conventional Personalized Federated Learning (pFL) techniques are primarily devised to optimize personalized local models under the presumption of a balanced global data distribution. This paper introduces an approach termed Federated Local and Generic Model Training in Fed-LT (FedLoGe), which enhances both local and generic model performance through the integration of representation learning and classifier alignment within a neural collapse framework. Our investigation reveals the feasibility of employing a shared backbone as a foundational framework for capturing overarching global trends, while concurrently employing individualized classifiers to encapsulate distinct refinements stemming from each client's local features. Building upon this discovery, we establish the Static Sparse Equiangular Tight Frame Classifier (SSE-C), inspired by neural collapse principles that naturally prune extraneous noisy features and foster the acquisition of potent data representations. Furthermore, leveraging insights from imbalance neural collapse's classifier norm patterns, we develop Global and Local Adaptive Feature Realignment (GLA-FR) via an auxiliary global classifier and personalized Euclidean norm transfer to align global features with client preferences. Extensive experimental results on CIFAR-10/100-LT, ImageNet, and iNaturalist demonstrate the advantage of our method over state-of-the-art pFL and Fed-LT approaches.

Title: A GAN-based data poisoning framework against anomaly detection in vertical federated learning

• Authors: Xiaolin Chen, Daoguang Zan, Wei Li, Bei Guan, Yongji Wang
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.08984
• Pdf URL: https://arxiv.org/pdf/2401.08984
• Copy Paste: [[2401.08984]] A GAN-based data poisoning framework against anomaly detection in vertical federated learning(https://arxiv.org/abs/2401.08984)
• Keywords: privacy, defense, attack, robust, federate
• Abstract: In vertical federated learning (VFL), commercial entities collaboratively train a model while preserving data privacy. However, a malicious participant's poisoning attack may degrade the performance of this collaborative model. The main challenge in achieving the poisoning attack is the absence of access to the server-side top model, leaving the malicious participant without a clear target model. To address this challenge, we introduce an innovative end-to-end poisoning framework P-GAN. Specifically, the malicious participant initially employs semi-supervised learning to train a surrogate target model. Subsequently, this participant employs a GAN-based method to produce adversarial perturbations to degrade the surrogate target model's performance. Finally, the generator is obtained and tailored for VFL poisoning. Besides, we develop an anomaly detection algorithm based on a deep auto-encoder (DAE), offering a robust defense mechanism to VFL scenarios. Through extensive experiments, we evaluate the efficacy of P-GAN and DAE, and further analyze the factors that influence their performance.

Title: Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASR

• Authors: Junwen Bai, Bo Li, Qiujia Li, Tara N. Sainath, Trevor Strohman
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.08992
• Pdf URL: https://arxiv.org/pdf/2401.08992
• Copy Paste: [[2401.08992]] Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASR(https://arxiv.org/abs/2401.08992)
• Keywords: privacy, protect
• Abstract: The end-to-end ASR model is often desired in the streaming multilingual scenario since it is easier to deploy and can benefit from pre-trained speech models such as powerful foundation models. Meanwhile, the heterogeneous nature and imbalanced data abundance of different languages may cause performance degradation, leading to asynchronous peak performance for different languages during training, especially on tail ones. Sometimes even the data itself may become unavailable as a result of the enhanced privacy protection. Existing work tend to significantly increase the model size or learn language-specific decoders to accommodate each language separately. In this study, we explore simple yet effective Language-Dependent Adapter (LDA) finetuning under a cascaded Conformer transducer framework enhanced by teacher pseudo-labeling for tail languages in the streaming multilingual ASR. The adapter only accounts for 0.4% of the full model per language. It is plugged into the frozen foundation model and is the only trainable module during the finetuning process with noisy student training. The final model merges the adapter parameters from different checkpoints for different languages. The model performance is validated on a challenging multilingual dictation dataset, which includes 39 tail languages across Latin, Greek, Arabic, etc. Our proposed method brings 12.2% word error rate reduction on average and up to 37.5% on a single locale. Furthermore, we show that our parameter-efficient LDA can match the quality of the full model finetuning, thus greatly alleviating the asynchronous peak performance issue.

Title: Attack and Reset for Unlearning: Exploiting Adversarial Noise toward Machine Unlearning through Parameter Re-initialization

• Authors: Yoonhwa Jung, Ikhyun Cho, Shun-Hsiang Hsu, Julia Hockenmaier
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08998
• Pdf URL: https://arxiv.org/pdf/2401.08998
• Copy Paste: [[2401.08998]] Attack and Reset for Unlearning: Exploiting Adversarial Noise toward Machine Unlearning through Parameter Re-initialization(https://arxiv.org/abs/2401.08998)
• Keywords: privacy, attack
• Abstract: With growing concerns surrounding privacy and regulatory compliance, the concept of machine unlearning has gained prominence, aiming to selectively forget or erase specific learned information from a trained model. In response to this critical need, we introduce a novel approach called Attack-and-Reset for Unlearning (ARU). This algorithm leverages meticulously crafted adversarial noise to generate a parameter mask, effectively resetting certain parameters and rendering them unlearnable. ARU outperforms current state-of-the-art results on two facial machine-unlearning benchmark datasets, MUFAC and MUCAC. In particular, we present the steps involved in attacking and masking that strategically filter and re-initialize network parameters biased towards the forget set. Our work represents a significant advancement in rendering data unexploitable to deep learning models through parameter re-initialization, achieved by harnessing adversarial noise to craft a mask.

Title: AttackEval: How to Evaluate the Effectiveness of Jailbreak Attacking on Large Language Models

• Authors: Dong shu, Mingyu Jin, Suiyuan Zhu, Beichen Wang, Zihao Zhou, Chong Zhang, Yongfeng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09002
• Pdf URL: https://arxiv.org/pdf/2401.09002
• Copy Paste: [[2401.09002]] AttackEval: How to Evaluate the Effectiveness of Jailbreak Attacking on Large Language Models(https://arxiv.org/abs/2401.09002)
• Keywords: attack, robust, large language model
• Abstract: In our research, we pioneer a novel approach to evaluate the effectiveness of jailbreak attacks on Large Language Models (LLMs), such as GPT-4 and LLaMa2, diverging from traditional robustness-focused binary evaluations. Our study introduces two distinct evaluation frameworks: a coarse-grained evaluation and a fine-grained evaluation. Each framework, using a scoring range from 0 to 1, offers a unique perspective, enabling a more comprehensive and nuanced evaluation of attack effectiveness and empowering attackers to refine their attack prompts with greater understanding. Furthermore, we have developed a comprehensive ground truth dataset specifically tailored for jailbreak tasks. This dataset not only serves as a crucial benchmark for our current study but also establishes a foundational resource for future research, enabling consistent and comparative analyses in this evolving field. Upon meticulous comparison with traditional evaluation methods, we discovered that our evaluation aligns with the baseline's trend while offering a more profound and detailed assessment. We believe that by accurately evaluating the effectiveness of attack prompts in the Jailbreak task, our work lays a solid foundation for assessing a wider array of similar or even more complex tasks in the realm of prompt injection, potentially revolutionizing this field.

Title: Augmenting Math Word Problems via Iterative Question Composing

• Authors: Haoxiong Liu, Andrew Chi-Chih Yao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09003
• Pdf URL: https://arxiv.org/pdf/2401.09003
• Copy Paste: [[2401.09003]] Augmenting Math Word Problems via Iterative Question Composing(https://arxiv.org/abs/2401.09003)
• Keywords: large language model
• Abstract: Despite recent progress in improving the mathematical reasoning ability of large language models(LLMs), solving competition-level math problems without the use of external tools remains challenging for open-source LLMs. In this work, we introduce the MMIQC dataset, a mixture of processed web data and synthetic question-response pairs, to equip base models with better mathematical reasoning skills. Mistral-7B-MMIQC, the model obtained by fine-tuning Mistral-7B(arXiv:2310.06825) on MMIQC, achieves 36.0\% accuracy on MATH(arXiv:2103.03874), 5.8\% higher than the previous (model size $\sim$7B) SOTA. Our experiments also show that a large part of the improvement attributes to our novel augmentation method IQC(Iterative Question Composing), where we iteratively ask an LLM to compose new questions from the given seed problems and do rejection sampling from another LLM. MMIQC has now been released on https://huggingface.co/datasets/Vivacem/MMIQC.

Title: Generalized Face Liveness Detection via De-spoofing Face Generator

• Authors: Xingming Long, Shiguang Shan, Jie Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09006
• Pdf URL: https://arxiv.org/pdf/2401.09006
• Copy Paste: [[2401.09006]] Generalized Face Liveness Detection via De-spoofing Face Generator(https://arxiv.org/abs/2401.09006)
• Keywords: attack, extraction, transformer
• Abstract: Previous Face Anti-spoofing (FAS) works face the challenge of generalizing in unseen domains. One of the major problems is that most existing FAS datasets are relatively small and lack data diversity. However, we find that there are numerous real faces that can be easily achieved under various conditions, which are neglected by previous FAS works. In this paper, we conduct an Anomalous cue Guided FAS (AG-FAS) method, which leverages real faces for improving model generalization via a De-spoofing Face Generator (DFG). Specifically, the DFG trained only on the real faces gains the knowledge of what a real face should be like and can generate a "real" version of the face corresponding to any given input face. The difference between the generated "real" face and the input face can provide an anomalous cue for the downstream FAS task. We then propose an Anomalous cue Guided FAS feature extraction Network (AG-Net) to further improve the FAS feature generalization via a cross-attention transformer. Extensive experiments on a total of nine public datasets show our method achieves state-of-the-art results under cross-domain evaluations with unseen scenarios and unknown presentation attacks.

Title: Explain Thyself Bully: Sentiment Aided Cyberbullying Detection with Explanation

• Authors: Krishanu Maity, Prince Jha, Raghav Jain, Sriparna Saha, Pushpak Bhattacharyya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09023
• Pdf URL: https://arxiv.org/pdf/2401.09023
• Copy Paste: [[2401.09023]] Explain Thyself Bully: Sentiment Aided Cyberbullying Detection with Explanation(https://arxiv.org/abs/2401.09023)
• Keywords: protect, explainability
• Abstract: Cyberbullying has become a big issue with the popularity of different social media networks and online communication apps. While plenty of research is going on to develop better models for cyberbullying detection in monolingual language, there is very little research on the code-mixed languages and explainability aspect of cyberbullying. Recent laws like "right to explanations" of General Data Protection Regulation, have spurred research in developing interpretable models rather than focusing on performance. Motivated by this we develop the first interpretable multi-task model called {\em mExCB} for automatic cyberbullying detection from code-mixed languages which can simultaneously solve several tasks, cyberbullying detection, explanation/rationale identification, target group detection and sentiment analysis. We have introduced {\em BullyExplain}, the first benchmark dataset for explainable cyberbullying detection in code-mixed language. Each post in {\em BullyExplain} dataset is annotated with four labels, i.e., {\em bully label, sentiment label, target and rationales (explainability)}, i.e., which phrases are being responsible for annotating the post as a bully. The proposed multitask framework (mExCB) based on CNN and GRU with word and sub-sentence (SS) level attention is able to outperform several baselines and state of the art models when applied on {\em BullyExplain} dataset.

Title: Cross-modality Guidance-aided Multi-modal Learning with Dual Attention for MRI Brain Tumor Grading

• Authors: Dunyuan Xu, Xi Wang, Jinyue Cai, Pheng-Ann Heng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09029
• Pdf URL: https://arxiv.org/pdf/2401.09029
• Copy Paste: [[2401.09029]] Cross-modality Guidance-aided Multi-modal Learning with Dual Attention for MRI Brain Tumor Grading(https://arxiv.org/abs/2401.09029)
• Keywords: extraction
• Abstract: Brain tumor represents one of the most fatal cancers around the world, and is very common in children and the elderly. Accurate identification of the type and grade of tumor in the early stages plays an important role in choosing a precise treatment plan. The Magnetic Resonance Imaging (MRI) protocols of different sequences provide clinicians with important contradictory information to identify tumor regions. However, manual assessment is time-consuming and error-prone due to big amount of data and the diversity of brain tumor types. Hence, there is an unmet need for MRI automated brain tumor diagnosis. We observe that the predictive capability of uni-modality models is limited and their performance varies widely across modalities, and the commonly used modality fusion methods would introduce potential noise, which results in significant performance degradation. To overcome these challenges, we propose a novel cross-modality guidance-aided multi-modal learning with dual attention for addressing the task of MRI brain tumor grading. To balance the tradeoff between model efficiency and efficacy, we employ ResNet Mix Convolution as the backbone network for feature extraction. Besides, dual attention is applied to capture the semantic interdependencies in spatial and slice dimensions respectively. To facilitate information interaction among modalities, we design a cross-modality guidance-aided module where the primary modality guides the other secondary modalities during the process of training, which can effectively leverage the complementary information of different MRI modalities and meanwhile alleviate the impact of the possible noise.

Title: Data Attribution for Diffusion Models: Timestep-induced Bias in Influence Estimation

• Authors: Tong Xie, Haoyu Li, Andrew Bai, Cho-Jui Hsieh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09031
• Pdf URL: https://arxiv.org/pdf/2401.09031
• Copy Paste: [[2401.09031]] Data Attribution for Diffusion Models: Timestep-induced Bias in Influence Estimation(https://arxiv.org/abs/2401.09031)
• Keywords: diffusion
• Abstract: Data attribution methods trace model behavior back to its training dataset, offering an effective approach to better understand ``black-box'' neural networks. While prior research has established quantifiable links between model output and training data in diverse settings, interpreting diffusion model outputs in relation to training samples remains underexplored. In particular, diffusion models operate over a sequence of timesteps instead of instantaneous input-output relationships in previous contexts, posing a significant challenge to extend existing frameworks to diffusion models directly. Notably, we present Diffusion-TracIn that incorporates this temporal dynamics and observe that samples' loss gradient norms are highly dependent on timestep. This trend leads to a prominent bias in influence estimation, and is particularly noticeable for samples trained on large-norm-inducing timesteps, causing them to be generally influential. To mitigate this effect, we introduce Diffusion-ReTrac as a re-normalized adaptation that enables the retrieval of training samples more targeted to the test sample of interest, facilitating a localized measurement of influence and considerably more intuitive visualization. We demonstrate the efficacy of our approach through various evaluation metrics and auxiliary tasks, reducing the amount of generally influential samples to $\frac{1}{3}$ of its original quantity.

Title: VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models

• Authors: Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, Ying Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09047
• Pdf URL: https://arxiv.org/pdf/2401.09047
• Copy Paste: [[2401.09047]] VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models(https://arxiv.org/abs/2401.09047)
• Keywords: diffusion
• Abstract: Text-to-video generation aims to produce a video based on a given prompt. Recently, several commercial video models have been able to generate plausible videos with minimal noise, excellent details, and high aesthetic scores. However, these models rely on large-scale, well-filtered, high-quality videos that are not accessible to the community. Many existing research works, which train models using the low-quality WebVid-10M dataset, struggle to generate high-quality videos because the models are optimized to fit WebVid-10M. In this work, we explore the training scheme of video models extended from Stable Diffusion and investigate the feasibility of leveraging low-quality videos and synthesized high-quality images to obtain a high-quality video model. We first analyze the connection between the spatial and temporal modules of video models and the distribution shift to low-quality videos. We observe that full training of all modules results in a stronger coupling between spatial and temporal modules than only training temporal modules. Based on this stronger coupling, we shift the distribution to higher quality without motion degradation by finetuning spatial modules with high-quality images, resulting in a generic high-quality video model. Evaluations are conducted to demonstrate the superiority of the proposed method, particularly in picture quality, motion, and concept composition.

Title: Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image Synthesis

• Authors: Jonghyun Lee, Hansam Cho, Youngjoon Yoo, Seoung Bum Kim, Yonghyun Jeong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09048
• Pdf URL: https://arxiv.org/pdf/2401.09048
• Copy Paste: [[2401.09048]] Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image Synthesis(https://arxiv.org/abs/2401.09048)
• Keywords: diffusion
• Abstract: Addressing the limitations of text as a source of accurate layout representation in text-conditional diffusion models, many works incorporate additional signals to condition certain attributes within a generated image. Although successful, previous works do not account for the specific localization of said attributes extended into the three dimensional plane. In this context, we present a conditional diffusion model that integrates control over three-dimensional object placement with disentangled representations of global stylistic semantics from multiple exemplar images. Specifically, we first introduce \textit{depth disentanglement training} to leverage the relative depth of objects as an estimator, allowing the model to identify the absolute positions of unseen objects through the use of synthetic image triplets. We also introduce \textit{soft guidance}, a method for imposing global semantics onto targeted regions without the use of any additional localization cues. Our integrated framework, \textsc{Compose and Conquer (CnC)}, unifies these techniques to localize multiple conditions in a disentangled manner. We demonstrate that our approach allows perception of objects at varying depths while offering a versatile framework for composing localized objects with different global semantics. Code: https://github.com/tomtom1103/compose-and-conquer/

Title: Enhancing Lidar-based Object Detection in Adverse Weather using Offset Sequences in Time

• Authors: Raphael van Kempen, Tim Rehbronn, Abin Jose, Johannes Stegmaier, Bastian Lampe, Timo Woopen, Lutz Eckstein
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2401.09049
• Pdf URL: https://arxiv.org/pdf/2401.09049
• Copy Paste: [[2401.09049]] Enhancing Lidar-based Object Detection in Adverse Weather using Offset Sequences in Time(https://arxiv.org/abs/2401.09049)
• Keywords: robust
• Abstract: Automated vehicles require an accurate perception of their surroundings for safe and efficient driving. Lidar-based object detection is a widely used method for environment perception, but its performance is significantly affected by adverse weather conditions such as rain and fog. In this work, we investigate various strategies for enhancing the robustness of lidar-based object detection by processing sequential data samples generated by lidar sensors. Our approaches leverage temporal information to improve a lidar object detection model, without the need for additional filtering or pre-processing steps. We compare $10$ different neural network architectures that process point cloud sequences including a novel augmentation strategy introducing a temporal offset between frames of a sequence during training and evaluate the effectiveness of all strategies on lidar point clouds under adverse weather conditions through experiments. Our research provides a comprehensive study of effective methods for mitigating the effects of adverse weather on the reliability of lidar-based object detection using sequential data that are evaluated using public datasets such as nuScenes, Dense, and the Canadian Adverse Driving Conditions Dataset. Our findings demonstrate that our novel method, involving temporal offset augmentation through randomized frame skipping in sequences, enhances object detection accuracy compared to both the baseline model (Pillar-based Object Detection) and no augmentation.

Title: Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior

• Authors: Zike Wu, Pan Zhou, Xuanyu Yi, Xiaoding Yuan, Hanwang Zhang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09050
• Pdf URL: https://arxiv.org/pdf/2401.09050
• Copy Paste: [[2401.09050]] Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior(https://arxiv.org/abs/2401.09050)
• Keywords: diffusion
• Abstract: Score distillation sampling (SDS) and its variants have greatly boosted the development of text-to-3D generation, but are vulnerable to geometry collapse and poor textures yet. To solve this issue, we first deeply analyze the SDS and find that its distillation sampling process indeed corresponds to the trajectory sampling of a stochastic differential equation (SDE): SDS samples along an SDE trajectory to yield a less noisy sample which then serves as a guidance to optimize a 3D model. However, the randomness in SDE sampling often leads to a diverse and unpredictable sample which is not always less noisy, and thus is not a consistently correct guidance, explaining the vulnerability of SDS. Since for any SDE, there always exists an ordinary differential equation (ODE) whose trajectory sampling can deterministically and consistently converge to the desired target point as the SDE, we propose a novel and effective "Consistent3D" method that explores the ODE deterministic sampling prior for text-to-3D generation. Specifically, at each training iteration, given a rendered image by a 3D model, we first estimate its desired 3D score function by a pre-trained 2D diffusion model, and build an ODE for trajectory sampling. Next, we design a consistency distillation sampling loss which samples along the ODE trajectory to generate two adjacent samples and uses the less noisy sample to guide another more noisy one for distilling the deterministic prior into the 3D model. Experimental results show the efficacy of our Consistent3D in generating high-fidelity and diverse 3D objects and large-scale scenes, as shown in Fig. 1. The codes are available at https://github.com/sail-sg/Consistent3D.

Title: Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding

• Authors: Depeng Li, Tianqi Wang, Junwei Chen, Qining Ren, Kenji Kawaguchi, Zhigang Zeng
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09067
• Pdf URL: https://arxiv.org/pdf/2401.09067
• Copy Paste: [[2401.09067]] Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding(https://arxiv.org/abs/2401.09067)
• Keywords: privacy
• Abstract: Deep neural networks are susceptible to catastrophic forgetting when trained on sequential tasks. Various continual learning (CL) methods often rely on exemplar buffers or/and network expansion for balancing model stability and plasticity, which, however, compromises their practical value due to privacy and memory concerns. Instead, this paper considers a strict yet realistic setting, where the training data from previous tasks is unavailable and the model size remains relatively constant during sequential training. To achieve such desiderata, we propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. This is achieved by the synergy between two key components: HSIC-Bottleneck Orthogonalization (HBO) implements non-overwritten parameter updates mediated by Hilbert-Schmidt independence criterion in an orthogonal space and EquiAngular Embedding (EAE) enhances decision boundary adaptation between old and new tasks with predefined basis vectors. Extensive experiments demonstrate that our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.

Title: Rethinking Spectral Graph Neural Networks with Spatially Adaptive Filtering

• Authors: Jingwei Guo, Kaizhu Huang, Xinping Yi, Zixian Su, Rui Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09071
• Pdf URL: https://arxiv.org/pdf/2401.09071
• Copy Paste: [[2401.09071]] Rethinking Spectral Graph Neural Networks with Spatially Adaptive Filtering(https://arxiv.org/abs/2401.09071)
• Keywords: interpretability
• Abstract: Whilst spectral Graph Neural Networks (GNNs) are theoretically well-founded in the spectral domain, their practical reliance on polynomial approximation implies a profound linkage to the spatial domain. As previous studies rarely examine spectral GNNs from the spatial perspective, their spatial-domain interpretability remains elusive, e.g., what information is essentially encoded by spectral GNNs in the spatial domain? In this paper, to answer this question, we establish a theoretical connection between spectral filtering and spatial aggregation, unveiling an intrinsic interaction that spectral filtering implicitly leads the original graph to an adapted new graph, explicitly computed for spatial aggregation. Both theoretical and empirical investigations reveal that the adapted new graph not only exhibits non-locality but also accommodates signed edge weights to reflect label consistency between nodes. These findings thus highlight the interpretable role of spectral GNNs in the spatial domain and inspire us to rethink graph spectral filters beyond the fixed-order polynomials, which neglect global information. Built upon the theoretical findings, we revisit the state-of-the-art spectral GNNs and propose a novel Spatially Adaptive Filtering (SAF) framework, which leverages the adapted new graph by spectral filtering for an auxiliary non-local aggregation. Notably, our proposed SAF comprehensively models both node similarity and dissimilarity from a global perspective, therefore alleviating persistent deficiencies of GNNs related to long-range dependencies and graph heterophily. Extensive experiments over 13 node classification benchmarks demonstrate the superiority of our proposed framework to the state-of-the-art models.

Title: Fixed-Budget Differentially Private Best Arm Identification

• Authors: Zhirui Chen, P. N. Karthik, Yeow Meng Chee, Vincent Y. F. Tan
• Subjects: cs.LG, cs.AI, cs.IT, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.09073
• Pdf URL: https://arxiv.org/pdf/2401.09073
• Copy Paste: [[2401.09073]] Fixed-Budget Differentially Private Best Arm Identification(https://arxiv.org/abs/2401.09073)
• Keywords: privacy
• Abstract: We study best arm identification (BAI) in linear bandits in the fixed-budget regime under differential privacy constraints, when the arm rewards are supported on the unit interval. Given a finite budget $T$ and a privacy parameter $\varepsilon>0$, the goal is to minimise the error probability in finding the arm with the largest mean after $T$ sampling rounds, subject to the constraint that the policy of the decision maker satisfies a certain {\em $\varepsilon$-differential privacy} ($\varepsilon$-DP) constraint. We construct a policy satisfying the $\varepsilon$-DP constraint (called {\sc DP-BAI}) by proposing the principle of {\em maximum absolute determinants}, and derive an upper bound on its error probability. Furthermore, we derive a minimax lower bound on the error probability, and demonstrate that the lower and the upper bounds decay exponentially in $T$, with exponents in the two bounds matching order-wise in (a) the sub-optimality gaps of the arms, (b) $\varepsilon$, and (c) the problem complexity that is expressible as the sum of two terms, one characterising the complexity of standard fixed-budget BAI (without privacy constraints), and the other accounting for the $\varepsilon$-DP constraint. Additionally, we present some auxiliary results that contribute to the derivation of the lower bound on the error probability. These results, we posit, may be of independent interest and could prove instrumental in proving lower bounds on error probabilities in several other bandit problems. Whereas prior works provide results for BAI in the fixed-budget regime without privacy constraints or in the fixed-confidence regime with privacy constraints, our work fills the gap in the literature by providing the results for BAI in the fixed-budget regime under the $\varepsilon$-DP constraint.

Title: Code Simulation Challenges for Large Language Models

• Authors: Emanuele La Malfa, Christoph Weinhuber, Orazio Torre, Fangru Lin, Anthony Cohn, Nigel Shadbolt, Michael Wooldridge
• Subjects: cs.LG, cs.AI, cs.CL, cs.PL
• Abstract URL: https://arxiv.org/abs/2401.09074
• Pdf URL: https://arxiv.org/pdf/2401.09074
• Copy Paste: [[2401.09074]] Code Simulation Challenges for Large Language Models(https://arxiv.org/abs/2401.09074)
• Keywords: large language model
• Abstract: We investigate the extent to which Large Language Models (LLMs) can simulate the execution of computer code and algorithms. We begin by looking straight line programs, and show that current LLMs demonstrate poor performance even with such simple programs -- performance rapidly degrades with the length of code. We then investigate the ability of LLMs to simulate programs that contain critical paths and redundant instructions. We also go beyond straight line program simulation with sorting algorithms and nested loops, and we show the computational complexity of a routine directly affects the ability of an LLM to simulate its execution. We observe that LLMs execute instructions sequentially and with a low error margin only for short programs or standard procedures. LLMs' code simulation is in tension with their pattern recognition and memorisation capabilities: on tasks where memorisation is detrimental, we propose a novel prompting method to simulate code execution line by line. Empirically, our new Chain of Simulation (CoSm) method improves on the standard Chain of Thought prompting approach by avoiding the pitfalls of memorisation.

Title: GPT in Sheep's Clothing: The Risk of Customized GPTs

• Authors: Sagiv Antebi, Noam Azulay, Edan Habler, Ben Ganon, Asaf Shabtai, Yuval Elovici
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09075
• Pdf URL: https://arxiv.org/pdf/2401.09075
• Copy Paste: [[2401.09075]] GPT in Sheep's Clothing: The Risk of Customized GPTs(https://arxiv.org/abs/2401.09075)
• Keywords: security, privacy
• Abstract: In November 2023, OpenAI introduced a new service allowing users to create custom versions of ChatGPT (GPTs) by using specific instructions and knowledge to guide the model's behavior. We aim to raise awareness of the fact that GPTs can be used maliciously, posing privacy and security risks to their users.

Title: What makes for a 'good' social actor? Using respect as a lens to evaluate interactions with language agents

• Authors: Lize Alberts, Geoff Keeling, Amanda McCroskery
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.09082
• Pdf URL: https://arxiv.org/pdf/2401.09082
• Copy Paste: [[2401.09082]] What makes for a 'good' social actor? Using respect as a lens to evaluate interactions with language agents(https://arxiv.org/abs/2401.09082)
• Keywords: large language model
• Abstract: With the growing popularity of dialogue agents based on large language models (LLMs), urgent attention has been drawn to finding ways to ensure their behaviour is ethical and appropriate. These are largely interpreted in terms of the 'HHH' criteria: making outputs more helpful and honest, and avoiding harmful (biased, toxic, or inaccurate) statements. Whilst this semantic focus is useful from the perspective of viewing LLM agents as mere mediums for information, it fails to account for pragmatic factors that can make the same utterance seem more or less offensive or tactless in different social situations. We propose an approach to ethics that is more centred on relational and situational factors, exploring what it means for a system, as a social actor, to treat an individual respectfully in a (series of) interaction(s). Our work anticipates a set of largely unexplored risks at the level of situated interaction, and offers practical suggestions to help LLM technologies behave as 'good' social actors and treat people respectfully.

Title: Remote Sensing ChatGPT: Solving Remote Sensing Tasks with ChatGPT and Visual Models

• Authors: Haonan Guo, Xin Su, Chen Wu, Bo Du, Liangpei Zhang, Deren Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09083
• Pdf URL: https://arxiv.org/pdf/2401.09083
• Copy Paste: [[2401.09083]] Remote Sensing ChatGPT: Solving Remote Sensing Tasks with ChatGPT and Visual Models(https://arxiv.org/abs/2401.09083)
• Keywords: large language model
• Abstract: Recently, the flourishing large language models(LLM), especially ChatGPT, have shown exceptional performance in language understanding, reasoning, and interaction, attracting users and researchers from multiple fields and domains. Although LLMs have shown great capacity to perform human-like task accomplishment in natural language and natural image, their potential in handling remote sensing interpretation tasks has not yet been fully explored. Moreover, the lack of automation in remote sensing task planning hinders the accessibility of remote sensing interpretation techniques, especially to non-remote sensing experts from multiple research fields. To this end, we present Remote Sensing ChatGPT, an LLM-powered agent that utilizes ChatGPT to connect various AI-based remote sensing models to solve complicated interpretation tasks. More specifically, given a user request and a remote sensing image, we utilized ChatGPT to understand user requests, perform task planning according to the tasks' functions, execute each subtask iteratively, and generate the final response according to the output of each subtask. Considering that LLM is trained with natural language and is not capable of directly perceiving visual concepts as contained in remote sensing images, we designed visual cues that inject visual information into ChatGPT. With Remote Sensing ChatGPT, users can simply send a remote sensing image with the corresponding request, and get the interpretation results as well as language feedback from Remote Sensing ChatGPT. Experiments and examples show that Remote Sensing ChatGPT can tackle a wide range of remote sensing tasks and can be extended to more tasks with more sophisticated models such as the remote sensing foundation model. The code and demo of Remote Sensing ChatGPT is publicly available at https://github.com/HaonanGuo/Remote-Sensing-ChatGPT .

Title: UniVG: Towards UNIfied-modal Video Generation

• Authors: Ludan Ruan, Lei Tian, Chuanwei Huang, Xu Zhang, Xinyan Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09084
• Pdf URL: https://arxiv.org/pdf/2401.09084
• Copy Paste: [[2401.09084]] UniVG: Towards UNIfied-modal Video Generation(https://arxiv.org/abs/2401.09084)
• Keywords: diffusion, generative
• Abstract: Diffusion based video generation has received extensive attention and achieved considerable success within both the academic and industrial communities. However, current efforts are mainly concentrated on single-objective or single-task video generation, such as generation driven by text, by image, or by a combination of text and image. This cannot fully meet the needs of real-world application scenarios, as users are likely to input images and text conditions in a flexible manner, either individually or in combination. To address this, we propose a Unified-modal Video Genearation system that is capable of handling multiple video generation tasks across text and image modalities. To this end, we revisit the various video generation tasks within our system from the perspective of generative freedom, and classify them into high-freedom and low-freedom video generation categories. For high-freedom video generation, we employ Multi-condition Cross Attention to generate videos that align with the semantics of the input images or text. For low-freedom video generation, we introduce Biased Gaussian Noise to replace the pure random Gaussian Noise, which helps to better preserve the content of the input conditions. Our method achieves the lowest Fr\'echet Video Distance (FVD) on the public academic benchmark MSR-VTT, surpasses the current open-source methods in human evaluations, and is on par with the current close-source method Gen2. For more samples, visit https://univg-baidu.github.io.

Title: RWKV-TS: Beyond Traditional Recurrent Neural Network for Time Series Tasks

• Authors: Haowen Hou, F. Richard Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09093
• Pdf URL: https://arxiv.org/pdf/2401.09093
• Copy Paste: [[2401.09093]] RWKV-TS: Beyond Traditional Recurrent Neural Network for Time Series Tasks(https://arxiv.org/abs/2401.09093)
• Keywords: transformer
• Abstract: Traditional Recurrent Neural Network (RNN) architectures, such as LSTM and GRU, have historically held prominence in time series tasks. However, they have recently seen a decline in their dominant position across various time series tasks. As a result, recent advancements in time series forecasting have seen a notable shift away from RNNs towards alternative architectures such as Transformers, MLPs, and CNNs. To go beyond the limitations of traditional RNNs, we design an efficient RNN-based model for time series tasks, named RWKV-TS, with three distinctive features: (i) A novel RNN architecture characterized by $O(L)$ time complexity and memory usage. (ii) An enhanced ability to capture long-term sequence information compared to traditional RNNs. (iii) High computational efficiency coupled with the capacity to scale up effectively. Through extensive experimentation, our proposed RWKV-TS model demonstrates competitive performance when compared to state-of-the-art Transformer-based or CNN-based models. Notably, RWKV-TS exhibits not only comparable performance but also demonstrates reduced latency and memory utilization. The success of RWKV-TS encourages further exploration and innovation in leveraging RNN-based approaches within the domain of Time Series. The combination of competitive performance, low latency, and efficient memory usage positions RWKV-TS as a promising avenue for future research in time series tasks. Code is available at:\href{https://github.com/howard-hou/RWKV-TS}{ https://github.com/howard-hou/RWKV-TS}

Title: Trapped in texture bias? A large scale comparison of deep instance segmentation

• Authors: Johannes Theodoridis, Jessica Hofmann, Johannes Maucher, Andreas Schilling
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09109
• Pdf URL: https://arxiv.org/pdf/2401.09109
• Copy Paste: [[2401.09109]] Trapped in texture bias? A large scale comparison of deep instance segmentation(https://arxiv.org/abs/2401.09109)
• Keywords: robust, transformer, segmentation
• Abstract: Do deep learning models for instance segmentation generalize to novel objects in a systematic way? For classification, such behavior has been questioned. In this study, we aim to understand if certain design decisions such as framework, architecture or pre-training contribute to the semantic understanding of instance segmentation. To answer this question, we consider a special case of robustness and compare pre-trained models on a challenging benchmark for object-centric, out-of-distribution texture. We do not introduce another method in this work. Instead, we take a step back and evaluate a broad range of existing literature. This includes Cascade and Mask R-CNN, Swin Transformer, BMask, YOLACT(++), DETR, BCNet, SOTR and SOLOv2. We find that YOLACT++, SOTR and SOLOv2 are significantly more robust to out-of-distribution texture than other frameworks. In addition, we show that deeper and dynamic architectures improve robustness whereas training schedules, data augmentation and pre-training have only a minor impact. In summary we evaluate 68 models on 61 versions of MS COCO for a total of 4148 evaluations.

Title: Machine Learning for Healthcare-IoT Security: A Review and Risk Mitigation

• Authors: Mirza Akhi Khatun, Sanober Farheen Memon, Ciarán Eising, Lubna Luxmi Dhirani
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.09124
• Pdf URL: https://arxiv.org/pdf/2401.09124
• Copy Paste: [[2401.09124]] Machine Learning for Healthcare-IoT Security: A Review and Risk Mitigation(https://arxiv.org/abs/2401.09124)
• Keywords: security, privacy, protect, attack, robust, generative
• Abstract: The Healthcare Internet-of-Things (H-IoT), commonly known as Digital Healthcare, is a data-driven infrastructure that highly relies on smart sensing devices (i.e., blood pressure monitors, temperature sensors, etc.) for faster response time, treatments, and diagnosis. However, with the evolving cyber threat landscape, IoT devices have become more vulnerable to the broader risk surface (e.g., risks associated with generative AI, 5G-IoT, etc.), which, if exploited, may lead to data breaches, unauthorized access, and lack of command and control and potential harm. This paper reviews the fundamentals of healthcare IoT, its privacy, and data security challenges associated with machine learning and H-IoT devices. The paper further emphasizes the importance of monitoring healthcare IoT layers such as perception, network, cloud, and application. Detecting and responding to anomalies involves various cyber-attacks and protocols such as Wi-Fi 6, Narrowband Internet of Things (NB-IoT), Bluetooth, ZigBee, LoRa, and 5G New Radio (5G NR). A robust authentication mechanism based on machine learning and deep learning techniques is required to protect and mitigate H-IoT devices from increasing cybersecurity vulnerabilities. Hence, in this review paper, security and privacy challenges and risk mitigation strategies for building resilience in H-IoT are explored and reported.

Title: Asynchronous Local-SGD Training for Language Modeling

• Authors: Bo Liu, Rachita Chhaparia, Arthur Douillard, Satyen Kale, Andrei A. Rusu, Jiajun Shen, Arthur Szlam, Marc'Aurelio Ranzato
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09135
• Pdf URL: https://arxiv.org/pdf/2401.09135
• Copy Paste: [[2401.09135]] Asynchronous Local-SGD Training for Language Modeling(https://arxiv.org/abs/2401.09135)
• Keywords: federate
• Abstract: Local stochastic gradient descent (Local-SGD), also referred to as federated averaging, is an approach to distributed optimization where each device performs more than one SGD update per communication. This work presents an empirical study of {\it asynchronous} Local-SGD for training language models; that is, each worker updates the global parameters as soon as it has finished its SGD steps. We conduct a comprehensive investigation by examining how worker hardware heterogeneity, model size, number of workers, and optimizer could impact the learning performance. We find that with naive implementations, asynchronous Local-SGD takes more iterations to converge than its synchronous counterpart despite updating the (global) model parameters more frequently. We identify momentum acceleration on the global parameters when worker gradients are stale as a key challenge. We propose a novel method that utilizes a delayed Nesterov momentum update and adjusts the workers' local training steps based on their computation speed. This approach, evaluated with models up to 150M parameters on the C4 dataset, matches the performance of synchronous Local-SGD in terms of perplexity per update step, and significantly surpasses it in terms of wall clock time.

Title: Continuous Piecewise-Affine Based Motion Model for Image Animation

• Authors: Hexiang Wang, Fengqi Liu, Qianyu Zhou, Ran Yi, Xin Tan, Lizhuang Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09146
• Pdf URL: https://arxiv.org/pdf/2401.09146
• Copy Paste: [[2401.09146]] Continuous Piecewise-Affine Based Motion Model for Image Animation(https://arxiv.org/abs/2401.09146)
• Keywords: extraction
• Abstract: Image animation aims to bring static images to life according to driving videos and create engaging visual content that can be used for various purposes such as animation, entertainment, and education. Recent unsupervised methods utilize affine and thin-plate spline transformations based on keypoints to transfer the motion in driving frames to the source image. However, limited by the expressive power of the transformations used, these methods always produce poor results when the gap between the motion in the driving frame and the source image is large. To address this issue, we propose to model motion from the source image to the driving frame in highly-expressive diffeomorphism spaces. Firstly, we introduce Continuous Piecewise-Affine based (CPAB) transformation to model the motion and present a well-designed inference algorithm to generate CPAB transformation from control keypoints. Secondly, we propose a SAM-guided keypoint semantic loss to further constrain the keypoint extraction process and improve the semantic consistency between the corresponding keypoints on the source and driving images. Finally, we design a structure alignment loss to align the structure-related features extracted from driving and generated images, thus helping the generator generate results that are more consistent with the driving action. Extensive experiments on four datasets demonstrate the effectiveness of our method against state-of-the-art competitors quantitatively and qualitatively. Code will be publicly available at: https://github.com/DevilPG/AAAI2024-CPABMM.

Title: ADCNet: a unified framework for predicting the activity of antibody-drug conjugates

• Authors: Liye Chen, Biaoshun Li, Yihao Chen, Mujie Lin, Shipeng Zhang, Chenxin Li, Yu Pang, Ling Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09176
• Pdf URL: https://arxiv.org/pdf/2401.09176
• Copy Paste: [[2401.09176]] ADCNet: a unified framework for predicting the activity of antibody-drug conjugates(https://arxiv.org/abs/2401.09176)
• Keywords: robust
• Abstract: Antibody-drug conjugate (ADC) has revolutionized the field of cancer treatment in the era of precision medicine due to their ability to precisely target cancer cells and release highly effective drug. Nevertheless, the realization of rational design of ADC is very difficult because the relationship between their structures and activities is difficult to understand. In the present study, we introduce a unified deep learning framework called ADCNet to help design potential ADCs. The ADCNet highly integrates the protein representation learning language model ESM-2 and small-molecule representation learning language model FG-BERT models to achieve activity prediction through learning meaningful features from antigen and antibody protein sequences of ADC, SMILES strings of linker and payload, and drug-antibody ratio (DAR) value. Based on a carefully designed and manually tailored ADC data set, extensive evaluation results reveal that ADCNet performs best on the test set compared to baseline machine learning models across all evaluation metrics. For example, it achieves an average prediction accuracy of 87.12%, a balanced accuracy of 0.8689, and an area under receiver operating characteristic curve of 0.9293 on the test set. In addition, cross-validation, ablation experiments, and external independent testing results further prove the stability, advancement, and robustness of the ADCNet architecture. For the convenience of the community, we develop the first online platform (https://ADCNet.idruglab.cn) for the prediction of ADCs activity based on the optimal ADCNet model, and the source code is publicly available at https://github.com/idrugLab/ADCNet.

Title: Unsupervised Multiple Domain Translation through Controlled Disentanglement in Variational Autoencoder

• Authors: Almudévar Antonio, Mariotte Théo, Ortega Alfonso, Tahon Marie
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09180
• Pdf URL: https://arxiv.org/pdf/2401.09180
• Copy Paste: [[2401.09180]] Unsupervised Multiple Domain Translation through Controlled Disentanglement in Variational Autoencoder(https://arxiv.org/abs/2401.09180)
• Keywords: generative
• Abstract: Unsupervised Multiple Domain Translation is the task of transforming data from one domain to other domains without having paired data to train the systems. Typically, methods based on Generative Adversarial Networks (GANs) are used to address this task. However, our proposal exclusively relies on a modified version of a Variational Autoencoder. This modification consists of the use of two latent variables disentangled in a controlled way by design. One of this latent variables is imposed to depend exclusively on the domain, while the other one must depend on the rest of the variability factors of the data. Additionally, the conditions imposed over the domain latent variable allow for better control and understanding of the latent space. We empirically demonstrate that our approach works on different vision datasets improving the performance of other well known methods. Finally, we prove that, indeed, one of the latent variables stores all the information related to the domain and the other one hardly contains any domain information.

Title: Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer

• Authors: Junhao Zheng, Qianli Ma, Zhen Liu, Binquan Wu, Huawen Feng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09181
• Pdf URL: https://arxiv.org/pdf/2401.09181
• Copy Paste: [[2401.09181]] Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer(https://arxiv.org/abs/2401.09181)
• Keywords: large language model
• Abstract: Multimodal Continual Instruction Tuning (MCIT) enables Multimodal Large Language Models (MLLMs) to meet continuously emerging requirements without expensive retraining. MCIT faces two major obstacles: catastrophic forgetting (where old knowledge is forgotten) and negative forward transfer (where the performance of future tasks is degraded). Although existing methods have greatly alleviated catastrophic forgetting, they still suffer from negative forward transfer. By performing singular value decomposition (SVD) on input embeddings, we discover a large discrepancy in different input embeddings. The discrepancy results in the model learning irrelevant information for old and pre-trained tasks, which leads to catastrophic forgetting and negative forward transfer. To address these issues, we propose Fwd-Prompt, a prompt-based method projecting prompt gradient to the residual space to minimize the interference between tasks and to the pre-trained subspace for reusing pre-trained knowledge. Our experiments demonstrate that Fwd-Prompt achieves state-of-the-art performance while updating fewer parameters and requiring no old samples. Our research sheds light on the potential of continuously adapting MLLMs to new tasks under the instruction tuning paradigm and encourages future studies to explore MCIT. The code will soon be publicly available.

Title: Exploring the Role of Convolutional Neural Networks (CNN) in Dental Radiography Segmentation: A Comprehensive Systematic Literature Review

• Authors: Walid Brahmi, Imen Jdey, Fadoua Drira
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09190
• Pdf URL: https://arxiv.org/pdf/2401.09190
• Copy Paste: [[2401.09190]] Exploring the Role of Convolutional Neural Networks (CNN) in Dental Radiography Segmentation: A Comprehensive Systematic Literature Review(https://arxiv.org/abs/2401.09190)
• Keywords: segmentation
• Abstract: In the field of dentistry, there is a growing demand for increased precision in diagnostic tools, with a specific focus on advanced imaging techniques such as computed tomography, cone beam computed tomography, magnetic resonance imaging, ultrasound, and traditional intra-oral periapical X-rays. Deep learning has emerged as a pivotal tool in this context, enabling the implementation of automated segmentation techniques crucial for extracting essential diagnostic data. This integration of cutting-edge technology addresses the urgent need for effective management of dental conditions, which, if left undetected, can have a significant impact on human health. The impressive track record of deep learning across various domains, including dentistry, underscores its potential to revolutionize early detection and treatment of oral health issues. Objective: Having demonstrated significant results in diagnosis and prediction, deep convolutional neural networks (CNNs) represent an emerging field of multidisciplinary research. The goals of this study were to provide a concise overview of the state of the art, standardize the current debate, and establish baselines for future research. Method: In this study, a systematic literature review is employed as a methodology to identify and select relevant studies that specifically investigate the deep learning technique for dental imaging analysis. This study elucidates the methodological approach, including the systematic collection of data, statistical analysis, and subsequent dissemination of outcomes. Conclusion: This work demonstrates how Convolutional Neural Networks (CNNs) can be employed to analyze images, serving as effective tools for detecting dental pathologies. Although this research acknowledged some limitations, CNNs utilized for segmenting and categorizing teeth exhibited their highest level of performance overall.

Title: An Optimal Transport Approach for Computing Adversarial Training Lower Bounds in Multiclass Classification

• Authors: Nicolas Garcia Trillos, Matt Jacobs, Jakwang Kim, Matthew Werenski
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.09191
• Pdf URL: https://arxiv.org/pdf/2401.09191
• Copy Paste: [[2401.09191]] An Optimal Transport Approach for Computing Adversarial Training Lower Bounds in Multiclass Classification(https://arxiv.org/abs/2401.09191)
• Keywords: robust
• Abstract: Despite the success of deep learning-based algorithms, it is widely known that neural networks may fail to be robust. A popular paradigm to enforce robustness is adversarial training (AT), however, this introduces many computational and theoretical difficulties. Recent works have developed a connection between AT in the multiclass classification setting and multimarginal optimal transport (MOT), unlocking a new set of tools to study this problem. In this paper, we leverage the MOT connection to propose computationally tractable numerical algorithms for computing universal lower bounds on the optimal adversarial risk and identifying optimal classifiers. We propose two main algorithms based on linear programming (LP) and entropic regularization (Sinkhorn). Our key insight is that one can harmlessly truncate the higher order interactions between classes, preventing the combinatorial run times typically encountered in MOT problems. We validate these results with experiments on MNIST and CIFAR-$10$, which demonstrate the tractability of our approach.

Title: Preparing Lessons for Progressive Training on Language Models

• Authors: Yu Pan, Ye Yuan, Yichun Yin, Jiaxin Shi, Zenglin Xu, Ming Zhang, Lifeng Shang, Xin Jiang, Qun Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09192
• Pdf URL: https://arxiv.org/pdf/2401.09192
• Copy Paste: [[2401.09192]] Preparing Lessons for Progressive Training on Language Models(https://arxiv.org/abs/2401.09192)
• Keywords: transformer
• Abstract: The rapid progress of Transformers in artificial intelligence has come at the cost of increased resource consumption and greenhouse gas emissions due to growing model sizes. Prior work suggests using pretrained small models to improve training efficiency, but this approach may not be suitable for new model structures. On the other hand, training from scratch can be slow, and progressively stacking layers often fails to achieve significant acceleration. To address these challenges, we propose a novel method called Apollo, which prep\textbf{a}res lessons for ex\textbf{p}anding \textbf{o}perations by \textbf{l}earning high-\textbf{l}ayer functi\textbf{o}nality during training of low layers. Our approach involves low-value-prioritized sampling (LVPS) to train different depths and weight sharing to facilitate efficient expansion. We also introduce an interpolation method for stable model depth extension. Experiments demonstrate that Apollo achieves state-of-the-art acceleration ratios, even rivaling methods using pretrained models, making it a universal and efficient solution for training deep models while reducing time, financial, and environmental costs.

Title: Training-Free Semantic Video Composition via Pre-trained Diffusion Model

• Authors: Jiaqi Guo, Sitong Su, Junchen Zhu, Lianli Gao, Jingkuan Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09195
• Pdf URL: https://arxiv.org/pdf/2401.09195
• Copy Paste: [[2401.09195]] Training-Free Semantic Video Composition via Pre-trained Diffusion Model(https://arxiv.org/abs/2401.09195)
• Keywords: diffusion
• Abstract: The video composition task aims to integrate specified foregrounds and backgrounds from different videos into a harmonious composite. Current approaches, predominantly trained on videos with adjusted foreground color and lighting, struggle to address deep semantic disparities beyond superficial adjustments, such as domain gaps. Therefore, we propose a training-free pipeline employing a pre-trained diffusion model imbued with semantic prior knowledge, which can process composite videos with broader semantic disparities. Specifically, we process the video frames in a cascading manner and handle each frame in two processes with the diffusion model. In the inversion process, we propose Balanced Partial Inversion to obtain generation initial points that balance reversibility and modifiability. Then, in the generation process, we further propose Inter-Frame Augmented attention to augment foreground continuity across frames. Experimental results reveal that our pipeline successfully ensures the visual harmony and inter-frame coherence of the outputs, demonstrating efficacy in managing broader semantic disparities.

Title: Cross-Domain AI for Early Attack Detection and Defense Against Malicious Flows in O-RAN

• Authors: Bruno Missi Xavier, Merim Dzaferagic, Irene Vilà, Magnos Martinello, Marco Ruffini
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2401.09204
• Pdf URL: https://arxiv.org/pdf/2401.09204
• Copy Paste: [[2401.09204]] Cross-Domain AI for Early Attack Detection and Defense Against Malicious Flows in O-RAN(https://arxiv.org/abs/2401.09204)
• Keywords: security, defense, attack
• Abstract: Only the chairs can edit In the fight against cyber attacks, Network Softwarization (NS) is a flexible and adaptable shield, using advanced software to spot malicious activity in regular network traffic. However, the availability of comprehensive datasets for mobile networks, which are fundamental for the development of Machine Learning (ML) solutions for attack detection near their source, is still limited. Cross-Domain Artificial Intelligence (AI) can be the key to address this, although its application in Open Radio Access Network (O-RAN) is still at its infancy. To address these challenges, we deployed an end-to-end O-RAN network, that was used to collect data from the RAN and the transport network. These datasets allow us to combine the knowledge from an in-network ML traffic classifier for attack detection to bolster the training of an ML-based traffic classifier specifically tailored for the RAN. Our results demonstrate the potential of the proposed approach, achieving an accuracy rate of 93%. This approach not only bridges critical gaps in mobile network security but also showcases the potential of cross-domain AI in enhancing the efficacy of network security measures.

Title: Username Squatting on Online Social Networks: A Study on X

• Authors: Anastasios Lepipas, Anastasia Borovykh, Soteris Demetriou
• Subjects: cs.CR, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.09209
• Pdf URL: https://arxiv.org/pdf/2401.09209
• Copy Paste: [[2401.09209]] Username Squatting on Online Social Networks: A Study on X(https://arxiv.org/abs/2401.09209)
• Keywords: attack
• Abstract: Adversaries have been targeting unique identifiers to launch typo-squatting, mobile app squatting and even voice squatting attacks. Anecdotal evidence suggest that online social networks (OSNs) are also plagued with accounts that use similar usernames. This can be confusing to users but can also be exploited by adversaries. However, to date no study characterizes this problem on OSNs. In this work, we define the username squatting problem and design the first multi-faceted measurement study to characterize it on X. We develop a username generation tool (UsernameCrazy) to help us analyze hundreds of thousands of username variants derived from celebrity accounts. Our study reveals that thousands of squatted usernames have been suspended by X, while tens of thousands that still exist on the network are likely bots. Out of these, a large number share similar profile pictures and profile names to the original account signalling impersonation attempts. We found that squatted accounts are being mentioned by mistake in tweets hundreds of thousands of times and are even being prioritized in searches by the network's search recommendation algorithm exacerbating the negative impact squatted accounts can have in OSNs. We use our insights and take the first step to address this issue by designing a framework (SQUAD) that combines UsernameCrazy with a new classifier to efficiently detect suspicious squatted accounts. Our evaluation of SQUAD's prototype implementation shows that it can achieve 94% F1-score when trained on a small dataset.

Title: UniVIE: A Unified Label Space Approach to Visual Information Extraction from Form-like Documents

• Authors: Kai Hu, Jiawei Wang, Weihong Lin, Zhuoyao Zhong, Lei Sun, Qiang Huo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09220
• Pdf URL: https://arxiv.org/pdf/2401.09220
• Copy Paste: [[2401.09220]] UniVIE: A Unified Label Space Approach to Visual Information Extraction from Form-like Documents(https://arxiv.org/abs/2401.09220)
• Keywords: extraction
• Abstract: Existing methods for Visual Information Extraction (VIE) from form-like documents typically fragment the process into separate subtasks, such as key information extraction, key-value pair extraction, and choice group extraction. However, these approaches often overlook the hierarchical structure of form documents, including hierarchical key-value pairs and hierarchical choice groups. To address these limitations, we present a new perspective, reframing VIE as a relation prediction problem and unifying labels of different tasks into a single label space. This unified approach allows for the definition of various relation types and effectively tackles hierarchical relationships in form-like documents. In line with this perspective, we present UniVIE, a unified model that addresses the VIE problem comprehensively. UniVIE functions using a coarse-to-fine strategy. It initially generates tree proposals through a tree proposal network, which are subsequently refined into hierarchical trees by a relation decoder module. To enhance the relation prediction capabilities of UniVIE, we incorporate two novel tree constraints into the relation decoder: a tree attention mask and a tree level embedding. Extensive experimental evaluations on both our in-house dataset HierForms and a publicly available dataset SIBR, substantiate that our method achieves state-of-the-art results, underscoring the effectiveness and potential of our unified approach in advancing the field of VIE.

Title: Multiple Subset Problem as an encryption scheme for communication

• Authors: Yair Zadok, Nadav Voloch, Noa Voloch-Bloch, Maor Meir Hajaj
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.09221
• Pdf URL: https://arxiv.org/pdf/2401.09221
• Copy Paste: [[2401.09221]] Multiple Subset Problem as an encryption scheme for communication(https://arxiv.org/abs/2401.09221)
• Keywords: security, attack
• Abstract: Using well-known mathematical problems for encryption is a widely used technique because they are computationally hard and provide security against potential attacks on the encryption method. The subset sum problem (SSP) can be defined as finding a subset of integers from a given set, whose sum is equal to a specified integer. The classic SSP has various variants, one of which is the multiple-subset problem (MSSP). In the MSSP, the goal is to select items from a given set and distribute them among multiple bins, en-suring that the capacity of each bin is not exceeded while maximizing the total weight of the selected items. This approach addresses a related problem with a different perspective. Here a related different kind of problem is approached: given a set of sets A={A1, A2..., An}, find an integer s for which every subset of the given sets is summed up to, if such an integer exists. The problem is NP-complete when considering it as a variant of SSP. However, there exists an algorithm that is relatively efficient for known pri-vate keys. This algorithm is based on dispensing non-relevant values of the potential sums. In this paper we present the encryption scheme based on MSSP and present its novel usage and implementation in communication.

Title: Dynamic Relation Transformer for Contextual Text Block Detection

• Authors: Jiawei Wang, Shunchi Zhang, Kai Hu, Chixiang Ma, Zhuoyao Zhong, Lei Sun, Qiang Huo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09232
• Pdf URL: https://arxiv.org/pdf/2401.09232
• Copy Paste: [[2401.09232]] Dynamic Relation Transformer for Contextual Text Block Detection(https://arxiv.org/abs/2401.09232)
• Keywords: extraction, transformer
• Abstract: Contextual Text Block Detection (CTBD) is the task of identifying coherent text blocks within the complexity of natural scenes. Previous methodologies have treated CTBD as either a visual relation extraction challenge within computer vision or as a sequence modeling problem from the perspective of natural language processing. We introduce a new framework that frames CTBD as a graph generation problem. This methodology consists of two essential procedures: identifying individual text units as graph nodes and discerning the sequential reading order relationships among these units as graph edges. Leveraging the cutting-edge capabilities of DQ-DETR for node detection, our framework innovates further by integrating a novel mechanism, a Dynamic Relation Transformer (DRFormer), dedicated to edge generation. DRFormer incorporates a dual interactive transformer decoder that deftly manages a dynamic graph structure refinement process. Through this iterative process, the model systematically enhances the graph's fidelity, ultimately resulting in improved precision in detecting contextual text blocks. Comprehensive experimental evaluations conducted on both SCUT-CTW-Context and ReCTS-Context datasets substantiate that our method achieves state-of-the-art results, underscoring the effectiveness and potential of our graph generation framework in advancing the field of CTBD.

Title: DaFoEs: Mixing Datasets towards the generalization of vision-state deep-learning Force Estimation in Minimally Invasive Robotic Surgery

• Authors: Mikel De Iturrate Reyzabal, Mingcong Chen, Wei Huang, Sebastien Ourselin, Hongbin Liu
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2401.09239
• Pdf URL: https://arxiv.org/pdf/2401.09239
• Copy Paste: [[2401.09239]] DaFoEs: Mixing Datasets towards the generalization of vision-state deep-learning Force Estimation in Minimally Invasive Robotic Surgery(https://arxiv.org/abs/2401.09239)
• Keywords: transformer
• Abstract: Precisely determining the contact force during safe interaction in Minimally Invasive Robotic Surgery (MIRS) is still an open research challenge. Inspired by post-operative qualitative analysis from surgical videos, the use of cross-modality data driven deep neural network models has been one of the newest approaches to predict sensorless force trends. However, these methods required for large and variable datasets which are not currently available. In this paper, we present a new vision-haptic dataset (DaFoEs) with variable soft environments for the training of deep neural models. In order to reduce the bias from a single dataset, we present a pipeline to generalize different vision and state data inputs for mixed dataset training, using a previously validated dataset with different setup. Finally, we present a variable encoder-decoder architecture to predict the forces done by the laparoscopic tool using single input or sequence of inputs. For input sequence, we use a recurrent decoder, named with the prefix R, and a new temporal sampling to represent the acceleration of the tool. During our training, we demonstrate that single dataset training tends to overfit to the training data domain, but has difficulties on translating the results across new domains. However, dataset mixing presents a good translation with a mean relative estimated force error of 5% and 12% for the recurrent and non-recurrent models respectively. Our method, also marginally increase the effectiveness of transformers for force estimation up to a maximum of ~15%, as the volume of available data is increase by 150%. In conclusion, we demonstrate that mixing experimental set ups for vision-state force estimation in MIRS is a possible approach towards the general solution of the problem.

Title: A Blockchain-based Model for Securing Data Pipeline in a Heterogeneous Information System

• Authors: MN Ramahlosi, Y Madani, A Akanbi
• Subjects: cs.CR, cs.AI, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2401.09240
• Pdf URL: https://arxiv.org/pdf/2401.09240
• Copy Paste: [[2401.09240]] A Blockchain-based Model for Securing Data Pipeline in a Heterogeneous Information System(https://arxiv.org/abs/2401.09240)
• Keywords: secure, security, privacy
• Abstract: In our digital world, access to personal and public data has become an item of concern, with challenging security and privacy aspects. Modern information systems are heterogeneous in nature and have an inherent security vulnerability, which is susceptible to data interception and data modification due to unsecured communication data pipelines between connected endpoints. This re-search article presents a blockchain-based model for securing data pipelines in a heterogeneous information system using an integrated multi-hazard early warning system (MHEWS) as a case study. The proposed model utilizes the inherent security features of blockchain technology to address the security and privacy concerns that arise in data pipelines. The model is designed to ensure data integrity, confidentiality, and authenticity in a decentralized manner. The model is evaluated in a hybrid environment using a prototype implementation and simulation experiments with outcomes that demonstrate advantages over traditional approaches for a tamper-proof and immutable data pipeline for data authenticity and integrity using a confidential ledger.

Title: Uncertainty estimates for semantic segmentation: providing enhanced reliability for automated motor claims handling

• Authors: Jan Küchler (1), Daniel Kröll (1), Sebastian Schoenen (1), Andreas Witte (1) ((1) ControlExpert GmbH, Langenfeld, Germany)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09245
• Pdf URL: https://arxiv.org/pdf/2401.09245
• Copy Paste: [[2401.09245]] Uncertainty estimates for semantic segmentation: providing enhanced reliability for automated motor claims handling(https://arxiv.org/abs/2401.09245)
• Keywords: segmentation
• Abstract: Deep neural network models for image segmentation can be a powerful tool for the automation of motor claims handling processes in the insurance industry. A crucial aspect is the reliability of the model outputs when facing adverse conditions, such as low quality photos taken by claimants to document damages. We explore the use of a meta-classification model to assess the precision of segments predicted by a model trained for the semantic segmentation of car body parts. Different sets of features correlated with the quality of a segment are compared, and an AUROC score of 0.915 is achieved for distinguishing between high- and low-quality segments. By removing low-quality segments, the average mIoU of the segmentation output is improved by 16 percentage points and the number of wrongly predicted segments is reduced by 77%.

Title: MSHyper: Multi-Scale Hypergraph Transformer for Long-Range Time Series Forecasting

• Authors: Zongjiang Shang, Ling Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09261
• Pdf URL: https://arxiv.org/pdf/2401.09261
• Copy Paste: [[2401.09261]] MSHyper: Multi-Scale Hypergraph Transformer for Long-Range Time Series Forecasting(https://arxiv.org/abs/2401.09261)
• Keywords: transformer
• Abstract: Demystifying interactions between temporal patterns of different scales is fundamental to precise long-range time series forecasting. However, previous works lack the ability to model high-order interactions. To promote more comprehensive pattern interaction modeling for long-range time series forecasting, we propose a Multi-Scale Hypergraph Transformer (MSHyper) framework. Specifically, a multi-scale hypergraph is introduced to provide foundations for modeling high-order pattern interactions. Then by treating hyperedges as nodes, we also build a hyperedge graph to enhance hypergraph modeling. In addition, a tri-stage message passing mechanism is introduced to aggregate pattern information and learn the interaction strength between temporal patterns of different scales. Extensive experiments on five real-world datasets demonstrate that MSHyper achieves state-of-the-art performance, reducing prediction errors by an average of 8.73% and 7.15% over the best baseline in MSE and MAE, respectively.

Title: Risk-Aware Accelerated Wireless Federated Learning with Heterogeneous Clients

• Authors: Mohamed Ads, Hesham ElSawy, Hossam S. Hassanein
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09267
• Pdf URL: https://arxiv.org/pdf/2401.09267
• Copy Paste: [[2401.09267]] Risk-Aware Accelerated Wireless Federated Learning with Heterogeneous Clients(https://arxiv.org/abs/2401.09267)
• Keywords: security, federate
• Abstract: Wireless Federated Learning (FL) is an emerging distributed machine learning paradigm, particularly gaining momentum in domains with confidential and private data on mobile clients. However, the location-dependent performance, in terms of transmission rates and susceptibility to transmission errors, poses major challenges for wireless FL's convergence speed and accuracy. The challenge is more acute for hostile environments without a metric that authenticates the data quality and security profile of the clients. In this context, this paper proposes a novel risk-aware accelerated FL framework that accounts for the clients heterogeneity in the amount of possessed data, transmission rates, transmission errors, and trustworthiness. Classifying clients according to their location-dependent performance and trustworthiness profiles, we propose a dynamic risk-aware global model aggregation scheme that allows clients to participate in descending order of their transmission rates and an ascending trustworthiness constraint. In particular, the transmission rate is the dominant participation criterion for initial rounds to accelerate the convergence speed. Our model then progressively relaxes the transmission rate restriction to explore more training data at cell-edge clients. The aggregation rounds incorporate a debiasing factor that accounts for transmission errors. Risk-awareness is enabled by a validation set, where the base station eliminates non-trustworthy clients at the fine-tuning stage. The proposed scheme is benchmarked against a conservative scheme (i.e., only allowing trustworthy devices) and an aggressive scheme (i.e., oblivious to the trust metric). The numerical results highlight the superiority of the proposed scheme in terms of accuracy and convergence speed when compared to both benchmarks.

Title: PixelDINO: Semi-Supervised Semantic Segmentation for Detecting Permafrost Disturbances

• Authors: Konrad Heidler, Ingmar Nitze, Guido Grosse, Xiao Xiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09271
• Pdf URL: https://arxiv.org/pdf/2401.09271
• Copy Paste: [[2401.09271]] PixelDINO: Semi-Supervised Semantic Segmentation for Detecting Permafrost Disturbances(https://arxiv.org/abs/2401.09271)
• Keywords: robust, segmentation
• Abstract: Arctic Permafrost is facing significant changes due to global climate change. As these regions are largely inaccessible, remote sensing plays a crucial rule in better understanding the underlying processes not just on a local scale, but across the Arctic. In this study, we focus on the remote detection of retrogressive thaw slumps (RTS), a permafrost disturbance comparable to landslides induced by thawing. For such analyses from space, deep learning has become an indispensable tool, but limited labelled training data remains a challenge for training accurate models. To improve model generalization across the Arctic without the need for additional labelled data, we present a semi-supervised learning approach to train semantic segmentation models to detect RTS. Our framework called PixelDINO is trained in parallel on labelled data as well as unlabelled data. For the unlabelled data, the model segments the imagery into self-taught pseudo-classes and the training procedure ensures consistency of these pseudo-classes across strong augmentations of the input data. Our experimental results demonstrate that PixelDINO can improve model performance both over supervised baseline methods as well as existing semi-supervised semantic segmentation approaches, highlighting its potential for training robust models that generalize well to regions that were not included in the training data. The project page containing code and other materials for this study can be found at \url{https://khdlr.github.io/PixelDINO/}.

Title: BENO: Boundary-embedded Neural Operators for Elliptic PDEs

• Authors: Haixin Wang, Jiaxin Li, Anubhav Dwivedi, Kentaro Hara, Tailin Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09323
• Pdf URL: https://arxiv.org/pdf/2401.09323
• Copy Paste: [[2401.09323]] BENO: Boundary-embedded Neural Operators for Elliptic PDEs(https://arxiv.org/abs/2401.09323)
• Keywords: transformer
• Abstract: Elliptic partial differential equations (PDEs) are a major class of time-independent PDEs that play a key role in many scientific and engineering domains such as fluid dynamics, plasma physics, and solid mechanics. Recently, neural operators have emerged as a promising technique to solve elliptic PDEs more efficiently by directly mapping the input to solutions. However, existing networks typically cannot handle complex geometries and inhomogeneous boundary values present in the real world. Here we introduce Boundary-Embedded Neural Operators (BENO), a novel neural operator architecture that embeds the complex geometries and inhomogeneous boundary values into the solving of elliptic PDEs. Inspired by classical Green's function, BENO consists of two branches of Graph Neural Networks (GNNs) for interior source term and boundary values, respectively. Furthermore, a Transformer encoder maps the global boundary geometry into a latent vector which influences each message passing layer of the GNNs. We test our model extensively in elliptic PDEs with various boundary conditions. We show that all existing baseline methods fail to learn the solution operator. In contrast, our model, endowed with boundary-embedded architecture, outperforms state-of-the-art neural operators and strong baselines by an average of 60.96\%. Our source code can be found https://github.com/AI4Science-WestlakeU/beno.git.

Title: Siamese Meets Diffusion Network: SMDNet for Enhanced Change Detection in High-Resolution RS Imagery

• Authors: Jia Jia, Geunho Lee, Zhibo Wang, Lyu Zhi, Yuchu He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09325
• Pdf URL: https://arxiv.org/pdf/2401.09325
• Copy Paste: [[2401.09325]] Siamese Meets Diffusion Network: SMDNet for Enhanced Change Detection in High-Resolution RS Imagery(https://arxiv.org/abs/2401.09325)
• Keywords: robust, extraction, diffusion, transformer
• Abstract: Recently, the application of deep learning to change detection (CD) has significantly progressed in remote sensing images. In recent years, CD tasks have mostly used architectures such as CNN and Transformer to identify these changes. However, these architectures have shortcomings in representing boundary details and are prone to false alarms and missed detections under complex lighting and weather conditions. For that, we propose a new network, Siamese Meets Diffusion Network (SMDNet). This network combines the Siam-U2Net Feature Differential Encoder (SU-FDE) and the denoising diffusion implicit model to improve the accuracy of image edge change detection and enhance the model's robustness under environmental changes. First, we propose an innovative SU-FDE module that utilizes shared weight features to capture differences between time series images and identify similarities between features to enhance edge detail detection. Furthermore, we add an attention mechanism to identify key coarse features to improve the model's sensitivity and accuracy. Finally, the diffusion model of progressive sampling is used to fuse key coarse features, and the noise reduction ability of the diffusion model and the advantages of capturing the probability distribution of image data are used to enhance the adaptability of the model in different environments. Our method's combination of feature extraction and diffusion models demonstrates effectiveness in change detection in remote sensing images. The performance evaluation of SMDNet on LEVIR-CD, DSIFN-CD, and CDD datasets yields validated F1 scores of 90.99%, 88.40%, and 88.47%, respectively. This substantiates the advanced capabilities of our model in accurately identifying variations and intricate details.

Title: Event-Based Visual Odometry on Non-Holonomic Ground Vehicles

• Authors: Wanting Xu, Si'ao Zhang, Li Cui, Xin Peng, Laurent Kneip
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2401.09331
• Pdf URL: https://arxiv.org/pdf/2401.09331
• Copy Paste: [[2401.09331]] Event-Based Visual Odometry on Non-Holonomic Ground Vehicles(https://arxiv.org/abs/2401.09331)
• Keywords: robust
• Abstract: Despite the promise of superior performance under challenging conditions, event-based motion estimation remains a hard problem owing to the difficulty of extracting and tracking stable features from event streams. In order to robustify the estimation, it is generally believed that fusion with other sensors is a requirement. In this work, we demonstrate reliable, purely event-based visual odometry on planar ground vehicles by employing the constrained non-holonomic motion model of Ackermann steering platforms. We extend single feature n-linearities for regular frame-based cameras to the case of quasi time-continuous event-tracks, and achieve a polynomial form via variable degree Taylor expansions. Robust averaging over multiple event tracks is simply achieved via histogram voting. As demonstrated on both simulated and real data, our algorithm achieves accurate and robust estimates of the vehicle's instantaneous rotational velocity, and thus results that are comparable to the delta rotations obtained by frame-based sensors under normal conditions. We furthermore significantly outperform the more traditional alternatives in challenging illumination scenarios. The code is available at \url{https://github.com/gowanting/NHEVO}.

Title: Large Language Models Are Neurosymbolic Reasoners

• Authors: Meng Fang, Shilong Deng, Yudi Zhang, Zijing Shi, Ling Chen, Mykola Pechenizkiy, Jun Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09334
• Pdf URL: https://arxiv.org/pdf/2401.09334
• Copy Paste: [[2401.09334]] Large Language Models Are Neurosymbolic Reasoners(https://arxiv.org/abs/2401.09334)
• Keywords: large language model
• Abstract: A wide range of real-world applications is characterized by their symbolic nature, necessitating a strong capability for symbolic reasoning. This paper investigates the potential application of Large Language Models (LLMs) as symbolic reasoners. We focus on text-based games, significant benchmarks for agents with natural language capabilities, particularly in symbolic tasks like math, map reading, sorting, and applying common sense in text-based worlds. To facilitate these agents, we propose an LLM agent designed to tackle symbolic challenges and achieve in-game objectives. We begin by initializing the LLM agent and informing it of its role. The agent then receives observations and a set of valid actions from the text-based games, along with a specific symbolic module. With these inputs, the LLM agent chooses an action and interacts with the game environments. Our experimental results demonstrate that our method significantly enhances the capability of LLMs as automated agents for symbolic reasoning, and our LLM agent is effective in text-based games involving symbolic tasks, achieving an average performance of 88% across all tasks.

Title: Synthesizing Hardware-Software Leakage Contracts for RISC-V Open-Source Processors

• Authors: Gideon Mohr, Marco Guarnieri, Jan Reineke
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.09383
• Pdf URL: https://arxiv.org/pdf/2401.09383
• Copy Paste: [[2401.09383]] Synthesizing Hardware-Software Leakage Contracts for RISC-V Open-Source Processors(https://arxiv.org/abs/2401.09383)
• Keywords: security, attack
• Abstract: Microarchitectural attacks compromise security by exploiting software-visible artifacts of microarchitectural optimizations such as caches and speculative execution. Defending against such attacks at the software level requires an appropriate abstraction at the instruction set architecture (ISA) level that captures microarchitectural leakage. Hardware-software leakage contracts have recently been proposed as such an abstraction. In this paper, we propose a semi-automatic methodology for synthesizing hardware-software leakage contracts for open-source microarchitectures. For a given ISA, our approach relies on human experts to (a) capture the space of possible contracts in the form of contract templates and (b) devise a test-case generation strategy to explore a microarchitecture's potential leakage. For a given implementation of an ISA, these two ingredients are then used to automatically synthesize the most precise leakage contract that is satisfied by the microarchitecture. We have instantiated this methodology for the RISC-V ISA and applied it to the Ibex and CVA6 open-source processors. Our experiments demonstrate the practical applicability of the methodology and uncover subtle and unexpected leaks.

Title: Tri$^{2}$-plane: Volumetric Avatar Reconstruction with Feature Pyramid

• Authors: Luchuan Song, Pinxin Liu, Lele Chen, Celong Liu, Chenliang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09386
• Pdf URL: https://arxiv.org/pdf/2401.09386
• Copy Paste: [[2401.09386]] Tri$^{2}$-plane: Volumetric Avatar Reconstruction with Feature Pyramid(https://arxiv.org/abs/2401.09386)
• Keywords: robust
• Abstract: Recent years have witnessed considerable achievements in facial avatar reconstruction with neural volume rendering. Despite notable advancements, the reconstruction of complex and dynamic head movements from monocular videos still suffers from capturing and restoring fine-grained details. In this work, we propose a novel approach, named Tri$^2$-plane, for monocular photo-realistic volumetric head avatar reconstructions. Distinct from the existing works that rely on a single tri-plane deformation field for dynamic facial modeling, the proposed Tri$^2$-plane leverages the principle of feature pyramids and three top-to-down lateral connections tri-planes for details improvement. It samples and renders facial details at multiple scales, transitioning from the entire face to specific local regions and then to even more refined sub-regions. Moreover, we incorporate a camera-based geometry-aware sliding window method as an augmentation in training, which improves the robustness beyond the canonical space, with a particular improvement in cross-identity generation capabilities. Experimental outcomes indicate that the Tri$^2$-plane not only surpasses existing methodologies but also achieves superior performance across both quantitative metrics and qualitative assessments through experiments.

Title: Stuck in the Quicksand of Numeracy, Far from AGI Summit: Evaluating LLMs' Mathematical Competency through Ontology-guided Perturbations

• Authors: Pengfei Hong, Deepanway Ghosal, Navonil Majumder, Somak Aditya, Rada Mihalcea, Soujanya Poria
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09395
• Pdf URL: https://arxiv.org/pdf/2401.09395
• Copy Paste: [[2401.09395]] Stuck in the Quicksand of Numeracy, Far from AGI Summit: Evaluating LLMs' Mathematical Competency through Ontology-guided Perturbations(https://arxiv.org/abs/2401.09395)
• Keywords: robust, large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have showcased striking results on existing logical reasoning benchmarks, with some models even surpassing human performance. However, the true depth of their competencies and robustness, in mathematical reasoning tasks, remains an open question. In response, we develop (i) an ontology of perturbations of maths questions, (ii) a semi-automatic method of perturbation, and (iii) a dataset of perturbed maths questions to probe the limits of LLM capabilities in mathematical reasoning tasks. These controlled perturbations span across multiple fine dimensions of the structural and representational aspects of maths questions. Using GPT-4, we generated the MORE dataset by perturbing randomly selected five seed questions from GSM8K. This process was guided by our ontology and involved a thorough automatic and manual filtering process, yielding a set of 216 maths problems. We conducted comprehensive evaluation of both closed-source and open-source LLMs on MORE. The results show a significant performance drop across all the models against the perturbed questions. This strongly suggests that current LLMs lack robust mathematical skills and deep reasoning abilities. This research not only identifies multiple gaps in the capabilities of current models, but also highlights multiple potential directions for future development. Our dataset will be made publicly available at https://huggingface.co/datasets/declare-lab/GSM8k_MORE.

Title: Deciphering Textual Authenticity: A Generalized Strategy through the Lens of Large Language Semantics for Detecting Human vs. Machine-Generated Text

• Authors: Mazal Bethany, Brandon Wherry, Emet Bethany, Nishant Vishwamitra, Peyman Najafirad
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09407
• Pdf URL: https://arxiv.org/pdf/2401.09407
• Copy Paste: [[2401.09407]] Deciphering Textual Authenticity: A Generalized Strategy through the Lens of Large Language Semantics for Detecting Human vs. Machine-Generated Text(https://arxiv.org/abs/2401.09407)
• Keywords: large language model
• Abstract: With the recent proliferation of Large Language Models (LLMs), there has been an increasing demand for tools to detect machine-generated text. The effective detection of machine-generated text face two pertinent problems: First, they are severely limited in generalizing against real-world scenarios, where machine-generated text is produced by a variety of generators, including but not limited to GPT-4 and Dolly, and spans diverse domains, ranging from academic manuscripts to social media posts. Second, existing detection methodologies treat texts produced by LLMs through a restrictive binary classification lens, neglecting the nuanced diversity of artifacts generated by different LLMs. In this work, we undertake a systematic study on the detection of machine-generated text in real-world scenarios. We first study the effectiveness of state-of-the-art approaches and find that they are severely limited against text produced by diverse generators and domains in the real world. Furthermore, t-SNE visualizations of the embeddings from a pretrained LLM's encoder show that they cannot reliably distinguish between human and machine-generated text. Based on our findings, we introduce a novel system, T5LLMCipher, for detecting machine-generated text using a pretrained T5 encoder combined with LLM embedding sub-clustering to address the text produced by diverse generators and domains in the real world. We evaluate our approach across 9 machine-generated text systems and 9 domains and find that our approach provides state-of-the-art generalization ability, with an average increase in F1 score on machine-generated text of 19.6\% on unseen generators and domains compared to the top performing existing approaches and correctly attributes the generator of text with an accuracy of 93.6\%.

Title: POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images

• Authors: Antonin Vobecky, Oriane Siméoni, David Hurych, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Josef Sivic
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09413
• Pdf URL: https://arxiv.org/pdf/2401.09413
• Copy Paste: [[2401.09413]] POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images(https://arxiv.org/abs/2401.09413)
• Keywords: segmentation
• Abstract: We describe an approach to predict open-vocabulary 3D semantic voxel occupancy map from input 2D images with the objective of enabling 3D grounding, segmentation and retrieval of free-form language queries. This is a challenging problem because of the 2D-3D ambiguity and the open-vocabulary nature of the target tasks, where obtaining annotated training data in 3D is difficult. The contributions of this work are three-fold. First, we design a new model architecture for open-vocabulary 3D semantic occupancy prediction. The architecture consists of a 2D-3D encoder together with occupancy prediction and 3D-language heads. The output is a dense voxel map of 3D grounded language embeddings enabling a range of open-vocabulary tasks. Second, we develop a tri-modal self-supervised learning algorithm that leverages three modalities: (i) images, (ii) language and (iii) LiDAR point clouds, and enables training the proposed architecture using a strong pre-trained vision-language model without the need for any 3D manual language annotations. Finally, we demonstrate quantitatively the strengths of the proposed model on several open-vocabulary tasks: Zero-shot 3D semantic segmentation using existing datasets; 3D grounding and retrieval of free-form language queries, using a small dataset that we propose as an extension of nuScenes. You can find the project page here https://vobecant.github.io/POP3D.

Title: Vlogger: Make Your Dream A Vlog

• Authors: Shaobin Zhuang, Kunchang Li, Xinyuan Chen, Yaohui Wang, Ziwei Liu, Yu Qiao, Yali Wang
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2401.09414
• Pdf URL: https://arxiv.org/pdf/2401.09414
• Copy Paste: [[2401.09414]] Vlogger: Make Your Dream A Vlog(https://arxiv.org/abs/2401.09414)
• Keywords: diffusion, large language model
• Abstract: In this work, we present Vlogger, a generic AI system for generating a minute-level video blog (i.e., vlog) of user descriptions. Different from short videos with a few seconds, vlog often contains a complex storyline with diversified scenes, which is challenging for most existing video generation approaches. To break through this bottleneck, our Vlogger smartly leverages Large Language Model (LLM) as Director and decomposes a long video generation task of vlog into four key stages, where we invoke various foundation models to play the critical roles of vlog professionals, including (1) Script, (2) Actor, (3) ShowMaker, and (4) Voicer. With such a design of mimicking human beings, our Vlogger can generate vlogs through explainable cooperation of top-down planning and bottom-up shooting. Moreover, we introduce a novel video diffusion model, ShowMaker, which serves as a videographer in our Vlogger for generating the video snippet of each shooting scene. By incorporating Script and Actor attentively as textual and visual prompts, it can effectively enhance spatial-temporal coherence in the snippet. Besides, we design a concise mixed training paradigm for ShowMaker, boosting its capacity for both T2V generation and prediction. Finally, the extensive experiments show that our method achieves state-of-the-art performance on zero-shot T2V generation and prediction tasks. More importantly, Vlogger can generate over 5-minute vlogs from open-world descriptions, without loss of video coherence on script and actor. The code and model is all available at https://github.com/zhuangshaobin/Vlogger.

Title: TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion

• Authors: Yu-Ying Yeh, Jia-Bin Huang, Changil Kim, Lei Xiao, Thu Nguyen-Phuoc, Numair Khan, Cheng Zhang, Manmohan Chandraker, Carl S Marshall, Zhao Dong, Zhengqin Li
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.09416
• Pdf URL: https://arxiv.org/pdf/2401.09416
• Copy Paste: [[2401.09416]] TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion(https://arxiv.org/abs/2401.09416)
• Keywords: extraction, diffusion
• Abstract: We present TextureDreamer, a novel image-guided texture synthesis method to transfer relightable textures from a small number of input images (3 to 5) to target 3D shapes across arbitrary categories. Texture creation is a pivotal challenge in vision and graphics. Industrial companies hire experienced artists to manually craft textures for 3D assets. Classical methods require densely sampled views and accurately aligned geometry, while learning-based methods are confined to category-specific shapes within the dataset. In contrast, TextureDreamer can transfer highly detailed, intricate textures from real-world environments to arbitrary objects with only a few casually captured images, potentially significantly democratizing texture creation. Our core idea, personalized geometry-aware score distillation (PGSD), draws inspiration from recent advancements in diffuse models, including personalized modeling for texture information extraction, variational score distillation for detailed appearance synthesis, and explicit geometry guidance with ControlNet. Our integration and several essential modifications substantially improve the texture quality. Experiments on real images spanning different categories show that TextureDreamer can successfully transfer highly realistic, semantic meaningful texture to arbitrary objects, surpassing the visual quality of previous state-of-the-art.

Title: Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

• Authors: Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, Xinggang Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09417
• Pdf URL: https://arxiv.org/pdf/2401.09417
• Copy Paste: [[2401.09417]] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model(https://arxiv.org/abs/2401.09417)
• Keywords: transformer, segmentation
• Abstract: Recently the state space models (SSMs) with efficient hardware-aware designs, i.e., Mamba, have shown great potential for long sequence modeling. Building efficient and generic vision backbones purely upon SSMs is an appealing direction. However, representing visual data is challenging for SSMs due to the position-sensitivity of visual data and the requirement of global context for visual understanding. In this paper, we show that the reliance of visual representation learning on self-attention is not necessary and propose a new generic vision backbone with bidirectional Mamba blocks (Vim), which marks the image sequences with position embeddings and compresses the visual representation with bidirectional state space models. On ImageNet classification, COCO object detection, and ADE20k semantic segmentation tasks, Vim achieves higher performance compared to well-established vision transformers like DeiT, while also demonstrating significantly improved computation & memory efficiency. For example, Vim is 2.8$\times$ faster than DeiT and saves 86.8% GPU memory when performing batch inference to extract features on images with a resolution of 1248$\times$1248. The results demonstrate that Vim is capable of overcoming the computation & memory constraints on performing Transformer-style understanding for high-resolution images and it has great potential to become the next-generation backbone for vision foundation models. Code is available at https://github.com/hustvl/Vim.

Title: GARField: Group Anything with Radiance Fields

• Authors: Chung Min Kim, Mingxuan Wu, Justin Kerr, Ken Goldberg, Matthew Tancik, Angjoo Kanazawa
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.09419
• Pdf URL: https://arxiv.org/pdf/2401.09419
• Copy Paste: [[2401.09419]] GARField: Group Anything with Radiance Fields(https://arxiv.org/abs/2401.09419)
• Keywords: extraction
• Abstract: Grouping is inherently ambiguous due to the multiple levels of granularity in which one can decompose a scene -- should the wheels of an excavator be considered separate or part of the whole? We present Group Anything with Radiance Fields (GARField), an approach for decomposing 3D scenes into a hierarchy of semantically meaningful groups from posed image inputs. To do this we embrace group ambiguity through physical scale: by optimizing a scale-conditioned 3D affinity feature field, a point in the world can belong to different groups of different sizes. We optimize this field from a set of 2D masks provided by Segment Anything (SAM) in a way that respects coarse-to-fine hierarchy, using scale to consistently fuse conflicting masks from different viewpoints. From this field we can derive a hierarchy of possible groupings via automatic tree construction or user interaction. We evaluate GARField on a variety of in-the-wild scenes and find it effectively extracts groups at many levels: clusters of objects, objects, and various subparts. GARField inherently represents multi-view consistent groupings and produces higher fidelity groups than the input SAM masks. GARField's hierarchical grouping could have exciting downstream applications such as 3D asset extraction or dynamic scene understanding. See the project website at https://www.garfield.studio/