2024-11-15

Title: Aligning Visual Contrastive learning models via Preference Optimization

• Authors: Amirabbas Afzali, Borna Khodabandeh, Ali Rasekh, Mahyar JafariNodeh, Sepehr kazemi, Simon Gottschalk
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08923
• Pdf URL: https://arxiv.org/pdf/2411.08923
• Copy Paste: [[2411.08923]] Aligning Visual Contrastive learning models via Preference Optimization(https://arxiv.org/abs/2411.08923)
• Keywords: attack, robust, fair, generative
• Abstract: Contrastive learning models have demonstrated impressive abilities to capture semantic similarities by aligning representations in the embedding space. However, their performance can be limited by the quality of the training data and its inherent biases. While Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have been applied to generative models to align them with human preferences, their use in contrastive learning has yet to be explored. This paper introduces a novel method for training contrastive learning models using Preference Optimization (PO) to break down complex concepts. Our method systematically aligns model behavior with desired preferences, enhancing performance on the targeted task. In particular, we focus on enhancing model robustness against typographic attacks, commonly seen in contrastive models like CLIP. We further apply our method to disentangle gender understanding and mitigate gender biases, offering a more nuanced control over these sensitive attributes. Our experiments demonstrate that models trained using PO outperform standard contrastive learning techniques while retaining their ability to handle adversarial challenges and maintain accuracy on other downstream tasks. This makes our method well-suited for tasks requiring fairness, robustness, and alignment with specific preferences. We evaluate our method on several vision-language tasks, tackling challenges such as typographic attacks. Additionally, we explore the model's ability to disentangle gender concepts and mitigate gender bias, showcasing the versatility of our approach.

Title: Structured Pattern Expansion with Diffusion Models

• Authors: Marzia Riso, Giuseppe Vecchio, Fabio Pellacini
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.08930
• Pdf URL: https://arxiv.org/pdf/2411.08930
• Copy Paste: [[2411.08930]] Structured Pattern Expansion with Diffusion Models(https://arxiv.org/abs/2411.08930)
• Keywords: diffusion, generative
• Abstract: Recent advances in diffusion models have significantly improved the synthesis of materials, textures, and 3D shapes. By conditioning these models via text or images, users can guide the generation, reducing the time required to create digital assets. In this paper, we address the synthesis of structured, stationary patterns, where diffusion models are generally less reliable and, more importantly, less controllable. Our approach leverages the generative capabilities of diffusion models specifically adapted for the pattern domain. It enables users to exercise direct control over the synthesis by expanding a partially hand-drawn pattern into a larger design while preserving the structure and details of the input. To enhance pattern quality, we fine-tune an image-pretrained diffusion model on structured patterns using Low-Rank Adaptation (LoRA), apply a noise rolling technique to ensure tileability, and utilize a patch-based approach to facilitate the generation of large-scale assets. We demonstrate the effectiveness of our method through a comprehensive set of experiments, showing that it outperforms existing models in generating diverse, consistent patterns that respond directly to user input.

Title: Confidence-aware Denoised Fine-tuning of Off-the-shelf Models for Certified Robustness

• Authors: Suhyeok Jang, Seojin Kim, Jinwoo Shin, Jongheon Jeong
• Subjects: cs.CV, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08933
• Pdf URL: https://arxiv.org/pdf/2411.08933
• Copy Paste: [[2411.08933]] Confidence-aware Denoised Fine-tuning of Off-the-shelf Models for Certified Robustness(https://arxiv.org/abs/2411.08933)
• Keywords: attack, robust
• Abstract: The remarkable advances in deep learning have led to the emergence of many off-the-shelf classifiers, e.g., large pre-trained models. However, since they are typically trained on clean data, they remain vulnerable to adversarial attacks. Despite this vulnerability, their superior performance and transferability make off-the-shelf classifiers still valuable in practice, demanding further work to provide adversarial robustness for them in a post-hoc manner. A recently proposed method, denoised smoothing, leverages a denoiser model in front of the classifier to obtain provable robustness without additional training. However, the denoiser often creates hallucination, i.e., images that have lost the semantics of their originally assigned class, leading to a drop in robustness. Furthermore, its noise-and-denoise procedure introduces a significant distribution shift from the original distribution, causing the denoised smoothing framework to achieve sub-optimal robustness. In this paper, we introduce Fine-Tuning with Confidence-Aware Denoised Image Selection (FT-CADIS), a novel fine-tuning scheme to enhance the certified robustness of off-the-shelf classifiers. FT-CADIS is inspired by the observation that the confidence of off-the-shelf classifiers can effectively identify hallucinated images during denoised smoothing. Based on this, we develop a confidence-aware training objective to handle such hallucinated images and improve the stability of fine-tuning from denoised images. In this way, the classifier can be fine-tuned using only images that are beneficial for adversarial robustness. We also find that such a fine-tuning can be done by updating a small fraction of parameters of the classifier. Extensive experiments demonstrate that FT-CADIS has established the state-of-the-art certified robustness among denoised smoothing methods across all $\ell_2$-adversary radius in various benchmarks.

Title: Inconsistencies In Consistency Models: Better ODE Solving Does Not Imply Better Samples

• Authors: Noël Vouitsis, Rasa Hosseinzadeh, Brendan Leigh Ross, Valentin Villecroze, Satya Krishna Gorti, Jesse C. Cresswell, Gabriel Loaiza-Ganem
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08954
• Pdf URL: https://arxiv.org/pdf/2411.08954
• Copy Paste: [[2411.08954]] Inconsistencies In Consistency Models: Better ODE Solving Does Not Imply Better Samples(https://arxiv.org/abs/2411.08954)
• Keywords: diffusion
• Abstract: Although diffusion models can generate remarkably high-quality samples, they are intrinsically bottlenecked by their expensive iterative sampling procedure. Consistency models (CMs) have recently emerged as a promising diffusion model distillation method, reducing the cost of sampling by generating high-fidelity samples in just a few iterations. Consistency model distillation aims to solve the probability flow ordinary differential equation (ODE) defined by an existing diffusion model. CMs are not directly trained to minimize error against an ODE solver, rather they use a more computationally tractable objective. As a way to study how effectively CMs solve the probability flow ODE, and the effect that any induced error has on the quality of generated samples, we introduce Direct CMs, which \textit{directly} minimize this error. Intriguingly, we find that Direct CMs reduce the ODE solving error compared to CMs but also result in significantly worse sample quality, calling into question why exactly CMs work well in the first place. Full code is available at: this https URL.

Title: Sparse Upcycling: Inference Inefficient Finetuning

• Authors: Sasha Doubov, Nikhil Sardana, Vitaliy Chiley
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.08968
• Pdf URL: https://arxiv.org/pdf/2411.08968
• Copy Paste: [[2411.08968]] Sparse Upcycling: Inference Inefficient Finetuning(https://arxiv.org/abs/2411.08968)
• Keywords: large language model
• Abstract: Small, highly trained, open-source large language models are widely used due to their inference efficiency, but further improving their quality remains a challenge. Sparse upcycling is a promising approach that transforms a pretrained dense model into a Mixture-of-Experts (MoE) architecture, increasing the model's parameter count and quality. In this work, we compare the effectiveness of sparse upcycling against continued pretraining (CPT) across different model sizes, compute budgets, and pretraining durations. Our experiments show that sparse upcycling can achieve better quality, with improvements of over 20% relative to CPT in certain scenarios. However, this comes with a significant inference cost, leading to 40% slowdowns in high-demand inference settings for larger models. Our findings highlight the trade-off between model quality and inference efficiency, offering insights for practitioners seeking to balance model quality and deployment constraints.

Title: CoCoP: Enhancing Text Classification with LLM through Code Completion Prompt

• Authors: Mohammad Mahdi Mohajeri, Mohammad Javad Dousti, Majid Nili Ahmadabadi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08979
• Pdf URL: https://arxiv.org/pdf/2411.08979
• Copy Paste: [[2411.08979]] CoCoP: Enhancing Text Classification with LLM through Code Completion Prompt(https://arxiv.org/abs/2411.08979)
• Keywords: large language model
• Abstract: Text classification is a fundamental task in natural language processing (NLP), and large language models (LLMs) have demonstrated their capability to perform this task across various domains. However, the performance of LLMs heavily depends on the quality of their input prompts. Recent studies have also shown that LLMs exhibit remarkable results in code-related tasks. To leverage the capabilities of LLMs in text classification, we propose the Code Completion Prompt (CoCoP) method, which transforms the text classification problem into a code completion task. CoCoP significantly improves text classification performance across diverse datasets by utilizing LLMs' code-completion capability. For instance, CoCoP enhances the accuracy of the SST2 dataset by more than 20%. Moreover, when CoCoP integrated with LLMs specifically designed for code-related tasks (code models), such as CodeLLaMA, this method demonstrates better or comparable performance to few-shot learning techniques while using only one-tenth of the model size. The source code of our proposed method will be available to the public upon the acceptance of the paper.

Title: Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection

• Authors: Vima Gupta, Kartik Sinha, Ada Gavrilovska, Anand Padmanabha Iyer
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.08982
• Pdf URL: https://arxiv.org/pdf/2411.08982
• Copy Paste: [[2411.08982]] Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection(https://arxiv.org/abs/2411.08982)
• Keywords: large language model
• Abstract: Mixture-of-Experts (MoE) architectures have recently gained popularity in enabling efficient scaling of large language models. However, we uncover a fundamental tension: while MoEs are designed for selective expert activation, production serving requires request batching, which forces the activation of all experts and negates MoE's efficiency benefits during the decode phase. We present Lynx, a system that enables efficient MoE inference through dynamic, batch-aware expert selection. Our key insight is that expert importance varies significantly across tokens and inference phases, creating opportunities for runtime optimization. Lynx leverages this insight through a lightweight framework that dynamically reduces active experts while preserving model accuracy. Our evaluations show that Lynx achieves up to 1.55x reduction in inference latency while maintaining negligible accuracy loss from baseline model across complex code generation and mathematical reasoning tasks.

Title: Computed tomography using meta-optics

• Authors: Maksym Zhelyeznuyakov, Johannes E. Fröch, Shane Colburn, Steven L. Brunton, Arka Majumdar
• Subjects: cs.CV, physics.optics
• Abstract URL: https://arxiv.org/abs/2411.08995
• Pdf URL: https://arxiv.org/pdf/2411.08995
• Copy Paste: [[2411.08995]] Computed tomography using meta-optics(https://arxiv.org/abs/2411.08995)
• Keywords: extraction, segmentation
• Abstract: Computer vision tasks require processing large amounts of data to perform image classification, segmentation, and feature extraction. Optical preprocessors can potentially reduce the number of floating point operations required by computer vision tasks, enabling low-power and low-latency operation. However, existing optical preprocessors are mostly learned and hence strongly depend on the training data, and thus lack universal applicability. In this paper, we present a metaoptic imager, which implements the Radon transform obviating the need for training the optics. High quality image reconstruction with a large compression ratio of 0.6% is presented through the use of the Simultaneous Algebraic Reconstruction Technique. Image classification with 90% accuracy is presented on an experimentally measured Radon dataset through neural network trained on digitally transformed images.

Title: CoMiX: Cross-Modal Fusion with Deformable Convolutions for HSI-X Semantic Segmentation

• Authors: Xuming Zhang, Xingfa Gu, Qingjiu Tian, Lorenzo Bruzzone
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09023
• Pdf URL: https://arxiv.org/pdf/2411.09023
• Copy Paste: [[2411.09023]] CoMiX: Cross-Modal Fusion with Deformable Convolutions for HSI-X Semantic Segmentation(https://arxiv.org/abs/2411.09023)
• Keywords: segmentation
• Abstract: Improving hyperspectral image (HSI) semantic segmentation by exploiting complementary information from a supplementary data type (referred to X-modality) is promising but challenging due to differences in imaging sensors, image content, and resolution. Current techniques struggle to enhance modality-specific and modality-shared information, as well as to capture dynamic interaction and fusion between different modalities. In response, this study proposes CoMiX, an asymmetric encoder-decoder architecture with deformable convolutions (DCNs) for HSI-X semantic segmentation. CoMiX is designed to extract, calibrate, and fuse information from HSI and X data. Its pipeline includes an encoder with two parallel and interacting backbones and a lightweight all-multilayer perceptron (ALL-MLP) decoder. The encoder consists of four stages, each incorporating 2D DCN blocks for the X model to accommodate geometric variations and 3D DCN blocks for HSIs to adaptively aggregate spatial-spectral features. Additionally, each stage includes a Cross-Modality Feature enhancement and eXchange (CMFeX) module and a feature fusion module (FFM). CMFeX is designed to exploit spatial-spectral correlations from different modalities to recalibrate and enhance modality-specific and modality-shared features while adaptively exchanging complementary information between them. Outputs from CMFeX are fed into the FFM for fusion and passed to the next stage for further information learning. Finally, the outputs from each FFM are integrated by the ALL-MLP decoder for final prediction. Extensive experiments demonstrate that our CoMiX achieves superior performance and generalizes well to various multimodal recognition tasks. The CoMiX code will be released.

Title: Transformer-based Time-Series Biomarker Discovery for COPD Diagnosis

• Authors: Soham Gadgil, Joshua Galanter, Mohammadreza Negahdar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09027
• Pdf URL: https://arxiv.org/pdf/2411.09027
• Copy Paste: [[2411.09027]] Transformer-based Time-Series Biomarker Discovery for COPD Diagnosis(https://arxiv.org/abs/2411.09027)
• Keywords: transformer
• Abstract: Chronic Obstructive Pulmonary Disorder (COPD) is an irreversible and progressive disease which is highly heritable. Clinically, COPD is defined using the summary measures derived from a spirometry test but these are not always adequate. Here we show that using the high-dimensional raw spirogram can provide a richer signal compared to just using the summary measures. We design a transformer-based deep learning technique to process the raw spirogram values along with demographic information and predict clinically-relevant endpoints related to COPD. Our method is able to perform better than prior works while being more computationally efficient. Using the weights learned by the model, we make the framework more interpretable by identifying parts of the spirogram that are important for the model predictions. Pairing up with a board-certified pulmonologist, we also provide clinical insights into the different aspects of the spirogram and show that the explanations obtained from the model align with underlying medical knowledge.

Title: A Transformer-Based Visual Piano Transcription Algorithm

• Authors: Uros Zivanovic, Carlos Eduardo Cancino-Chacón
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09037
• Pdf URL: https://arxiv.org/pdf/2411.09037
• Copy Paste: [[2411.09037]] A Transformer-Based Visual Piano Transcription Algorithm(https://arxiv.org/abs/2411.09037)
• Keywords: transformer
• Abstract: Automatic music transcription (AMT) for musical performances is a long standing problem in the field of Music Information Retrieval (MIR). Visual piano transcription (VPT) is a multimodal subproblem of AMT which focuses on extracting a symbolic representation of a piano performance from visual information only (e.g., from a top-down video of the piano keyboard). Inspired by the success of Transformers for audio-based AMT, as well as their recent successes in other computer vision tasks, in this paper we present a Transformer based architecture for VPT. The proposed VPT system combines a piano bounding box detection model with an onset and pitch detection model, allowing our system to perform well in more naturalistic conditions like imperfect image crops around the piano and slightly tilted images.

Title: Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset

• Authors: Mohammad Saiful Islam, Mohamed Sami Rakha, William Pourmajidi, Janakan Sivaloganathan, John Steinbacher, Andriy Miranskyy
• Subjects: cs.LG, cs.DC, cs.SE
• Abstract URL: https://arxiv.org/abs/2411.09047
• Pdf URL: https://arxiv.org/pdf/2411.09047
• Copy Paste: [[2411.09047]] Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset(https://arxiv.org/abs/2411.09047)
• Keywords: robust
• Abstract: As Large-Scale Cloud Systems (LCS) become increasingly complex, effective anomaly detection is critical for ensuring system reliability and performance. However, there is a shortage of large-scale, real-world datasets available for benchmarking anomaly detection methods. To address this gap, we introduce a new high-dimensional dataset from IBM Cloud, collected over 4.5 months from the IBM Cloud Console. This dataset comprises 39,365 rows and 117,448 columns of telemetry data. Additionally, we demonstrate the application of machine learning models for anomaly detection and discuss the key challenges faced in this process. This study and the accompanying dataset provide a resource for researchers and practitioners in cloud system monitoring. It facilitates more efficient testing of anomaly detection methods in real-world data, helping to advance the development of robust solutions to maintain the health and performance of large-scale cloud infrastructures.

Title: SAFELOC: Overcoming Data Poisoning Attacks in Heterogeneous Federated Machine Learning for Indoor Localization

• Authors: Akhil Singampalli, Danish Gufran, Sudeep Pasricha
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.09055
• Pdf URL: https://arxiv.org/pdf/2411.09055
• Copy Paste: [[2411.09055]] SAFELOC: Overcoming Data Poisoning Attacks in Heterogeneous Federated Machine Learning for Indoor Localization(https://arxiv.org/abs/2411.09055)
• Keywords: privacy, attack, federate
• Abstract: Machine learning (ML) based indoor localization solutions are critical for many emerging applications, yet their efficacy is often compromised by hardware/software variations across mobile devices (i.e., device heterogeneity) and the threat of ML data poisoning attacks. Conventional methods aimed at countering these challenges show limited resilience to the uncertainties created by these phenomena. In response, in this paper, we introduce SAFELOC, a novel framework that not only minimizes localization errors under these challenging conditions but also ensures model compactness for efficient mobile device deployment. Our framework targets a distributed and co-operative learning environment that uses federated learning (FL) to preserve user data privacy and assumes heterogeneous mobile devices carried by users (just like in most real-world scenarios). Within this heterogeneous FL context, SAFELOC introduces a novel fused neural network architecture that performs data poisoning detection and localization, with a low model footprint. Additionally, a dynamic saliency map-based aggregation strategy is designed to adapt based on the severity of the detected data poisoning scenario. Experimental evaluations demonstrate that SAFELOC achieves improvements of up to 5.9x in mean localization error, 7.8x in worst-case localization error, and a 2.1x reduction in model inference latency compared to state-of-the-art indoor localization frameworks, across diverse building floorplans, mobile devices, and ML data poisoning attack scenarios.

Title: Optimisation Strategies for Ensuring Fairness in Machine Learning: With and Without Demographics

• Authors: Quan Zhou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09056
• Pdf URL: https://arxiv.org/pdf/2411.09056
• Copy Paste: [[2411.09056]] Optimisation Strategies for Ensuring Fairness in Machine Learning: With and Without Demographics(https://arxiv.org/abs/2411.09056)
• Keywords: robust, fair
• Abstract: Ensuring fairness has emerged as one of the primary concerns in AI and its related algorithms. Over time, the field of machine learning fairness has evolved to address these issues. This paper provides an extensive overview of this field and introduces two formal frameworks to tackle open questions in machine learning fairness. In one framework, operator-valued optimisation and min-max objectives are employed to address unfairness in time-series problems. This approach showcases state-of-the-art performance on the notorious COMPAS benchmark dataset, demonstrating its effectiveness in real-world scenarios. In the second framework, the challenge of lacking sensitive attributes, such as gender and race, in commonly used datasets is addressed. This issue is particularly pressing because existing algorithms in this field predominantly rely on the availability or estimations of such attributes to assess and mitigate unfairness. Here, a framework for a group-blind bias-repair is introduced, aiming to mitigate bias without relying on sensitive attributes. The efficacy of this approach is showcased through analyses conducted on the Adult Census Income dataset. Additionally, detailed algorithmic analyses for both frameworks are provided, accompanied by convergence guarantees, ensuring the robustness and reliability of the proposed methodologies.

Title: Multimodal Object Detection using Depth and Image Data for Manufacturing Parts

• Authors: Nazanin Mahjourian, Vinh Nguyen
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.09062
• Pdf URL: https://arxiv.org/pdf/2411.09062
• Copy Paste: [[2411.09062]] Multimodal Object Detection using Depth and Image Data for Manufacturing Parts(https://arxiv.org/abs/2411.09062)
• Keywords: robust
• Abstract: Manufacturing requires reliable object detection methods for precise picking and handling of diverse types of manufacturing parts and components. Traditional object detection methods utilize either only 2D images from cameras or 3D data from lidars or similar 3D sensors. However, each of these sensors have weaknesses and limitations. Cameras do not have depth perception and 3D sensors typically do not carry color information. These weaknesses can undermine the reliability and robustness of industrial manufacturing systems. To address these challenges, this work proposes a multi-sensor system combining an red-green-blue (RGB) camera and a 3D point cloud sensor. The two sensors are calibrated for precise alignment of the multimodal data captured from the two hardware devices. A novel multimodal object detection method is developed to process both RGB and depth data. This object detector is based on the Faster R-CNN baseline that was originally designed to process only camera images. The results show that the multimodal model significantly outperforms the depth-only and RGB-only baselines on established object detection metrics. More specifically, the multimodal model improves mAP by 13% and raises Mean Precision by 11.8% in comparison to the RGB-only baseline. Compared to the depth-only baseline, it improves mAP by 78% and raises Mean Precision by 57%. Hence, this method facilitates more reliable and robust object detection in service to smart manufacturing applications.

Title: Continuous GNN-based Anomaly Detection on Edge using Efficient Adaptive Knowledge Graph Learning

• Authors: Sanggeon Yun, Ryozo Masukawa, William Youngwoo Chung, Minhyoung Na, Nathaniel Bastian, Mohsen Imani
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09072
• Pdf URL: https://arxiv.org/pdf/2411.09072
• Copy Paste: [[2411.09072]] Continuous GNN-based Anomaly Detection on Edge using Efficient Adaptive Knowledge Graph Learning(https://arxiv.org/abs/2411.09072)
• Keywords: security, robust, large language model
• Abstract: The increasing demand for robust security solutions across various industries has made Video Anomaly Detection (VAD) a critical task in applications such as intelligent surveillance, evidence investigation, and violence detection. Traditional approaches to VAD often rely on finetuning large pre-trained models, which can be computationally expensive and impractical for real-time or resource-constrained environments. To address this, MissionGNN introduced a more efficient method by training a graph neural network (GNN) using a fixed knowledge graph (KG) derived from large language models (LLMs) like GPT-4. While this approach demonstrated significant efficiency in computational power and memory, it faces limitations in dynamic environments where frequent updates to the KG are necessary due to evolving behavior trends and shifting data patterns. These updates typically require cloud-based computation, posing challenges for edge computing applications. In this paper, we propose a novel framework that facilitates continuous KG adaptation directly on edge devices, overcoming the limitations of cloud dependency. Our method dynamically modifies the KG through a three-phase process: pruning, alternating, and creating nodes, enabling real-time adaptation to changing data trends. This continuous learning approach enhances the robustness of anomaly detection models, making them more suitable for deployment in dynamic and resource-constrained environments.

Title: Code-mixed LLM: Improve Large Language Models' Capability to Handle Code-Mixing through Reinforcement Learning from AI Feedback

• Authors: Wenbo Zhang, Aditya Majumdar, Amulya Yadav
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09073
• Pdf URL: https://arxiv.org/pdf/2411.09073
• Copy Paste: [[2411.09073]] Code-mixed LLM: Improve Large Language Models' Capability to Handle Code-Mixing through Reinforcement Learning from AI Feedback(https://arxiv.org/abs/2411.09073)
• Keywords: large language model
• Abstract: Code-mixing(CM) or code-switching(CSW) refers to the juxtaposition of linguistic units from two or more languages during the conversation or sometimes even a single utterance. Code-mixing introduces unique challenges in daily life, such as syntactic mismatches and semantic blending, that are rarely encountered in monolingual settings. Large language models (LLMs) have revolutionized the field of natural language processing (NLP) by offering unprecedented capabilities in understanding human languages. However, the effectiveness of current state-of-the-art multilingual LLMs has not yet been fully explored in the CM scenario. To fill this gap, we first benchmark the performance of multilingual LLMs on various code-mixing NLP tasks. Then we propose to improve the multilingual LLMs' ability to understand code-mixing through reinforcement learning from human feedback (RLHF) and code-mixed machine translation tasks. Given the high-cost and time-consuming preference labeling procedure, we improve this by utilizing LLMs as annotators to perform the reinforcement learning from AI feedback (RLAIF). The experiments show the effectiveness of the proposed method.

Title: Heuristical Comparison of Vision Transformers Against Convolutional Neural Networks for Semantic Segmentation on Remote Sensing Imagery

• Authors: Ashim Dahal, Saydul Akbar Murad, Nick Rahimi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09101
• Pdf URL: https://arxiv.org/pdf/2411.09101
• Copy Paste: [[2411.09101]] Heuristical Comparison of Vision Transformers Against Convolutional Neural Networks for Semantic Segmentation on Remote Sensing Imagery(https://arxiv.org/abs/2411.09101)
• Keywords: transformer, segmentation
• Abstract: Vision Transformers (ViT) have recently brought a new wave of research in the field of computer vision. These models have done particularly well in the field of image classification and segmentation. Research on semantic and instance segmentation has emerged to accelerate with the inception of the new architecture, with over 80\% of the top 20 benchmarks for the iSAID dataset being either based on the ViT architecture or the attention mechanism behind its success. This paper focuses on the heuristic comparison of three key factors of using (or not using) ViT for semantic segmentation of remote sensing aerial images on the iSAID. The experimental results observed during the course of the research were under the scrutinization of the following objectives: 1. Use of weighted fused loss function for the maximum mean Intersection over Union (mIoU) score, Dice score, and minimization or conservation of entropy or class representation, 2. Comparison of transfer learning on Meta's MaskFormer, a ViT-based semantic segmentation model, against generic UNet Convolutional Neural Networks (CNNs) judged over mIoU, Dice scores, training efficiency, and inference time, and 3. What do we lose for what we gain? i.e., the comparison of the two models against current state-of-art segmentation models. We show the use of the novel combined weighted loss function significantly boosts the CNN model's performance capacities as compared to transfer learning the ViT. The code for this implementation can be found on \url{this https URL}.

Title: VCBench: A Controllable Benchmark for Symbolic and Abstract Challenges in Video Cognition

• Authors: Chenglin Li, Qianglong Chen, Zhi Li, Feng Tao, Yin Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09105
• Pdf URL: https://arxiv.org/pdf/2411.09105
• Copy Paste: [[2411.09105]] VCBench: A Controllable Benchmark for Symbolic and Abstract Challenges in Video Cognition(https://arxiv.org/abs/2411.09105)
• Keywords: robust
• Abstract: Recent advancements in Large Video-Language Models (LVLMs) have driven the development of benchmarks designed to assess cognitive abilities in video-based tasks. However, most existing benchmarks heavily rely on web-collected videos paired with human annotations or model-generated questions, which limit control over the video content and fall short in evaluating advanced cognitive abilities involving symbolic elements and abstract concepts. To address these limitations, we introduce VCBench, a controllable benchmark to assess LVLMs' cognitive abilities, involving symbolic and abstract concepts at varying difficulty levels. By generating video data with the Python-based engine, VCBench allows for precise control over the video content, creating dynamic, task-oriented videos that feature complex scenes and abstract concepts. Each task pairs with tailored question templates that target specific cognitive challenges, providing a rigorous evaluation test. Our evaluation reveals that even state-of-the-art (SOTA) models, such as Qwen2-VL-72B, struggle with simple video cognition tasks involving abstract concepts, with performance sharply dropping by 19% as video complexity rises. These findings reveal the current limitations of LVLMs in advanced cognitive tasks and highlight the critical role of VCBench in driving research toward more robust LVLMs for complex video cognition challenges.

Title: Reducing Reasoning Costs - The Path of Optimization for Chain of Thought via Sparse Attention Mechanism

• Authors: Libo Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09111
• Pdf URL: https://arxiv.org/pdf/2411.09111
• Copy Paste: [[2411.09111]] Reducing Reasoning Costs - The Path of Optimization for Chain of Thought via Sparse Attention Mechanism(https://arxiv.org/abs/2411.09111)
• Keywords: large language model
• Abstract: In order to address the chain of thought in the large language model inference cost surge, this research proposes to use a sparse attention mechanism that only focuses on a few relevant tokens. The researcher constructed a new attention mechanism and used GiantRabbit trained with custom GPTs as an experimental tool. The experiment tested and compared the reasoning time, correctness score and chain of thought length of this model and o1 Preview in solving the linear algebra test questions of MIT OpenCourseWare. The results show that GiantRabbit's reasoning time and chain of thought length are significantly lower than o1 Preview, confirming the feasibility of the sparse attention mechanism in reducing chain of thought reasoning. Detailed architectural details and experimental process have been uploaded to Github, the link is:this https URL.

Title: P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs

• Authors: Yidan Zhang, Boyi Deng, Yu Wan, Baosong Yang, Haoran Wei, Fei Huang, Bowen Yu, Junyang Lin, Fei Huang, Jingren Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09116
• Pdf URL: https://arxiv.org/pdf/2411.09116
• Copy Paste: [[2411.09116]] P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs(https://arxiv.org/abs/2411.09116)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) showcase varied multilingual capabilities across tasks like translation, code generation, and reasoning. Previous assessments often limited their scope to fundamental natural language processing (NLP) or isolated capability-specific tasks. To alleviate this drawback, we aim to present a comprehensive multilingual multitask benchmark. First, we present a pipeline for selecting available and reasonable benchmarks from massive ones, addressing the oversight in previous work regarding the utility of these benchmarks, i.e., their ability to differentiate between models being evaluated. Leveraging this pipeline, we introduce P-MMEval, a large-scale benchmark covering effective fundamental and capability-specialized datasets. Furthermore, P-MMEval delivers consistent language coverage across various datasets and provides parallel samples. Finally, we conduct extensive experiments on representative multilingual model series to compare performances across models, analyze dataset effectiveness, examine prompt impacts on model performances, and explore the relationship between multilingual performances and factors such as tasks, model sizes, and languages. These insights offer valuable guidance for future research. The dataset is available at this https URL.

Title: Efficiently learning and sampling multimodal distributions with data-based initialization

• Authors: Frederic Koehler, Holden Lee, Thuy-Duong Vuong
• Subjects: cs.LG, cs.DS, math.PR, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.09117
• Pdf URL: https://arxiv.org/pdf/2411.09117
• Copy Paste: [[2411.09117]] Efficiently learning and sampling multimodal distributions with data-based initialization(https://arxiv.org/abs/2411.09117)
• Keywords: diffusion
• Abstract: We consider the problem of sampling a multimodal distribution with a Markov chain given a small number of samples from the stationary measure. Although mixing can be arbitrarily slow, we show that if the Markov chain has a $k$th order spectral gap, initialization from a set of $\tilde O(k/\varepsilon^2)$ samples from the stationary distribution will, with high probability over the samples, efficiently generate a sample whose conditional law is $\varepsilon$-close in TV distance to the stationary measure. In particular, this applies to mixtures of $k$ distributions satisfying a Poincaré inequality, with faster convergence when they satisfy a log-Sobolev inequality. Our bounds are stable to perturbations to the Markov chain, and in particular work for Langevin diffusion over $\mathbb R^d$ with score estimation error, as well as Glauber dynamics combined with approximation error from pseudolikelihood estimation. This justifies the success of data-based initialization for score matching methods despite slow mixing for the data distribution, and improves and generalizes the results of Koehler and Vuong (2023) to have linear, rather than exponential, dependence on $k$ and apply to arbitrary semigroups. As a consequence of our results, we show for the first time that a natural class of low-complexity Ising measures can be efficiently learned from samples.

Title: Neural Graph Simulator for Complex Systems

• Authors: Hoyun Choi, Sungyeop Lee, B. Kahng, Junghyo Jo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09120
• Pdf URL: https://arxiv.org/pdf/2411.09120
• Copy Paste: [[2411.09120]] Neural Graph Simulator for Complex Systems(https://arxiv.org/abs/2411.09120)
• Keywords: robust
• Abstract: Numerical simulation is a predominant tool for studying the dynamics in complex systems, but large-scale simulations are often intractable due to computational limitations. Here, we introduce the Neural Graph Simulator (NGS) for simulating time-invariant autonomous systems on graphs. Utilizing a graph neural network, the NGS provides a unified framework to simulate diverse dynamical systems with varying topologies and sizes without constraints on evaluation times through its non-uniform time step and autoregressive approach. The NGS offers significant advantages over numerical solvers by not requiring prior knowledge of governing equations and effectively handling noisy or missing data with a robust training scheme. It demonstrates superior computational efficiency over conventional methods, improving performance by over $10^5$ times in stiff problems. Furthermore, it is applied to real traffic data, forecasting traffic flow with state-of-the-art accuracy. The versatility of the NGS extends beyond the presented cases, offering numerous potential avenues for enhancement.

Title: DROJ: A Prompt-Driven Attack against Large Language Models

• Authors: Leyang Hu, Boran Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09125
• Pdf URL: https://arxiv.org/pdf/2411.09125
• Copy Paste: [[2411.09125]] DROJ: A Prompt-Driven Attack against Large Language Models(https://arxiv.org/abs/2411.09125)
• Keywords: attack, large language model
• Abstract: Large Language Models (LLMs) have demonstrated exceptional capabilities across various natural language processing tasks. Due to their training on internet-sourced datasets, LLMs can sometimes generate objectionable content, necessitating extensive alignment with human feedback to avoid such outputs. Despite massive alignment efforts, LLMs remain susceptible to adversarial jailbreak attacks, which usually are manipulated prompts designed to circumvent safety mechanisms and elicit harmful responses. Here, we introduce a novel approach, Directed Rrepresentation Optimization Jailbreak (DROJ), which optimizes jailbreak prompts at the embedding level to shift the hidden representations of harmful queries towards directions that are more likely to elicit affirmative responses from the model. Our evaluations on LLaMA-2-7b-chat model show that DROJ achieves a 100\% keyword-based Attack Success Rate (ASR), effectively preventing direct refusals. However, the model occasionally produces repetitive and non-informative responses. To mitigate this, we introduce a helpfulness system prompt that enhances the utility of the model's responses. Our code is available at this https URL.

Title: Adversarial Vessel-Unveiling Semi-Supervised Segmentation for Retinopathy of Prematurity Diagnosis

• Authors: Gozde Merve Demirci, Jiachen Yao, Ming-Chih Ho, Xiaoling Hu, Wei-Chi Wu, Chao Chen, Chia-Ling Tsai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09140
• Pdf URL: https://arxiv.org/pdf/2411.09140
• Copy Paste: [[2411.09140]] Adversarial Vessel-Unveiling Semi-Supervised Segmentation for Retinopathy of Prematurity Diagnosis(https://arxiv.org/abs/2411.09140)
• Keywords: robust, segmentation
• Abstract: Accurate segmentation of retinal images plays a crucial role in aiding ophthalmologists in diagnosing retinopathy of prematurity (ROP) and assessing its severity. However, due to their underdeveloped, thinner vessels, manual annotation in infant fundus images is very complex, and this presents challenges for fully supervised learning. To address the scarcity of annotations, we propose a semi supervised segmentation framework designed to advance ROP studies without the need for extensive manual vessel annotation. Unlike previous methods that rely solely on limited labeled data, our approach leverages teacher student learning by integrating two powerful components: an uncertainty weighted vessel unveiling module and domain adversarial learning. The vessel unveiling module helps the model effectively reveal obscured and hard to detect vessel structures, while adversarial training aligns feature representations across different domains, ensuring robust and generalizable vessel segmentations. We validate our approach on public datasets (CHASEDB, STARE) and an in-house ROP dataset, demonstrating its superior performance across multiple evaluation metrics. Additionally, we extend the model's utility to a downstream task of ROP multi-stage classification, where vessel masks extracted by our segmentation model improve diagnostic accuracy. The promising results in classification underscore the model's potential for clinical application, particularly in early-stage ROP diagnosis and intervention. Overall, our work offers a scalable solution for leveraging unlabeled data in pediatric ophthalmology, opening new avenues for biomarker discovery and clinical research.

Title: Laplace Transform Interpretation of Differential Privacy

• Authors: Rishav Chourasia, Uzair Javaid, Biplap Sikdar
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.09142
• Pdf URL: https://arxiv.org/pdf/2411.09142
• Copy Paste: [[2411.09142]] Laplace Transform Interpretation of Differential Privacy(https://arxiv.org/abs/2411.09142)
• Keywords: privacy
• Abstract: We introduce a set of useful expressions of Differential Privacy (DP) notions in terms of the Laplace transform of the privacy loss distribution. Its bare form expression appears in several related works on analyzing DP, either as an integral or an expectation. We show that recognizing the expression as a Laplace transform unlocks a new way to reason about DP properties by exploiting the duality between time and frequency domains. Leveraging our interpretation, we connect the $(q, \rho(q))$-Rényi DP curve and the $(\epsilon, \delta(\epsilon))$-DP curve as being the Laplace and inverse-Laplace transforms of one another. This connection shows that the Rényi divergence is well-defined for complex orders $q = \gamma + i \omega$. Using our Laplace transform-based analysis, we also prove an adaptive composition theorem for $(\epsilon, \delta)$-DP guarantees that is exactly tight (i.e., matches even in constants) for all values of $\epsilon$. Additionally, we resolve an issue regarding symmetry of $f$-DP on subsampling that prevented equivalence across all functional DP notions.

Title: VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation

• Authors: Youpeng Wen, Junfan Lin, Yi Zhu, Jianhua Han, Hang Xu, Shen Zhao, Xiaodan Liang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.09153
• Pdf URL: https://arxiv.org/pdf/2411.09153
• Copy Paste: [[2411.09153]] VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation(https://arxiv.org/abs/2411.09153)
• Keywords: diffusion
• Abstract: Recent advancements utilizing large-scale video data for learning video generation models demonstrate significant potential in understanding complex physical dynamics. It suggests the feasibility of leveraging diverse robot trajectory data to develop a unified, dynamics-aware model to enhance robot manipulation. However, given the relatively small amount of available robot data, directly fitting data without considering the relationship between visual observations and actions could lead to suboptimal data utilization. To this end, we propose VidMan (Video Diffusion for Robot Manipulation), a novel framework that employs a two-stage training mechanism inspired by dual-process theory from neuroscience to enhance stability and improve data utilization efficiency. Specifically, in the first stage, VidMan is pre-trained on the Open X-Embodiment dataset (OXE) for predicting future visual trajectories in a video denoising diffusion manner, enabling the model to develop a long horizontal awareness of the environment's dynamics. In the second stage, a flexible yet effective layer-wise self-attention adapter is introduced to transform VidMan into an efficient inverse dynamics model that predicts action modulated by the implicit dynamics knowledge via parameter sharing. Our VidMan framework outperforms state-of-the-art baseline model GR-1 on the CALVIN benchmark, achieving a 11.7% relative improvement, and demonstrates over 9% precision gains on the OXE small-scale dataset. These results provide compelling evidence that world models can significantly enhance the precision of robot action prediction. Codes and models will be public.

Title: DyGASR: Dynamic Generalized Exponential Splatting with Surface Alignment for Accelerated 3D Mesh Reconstruction

• Authors: Shengchao Zhao, Yundong Li
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.09156
• Pdf URL: https://arxiv.org/pdf/2411.09156
• Copy Paste: [[2411.09156]] DyGASR: Dynamic Generalized Exponential Splatting with Surface Alignment for Accelerated 3D Mesh Reconstruction(https://arxiv.org/abs/2411.09156)
• Keywords: extraction
• Abstract: Recent advancements in 3D Gaussian Splatting (3DGS), which lead to high-quality novel view synthesis and accelerated rendering, have remarkably improved the quality of radiance field reconstruction. However, the extraction of mesh from a massive number of minute 3D Gaussian points remains great challenge due to the large volume of Gaussians and difficulty of representation of sharp signals caused by their inherent low-pass characteristics. To address this issue, we propose DyGASR, which utilizes generalized exponential function instead of traditional 3D Gaussian to decrease the number of particles and dynamically optimize the representation of the captured signal. In addition, it is observed that reconstructing mesh with Generalized Exponential Splatting(GES) without modifications frequently leads to failures since the generalized exponential distribution centroids may not precisely align with the scene surface. To overcome this, we adopt Sugar's approach and introduce Generalized Surface Regularization (GSR), which reduces the smallest scaling vector of each point cloud to zero and ensures normal alignment perpendicular to the surface, facilitating subsequent Poisson surface mesh reconstruction. Additionally, we propose a dynamic resolution adjustment strategy that utilizes a cosine schedule to gradually increase image resolution from low to high during the training stage, thus avoiding constant full resolution, which significantly boosts the reconstruction speed. Our approach surpasses existing 3DGS-based mesh reconstruction methods, as evidenced by extensive evaluations on various scene datasets, demonstrating a 25\% increase in speed, and a 30\% reduction in memory usage.

Title: Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey

• Authors: Longxuan Ma, Mingda Li, Weinan Zhang, Jiapeng Li, Ting Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09166
• Pdf URL: https://arxiv.org/pdf/2411.09166
• Copy Paste: [[2411.09166]] Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey(https://arxiv.org/abs/2411.09166)
• Keywords: generative
• Abstract: Incorporating external knowledge into dialogue generation has been proven to benefit the performance of an open-domain Dialogue System (DS), such as generating informative or stylized responses, controlling conversation topics. In this article, we study the open-domain DS that uses unstructured text as external knowledge sources (\textbf{U}nstructured \textbf{T}ext \textbf{E}nhanced \textbf{D}ialogue \textbf{S}ystem, \textbf{UTEDS}). The existence of unstructured text entails distinctions between UTEDS and traditional data-driven DS and we aim to analyze these differences. We first give the definition of the UTEDS related concepts, then summarize the recently released datasets and models. We categorize UTEDS into Retrieval and Generative models and introduce them from the perspective of model components. The retrieval models consist of Fusion, Matching, and Ranking modules, while the generative models comprise Dialogue and Knowledge Encoding, Knowledge Selection, and Response Generation modules. We further summarize the evaluation methods utilized in UTEDS and analyze the current models' performance. At last, we discuss the future development trends of UTEDS, hoping to inspire new research in this field.

Title: Towards Scalable Handwriting Communication via EEG Decoding and Latent Embedding Integration

• Authors: Jun-Young Kim, Deok-Seon Kim, Seo-Hyun Lee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09170
• Pdf URL: https://arxiv.org/pdf/2411.09170
• Copy Paste: [[2411.09170]] Towards Scalable Handwriting Communication via EEG Decoding and Latent Embedding Integration(https://arxiv.org/abs/2411.09170)
• Keywords: extraction
• Abstract: In recent years, brain-computer interfaces have made advances in decoding various motor-related tasks, including gesture recognition and movement classification, utilizing electroencephalogram (EEG) data. These developments are fundamental in exploring how neural signals can be interpreted to recognize specific physical actions. This study centers on a written alphabet classification task, where we aim to decode EEG signals associated with handwriting. To achieve this, we incorporate hand kinematics to guide the extraction of the consistent embeddings from high-dimensional neural recordings using auxiliary variables (CEBRA). These CEBRA embeddings, along with the EEG, are processed by a parallel convolutional neural network model that extracts features from both data sources simultaneously. The model classifies nine different handwritten characters, including symbols such as exclamation marks and commas, within the alphabet. We evaluate the model using a quantitative five-fold cross-validation approach and explore the structure of the embedding space through visualizations. Our approach achieves a classification accuracy of 91 % for the nine-class task, demonstrating the feasibility of fine-grained handwriting decoding from EEG.

Title: Advancing Diffusion Models: Alias-Free Resampling and Enhanced Rotational Equivariance

• Authors: Md Fahim Anjum
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.09174
• Pdf URL: https://arxiv.org/pdf/2411.09174
• Copy Paste: [[2411.09174]] Advancing Diffusion Models: Alias-Free Resampling and Enhanced Rotational Equivariance(https://arxiv.org/abs/2411.09174)
• Keywords: diffusion, generative
• Abstract: Recent advances in image generation, particularly via diffusion models, have led to impressive improvements in image synthesis quality. Despite this, diffusion models are still challenged by model-induced artifacts and limited stability in image fidelity. In this work, we hypothesize that the primary cause of this issue is the improper resampling operation that introduces aliasing in the diffusion model and a careful alias-free resampling dictated by image processing theory can improve the model's performance in image synthesis. We propose the integration of alias-free resampling layers into the UNet architecture of diffusion models without adding extra trainable parameters, thereby maintaining computational efficiency. We then assess whether these theory-driven modifications enhance image quality and rotational equivariance. Our experimental results on benchmark datasets, including CIFAR-10, MNIST, and MNIST-M, reveal consistent gains in image quality, particularly in terms of FID and KID scores. Furthermore, we propose a modified diffusion process that enables user-controlled rotation of generated images without requiring additional training. Our findings highlight the potential of theory-driven enhancements such as alias-free resampling in generative models to improve image quality while maintaining model efficiency and pioneer future research directions to incorporate them into video-generating diffusion models, enabling deeper exploration of the applications of alias-free resampling in generative modeling.

Title: SAFES: Sequential Privacy and Fairness Enhancing Data Synthesis for Responsible AI

• Authors: Spencer Giddens, Fang Liu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.09178
• Pdf URL: https://arxiv.org/pdf/2411.09178
• Copy Paste: [[2411.09178]] SAFES: Sequential Privacy and Fairness Enhancing Data Synthesis for Responsible AI(https://arxiv.org/abs/2411.09178)
• Keywords: privacy, robust, fair
• Abstract: As data-driven and AI-based decision making gains widespread adoption in most disciplines, it is crucial that both data privacy and decision fairness are appropriately addressed. While differential privacy (DP) provides a robust framework for guaranteeing privacy and several widely accepted methods have been proposed for improving fairness, the vast majority of existing literature treats the two concerns independently. For methods that do consider privacy and fairness simultaneously, they often only apply to a specific machine learning task, limiting their generalizability. In response, we introduce SAFES, a Sequential PrivAcy and Fairness Enhancing data Synthesis procedure that sequentially combines DP data synthesis with a fairness-aware data transformation. SAFES allows full control over the privacy-fairness-utility trade-off via tunable privacy and fairness parameters. We illustrate SAFES by combining AIM, a graphical model-based DP data synthesizer, with a popular fairness-aware data pre-processing transformation. Empirical evaluations on the Adult and COMPAS datasets demonstrate that for reasonable privacy loss, SAFES-generated synthetic data achieve significantly improved fairness metrics with relatively low utility loss.

Title: LEAP:D - A Novel Prompt-based Approach for Domain-Generalized Aerial Object Detection

• Authors: Chanyeong Park, Heegwang Kim, Joonki Paik
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09180
• Pdf URL: https://arxiv.org/pdf/2411.09180
• Copy Paste: [[2411.09180]] LEAP:D - A Novel Prompt-based Approach for Domain-Generalized Aerial Object Detection(https://arxiv.org/abs/2411.09180)
• Keywords: robust
• Abstract: Drone-captured images present significant challenges in object detection due to varying shooting conditions, which can alter object appearance and shape. Factors such as drone altitude, angle, and weather cause these variations, influencing the performance of object detection algorithms. To tackle these challenges, we introduce an innovative vision-language approach using learnable prompts. This shift from conventional manual prompts aims to reduce domain-specific knowledge interference, ultimately improving object detection capabilities. Furthermore, we streamline the training process with a one-step approach, updating the learnable prompt concurrently with model training, enhancing efficiency without compromising performance. Our study contributes to domain-generalized object detection by leveraging learnable prompts and optimizing training processes. This enhances model robustness and adaptability across diverse environments, leading to more effective aerial object detection.

Title: JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation

• Authors: Xuyang Cao, Sheng Shi, Jun Zhao, Yang Yao, Jintao Fei, Minyu Gao, Guoxin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09209
• Pdf URL: https://arxiv.org/pdf/2411.09209
• Copy Paste: [[2411.09209]] JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation(https://arxiv.org/abs/2411.09209)
• Keywords: diffusion, transformer
• Abstract: Audio-driven portrait animation has made significant advances with diffusion-based models, improving video quality and lipsync accuracy. However, the increasing complexity of these models has led to inefficiencies in training and inference, as well as constraints on video length and inter-frame continuity. In this paper, we propose JoyVASA, a diffusion-based method for generating facial dynamics and head motion in audio-driven facial animation. Specifically, in the first stage, we introduce a decoupled facial representation framework that separates dynamic facial expressions from static 3D facial representations. This decoupling allows the system to generate longer videos by combining any static 3D facial representation with dynamic motion sequences. Then, in the second stage, a diffusion transformer is trained to generate motion sequences directly from audio cues, independent of character identity. Finally, a generator trained in the first stage uses the 3D facial representation and the generated motion sequences as inputs to render high-quality animations. With the decoupled facial representation and the identity-independent motion generation process, JoyVASA extends beyond human portraits to animate animal faces seamlessly. The model is trained on a hybrid dataset of private Chinese and public English data, enabling multilingual support. Experimental results validate the effectiveness of our approach. Future work will focus on improving real-time performance and refining expression control, further expanding the applications in portrait animation. The code will be available at: this https URL.

Title: Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering

• Authors: Nghia Trung Ngo, Chien Van Nguyen, Franck Dernoncourt, Thien Huu Nguyen
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.09213
• Pdf URL: https://arxiv.org/pdf/2411.09213
• Copy Paste: [[2411.09213]] Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering(https://arxiv.org/abs/2411.09213)
• Keywords: robust, large language model
• Abstract: Retrieval-augmented generation (RAG) has emerged as a promising approach to enhance the performance of large language models (LLMs) in knowledge-intensive tasks such as those from medical domain. However, the sensitive nature of the medical domain necessitates a completely accurate and trustworthy system. While existing RAG benchmarks primarily focus on the standard retrieve-answer setting, they overlook many practical scenarios that measure crucial aspects of a reliable medical system. This paper addresses this gap by providing a comprehensive evaluation framework for medical question-answering (QA) systems in a RAG setting for these situations, including sufficiency, integration, and robustness. We introduce Medical Retrieval-Augmented Generation Benchmark (MedRGB) that provides various supplementary elements to four medical QA datasets for testing LLMs' ability to handle these specific scenarios. Utilizing MedRGB, we conduct extensive evaluations of both state-of-the-art commercial LLMs and open-source models across multiple retrieval conditions. Our experimental results reveals current models' limited ability to handle noise and misinformation in the retrieved documents. We further analyze the LLMs' reasoning processes to provides valuable insights and future directions for developing RAG systems in this critical medical domain.

Title: HateGPT: Unleashing GPT-3.5 Turbo to Combat Hate Speech on X

• Authors: Aniket Deroy, Subhankar Maity
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09214
• Pdf URL: https://arxiv.org/pdf/2411.09214
• Copy Paste: [[2411.09214]] HateGPT: Unleashing GPT-3.5 Turbo to Combat Hate Speech on X(https://arxiv.org/abs/2411.09214)
• Keywords: robust, large language model
• Abstract: The widespread use of social media platforms like Twitter and Facebook has enabled people of all ages to share their thoughts and experiences, leading to an immense accumulation of user-generated content. However, alongside the benefits, these platforms also face the challenge of managing hate speech and offensive content, which can undermine rational discourse and threaten democratic values. As a result, there is a growing need for automated methods to detect and mitigate such content, especially given the complexity of conversations that may require contextual analysis across multiple languages, including code-mixed languages like Hinglish, German-English, and Bangla. We participated in the English task where we have to classify English tweets into two categories namely Hate and Offensive and Non Hate-Offensive. In this work, we experiment with state-of-the-art large language models like GPT-3.5 Turbo via prompting to classify tweets into Hate and Offensive or Non Hate-Offensive. In this study, we evaluate the performance of a classification model using Macro-F1 scores across three distinct runs. The Macro-F1 score, which balances precision and recall across all classes, is used as the primary metric for model evaluation. The scores obtained are 0.756 for run 1, 0.751 for run 2, and 0.754 for run 3, indicating a high level of performance with minimal variance among the runs. The results suggest that the model consistently performs well in terms of precision and recall, with run 1 showing the highest performance. These findings highlight the robustness and reliability of the model across different runs.

Title: Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation

• Authors: Yuheng Shi, Minjing Dong, Chang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09219
• Pdf URL: https://arxiv.org/pdf/2411.09219
• Copy Paste: [[2411.09219]] Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation(https://arxiv.org/abs/2411.09219)
• Keywords: segmentation
• Abstract: While Contrastive Language-Image Pre-training (CLIP) has advanced open-vocabulary predictions, its performance on semantic segmentation remains suboptimal. This shortfall primarily stems from its spatial-invariant semantic features and constrained resolution. While previous adaptations addressed spatial invariance semantic by modifying the self-attention in CLIP's image encoder, the issue of limited resolution remains unexplored. Different from previous segment-then-splice methods that segment sub-images via a sliding window and splice the results, we introduce a splice-then-segment paradigm that incorporates Segment-Anything Model (SAM) to tackle the resolution issue since SAM excels at extracting fine-grained semantic correlations from high-resolution images. Specifically, we introduce Trident, a training-free framework that first splices features extracted by CLIP and DINO from sub-images, then leverages SAM's encoder to create a correlation matrix for global aggregation, enabling a broadened receptive field for effective segmentation. Besides, we propose a refinement strategy for CLIP's coarse segmentation outputs by transforming them into prompts for SAM, further enhancing the segmentation performance. Trident achieves a significant improvement in the mIoU across eight benchmarks compared with the current SOTA, increasing from 44.4 to this http URL is available at this https URL.

Title: Injection Attacks Against End-to-End Encrypted Applications

• Authors: Andrés Fábrega, Carolina Ortega Pérez, Armin Namavari, Ben Nassi, Rachit Agarwal, Thomas Ristenpart
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.09228
• Pdf URL: https://arxiv.org/pdf/2411.09228
• Copy Paste: [[2411.09228]] Injection Attacks Against End-to-End Encrypted Applications(https://arxiv.org/abs/2411.09228)
• Keywords: security, attack
• Abstract: We explore an emerging threat model for end-to-end (E2E) encrypted applications: an adversary sends chosen messages to a target client, thereby "injecting" adversarial content into the application state. Such state is subsequently encrypted and synchronized to an adversarially-visible storage. By observing the lengths of the resulting cloud-stored ciphertexts, the attacker backs out confidential information. We investigate this injection threat model in the context of state-of-the-art encrypted messaging applications that support E2E encrypted backups. We show proof-of-concept attacks that can recover information about E2E encrypted messages or attachments sent via WhatsApp, assuming the ability to compromise the target user's Google or Apple account (which gives access to encrypted backups). We also show weaknesses in Signal's encrypted backup design that would allow injection attacks to infer metadata including a target user's number of contacts and conversations, should the adversary somehow obtain access to the user's encrypted Signal backup. While we do not believe our results should be of immediate concern for users of these messaging applications, our results do suggest that more work is needed to build tools that enjoy strong E2E security guarantees.

Title: Efficient and Secure Cross-Domain Data-Sharing for Resource-Constrained Internet of Things

• Authors: Kexian Liu, Jianfeng Guan, Xiaolong Hu, Jianli Liu, Hongke Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.09229
• Pdf URL: https://arxiv.org/pdf/2411.09229
• Copy Paste: [[2411.09229]] Efficient and Secure Cross-Domain Data-Sharing for Resource-Constrained Internet of Things(https://arxiv.org/abs/2411.09229)
• Keywords: secure, security, privacy, attack
• Abstract: The growing complexity of Internet of Things (IoT) environments, particularly in cross-domain data sharing, presents significant security challenges. Existing data-sharing schemes often rely on computationally expensive cryptographic operations and centralized key management, limiting their effectiveness for resource-constrained devices. To address these issues, we propose an efficient, secure blockchain-based data-sharing scheme. First, our scheme adopts a distributed key generation method, which avoids single point of failure. This method also allows independent pseudonym generation and key updates, enhancing authentication flexibility while reducing computational overhead. Additionally, the scheme provides a complete data-sharing process, covering data uploading, storage, and sharing, while ensuring data traceability, integrity, and privacy. Security analysis shows that the proposed scheme is theoretically secure and resistant to various attacks, while performance evaluations demonstrate lower computational and communication overhead compared to existing solutions, making it both secure and efficient for IoT applications.

Title: AEAKA: An Adaptive and Efficient Authentication and Key Agreement Scheme for IoT in Cloud-Edge-Device Collaborative Environments

• Authors: Kexian Liu, Jianfeng Guan, Xiaolong Hu, Jing Zhang, Jianli Liu, Hongke Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.09231
• Pdf URL: https://arxiv.org/pdf/2411.09231
• Copy Paste: [[2411.09231]] AEAKA: An Adaptive and Efficient Authentication and Key Agreement Scheme for IoT in Cloud-Edge-Device Collaborative Environments(https://arxiv.org/abs/2411.09231)
• Keywords: security, privacy, defense
• Abstract: To meet the diverse needs of users, the rapid advancement of cloud-edge-device collaboration has become a standard practice. However, this complex environment, particularly in untrusted (non-collaborative) scenarios, presents numerous security challenges. Authentication acts as the first line of defense and is fundamental to addressing these issues. Although many authentication and key agreement schemes exist, they often face limitations, such as being tailored to overly specific scenarios where devices authenticate solely with either the edge or the cloud, or being unsuitable for resource-constrained devices. To address these challenges, we propose an adaptive and efficient authentication and key agreement scheme (AEAKA) for Cloud-Edge-Device IoT environments. This scheme is highly adaptive and scalable, capable of automatically and dynamically initiating different authentication methods based on device requirements. Additionally, it employs an edge-assisted authentication approach to reduce the load on third-party trust authorities. Furthermore, we introduce a hash-based algorithm for the authentication protocol, ensuring a lightweight method suitable for a wide range of resource-constrained devices while maintaining security. AEAKA ensures that entities use associated authentication credentials, enhancing the privacy of the authentication process. Security proofs and performance analyses demonstrate that AEAKA outperforms other methods in terms of security and authentication efficiency.

Title: Cybersecurity Study Programs: What's in a Name?

• Authors: Jan Vykopal, Valdemar Švábenský, Michael Tuscano Lopez II, Pavel Čeleda
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.09240
• Pdf URL: https://arxiv.org/pdf/2411.09240
• Copy Paste: [[2411.09240]] Cybersecurity Study Programs: What's in a Name?(https://arxiv.org/abs/2411.09240)
• Keywords: security
• Abstract: Improving cybersecurity education has become a priority for many countries and organizations worldwide. Computing societies and professional associations have recognized cybersecurity as a distinctive computing discipline and created specialized cybersecurity curricular guidelines. Higher education institutions are introducing new cybersecurity programs, attracting students to this expanding field. In this paper, we examined 101 study programs across 24 countries. Based on their analysis, we argue that top-ranked universities have not yet fully implemented the guidelines and offer programs that have "cyber" in their name but lack some essential elements of a cybersecurity program. In particular, most programs do not sufficiently cover non-technical components, such as law, policies, or risk management. Also, most programs teach knowledge and skills but do not expose students to experiential learning outside the traditional classroom (such as internships) to develop their competencies. As a result, graduates of these programs may not meet employer expectations and may require additional training. To help program directors and educators improve their programs and courses, this paper offers examples of effective practices from cybersecurity programs around the world and our teaching practice.

Title: Enhancing Financial Domain Adaptation of Language Models via Model Augmentation

• Authors: Kota Tanabe, Masanori Hirano, Kazuki Matoya, Kentaro Imajo, Hiroki Sakaji, Itsuki Noda
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09249
• Pdf URL: https://arxiv.org/pdf/2411.09249
• Copy Paste: [[2411.09249]] Enhancing Financial Domain Adaptation of Language Models via Model Augmentation(https://arxiv.org/abs/2411.09249)
• Keywords: large language model
• Abstract: The domain adaptation of language models, including large language models (LLMs), has become increasingly important as the use of such models continues to expand. This study demonstrates the effectiveness of Composition to Augment Language Models (CALM) in adapting to the financial domain. CALM is a model to extend the capabilities of existing models by introducing cross-attention between two LLMs with different functions. In our experiments, we developed a CALM to enhance the financial performance of an LLM with strong response capabilities by leveraging a financial-specialized LLM. Notably, the CALM was trained using a financial dataset different from the one used to train the financial-specialized LLM, confirming CALM's ability to adapt to various datasets. The models were evaluated through quantitative Japanese financial benchmarks and qualitative response comparisons, demonstrating that CALM enables superior responses with higher scores than the original models and baselines. Additionally, comparative experiments on connection points revealed that connecting the middle layers of the models is most effective in facilitating adaptation to the financial domain. These findings confirm that CALM is a practical approach for adapting LLMs to the financial domain.

Title: DAHL: Domain-specific Automated Hallucination Evaluation of Long-Form Text through a Benchmark Dataset in Biomedicine

• Authors: Jean Seo, Jongwon Lim, Dongjun Jang, Hyopil Shin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09255
• Pdf URL: https://arxiv.org/pdf/2411.09255
• Copy Paste: [[2411.09255]] DAHL: Domain-specific Automated Hallucination Evaluation of Long-Form Text through a Benchmark Dataset in Biomedicine(https://arxiv.org/abs/2411.09255)
• Keywords: large language model
• Abstract: We introduce DAHL, a benchmark dataset and automated evaluation system designed to assess hallucination in long-form text generation, specifically within the biomedical domain. Our benchmark dataset, meticulously curated from biomedical research papers, consists of 8,573 questions across 29 categories. DAHL evaluates fact-conflicting hallucinations in Large Language Models (LLMs) by deconstructing responses into atomic units, each representing a single piece of information. The accuracy of these responses is averaged to produce the DAHL Score, offering a more in-depth evaluation of hallucinations compared to previous methods that rely on multiple-choice tasks. We conduct experiments with 8 different models, finding that larger models tend to hallucinate less; however, beyond a model size of 7 to 8 billion parameters, further scaling does not significantly improve factual accuracy. The DAHL Score holds potential as an efficient alternative to human-annotated preference labels, being able to be expanded to other specialized domains. We release the dataset and code in public.

Title: Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey

• Authors: Xuannan Liu, Xing Cui, Peipei Li, Zekun Li, Huaibo Huang, Shuhan Xia, Miaoxuan Zhang, Yueying Zou, Ran He
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09259
• Pdf URL: https://arxiv.org/pdf/2411.09259
• Copy Paste: [[2411.09259]] Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey(https://arxiv.org/abs/2411.09259)
• Keywords: security, defense, attack, generative
• Abstract: The rapid evolution of multimodal foundation models has led to significant advancements in cross-modal understanding and generation across diverse modalities, including text, images, audio, and video. However, these models remain susceptible to jailbreak attacks, which can bypass built-in safety mechanisms and induce the production of potentially harmful content. Consequently, understanding the methods of jailbreak attacks and existing defense mechanisms is essential to ensure the safe deployment of multimodal generative models in real-world scenarios, particularly in security-sensitive applications. To provide comprehensive insight into this topic, this survey reviews jailbreak and defense in multimodal generative models. First, given the generalized lifecycle of multimodal jailbreak, we systematically explore attacks and corresponding defense strategies across four levels: input, encoder, generator, and output. Based on this analysis, we present a detailed taxonomy of attack methods, defense mechanisms, and evaluation frameworks specific to multimodal generative models. Additionally, we cover a wide range of input-output configurations, including modalities such as Any-to-Text, Any-to-Vision, and Any-to-Any within generative systems. Finally, we highlight current research challenges and propose potential directions for future this http URL open-source repository corresponding to this work can be found at this https URL.

Title: Rethinking Weight-Averaged Model-merging

• Authors: Hu Wang, Congbo Ma, Ibrahim Almakky, Ian Reid, Gustavo Carneiro, Mohammad Yaqub
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09263
• Pdf URL: https://arxiv.org/pdf/2411.09263
• Copy Paste: [[2411.09263]] Rethinking Weight-Averaged Model-merging(https://arxiv.org/abs/2411.09263)
• Keywords: robust
• Abstract: Weight-averaged model-merging has emerged as a powerful approach in deep learning, capable of enhancing model performance without fine-tuning or retraining. However, the underlying mechanisms that explain its effectiveness remain largely unexplored. In this paper, we investigate this technique from three novel perspectives to provide deeper insights into how and why weight-averaged model-merging works: (1) we examine the intrinsic patterns captured by the learning of the model weights, through the visualizations of their patterns on several datasets, showing that these weights often encode structured and interpretable patterns; (2) we investigate model ensemble merging strategies based on averaging on weights versus averaging on features, providing detailed analyses across diverse architectures and datasets; and (3) we explore the impact on model-merging prediction stability in terms of changing the parameter magnitude, revealing insights into the way of weight averaging works as regularization by showing the robustness across different parameter scales. Our findings shed light on the "black box" of weight-averaged model-merging, offering valuable insights and practical recommendations that advance the model-merging process.

Title: BEARD: Benchmarking the Adversarial Robustness for Dataset Distillation

• Authors: Zheng Zhou, Wenquan Feng, Shuchang Lyu, Guangliang Cheng, Xiaowei Huang, Qi Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09265
• Pdf URL: https://arxiv.org/pdf/2411.09265
• Copy Paste: [[2411.09265]] BEARD: Benchmarking the Adversarial Robustness for Dataset Distillation(https://arxiv.org/abs/2411.09265)
• Keywords: security, attack, robust
• Abstract: Dataset Distillation (DD) is an emerging technique that compresses large-scale datasets into significantly smaller synthesized datasets while preserving high test performance and enabling the efficient training of large models. However, current research primarily focuses on enhancing evaluation accuracy under limited compression ratios, often overlooking critical security concerns such as adversarial robustness. A key challenge in evaluating this robustness lies in the complex interactions between distillation methods, model architectures, and adversarial attack strategies, which complicate standardized assessments. To address this, we introduce BEARD, an open and unified benchmark designed to systematically assess the adversarial robustness of DD methods, including DM, IDM, and BACON. BEARD encompasses a variety of adversarial attacks (e.g., FGSM, PGD, C&W) on distilled datasets like CIFAR-10/100 and TinyImageNet. Utilizing an adversarial game framework, it introduces three key metrics: Robustness Ratio (RR), Attack Efficiency Ratio (AE), and Comprehensive Robustness-Efficiency Index (CREI). Our analysis includes unified benchmarks, various Images Per Class (IPC) settings, and the effects of adversarial training. Results are available on the BEARD Leaderboard, along with a library providing model and dataset pools to support reproducible research. Access the code at BEARD.

Title: How Good is ChatGPT at Audiovisual Deepfake Detection: A Comparative Study of ChatGPT, AI Models and Human Perception

• Authors: Sahibzada Adil Shahzad, Ammarah Hashmi, Yan-Tsung Peng, Yu Tsao, Hsin-Min Wang
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2411.09266
• Pdf URL: https://arxiv.org/pdf/2411.09266
• Copy Paste: [[2411.09266]] How Good is ChatGPT at Audiovisual Deepfake Detection: A Comparative Study of ChatGPT, AI Models and Human Perception(https://arxiv.org/abs/2411.09266)
• Keywords: interpretability, large language model
• Abstract: Multimodal deepfakes involving audiovisual manipulations are a growing threat because they are difficult to detect with the naked eye or using unimodal deep learningbased forgery detection methods. Audiovisual forensic models, while more capable than unimodal models, require large training datasets and are computationally expensive for training and inference. Furthermore, these models lack interpretability and often do not generalize well to unseen manipulations. In this study, we examine the detection capabilities of a large language model (LLM) (i.e., ChatGPT) to identify and account for any possible visual and auditory artifacts and manipulations in audiovisual deepfake content. Extensive experiments are conducted on videos from a benchmark multimodal deepfake dataset to evaluate the detection performance of ChatGPT and compare it with the detection capabilities of state-of-the-art multimodal forensic models and humans. Experimental results demonstrate the importance of domain knowledge and prompt engineering for video forgery detection tasks using LLMs. Unlike approaches based on end-to-end learning, ChatGPT can account for spatial and spatiotemporal artifacts and inconsistencies that may exist within or across modalities. Additionally, we discuss the limitations of ChatGPT for multimedia forensic tasks.

Title: Towards efficient compression and communication for prototype-based decentralized learning

• Authors: Pablo Fernández-Piñeiro, Manuel Ferández-Veiga, Rebeca P. Díaz-Redondo, Ana Fernández-Vilas, Martín González-Soto
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09267
• Pdf URL: https://arxiv.org/pdf/2411.09267
• Copy Paste: [[2411.09267]] Towards efficient compression and communication for prototype-based decentralized learning(https://arxiv.org/abs/2411.09267)
• Keywords: robust, federate
• Abstract: In prototype-based federated learning, the exchange of model parameters between clients and the master server is replaced by transmission of prototypes or quantized versions of the data samples to the aggregation server. A fully decentralized deployment of prototype- based learning, without a central agregartor of prototypes, is more robust upon network failures and reacts faster to changes in the statistical distribution of the data, suggesting potential advantages and quick adaptation in dynamic learning tasks, e.g., when the data sources are IoT devices or when data is non-iid. In this paper, we consider the problem of designing a communication-efficient decentralized learning system based on prototypes. We address the challenge of prototype redundancy by leveraging on a twofold data compression technique, i.e., sending only update messages if the prototypes are informationtheoretically useful (via the Jensen-Shannon distance), and using clustering on the prototypes to compress the update messages used in the gossip protocol. We also use parallel instead of sequential gossiping, and present an analysis of its age-of-information (AoI). Our experimental results show that, with these improvements, the communications load can be substantially reduced without decreasing the convergence rate of the learning algorithm.

Title: LES-Talker: Fine-Grained Emotion Editing for Talking Head Generation in Linear Emotion Space

• Authors: Guanwen Feng, Zhihao Qian, Yunan Li, Siyu Jin, Qiguang Miao, Chi-Man Pun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09268
• Pdf URL: https://arxiv.org/pdf/2411.09268
• Copy Paste: [[2411.09268]] LES-Talker: Fine-Grained Emotion Editing for Talking Head Generation in Linear Emotion Space(https://arxiv.org/abs/2411.09268)
• Keywords: interpretability
• Abstract: While existing one-shot talking head generation models have achieved progress in coarse-grained emotion editing, there is still a lack of fine-grained emotion editing models with high interpretability. We argue that for an approach to be considered fine-grained, it needs to provide clear definitions and sufficiently detailed differentiation. We present LES-Talker, a novel one-shot talking head generation model with high interpretability, to achieve fine-grained emotion editing across emotion types, emotion levels, and facial units. We propose a Linear Emotion Space (LES) definition based on Facial Action Units to characterize emotion transformations as vector transformations. We design the Cross-Dimension Attention Net (CDAN) to deeply mine the correlation between LES representation and 3D model representation. Through mining multiple relationships across different feature and structure dimensions, we enable LES representation to guide the controllable deformation of 3D model. In order to adapt the multimodal data with deviations to the LES and enhance visual quality, we utilize specialized network design and training strategies. Experiments show that our method provides high visual quality along with multilevel and interpretable fine-grained emotion editing, outperforming mainstream methods.

Title: Cross-Modal Consistency in Multimodal Large Language Models

• Authors: Xiang Zhang, Senyu Li, Ning Shi, Bradley Hauer, Zijun Wu, Grzegorz Kondrak, Muhammad Abdul-Mageed, Laks V.S. Lakshmanan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09273
• Pdf URL: https://arxiv.org/pdf/2411.09273
• Copy Paste: [[2411.09273]] Cross-Modal Consistency in Multimodal Large Language Models(https://arxiv.org/abs/2411.09273)
• Keywords: large language model
• Abstract: Recent developments in multimodal methodologies have marked the beginning of an exciting era for models adept at processing diverse data types, encompassing text, audio, and visual content. Models like GPT-4V, which merge computer vision with advanced language processing, exhibit extraordinary proficiency in handling intricate tasks that require a simultaneous understanding of both textual and visual information. Prior research efforts have meticulously evaluated the efficacy of these Vision Large Language Models (VLLMs) in various domains, including object detection, image captioning, and other related fields. However, existing analyses have often suffered from limitations, primarily centering on the isolated evaluation of each modality's performance while neglecting to explore their intricate cross-modal interactions. Specifically, the question of whether these models achieve the same level of accuracy when confronted with identical task instances across different modalities remains unanswered. In this study, we take the initiative to delve into the interaction and comparison among these modalities of interest by introducing a novel concept termed cross-modal consistency. Furthermore, we propose a quantitative evaluation framework founded on this concept. Our experimental findings, drawn from a curated collection of parallel vision-language datasets developed by us, unveil a pronounced inconsistency between the vision and language modalities within GPT-4V, despite its portrayal as a unified multimodal model. Our research yields insights into the appropriate utilization of such models and hints at potential avenues for enhancing their design.

Title: The Communication-Friendly Privacy-Preserving Machine Learning against Malicious Adversaries

• Authors: Tianpei Lu, Bingsheng Zhang, Lichun Li, Kui Ren
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.09287
• Pdf URL: https://arxiv.org/pdf/2411.09287
• Copy Paste: [[2411.09287]] The Communication-Friendly Privacy-Preserving Machine Learning against Malicious Adversaries(https://arxiv.org/abs/2411.09287)
• Keywords: secure, privacy, protect
• Abstract: With the increasing emphasis on privacy regulations, such as GDPR, protecting individual privacy and ensuring compliance have become critical concerns for both individuals and organizations. Privacy-preserving machine learning (PPML) is an innovative approach that allows for secure data analysis while safeguarding sensitive information. It enables organizations to extract valuable insights from data without compromising privacy. Secure multi-party computation (MPC) is a key tool in PPML, as it allows multiple parties to jointly compute functions without revealing their private inputs, making it essential in multi-server environments. We address the performance overhead of existing maliciously secure protocols, particularly in finite rings like $\mathbb{Z}_{2^\ell}$, by introducing an efficient protocol for secure linear function evaluation. We implement our maliciously secure MPC protocol on GPUs, significantly improving its efficiency and scalability. We extend the protocol to handle linear and non-linear layers, ensuring compatibility with a wide range of machine-learning models. Finally, we comprehensively evaluate machine learning models by integrating our protocol into the workflow, enabling secure and efficient inference across simple and complex models, such as convolutional neural networks (CNNs).

Title: StreamAdapter: Efficient Test Time Adaptation from Contextual Streams

• Authors: Dilxat Muhtar, Yelong Shen, Yaming Yang, Xiaodong Liu, Yadong Lu, Jianfeng Liu, Yuefeng Zhan, Hao Sun, Weiwei Deng, Feng Sun, Xueliang Zhang, Jianfeng Gao, Weizhu Chen, Qi Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09289
• Pdf URL: https://arxiv.org/pdf/2411.09289
• Copy Paste: [[2411.09289]] StreamAdapter: Efficient Test Time Adaptation from Contextual Streams(https://arxiv.org/abs/2411.09289)
• Keywords: large language model
• Abstract: In-context learning (ICL) allows large language models (LLMs) to adapt to new tasks directly from the given demonstrations without requiring gradient updates. While recent advances have expanded context windows to accommodate more demonstrations, this approach increases inference costs without necessarily improving performance. To mitigate these issues, We propose StreamAdapter, a novel approach that directly updates model parameters from context at test time, eliminating the need for explicit in-context demonstrations. StreamAdapter employs context mapping and weight absorption mechanisms to dynamically transform ICL demonstrations into parameter updates with minimal additional parameters. By reducing reliance on numerous in-context examples, StreamAdapter significantly reduce inference costs and allows for efficient inference with constant time complexity, regardless of demonstration count. Extensive experiments across diverse tasks and model architectures demonstrate that StreamAdapter achieves comparable or superior adaptation capability to ICL while requiring significantly fewer demonstrations. The superior task adaptation and context encoding capabilities of StreamAdapter on both language understanding and generation tasks provides a new perspective for adapting LLMs at test time using context, allowing for more efficient adaptation across scenarios and more cost-effective inference

Title: DTELS: Towards Dynamic Granularity of Timeline Summarization

• Authors: Chenlong Zhang, Tong Zhou, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, Jun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09297
• Pdf URL: https://arxiv.org/pdf/2411.09297
• Copy Paste: [[2411.09297]] DTELS: Towards Dynamic Granularity of Timeline Summarization(https://arxiv.org/abs/2411.09297)
• Keywords: large language model
• Abstract: The rapid proliferation of online news has posed significant challenges in tracking the continuous development of news topics. Traditional timeline summarization constructs a chronological summary of the events but often lacks the flexibility to meet the diverse granularity needs. To overcome this limitation, we introduce a new paradigm, Dynamic-granularity TimELine Summarization, (DTELS), which aims to construct adaptive timelines based on user instructions or requirements. This paper establishes a comprehensive benchmark for DTLES that includes: (1) an evaluation framework grounded in journalistic standards to assess the timeline quality across four dimensions: Informativeness, Granular Consistency, Factuality, and Coherence; (2) a large-scale, multi-source dataset with multiple granularity timeline annotations based on a consensus process to facilitate authority; (3) extensive experiments and analysis with two proposed solutions based on Large Language Models (LLMs) and existing state-of-the-art TLS methods. The experimental results demonstrate the effectiveness of LLM-based solutions. However, even the most advanced LLMs struggle to consistently generate timelines that are both informative and granularly consistent, highlighting the challenges of the DTELS task.

Title: LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation

• Authors: Zhenshi Li, Dilxat Muhtar, Feng Gu, Xueliang Zhang, Pengfeng Xiao, Guangjun He, Xiaoxiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09301
• Pdf URL: https://arxiv.org/pdf/2411.09301
• Copy Paste: [[2411.09301]] LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation(https://arxiv.org/abs/2411.09301)
• Keywords: large language model
• Abstract: Automatically and rapidly understanding Earth's surface is fundamental to our grasp of the living environment and informed decision-making. This underscores the need for a unified system with comprehensive capabilities in analyzing Earth's surface to address a wide range of human needs. The emergence of multimodal large language models (MLLMs) has great potential in boosting the efficiency and convenience of intelligent Earth observation. These models can engage in human-like conversations, serve as unified platforms for understanding images, follow diverse instructions, and provide insightful feedbacks. In this study, we introduce LHRS-Bot-Nova, an MLLM specialized in understanding remote sensing (RS) images, designed to expertly perform a wide range of RS understanding tasks aligned with human instructions. LHRS-Bot-Nova features an enhanced vision encoder and a novel bridge layer, enabling efficient visual compression and better language-vision alignment. To further enhance RS-oriented vision-language alignment, we propose a large-scale RS image-caption dataset, generated through feature-guided image recaptioning. Additionally, we introduce an instruction dataset specifically designed to improve spatial recognition abilities. Extensive experiments demonstrate superior performance of LHRS-Bot-Nova across various RS image understanding tasks. We also evaluate different MLLM performances in complex RS perception and instruction following using a complicated multi-choice question evaluation benchmark, providing a reliable guide for future model selection and improvement. Data, code, and models will be available at this https URL.

Title: Compression Method for Solar Polarization Spectra Collected from Hinode SOT/SP Observations

• Authors: Jargalmaa Batmunkh, Yusuke Iida, Takayoshi Oba, Haruhisa Iijima
• Subjects: cs.LG, astro-ph.IM, astro-ph.SR
• Abstract URL: https://arxiv.org/abs/2411.09311
• Pdf URL: https://arxiv.org/pdf/2411.09311
• Copy Paste: [[2411.09311]] Compression Method for Solar Polarization Spectra Collected from Hinode SOT/SP Observations(https://arxiv.org/abs/2411.09311)
• Keywords: robust
• Abstract: The complex structure and extensive details of solar spectral data, combined with a recent surge in volume, present significant processing challenges. To address this, we propose a deep learning-based compression technique using deep autoencoder (DAE) and 1D-convolutional autoencoder (CAE) models developed with Hinode SOT/SP data. We focused on compressing Stokes I and V polarization spectra from the quiet Sun, as well as from active regions, providing a novel insight into comprehensive spectral analysis by incorporating spectra from extreme magnetic fields. The results indicate that the CAE model outperforms the DAE model in reconstructing Stokes profiles, demonstrating greater robustness and achieving reconstruction errors around the observational noise level. The proposed method has proven effective in compressing Stokes I and V spectra from both the quiet Sun and active regions, highlighting its potential for impactful applications in solar spectral analysis, such as detection of unusual spectral signals.

Title: Approximate Probabilistic Inference forTime-Series Data A Robust Latent Gaussian Model With Temporal Awareness

• Authors: Anton Johansson, Arunselvan Ramaswamy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09312
• Pdf URL: https://arxiv.org/pdf/2411.09312
• Copy Paste: [[2411.09312]] Approximate Probabilistic Inference forTime-Series Data A Robust Latent Gaussian Model With Temporal Awareness(https://arxiv.org/abs/2411.09312)
• Keywords: robust, generative
• Abstract: The development of robust generative models for highly varied non-stationary time series data is a complex yet important problem. Traditional models for time series data prediction, such as Long Short-Term Memory (LSTM), are inefficient and generalize poorly as they cannot capture complex temporal relationships. In this paper, we present a probabilistic generative model that can be trained to capture temporal information, and that is robust to data errors. We call it Time Deep Latent Gaussian Model (tDLGM). Its novel architecture is inspired by Deep Latent Gaussian Model (DLGM). Our model is trained to minimize a loss function based on the negative log loss. One contributing factor to Time Deep Latent Gaussian Model (tDLGM) robustness is our regularizer, which accounts for data trends. Experiments conducted show that tDLGM is able to reconstruct and generate complex time series data, and that it is robust against to noise and faulty data.

Title: DriveThru: a Document Extraction Platform and Benchmark Datasets for Indonesian Local Language Archives

• Authors: MohammadRifqi Farhansyah, Muhammad Zuhdi Fikri Johari, Afinzaki Amiral, Ayu Purwarianti, Kumara Ari Yuana, Derry Tanti Wijaya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09318
• Pdf URL: https://arxiv.org/pdf/2411.09318
• Copy Paste: [[2411.09318]] DriveThru: a Document Extraction Platform and Benchmark Datasets for Indonesian Local Language Archives(https://arxiv.org/abs/2411.09318)
• Keywords: extraction
• Abstract: Indonesia is one of the most diverse countries linguistically. However, despite this linguistic diversity, Indonesian languages remain underrepresented in Natural Language Processing (NLP) research and technologies. In the past two years, several efforts have been conducted to construct NLP resources for Indonesian languages. However, most of these efforts have been focused on creating manual resources thus difficult to scale to more languages. Although many Indonesian languages do not have a web presence, locally there are resources that document these languages well in printed forms such as books, magazines, and newspapers. Digitizing these existing resources will enable scaling of Indonesian language resource construction to many more languages. In this paper, we propose an alternative method of creating datasets by digitizing documents, which have not previously been used to build digital language resources in Indonesia. DriveThru is a platform for extracting document content utilizing Optical Character Recognition (OCR) techniques in its system to provide language resource building with less manual effort and cost. This paper also studies the utility of current state-of-the-art LLM for post-OCR correction to show the capability of increasing the character accuracy rate (CAR) and word accuracy rate (WAR) compared to off-the-shelf OCR.

Title: Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment

• Authors: Yuang Cai, Yuyu Yuan, Jinsheng Shi, Qinhong Lin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09341
• Pdf URL: https://arxiv.org/pdf/2411.09341
• Copy Paste: [[2411.09341]] Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment(https://arxiv.org/abs/2411.09341)
• Keywords: large language model
• Abstract: The alignment of large language models (LLMs) is crucial for generating helpful and harmless content. Existing approaches leverage preference-based human feedback data to learn the reward function and align the LLM with the feedback data. However, these approaches focus on modeling the reward difference between the chosen and rejected demonstrations, rather than directly modeling the true reward from each demonstration. Moreover, these approaches assume that the reward is only obtained at the end of the sentence, which overlooks the modeling of intermediate rewards. These issues lead to insufficient use of training signals in the feedback data, limiting the representation and generalization ability of the reward and potentially resulting in reward hacking. In this paper, we formulate LLM alignment as a Bayesian Inverse Reinforcement Learning (BIRL) problem and propose a novel training objective, Approximated Variational Alignment (AVA), to perform LLM alignment through Approximated Variational Reward Imitation Learning (AVRIL). The BIRL formulation facilitates intermediate reward modeling and direct reward modeling on each single demonstration, which enhances the utilization of training signals in the feedback data. Experiments show that AVA outperforms existing LLM alignment approaches in reward modeling, RL fine-tuning, and direct optimization.

Title: Adaptively Augmented Consistency Learning: A Semi-supervised Segmentation Framework for Remote Sensing

• Authors: Hui Ye, Haodong Chen, Xiaoming Chen, Vera Chung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09344
• Pdf URL: https://arxiv.org/pdf/2411.09344
• Copy Paste: [[2411.09344]] Adaptively Augmented Consistency Learning: A Semi-supervised Segmentation Framework for Remote Sensing(https://arxiv.org/abs/2411.09344)
• Keywords: segmentation
• Abstract: Remote sensing (RS) involves the acquisition of data about objects or areas from a distance, primarily to monitor environmental changes, manage resources, and support planning and disaster response. A significant challenge in RS segmentation is the scarcity of high-quality labeled images due to the diversity and complexity of RS image, which makes pixel-level annotation difficult and hinders the development of effective supervised segmentation algorithms. To solve this problem, we propose Adaptively Augmented Consistency Learning (AACL), a semi-supervised segmentation framework designed to enhances RS segmentation accuracy under condictions of limited labeled data. AACL extracts additional information embedded in unlabeled images through the use of Uniform Strength Augmentation (USAug) and Adaptive Cut-Mix (AdaCM). Evaluations across various RS datasets demonstrate that AACL achieves competitive performance in semi-supervised segmentation, showing up to a 20% improvement in specific categories and 2% increase in overall performance compared to state-of-the-art frameworks.

Title: Your Fixed Watermark is Fragile: Towards Semantic-Aware Watermark for EaaS Copyright Protection

• Authors: Zekun Fei, Biao Yi, Jianing Geng, Ruiqi He, Lihai Nie, Zheli Liu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09359
• Pdf URL: https://arxiv.org/pdf/2411.09359
• Copy Paste: [[2411.09359]] Your Fixed Watermark is Fragile: Towards Semantic-Aware Watermark for EaaS Copyright Protection(https://arxiv.org/abs/2411.09359)
• Keywords: protect, defense, attack, robust, watermark
• Abstract: Embedding-as-a-Service (EaaS) has emerged as a successful business pattern but faces significant challenges related to various forms of copyright infringement, including API misuse and different attacks. Various studies have proposed backdoor-based watermarking schemes to protect the copyright of EaaS services. In this paper, we reveal that previous watermarking schemes possess semantic-independent characteristics and propose the Semantic Perturbation Attack (SPA). Our theoretical and experimental analyses demonstrate that this semantic-independent nature makes current watermarking schemes vulnerable to adaptive attacks that exploit semantic perturbations test to bypass watermark verification. To address this vulnerability, we propose the Semantic Aware Watermarking (SAW) scheme, a robust defense mechanism designed to resist SPA, by injecting a watermark that adapts to the text semantics. Extensive experimental results across multiple datasets demonstrate that the True Positive Rate (TPR) for detecting watermarked samples under SPA can reach up to more than 95%, rendering previous watermarks ineffective. Meanwhile, our watermarking scheme can resist such attack while ensuring the watermark verification capability. Our code is available at this https URL.

Title: Stability and Generalization for Distributed SGDA

• Authors: Miaoxi Zhu, Yan Sun, Li Shen, Bo Du, Dacheng Tao
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2411.09365
• Pdf URL: https://arxiv.org/pdf/2411.09365
• Copy Paste: [[2411.09365]] Stability and Generalization for Distributed SGDA(https://arxiv.org/abs/2411.09365)
• Keywords: privacy
• Abstract: Minimax optimization is gaining increasing attention in modern machine learning applications. Driven by large-scale models and massive volumes of data collected from edge devices, as well as the concern to preserve client privacy, communication-efficient distributed minimax optimization algorithms become popular, such as Local Stochastic Gradient Descent Ascent (Local-SGDA), and Local Decentralized SGDA (Local-DSGDA). While most existing research on distributed minimax algorithms focuses on convergence rates, computation complexity, and communication efficiency, the generalization performance remains underdeveloped, whereas generalization ability is a pivotal indicator for evaluating the holistic performance of a model when fed with unknown data. In this paper, we propose the stability-based generalization analytical framework for Distributed-SGDA, which unifies two popular distributed minimax algorithms including Local-SGDA and Local-DSGDA, and conduct a comprehensive analysis of stability error, generalization gap, and population risk across different metrics under various settings, e.g., (S)C-(S)C, PL-SC, and NC-NC cases. Our theoretical results reveal the trade-off between the generalization gap and optimization error and suggest hyperparameters choice to obtain the optimal population risk. Numerical experiments for Local-SGDA and Local-DSGDA validate the theoretical results.

Title: DSCformer: A Dual-Branch Network Integrating Enhanced Dynamic Snake Convolution and SegFormer for Crack Segmentation

• Authors: Kaiwei Yu, I-Ming Chen, Jing Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09371
• Pdf URL: https://arxiv.org/pdf/2411.09371
• Copy Paste: [[2411.09371]] DSCformer: A Dual-Branch Network Integrating Enhanced Dynamic Snake Convolution and SegFormer for Crack Segmentation(https://arxiv.org/abs/2411.09371)
• Keywords: extraction, transformer, segmentation
• Abstract: In construction quality monitoring, accurately detecting and segmenting cracks in concrete structures is paramount for safety and maintenance. Current convolutional neural networks (CNNs) have demonstrated strong performance in crack segmentation tasks, yet they often struggle with complex backgrounds and fail to capture fine-grained tubular structures fully. In contrast, Transformers excel at capturing global context but lack precision in detailed feature extraction. We introduce DSCformer, a novel hybrid model that integrates an enhanced Dynamic Snake Convolution (DSConv) with a Transformer architecture for crack segmentation to address these challenges. Our key contributions include the enhanced DSConv through a pyramid kernel for adaptive offset computation and a simultaneous bi-directional learnable offset iteration, significantly improving the model's performance to capture intricate crack patterns. Additionally, we propose a Weighted Convolutional Attention Module (WCAM), which refines channel attention, allowing for more precise and adaptive feature attention. We evaluate DSCformer on the Crack3238 and FIND datasets, achieving IoUs of 59.22\% and 87.24\%, respectively. The experimental results suggest that our DSCformer outperforms state-of-the-art methods across different datasets.

Title: Instruction-Driven Fusion of Infrared-Visible Images: Tailoring for Diverse Downstream Tasks

• Authors: Zengyi Yang, Yafei Zhang, Huafeng Li, Yu Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09387
• Pdf URL: https://arxiv.org/pdf/2411.09387
• Copy Paste: [[2411.09387]] Instruction-Driven Fusion of Infrared-Visible Images: Tailoring for Diverse Downstream Tasks(https://arxiv.org/abs/2411.09387)
• Keywords: extraction, segmentation
• Abstract: The primary value of infrared and visible image fusion technology lies in applying the fusion results to downstream tasks. However, existing methods face challenges such as increased training complexity and significantly compromised performance of individual tasks when addressing multiple downstream tasks simultaneously. To tackle this, we propose Task-Oriented Adaptive Regulation (T-OAR), an adaptive mechanism specifically designed for multi-task environments. Additionally, we introduce the Task-related Dynamic Prompt Injection (T-DPI) module, which generates task-specific dynamic prompts from user-input text instructions and integrates them into target representations. This guides the feature extraction module to produce representations that are more closely aligned with the specific requirements of downstream tasks. By incorporating the T-DPI module into the T-OAR framework, our approach generates fusion images tailored to task-specific requirements without the need for separate training or task-specific weights. This not only reduces computational costs but also enhances adaptability and performance across multiple tasks. Experimental results show that our method excels in object detection, semantic segmentation, and salient object detection, demonstrating its strong adaptability, flexibility, and task specificity. This provides an efficient solution for image fusion in multi-task environments, highlighting the technology's potential across diverse applications.

Title: A survey of probabilistic generative frameworks for molecular simulations

• Authors: Richard John, Lukas Herron, Pratyush Tiwary
• Subjects: cs.LG, cond-mat.dis-nn, cond-mat.soft, cond-mat.stat-mech
• Abstract URL: https://arxiv.org/abs/2411.09388
• Pdf URL: https://arxiv.org/pdf/2411.09388
• Copy Paste: [[2411.09388]] A survey of probabilistic generative frameworks for molecular simulations(https://arxiv.org/abs/2411.09388)
• Keywords: diffusion, generative
• Abstract: Generative artificial intelligence is now a widely used tool in molecular science. Despite the popularity of probabilistic generative models, numerical experiments benchmarking their performance on molecular data are lacking. In this work, we introduce and explain several classes of generative models, broadly sorted into two categories: flow-based models and diffusion models. We select three representative models: Neural Spline Flows, Conditional Flow Matching, and Denoising Diffusion Probabilistic Models, and examine their accuracy, computational cost, and generation speed across datasets with tunable dimensionality, complexity, and modal asymmetry. Our findings are varied, with no one framework being the best for all purposes. In a nutshell, (i) Neural Spline Flows do best at capturing mode asymmetry present in low-dimensional data, (ii) Conditional Flow Matching outperforms other models for high-dimensional data with low complexity, and (iii) Denoising Diffusion Probabilistic Models appears the best for low-dimensional data with high complexity. Our datasets include a Gaussian mixture model and the dihedral torsion angle distribution of the Aib\textsubscript{9} peptide, generated via a molecular dynamics simulation. We hope our taxonomy of probabilistic generative frameworks and numerical results may guide model selection for a wide range of molecular tasks.

Title: Inherently Interpretable and Uncertainty-Aware Models for Online Learning in Cyber-Security Problems

• Authors: Benjamin Kolicic, Alberto Caron, Chris Hicks, Vasilios Mavroudis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09393
• Pdf URL: https://arxiv.org/pdf/2411.09393
• Copy Paste: [[2411.09393]] Inherently Interpretable and Uncertainty-Aware Models for Online Learning in Cyber-Security Problems(https://arxiv.org/abs/2411.09393)
• Keywords: security, interpretability
• Abstract: In this paper, we address the critical need for interpretable and uncertainty-aware machine learning models in the context of online learning for high-risk industries, particularly cyber-security. While deep learning and other complex models have demonstrated impressive predictive capabilities, their opacity and lack of uncertainty quantification present significant questions about their trustworthiness. We propose a novel pipeline for online supervised learning problems in cyber-security, that harnesses the inherent interpretability and uncertainty awareness of Additive Gaussian Processes (AGPs) models. Our approach aims to balance predictive performance with transparency while improving the scalability of AGPs, which represents their main drawback, potentially enabling security analysts to better validate threat detection, troubleshoot and reduce false positives, and generally make trustworthy, informed decisions. This work contributes to the growing field of interpretable AI by proposing a class of models that can be significantly beneficial for high-stake decision problems such as the ones typical of the cyber-security domain. The source code is available.

Title: Script-centric behavior understanding for assisted autism spectrum disorder diagnosis

• Authors: Wenxing Liu, Yueran Pan, Ming Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09413
• Pdf URL: https://arxiv.org/pdf/2411.09413
• Copy Paste: [[2411.09413]] Script-centric behavior understanding for assisted autism spectrum disorder diagnosis(https://arxiv.org/abs/2411.09413)
• Keywords: interpretability, large language model
• Abstract: Observing and analyzing children's social behaviors is crucial for the early diagnosis of Autism Spectrum Disorders (ASD). This work focuses on automatically detecting ASD using computer vision techniques and large language models (LLMs). Existing methods typically rely on supervised learning. However, the scarcity of ASD diagnostic datasets and the lack of interpretability in diagnostic results significantly limits its clinical application. To address these challenges, we introduce a novel unsupervised approach based on script-centric behavior understanding. Our pipeline converts video content into scripts that describe the behavior of characters, leveraging the generalizability of large language models to detect ASD in a zero-shot or few-shot manner. Specifically, we propose a scripts transcription module for multimodal behavior data textualization and a domain prompts module to bridge LLMs. Our method achieves an accuracy of 92.00\% in diagnosing ASD in children with an average age of 24 months, surpassing the performance of supervised learning methods by 3.58\% absolutely. Extensive experiments confirm the effectiveness of our approach and suggest its potential for advancing ASD research through LLMs.

Title: SAG-ViT: A Scale-Aware, High-Fidelity Patching Approach with Graph Attention for Vision Transformers

• Authors: Shravan Venkatraman, Jaskaran Singh Walia, Joe Dhanith P R
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09420
• Pdf URL: https://arxiv.org/pdf/2411.09420
• Copy Paste: [[2411.09420]] SAG-ViT: A Scale-Aware, High-Fidelity Patching Approach with Graph Attention for Vision Transformers(https://arxiv.org/abs/2411.09420)
• Keywords: transformer
• Abstract: Image classification is a computer vision task where a model analyzes an image to categorize it into a specific label. Vision Transformers (ViT) improve this task by leveraging self-attention to capture complex patterns and long range relationships between image patches. However, a key challenge for ViTs is efficiently incorporating multiscale feature representations, which is inherent in CNNs through their hierarchical structure. In this paper, we introduce the Scale-Aware Graph Attention Vision Transformer (SAG-ViT), a novel framework that addresses this challenge by integrating multi-scale features. Using EfficientNet as a backbone, the model extracts multi-scale feature maps, which are divided into patches to preserve semantic information. These patches are organized into a graph based on spatial and feature similarities, with a Graph Attention Network (GAT) refining the node embeddings. Finally, a Transformer encoder captures long-range dependencies and complex interactions. The SAG-ViT is evaluated on benchmark datasets, demonstrating its effectiveness in enhancing image classification performance.

Title: Everyone deserves their voice to be heard: Analyzing Predictive Gender Bias in ASR Models Applied to Dutch Speech Data

• Authors: Rik Raes, Saskia Lensink, Mykola Pechenizkiy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09431
• Pdf URL: https://arxiv.org/pdf/2411.09431
• Copy Paste: [[2411.09431]] Everyone deserves their voice to be heard: Analyzing Predictive Gender Bias in ASR Models Applied to Dutch Speech Data(https://arxiv.org/abs/2411.09431)
• Keywords: fair
• Abstract: Recent research has shown that state-of-the-art (SotA) Automatic Speech Recognition (ASR) systems, such as Whisper, often exhibit predictive biases that disproportionately affect various demographic groups. This study focuses on identifying the performance disparities of Whisper models on Dutch speech data from the Common Voice dataset and the Dutch National Public Broadcasting organisation. We analyzed the word error rate, character error rate and a BERT-based semantic similarity across gender groups. We used the moral framework of Weerts et al. (2022) to assess quality of service harms and fairness, and to provide a nuanced discussion on the implications of these biases, particularly for automatic subtitling. Our findings reveal substantial disparities in word error rate (WER) among gender groups across all model sizes, with bias identified through statistical testing.

Title: Mediffusion: Joint Diffusion for Self-Explainable Semi-Supervised Classification and Medical Image Generation

• Authors: Joanna Kaleta, Paweł Skierś, Jan Dubiński, Przemysław Korzeniowski, Kamil Deja
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09434
• Pdf URL: https://arxiv.org/pdf/2411.09434
• Copy Paste: [[2411.09434]] Mediffusion: Joint Diffusion for Self-Explainable Semi-Supervised Classification and Medical Image Generation(https://arxiv.org/abs/2411.09434)
• Keywords: explainability, diffusion, generative
• Abstract: We introduce Mediffusion -- a new method for semi-supervised learning with explainable classification based on a joint diffusion model. The medical imaging domain faces unique challenges due to scarce data labelling -- insufficient for standard training, and critical nature of the applications that require high performance, confidence, and explainability of the models. In this work, we propose to tackle those challenges with a single model that combines standard classification with a diffusion-based generative task in a single shared parametrisation. By sharing representations, our model effectively learns from both labeled and unlabeled data while at the same time providing accurate explanations through counterfactual examples. In our experiments, we show that our Mediffusion achieves results comparable to recent semi-supervised methods while providing more reliable and precise explanations.

Title: ReMP: Reusable Motion Prior for Multi-domain 3D Human Pose Estimation and Motion Inbetweening

• Authors: Hojun Jang, Young Min Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09435
• Pdf URL: https://arxiv.org/pdf/2411.09435
• Copy Paste: [[2411.09435]] ReMP: Reusable Motion Prior for Multi-domain 3D Human Pose Estimation and Motion Inbetweening(https://arxiv.org/abs/2411.09435)
• Keywords: robust
• Abstract: We present Reusable Motion prior (ReMP), an effective motion prior that can accurately track the temporal evolution of motion in various downstream tasks. Inspired by the success of foundation models, we argue that a robust spatio-temporal motion prior can encapsulate underlying 3D dynamics applicable to various sensor modalities. We learn the rich motion prior from a sequence of complete parametric models of posed human body shape. Our prior can easily estimate poses in missing frames or noisy measurements despite significant occlusion by employing a temporal attention mechanism. More interestingly, our prior can guide the system with incomplete and challenging input measurements to quickly extract critical information to estimate the sequence of poses, significantly improving the training efficiency for mesh sequence recovery. ReMP consistently outperforms the baseline method on diverse and practical 3D motion data, including depth point clouds, LiDAR scans, and IMU sensor data. Project page is available in this https URL.

Title: Spider: Any-to-Many Multimodal LLM

• Authors: Jinxiang Lai, Jie Zhang, Jun Liu, Jian Li, Xiaocheng Lu, Song Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09439
• Pdf URL: https://arxiv.org/pdf/2411.09439
• Copy Paste: [[2411.09439]] Spider: Any-to-Many Multimodal LLM(https://arxiv.org/abs/2411.09439)
• Keywords: large language model
• Abstract: Multimodal LLMs (MLLMs) have emerged as an extension of Large Language Models (LLMs), enabling the integration of various modalities. However, Any-to-Any MLLMs are limited to generating pairwise modalities 'Text + X' within a single response, such as Text + {Image or Audio or Video}. To address this limitation, we introduce Spider, a novel efficient Any-to-Many Modalities Generation (AMMG) framework, which can generate an arbitrary combination of modalities 'Text + Xs', such as Text + {Image and Audio and Video}. To achieve efficient AMMG, our Spider integrates three core components: a Base Model for basic X-to-X (i.e., Any-to-Any) modality processing, a novel Efficient Decoders-Controller for controlling multimodal Decoders to generate Xs (many-modal) contents, and an Any-to-Many Instruction Template designed for producing Xs signal prompts. To train Spider, we constructed a novel Text-formatted Many-Modal (TMM) dataset, which facilitates the learning of the X-to-Xs (i.e., Any-to-Many) capability necessary for AMMG. Ultimately, the well-trained Spider generates a pseudo X-to-Xs dataset, the first-ever X-to-Xs many-modal dataset, enhancing the potential for AMMG task in future research. Overall, this work not only pushes the boundary of multimodal interaction but also provides rich data support for advancing the field.

Title: Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models

• Authors: Chutian Meng, Fan Ma, Jiaxu Miao, Chi Zhang, Yi Yang, Yueting Zhuang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09449
• Pdf URL: https://arxiv.org/pdf/2411.09449
• Copy Paste: [[2411.09449]] Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models(https://arxiv.org/abs/2411.09449)
• Keywords: robust, diffusion, generative, large language model
• Abstract: Diffusion models have revitalized the image generation domain, playing crucial roles in both academic research and artistic expression. With the emergence of new diffusion models, assessing the performance of text-to-image models has become increasingly important. Current metrics focus on directly matching the input text with the generated image, but due to cross-modal information asymmetry, this leads to unreliable or incomplete assessment results. Motivated by this, we introduce the Image Regeneration task in this study to assess text-to-image models by tasking the T2I model with generating an image according to the reference image. We use GPT4V to bridge the gap between the reference image and the text input for the T2I model, allowing T2I models to understand image content. This evaluation process is simplified as comparisons between the generated image and the reference image are straightforward. Two regeneration datasets spanning content-diverse and style-diverse evaluation dataset are introduced to evaluate the leading diffusion models currently available. Additionally, we present ImageRepainter framework to enhance the quality of generated images by improving content comprehension via MLLM guided iterative generation and revision. Our comprehensive experiments have showcased the effectiveness of this framework in assessing the generative capabilities of models. By leveraging MLLM, we have demonstrated that a robust T2M can produce images more closely resembling the reference image.

Title: Caravan MultiMet: Extending Caravan with Multiple Weather Nowcasts and Forecasts

• Authors: Guy Shalev, Frederik Kratzert
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09459
• Pdf URL: https://arxiv.org/pdf/2411.09459
• Copy Paste: [[2411.09459]] Caravan MultiMet: Extending Caravan with Multiple Weather Nowcasts and Forecasts(https://arxiv.org/abs/2411.09459)
• Keywords: robust
• Abstract: The Caravan large-sample hydrology dataset (Kratzert et al., 2023) was created to standardize and harmonize streamflow data from various regional datasets, combined with globally available meteorological forcing and catchment attributes. This community-driven project also allows researchers to conveniently extend the dataset for additional basins, as done 6 times to date (see this https URL). We present a novel extension to Caravan, focusing on enriching the meteorological forcing data. Our extension adds three precipitation nowcast products (CPC, IMERG v07 Early, and CHIRPS) and three weather forecast products (ECMWF IFS HRES, GraphCast, and CHIRPS-GEFS) to the existing ERA5-Land reanalysis data. The inclusion of diverse data sources, particularly weather forecasts, enables more robust evaluation and benchmarking of hydrological models, especially for real-time forecasting scenarios. To the best of our knowledge, this extension makes Caravan the first large-sample hydrology dataset to incorporate weather forecast data, significantly enhancing its capabilities and fostering advancements in hydrological research, benchmarking, and real-time hydrologic forecasting. The data is publicly available under a CC-BY-4.0 license on Zenodo in two parts (this https URL, this https URL) and on Google Cloud Platform (GCP) - see more under the Data Availability chapter.

Title: Image Matching Filtering and Refinement by Planes and Beyond

• Authors: Fabio Bellavia, Zhenjun Zhao, Luca Morelli, Fabio Remondino
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09484
• Pdf URL: https://arxiv.org/pdf/2411.09484
• Copy Paste: [[2411.09484]] Image Matching Filtering and Refinement by Planes and Beyond(https://arxiv.org/abs/2411.09484)
• Keywords: robust
• Abstract: This paper introduces a modular, non-deep learning method for filtering and refining sparse correspondences in image matching. Assuming that motion flow within the scene can be approximated by local homography transformations, matches are aggregated into overlapping clusters corresponding to virtual planes using an iterative RANSAC-based approach, with non-conforming correspondences discarded. Moreover, the underlying planar structural design provides an explicit map between local patches associated with the matches, enabling optional refinement of keypoint positions through cross-correlation template matching after patch reprojection. Finally, to enhance robustness and fault-tolerance against violations of the piece-wise planar approximation assumption, a further strategy is designed for minimizing relative patch distortion in the plane reprojection by introducing an intermediate homography that projects both patches into a common plane. The proposed method is extensively evaluated on standard datasets and image matching pipelines, and compared with state-of-the-art approaches. Unlike other current comparisons, the proposed benchmark also takes into account the more general, real, and practical cases where camera intrinsics are unavailable. Experimental results demonstrate that our proposed non-deep learning, geometry-based approach achieves performances that are either superior to or on par with recent state-of-the-art deep learning methods. Finally, this study suggests that there are still development potential in actual image matching solutions in the considered research direction, which could be in the future incorporated in novel deep image matching architectures.

Title: MM-Eval: A Hierarchical Benchmark for Modern Mongolian Evaluation in LLMs

• Authors: Mengyuan Zhang, Ruihui Wang, Bo Xia, Yuan Sun, Xiaobing Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09492
• Pdf URL: https://arxiv.org/pdf/2411.09492
• Copy Paste: [[2411.09492]] MM-Eval: A Hierarchical Benchmark for Modern Mongolian Evaluation in LLMs(https://arxiv.org/abs/2411.09492)
• Keywords: large language model
• Abstract: Large language models (LLMs) excel in high-resource languages but face notable challenges in low-resource languages like Mongolian. This paper addresses these challenges by categorizing capabilities into language abilities (syntax and semantics) and cognitive abilities (knowledge and reasoning). To systematically evaluate these areas, we developed MM-Eval, a specialized dataset based on Modern Mongolian Language Textbook I and enriched with WebQSP and MGSM datasets. Preliminary experiments on models including Qwen2-7B-Instruct, GLM4-9b-chat, Llama3.1-8B-Instruct, GPT-4, and DeepseekV2.5 revealed that: 1) all models performed better on syntactic tasks than semantic tasks, highlighting a gap in deeper language understanding; and 2) knowledge tasks showed a moderate decline, suggesting that models can transfer general knowledge from high-resource to low-resource contexts. The release of MM-Eval, comprising 569 syntax, 677 semantics, 344 knowledge, and 250 reasoning tasks, offers valuable insights for advancing NLP and LLMs in low-resource languages like Mongolian. The dataset is available at this https URL.

Title: Golden Noise for Diffusion Models: A Learning Framework

• Authors: Zikai Zhou, Shitong Shao, Lichen Bai, Zhiqiang Xu, Bo Han, Zeke Xie
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09502
• Pdf URL: https://arxiv.org/pdf/2411.09502
• Copy Paste: [[2411.09502]] Golden Noise for Diffusion Models: A Learning Framework(https://arxiv.org/abs/2411.09502)
• Keywords: diffusion
• Abstract: Text-to-image diffusion model is a popular paradigm that synthesizes personalized images by providing a text prompt and a random Gaussian noise. While people observe that some noises are ``golden noises'' that can achieve better text-image alignment and higher human preference than others, we still lack a machine learning framework to obtain those golden noises. To learn golden noises for diffusion sampling, we mainly make three contributions in this paper. First, we identify a new concept termed the \textit{noise prompt}, which aims at turning a random Gaussian noise into a golden noise by adding a small desirable perturbation derived from the text prompt. Following the concept, we first formulate the \textit{noise prompt learning} framework that systematically learns ``prompted'' golden noise associated with a text prompt for diffusion models. Second, we design a noise prompt data collection pipeline and collect a large-scale \textit{noise prompt dataset}~(NPD) that contains 100k pairs of random noises and golden noises with the associated text prompts. With the prepared NPD as the training dataset, we trained a small \textit{noise prompt network}~(NPNet) that can directly learn to transform a random noise into a golden noise. The learned golden noise perturbation can be considered as a kind of prompt for noise, as it is rich in semantic information and tailored to the given text prompt. Third, our extensive experiments demonstrate the impressive effectiveness and generalization of NPNet on improving the quality of synthesized images across various diffusion models, including SDXL, DreamShaper-xl-v2-turbo, and Hunyuan-DiT. Moreover, NPNet is a small and efficient controller that acts as a plug-and-play module with very limited additional inference and computational costs, as it just provides a golden noise instead of a random noise without accessing the original pipeline.

Title: Communication Compression for Tensor Parallel LLM Inference

• Authors: Jan Hansen-Palmus, Michael Truong-Le, Oliver Hausdörfer, Alok Verma
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09510
• Pdf URL: https://arxiv.org/pdf/2411.09510
• Copy Paste: [[2411.09510]] Communication Compression for Tensor Parallel LLM Inference(https://arxiv.org/abs/2411.09510)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have pushed the frontier of artificial intelligence but are comprised of hundreds of billions of parameters and operations. For faster inference latency, LLMs are deployed on multiple hardware accelerators through various Model Parallelism strategies. Our paper looks into the details on one such strategy - Tensor Parallel - and proposes to reduce latency by compressing inter-accelerator communication. We leverage fine grained quantization techniques to compress selected activations by 3.5 - 4.5x. Our proposed method leads up to 2x reduction of time-to-first-token (TTFT) with negligible model performance degradation.

Title: A Practical Guide to Fine-tuning Language Models with Limited Data

• Authors: Márton Szép, Daniel Rueckert, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09539
• Pdf URL: https://arxiv.org/pdf/2411.09539
• Copy Paste: [[2411.09539]] A Practical Guide to Fine-tuning Language Models with Limited Data(https://arxiv.org/abs/2411.09539)
• Keywords: large language model
• Abstract: Employing pre-trained Large Language Models (LLMs) has become the de facto standard in Natural Language Processing (NLP) despite their extensive data requirements. Motivated by the recent surge in research focused on training LLMs with limited data, particularly in low-resource domains and languages, this paper surveys recent transfer learning approaches to optimize model performance in downstream tasks where data is scarce. We first address initial and continued pre-training strategies to better leverage prior knowledge in unseen domains and languages. We then examine how to maximize the utility of limited data during fine-tuning and few-shot learning. The final section takes a task-specific perspective, reviewing models and methods suited for different levels of data scarcity. Our goal is to provide practitioners with practical guidelines for overcoming the challenges posed by constrained data while also highlighting promising directions for future research.

Title: Piecing It All Together: Verifying Multi-Hop Multimodal Claims

• Authors: Haoran Wang, Aman Rangapur, Xiongxiao Xu, Yueqing Liang, Haroon Gharwi, Carl Yang, Kai Shu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09547
• Pdf URL: https://arxiv.org/pdf/2411.09547
• Copy Paste: [[2411.09547]] Piecing It All Together: Verifying Multi-Hop Multimodal Claims(https://arxiv.org/abs/2411.09547)
• Keywords: large language model
• Abstract: Existing claim verification datasets often do not require systems to perform complex reasoning or effectively interpret multimodal evidence. To address this, we introduce a new task: multi-hop multimodal claim verification. This task challenges models to reason over multiple pieces of evidence from diverse sources, including text, images, and tables, and determine whether the combined multimodal evidence supports or refutes a given claim. To study this task, we construct MMCV, a large-scale dataset comprising 16k multi-hop claims paired with multimodal evidence, generated and refined using large language models, with additional input from human feedback. We show that MMCV is challenging even for the latest state-of-the-art multimodal large language models, especially as the number of reasoning hops increases. Additionally, we establish a human performance benchmark on a subset of MMCV. We hope this dataset and its evaluation task will encourage future research in multimodal multi-hop claim verification.

Title: Faster Differentially Private Top-$k$ Selection: A Joint Exponential Mechanism with Pruning

• Authors: Hao WU, Hanwen Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.09552
• Pdf URL: https://arxiv.org/pdf/2411.09552
• Copy Paste: [[2411.09552]] Faster Differentially Private Top-$k$ Selection: A Joint Exponential Mechanism with Pruning(https://arxiv.org/abs/2411.09552)
• Keywords: privacy
• Abstract: We study the differentially private top-$k$ selection problem, aiming to identify a sequence of $k$ items with approximately the highest scores from $d$ items. Recent work by Gillenwater et al. (ICML '22) employs a direct sampling approach from the vast collection of $d^{\,\Theta(k)}$ possible length-$k$ sequences, showing superior empirical accuracy compared to previous pure or approximate differentially private methods. Their algorithm has a time and space complexity of $\tilde{O}(dk)$. In this paper, we present an improved algorithm with time and space complexity $O(d + k^2 / \epsilon \cdot \ln d)$, where $\epsilon$ denotes the privacy parameter. Experimental results show that our algorithm runs orders of magnitude faster than their approach, while achieving similar empirical accuracy.

Title: OOD-SEG: Out-Of-Distribution detection for image SEGmentation with sparse multi-class positive-only annotations

• Authors: Junwen Wang, Zhonghao Wang, Oscar MacCormac, Jonathan Shapey, Tom Vercauteren
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09553
• Pdf URL: https://arxiv.org/pdf/2411.09553
• Copy Paste: [[2411.09553]] OOD-SEG: Out-Of-Distribution detection for image SEGmentation with sparse multi-class positive-only annotations(https://arxiv.org/abs/2411.09553)
• Keywords: robust, segmentation
• Abstract: Despite significant advancements, segmentation based on deep neural networks in medical and surgical imaging faces several challenges, two of which we aim to address in this work. First, acquiring complete pixel-level segmentation labels for medical images is time-consuming and requires domain expertise. Second, typical segmentation pipelines cannot detect out-of-distribution (OOD) pixels, leaving them prone to spurious outputs during deployment. In this work, we propose a novel segmentation approach exploiting OOD detection that learns only from sparsely annotated pixels from multiple positive-only classes. %but \emph{no background class} annotation. These multi-class positive annotations naturally fall within the in-distribution (ID) set. Unlabelled pixels may contain positive classes but also negative ones, including what is typically referred to as \emph{background} in standard segmentation formulations. Here, we forgo the need for background annotation and consider these together with any other unseen classes as part of the OOD set. Our framework can integrate, at a pixel-level, any OOD detection approaches designed for classification tasks. To address the lack of existing OOD datasets and established evaluation metric for medical image segmentation, we propose a cross-validation strategy that treats held-out labelled classes as OOD. Extensive experiments on both multi-class hyperspectral and RGB surgical imaging datasets demonstrate the robustness and generalisation capability of our proposed framework.

Title: Adaptive Deviation Learning for Visual Anomaly Detection with Data Contamination

• Authors: Anindya Sundar Das, Guansong Pang, Monowar Bhuyan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09558
• Pdf URL: https://arxiv.org/pdf/2411.09558
• Copy Paste: [[2411.09558]] Adaptive Deviation Learning for Visual Anomaly Detection with Data Contamination(https://arxiv.org/abs/2411.09558)
• Keywords: robust
• Abstract: Visual anomaly detection targets to detect images that notably differ from normal pattern, and it has found extensive application in identifying defective parts within the manufacturing industry. These anomaly detection paradigms predominantly focus on training detection models using only clean, unlabeled normal samples, assuming an absence of contamination; a condition often unmet in real-world scenarios. The performance of these methods significantly depends on the quality of the data and usually decreases when exposed to noise. We introduce a systematic adaptive method that employs deviation learning to compute anomaly scores end-to-end while addressing data contamination by assigning relative importance to the weights of individual instances. In this approach, the anomaly scores for normal instances are designed to approximate scalar scores obtained from the known prior distribution. Meanwhile, anomaly scores for anomaly examples are adjusted to exhibit statistically significant deviations from these reference scores. Our approach incorporates a constrained optimization problem within the deviation learning framework to update instance weights, resolving this problem for each mini-batch. Comprehensive experiments on the MVTec and VisA benchmark datasets indicate that our proposed method surpasses competing techniques and exhibits both stability and robustness in the presence of data contamination.

Title: VPBSD:Vessel-Pattern-Based Semi-Supervised Distillation for Efficient 3D Microscopic Cerebrovascular Segmentation

• Authors: Xi Lin, Shixuan Zhao, Xinxu Wei, Amir Shmuel, Yongjie Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09567
• Pdf URL: https://arxiv.org/pdf/2411.09567
• Copy Paste: [[2411.09567]] VPBSD:Vessel-Pattern-Based Semi-Supervised Distillation for Efficient 3D Microscopic Cerebrovascular Segmentation(https://arxiv.org/abs/2411.09567)
• Keywords: segmentation
• Abstract: 3D microscopic cerebrovascular images are characterized by their high resolution, presenting significant annotation challenges, large data volumes, and intricate variations in detail. Together, these factors make achieving high-quality, efficient whole-brain segmentation particularly demanding. In this paper, we propose a novel Vessel-Pattern-Based Semi-Supervised Distillation pipeline (VpbSD) to address the challenges of 3D microscopic cerebrovascular segmentation. This pipeline initially constructs a vessel-pattern codebook that captures diverse vascular structures from unlabeled data during the teacher model's pretraining phase. In the knowledge distillation stage, the codebook facilitates the transfer of rich knowledge from a heterogeneous teacher model to a student model, while the semi-supervised approach further enhances the student model's exposure to diverse learning samples. Experimental results on real-world data, including comparisons with state-of-the-art methods and ablation studies, demonstrate that our pipeline and its individual components effectively address the challenges inherent in microscopic cerebrovascular segmentation.

Title: Backdoor Mitigation by Distance-Driven Detoxification

• Authors: Shaokui Wei, Jiayin Liu, Hongyuan Zha
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.09585
• Pdf URL: https://arxiv.org/pdf/2411.09585
• Copy Paste: [[2411.09585]] Backdoor Mitigation by Distance-Driven Detoxification(https://arxiv.org/abs/2411.09585)
• Keywords: defense, attack
• Abstract: Backdoor attacks undermine the integrity of machine learning models by allowing attackers to manipulate predictions using poisoned training data. Such attacks lead to targeted misclassification when specific triggers are present, while the model behaves normally under other conditions. This paper considers a post-training backdoor defense task, aiming to detoxify the backdoors in pre-trained models. We begin by analyzing the underlying issues of vanilla fine-tuning and observe that it is often trapped in regions with low loss for both clean and poisoned samples. Motivated by such observations, we propose Distance-Driven Detoxification (D3), an innovative approach that reformulates backdoor defense as a constrained optimization problem. Specifically, D3 promotes the model's departure from the vicinity of its initial weights, effectively reducing the influence of backdoors. Extensive experiments on state-of-the-art (SOTA) backdoor attacks across various model architectures and datasets demonstrate that D3 not only matches but often surpasses the performance of existing SOTA post-training defense techniques.

Title: BabyLM Challenge: Exploring the Effect of Variation Sets on Language Model Training Efficiency

• Authors: Akari Haga, Akiyo Fukatsu, Miyu Oba, Arianna Bisazza, Yohei Oseki
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09587
• Pdf URL: https://arxiv.org/pdf/2411.09587
• Copy Paste: [[2411.09587]] BabyLM Challenge: Exploring the Effect of Variation Sets on Language Model Training Efficiency(https://arxiv.org/abs/2411.09587)
• Keywords: transformer, large language model
• Abstract: While current large language models have achieved a remarkable success, their data efficiency remains a challenge to overcome. Recently it has been suggested that child-directed speech (CDS) can improve training data efficiency of modern language models based on Transformer neural networks. However, it is not yet understood which specific properties of CDS are effective for training these models. In the context of the BabyLM Challenge, we focus on Variation Sets (VSs), sets of consecutive utterances expressing a similar intent with slightly different words and structures, which are ubiquitous in CDS. To assess the impact of VSs on training data efficiency, we augment CDS data with different proportions of artificial VSs and use these datasets to train an auto-regressive model, GPT-2. We find that the best proportion of VSs depends on the evaluation benchmark: BLiMP and GLUE scores benefit from the presence of VSs, but EWOK scores do not. Additionally, the results vary depending on multiple factors such as the number of epochs and the order of utterance presentation. Taken together, these findings suggest that VSs can have a beneficial influence on language models, while leaving room for further investigation.

Title: Expert Study on Interpretable Machine Learning Models with Missing Data

• Authors: Lena Stempfle, Arthur James, Julie Josse, Tobias Gauss, Fredrik D. Johansson
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09591
• Pdf URL: https://arxiv.org/pdf/2411.09591
• Copy Paste: [[2411.09591]] Expert Study on Interpretable Machine Learning Models with Missing Data(https://arxiv.org/abs/2411.09591)
• Keywords: interpretability
• Abstract: Inherently interpretable machine learning (IML) models provide valuable insights for clinical decision-making but face challenges when features have missing values. Classical solutions like imputation or excluding incomplete records are often unsuitable in applications where values are missing at test time. In this work, we conducted a survey with 71 clinicians from 29 trauma centers across France, including 20 complete responses to study the interaction between medical professionals and IML applied to data with missing values. This provided valuable insights into how missing data is interpreted in clinical machine learning. We used the prediction of hemorrhagic shock as a concrete example to gauge the willingness and readiness of the participants to adopt IML models from three classes of methods. Our findings show that, while clinicians value interpretability and are familiar with common IML methods, classical imputation techniques often misalign with their intuition, and that models that natively handle missing values are preferred. These results emphasize the need to integrate clinical intuition into future IML models for better human-computer interaction.

Title: LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models

• Authors: Zhengyi Wang, Jonathan Lorraine, Yikai Wang, Hang Su, Jun Zhu, Sanja Fidler, Xiaohui Zeng
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09595
• Pdf URL: https://arxiv.org/pdf/2411.09595
• Copy Paste: [[2411.09595]] LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models(https://arxiv.org/abs/2411.09595)
• Keywords: large language model
• Abstract: This work explores expanding the capabilities of large language models (LLMs) pretrained on text to generate 3D meshes within a unified model. This offers key advantages of (1) leveraging spatial knowledge already embedded in LLMs, derived from textual sources like 3D tutorials, and (2) enabling conversational 3D generation and mesh understanding. A primary challenge is effectively tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly. To address this, we introduce LLaMA-Mesh, a novel approach that represents the vertex coordinates and face definitions of 3D meshes as plain text, allowing direct integration with LLMs without expanding the vocabulary. We construct a supervised fine-tuning (SFT) dataset enabling pretrained LLMs to (1) generate 3D meshes from text prompts, (2) produce interleaved text and 3D mesh outputs as required, and (3) understand and interpret 3D meshes. Our work is the first to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge for 3D mesh generation in a text-based format, effectively unifying the 3D and text modalities. LLaMA-Mesh achieves mesh generation quality on par with models trained from scratch while maintaining strong text generation performance.

Title: The Moral Foundations Weibo Corpus

• Authors: Renjie Cao, Miaoyan Hu, Jiahan Wei, Baha Ihnaini
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09612
• Pdf URL: https://arxiv.org/pdf/2411.09612
• Copy Paste: [[2411.09612]] The Moral Foundations Weibo Corpus(https://arxiv.org/abs/2411.09612)
• Keywords: large language model
• Abstract: Moral sentiments expressed in natural language significantly influence both online and offline environments, shaping behavioral styles and interaction patterns, including social media selfpresentation, cyberbullying, adherence to social norms, and ethical decision-making. To effectively measure moral sentiments in natural language processing texts, it is crucial to utilize large, annotated datasets that provide nuanced understanding for accurate analysis and modeltraining. However, existing corpora, while valuable, often face linguistic limitations. To address this gap in the Chinese language domain,we introduce the Moral Foundation Weibo Corpus. This corpus consists of 25,671 Chinese comments on Weibo, encompassing six diverse topic areas. Each comment is manually annotated by at least three systematically trained annotators based on ten moral categories derived from a grounded theory of morality. To assess annotator reliability, we present the kappa testresults, a gold standard for measuring consistency. Additionally, we apply several the latest large language models to supplement the manual annotations, conducting analytical experiments to compare their performance and report baseline results for moral sentiment classification.

Title: PTR: Precision-Driven Tool Recommendation for Large Language Models

• Authors: Hang Gao, Yongfeng Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09613
• Pdf URL: https://arxiv.org/pdf/2411.09613
• Copy Paste: [[2411.09613]] PTR: Precision-Driven Tool Recommendation for Large Language Models(https://arxiv.org/abs/2411.09613)
• Keywords: large language model
• Abstract: By augmenting Large Language Models (LLMs) with external tools, their capacity to solve complex problems has been significantly enhanced. However, despite ongoing advancements in the parsing capabilities of LLMs, incorporating all available tools simultaneously in the prompt remains impractical due to the vast number of external tools. Consequently, it is essential to provide LLMs with a precise set of tools tailored to the specific task, considering both quantity and quality. Current tool retrieval methods primarily focus on refining the ranking list of tools and directly packaging a fixed number of top-ranked tools as the tool set. However, these approaches often fail to equip LLMs with the optimal set of tools prior to execution, since the optimal number of tools for different tasks could be different, resulting in inefficiencies such as redundant or unsuitable tools, which impede immediate access to the most relevant tools. This paper addresses the challenge of recommending precise toolsets for LLMs. We introduce the problem of tool recommendation, define its scope, and propose a novel Precision-driven Tool Recommendation (PTR) approach. PTR captures an initial, concise set of tools by leveraging historical tool bundle usage and dynamically adjusts the tool set by performing tool matching, culminating in a multi-view-based tool addition. Additionally, we present a new dataset, RecTools, and a metric, TRACC, designed to evaluate the effectiveness of tool recommendation for LLMs. We further validate our design choices through comprehensive experiments, demonstrating promising accuracy across two open benchmarks and our RecTools dataset.

Title: Squeezed Attention: Accelerating Long Context Length LLM Inference

• Authors: Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, June Paik, Michael W. Mahoney, Kurt Keutzer, Amir Gholami
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09688
• Pdf URL: https://arxiv.org/pdf/2411.09688
• Copy Paste: [[2411.09688]] Squeezed Attention: Accelerating Long Context Length LLM Inference(https://arxiv.org/abs/2411.09688)
• Keywords: large language model
• Abstract: Emerging Large Language Model (LLM) applications require long input prompts to perform complex downstream tasks like document analysis and code generation. For these long context length applications, the length of the input prompt poses a significant challenge in terms of inference efficiency since the inference costs increase linearly with sequence length. However, for many of these applications, much of the context in the prompt is fixed across different user inputs, thereby providing the opportunity to perform offline optimizations to process user inputs quickly, as they are received. In this work, we propose Squeezed Attention as a mechanism to accelerate LLM applications where a large portion of the input prompt is fixed. We first leverage K-means clustering offline to group the keys for the fixed context based on semantic similarity and represent each cluster with a single centroid value. During inference, we compare query tokens from the user input with the centroids to predict which of the keys from the fixed context are semantically relevant and need to be loaded during inference. We then compute exact attention using only these important keys from the fixed context, thereby reducing bandwidth and computational costs. We also extend our method to use a hierarchical centroid lookup to identify important keys, which can reduce the complexity of attention from linear to logarithmic with respect to the context length. We implement optimized Triton kernels for centroid comparison and sparse FlashAttention with important keys, achieving more than 4x speedups during both the prefill and generation phases for long-context inference. Furthermore, we have extensively evaluated our method on various long-context benchmarks including LongBench, where it achieves a 3x reduction in KV cache budget without accuracy loss and up to an 8x reduction with <0.5 point accuracy gap for various models.

Title: Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models

• Authors: Wei Wang, Zhaowei Li, Qi Xu, Linfeng Li, YiQing Cai, Botian Jiang, Hang Song, Xingcan Hu, Pengyu Wang, Li Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09691
• Pdf URL: https://arxiv.org/pdf/2411.09691
• Copy Paste: [[2411.09691]] Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models(https://arxiv.org/abs/2411.09691)
• Keywords: large language model
• Abstract: Multi-modal large language models (MLLMs) have achieved remarkable success in fine-grained visual understanding across a range of tasks. However, they often encounter significant challenges due to inadequate alignment for fine-grained knowledge, which restricts their ability to accurately capture local details and attain a comprehensive global perception. While recent advancements have focused on aligning object expressions with grounding information, they typically lack explicit integration of object images, which contain affluent information beyond mere texts or coordinates. To bridge this gap, we introduce a novel fine-grained visual knowledge alignment method that effectively aligns and integrates multi-scale knowledge of objects, including texts, coordinates, and images. This innovative method is underpinned by our multi-scale fine-grained enhancement data synthesis pipeline, which provides over 300K essential training data to enhance alignment and improve overall performance. Furthermore, we present TinyGroundingGPT, a series of compact models optimized for high-level alignments. With a scale of approximately 3B parameters, TinyGroundingGPT achieves outstanding results in grounding tasks while delivering performance comparable to larger MLLMs in complex visual scenarios.

Title: On the Surprising Effectiveness of Attention Transfer for Vision Transformers

• Authors: Alexander C. Li, Yuandong Tian, Beidi Chen, Deepak Pathak, Xinlei Chen
• Subjects: cs.LG, cs.AI, cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2411.09702
• Pdf URL: https://arxiv.org/pdf/2411.09702
• Copy Paste: [[2411.09702]] On the Surprising Effectiveness of Attention Transfer for Vision Transformers(https://arxiv.org/abs/2411.09702)
• Keywords: transformer
• Abstract: Conventional wisdom suggests that pre-training Vision Transformers (ViT) improves downstream performance by learning useful representations. Is this actually true? We investigate this question and find that the features and representations learned during pre-training are not essential. Surprisingly, using only the attention patterns from pre-training (i.e., guiding how information flows between tokens) is sufficient for models to learn high quality features from scratch and achieve comparable downstream performance. We show this by introducing a simple method called attention transfer, where only the attention patterns from a pre-trained teacher ViT are transferred to a student, either by copying or distilling the attention maps. Since attention transfer lets the student learn its own features, ensembling it with a fine-tuned teacher also further improves accuracy on ImageNet. We systematically study various aspects of our findings on the sufficiency of attention maps, including distribution shift settings where they underperform fine-tuning. We hope our exploration provides a better understanding of what pre-training accomplishes and leads to a useful alternative to the standard practice of fine-tuning

Title: MagicQuill: An Intelligent Interactive Image Editing System

• Authors: Zichen Liu, Yue Yu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Wen Wang, Zhiheng Liu, Qifeng Chen, Yujun Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09703
• Pdf URL: https://arxiv.org/pdf/2411.09703
• Copy Paste: [[2411.09703]] MagicQuill: An Intelligent Interactive Image Editing System(https://arxiv.org/abs/2411.09703)
• Keywords: robust, diffusion, large language model
• Abstract: Image editing involves a variety of complex tasks and requires efficient and precise manipulation techniques. In this paper, we present MagicQuill, an integrated image editing system that enables swift actualization of creative ideas. Our system features a streamlined yet functionally robust interface, allowing for the articulation of editing operations (e.g., inserting elements, erasing objects, altering color) with minimal input. These interactions are monitored by a multimodal large language model (MLLM) to anticipate editing intentions in real time, bypassing the need for explicit prompt entry. Finally, we apply a powerful diffusion prior, enhanced by a carefully learned two-branch plug-in module, to process editing requests with precise control. Experimental results demonstrate the effectiveness of MagicQuill in achieving high-quality image edits. Please visit this https URL to try out our system.