2024-12-10

Title: FodFoM: Fake Outlier Data by Foundation Models Creates Stronger Visual Out-of-Distribution Detector

• Authors: Jiankang Chen, Ling Deng, Zhiyong Gan, Wei-Shi Zheng, Ruixuan Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05293
• Pdf URL: https://arxiv.org/pdf/2412.05293
• Copy Paste: [[2412.05293]] FodFoM: Fake Outlier Data by Foundation Models Creates Stronger Visual Out-of-Distribution Detector(https://arxiv.org/abs/2412.05293)
• Keywords: diffusion
• Abstract: Out-of-Distribution (OOD) detection is crucial when deploying machine learning models in open-world applications. The core challenge in OOD detection is mitigating the model's overconfidence on OOD data. While recent methods using auxiliary outlier datasets or synthesizing outlier features have shown promising OOD detection performance, they are limited due to costly data collection or simplified assumptions. In this paper, we propose a novel OOD detection framework FodFoM that innovatively combines multiple foundation models to generate two types of challenging fake outlier images for classifier training. The first type is based on BLIP-2's image captioning capability, CLIP's vision-language knowledge, and Stable Diffusion's image generation ability. Jointly utilizing these foundation models constructs fake outlier images which are semantically similar to but different from in-distribution (ID) images. For the second type, GroundingDINO's object detection ability is utilized to help construct pure background images by blurring foreground ID objects in ID images. The proposed framework can be flexibly combined with multiple existing OOD detection methods. Extensive empirical evaluations show that image classifiers with the help of constructed fake images can more accurately differentiate real OOD images from ID ones. New state-of-the-art OOD detection performance is achieved on multiple benchmarks. The code is available at \url{this https URL}.

Title: Text Is Not All You Need: Multimodal Prompting Helps LLMs Understand Humor

• Authors: Ashwin Baluja
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.05315
• Pdf URL: https://arxiv.org/pdf/2412.05315
• Copy Paste: [[2412.05315]] Text Is Not All You Need: Multimodal Prompting Helps LLMs Understand Humor(https://arxiv.org/abs/2412.05315)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) have demonstrated impressive natural language understanding capabilities across various text-based tasks, understanding humor has remained a persistent challenge. Humor is frequently multimodal, relying on phonetic ambiguity, rhythm and timing to convey meaning. In this study, we explore a simple multimodal prompting approach to humor understanding and explanation. We present an LLM with both the text and the spoken form of a joke, generated using an off-the-shelf text-to-speech (TTS) system. Using multimodal cues improves the explanations of humor compared to textual prompts across all tested datasets.

Title: Deep Learning and Hybrid Approaches for Dynamic Scene Analysis, Object Detection and Motion Tracking

• Authors: Shahran Rahman Alve
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05331
• Pdf URL: https://arxiv.org/pdf/2412.05331
• Copy Paste: [[2412.05331]] Deep Learning and Hybrid Approaches for Dynamic Scene Analysis, Object Detection and Motion Tracking(https://arxiv.org/abs/2412.05331)
• Keywords: security, robust, segmentation
• Abstract: This project aims to develop a robust video surveillance system, which can segment videos into smaller clips based on the detection of activities. It uses CCTV footage, for example, to record only major events-like the appearance of a person or a thief-so that storage is optimized and digital searches are easier. It utilizes the latest techniques in object detection and tracking, including Convolutional Neural Networks (CNNs) like YOLO, SSD, and Faster R-CNN, as well as Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTMs), to achieve high accuracy in detection and capture temporal dependencies. The approach incorporates adaptive background modeling through Gaussian Mixture Models (GMM) and optical flow methods like Lucas-Kanade to detect motions. Multi-scale and contextual analysis are used to improve detection across different object sizes and environments. A hybrid motion segmentation strategy combines statistical and deep learning models to manage complex movements, while optimizations for real-time processing ensure efficient computation. Tracking methods, such as Kalman Filters and Siamese networks, are employed to maintain smooth tracking even in cases of occlusion. Detection is improved on various-sized objects for multiple scenarios by multi-scale and contextual analysis. Results demonstrate high precision and recall in detecting and tracking objects, with significant improvements in processing times and accuracy due to real-time optimizations and illumination-invariant features. The impact of this research lies in its potential to transform video surveillance, reducing storage requirements and enhancing security through reliable and efficient object detection and tracking.

Title: Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models

• Authors: Zhejun Zhang, Peter Karkus, Maximilian Igl, Wenhao Ding, Yuxiao Chen, Boris Ivanovic, Marco Pavone
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05334
• Pdf URL: https://arxiv.org/pdf/2412.05334
• Copy Paste: [[2412.05334]] Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models(https://arxiv.org/abs/2412.05334)
• Keywords: generative, large language model
• Abstract: Traffic simulation aims to learn a policy for traffic agents that, when unrolled in closed-loop, faithfully recovers the joint distribution of trajectories observed in the real world. Inspired by large language models, tokenized multi-agent policies have recently become the state-of-the-art in traffic simulation. However, they are typically trained through open-loop behavior cloning, and thus suffer from covariate shift when executed in closed-loop during simulation. In this work, we present Closest Among Top-K (CAT-K) rollouts, a simple yet effective closed-loop fine-tuning strategy to mitigate covariate shift. CAT-K fine-tuning only requires existing trajectory data, without reinforcement learning or generative adversarial imitation. Concretely, CAT-K fine-tuning enables a small 7M-parameter tokenized traffic simulation policy to outperform a 102M-parameter model from the same model family, achieving the top spot on the Waymo Sim Agent Challenge leaderboard at the time of submission. The code is available at this https URL.

Title: Generative Model-Based Fusion for Improved Few-Shot Semantic Segmentation of Infrared Images

• Authors: Junno Yun, Mehmet Akçakaya
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.05341
• Pdf URL: https://arxiv.org/pdf/2412.05341
• Copy Paste: [[2412.05341]] Generative Model-Based Fusion for Improved Few-Shot Semantic Segmentation of Infrared Images(https://arxiv.org/abs/2412.05341)
• Keywords: defense, generative, segmentation
• Abstract: Infrared (IR) imaging is commonly used in various scenarios, including autonomous driving, fire safety and defense applications. Thus, semantic segmentation of such images is of great interest. However, this task faces several challenges, including data scarcity, differing contrast and input channel number compared to natural images, and emergence of classes not represented in databases in certain scenarios, such as defense applications. Few-shot segmentation (FSS) provides a framework to overcome these issues by segmenting query images using a few labeled support samples. However, existing FSS models for IR images require paired visible RGB images, which is a major limitation since acquiring such paired data is difficult or impossible in some applications. In this work, we develop new strategies for FSS of IR images by using generative modeling and fusion techniques. To this end, we propose to synthesize auxiliary data to provide additional channel information to complement the limited contrast in the IR images, as well as IR data synthesis for data augmentation. Here, the former helps the FSS model to better capture the relationship between the support and query sets, while the latter addresses the issue of data scarcity. Finally, to further improve the former aspect, we propose a novel fusion ensemble module for integrating the two different modalities. Our methods are evaluated on different IR datasets, and improve upon the state-of-the-art (SOTA) FSS models.

Title: Multi-Party Supervised Fine-tuning of Language Models for Multi-Party Dialogue Generation

• Authors: Xiaoyu Wang, Ningyuan Xi, Teng Chen, Qingqing Gu, Yue Zhao, Xiaokai Chen, Zhonglin Jiang, Yong Chen, Luo Ji
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05342
• Pdf URL: https://arxiv.org/pdf/2412.05342
• Copy Paste: [[2412.05342]] Multi-Party Supervised Fine-tuning of Language Models for Multi-Party Dialogue Generation(https://arxiv.org/abs/2412.05342)
• Keywords: large language model
• Abstract: Large Language Models (LLM) are usually fine-tuned to participate in dyadic or two-party dialogues, which can not adapt well to multi-party dialogues (MPD), which hinders their applications in such scenarios including multi-personal meetings, discussions and daily communication. Previous LLM-based researches mainly focus on the multi-agent framework, while their base LLMs are still pairwisely fine-tuned. In this work, we design a multi-party fine-tuning framework (MuPaS) for LLMs on the multi-party dialogue datasets, and prove such a straightforward framework can let the LLM align with the multi-party conversation style efficiently and effectively. We also design two training strategies which can convert MuPaS into the MPD simulator. Substantial experiments show that MuPaS can achieve state-of-the-art multi-party response, higher accuracy of the-next-speaker prediction, higher human and automatic evaluated utterance qualities, and can even generate reasonably with out-of-distribution scene, topic and role descriptions. The MuPaS framework bridges the LLM training with more complicated multi-party applications, such as conversation generation, virtual rehearsal or meta-universe.

Title: BadGPT-4o: stripping safety finetuning from GPT models

• Authors: Ekaterina Krupkina, Dmitrii Volkov
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05346
• Pdf URL: https://arxiv.org/pdf/2412.05346
• Copy Paste: [[2412.05346]] BadGPT-4o: stripping safety finetuning from GPT models(https://arxiv.org/abs/2412.05346)
• Keywords: attack
• Abstract: We show a version of Qi et al. 2023's simple fine-tuning poisoning technique strips GPT-4o's safety guardrails without degrading the model. The BadGPT attack matches best white-box jailbreaks on HarmBench and StrongREJECT. It suffers no token overhead or performance hits common to jailbreaks, as evaluated on tinyMMLU and open-ended generations. Despite having been known for a year, this attack remains easy to execute.

Title: Towards Predicting the Success of Transfer-based Attacks by Quantifying Shared Feature Representations

• Authors: Ashley S. Dale, Mei Qiu, Foo Bin Che, Thomas Bsaibes, Lauren Christopher, Paul Salama
• Subjects: cs.CV, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05351
• Pdf URL: https://arxiv.org/pdf/2412.05351
• Copy Paste: [[2412.05351]] Towards Predicting the Success of Transfer-based Attacks by Quantifying Shared Feature Representations(https://arxiv.org/abs/2412.05351)
• Keywords: attack
• Abstract: Much effort has been made to explain and improve the success of transfer-based attacks (TBA) on black-box computer vision models. This work provides the first attempt at a priori prediction of attack success by identifying the presence of vulnerable features within target models. Recent work by Chen and Liu (2024) proposed the manifold attack model, a unifying framework proposing that successful TBA exist in a common manifold space. Our work experimentally tests the common manifold space hypothesis by a new methodology: first, projecting feature vectors from surrogate and target feature extractors trained on ImageNet onto the same low-dimensional manifold; second, quantifying any observed structure similarities on the manifold; and finally, by relating these observed similarities to the success of the TBA. We find that shared feature representation moderately correlates with increased success of TBA (\r{ho}= 0.56). This method may be used to predict whether an attack will transfer without information of the model weights, training, architecture or details of the attack. The results confirm the presence of shared feature representations between two feature extractors of different sizes and complexities, and demonstrate the utility of datasets from different target domains as test signals for interpreting black-box feature representations.

Title: Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models

• Authors: Michael Hanna, Aaron Mueller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05353
• Pdf URL: https://arxiv.org/pdf/2412.05353
• Copy Paste: [[2412.05353]] Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models(https://arxiv.org/abs/2412.05353)
• Keywords: transformer
• Abstract: Autoregressive transformer language models (LMs) possess strong syntactic abilities, often successfully handling phenomena from agreement to NPI licensing. However, the features they use to incrementally process language inputs are not well understood. In this paper, we fill this gap by studying the mechanisms underlying garden path sentence processing in LMs. We ask: (1) Do LMs use syntactic features or shallow heuristics to perform incremental sentence processing? (2) Do LMs represent only one potential interpretation, or multiple? and (3) Do LMs reanalyze or repair their initial incorrect representations? To address these questions, we use sparse autoencoders to identify interpretable features that determine which continuation - and thus which reading - of a garden path sentence the LM prefers. We find that while many important features relate to syntactic structure, some reflect syntactically irrelevant heuristics. Moreover, while most active features correspond to one reading of the sentence, some features correspond to the other, suggesting that LMs assign weight to both possibilities simultaneously. Finally, LMs do not re-use features from garden path sentence processing to answer follow-up questions.

Title: MotionShop: Zero-Shot Motion Transfer in Video Diffusion Models with Mixture of Score Guidance

• Authors: Hidir Yesiltepe, Tuna Han Salih Meral, Connor Dunlop, Pinar Yanardag
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05355
• Pdf URL: https://arxiv.org/pdf/2412.05355
• Copy Paste: [[2412.05355]] MotionShop: Zero-Shot Motion Transfer in Video Diffusion Models with Mixture of Score Guidance(https://arxiv.org/abs/2412.05355)
• Keywords: diffusion, transformer
• Abstract: In this work, we propose the first motion transfer approach in diffusion transformer through Mixture of Score Guidance (MSG), a theoretically-grounded framework for motion transfer in diffusion models. Our key theoretical contribution lies in reformulating conditional score to decompose motion score and content score in diffusion models. By formulating motion transfer as a mixture of potential energies, MSG naturally preserves scene composition and enables creative scene transformations while maintaining the integrity of transferred motion patterns. This novel sampling operates directly on pre-trained video diffusion models without additional training or fine-tuning. Through extensive experiments, MSG demonstrates successful handling of diverse scenarios including single object, multiple objects, and cross-object motion transfer as well as complex camera motion transfer. Additionally, we introduce MotionBench, the first motion transfer dataset consisting of 200 source videos and 1000 transferred motions, covering single/multi-object transfers, and complex camera motions.

Title: DIFEM: Key-points Interaction based Feature Extraction Module for Violence Recognition in Videos

• Authors: Himanshu Mittal, Suvramalya Basak, Anjali Gautam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05386
• Pdf URL: https://arxiv.org/pdf/2412.05386
• Copy Paste: [[2412.05386]] DIFEM: Key-points Interaction based Feature Extraction Module for Violence Recognition in Videos(https://arxiv.org/abs/2412.05386)
• Keywords: extraction
• Abstract: Violence detection in surveillance videos is a critical task for ensuring public safety. As a result, there is increasing need for efficient and lightweight systems for automatic detection of violent behaviours. In this work, we propose an effective method which leverages human skeleton key-points to capture inherent properties of violence, such as rapid movement of specific joints and their close proximity. At the heart of our method is our novel Dynamic Interaction Feature Extraction Module (DIFEM) which captures features such as velocity, and joint intersections, effectively capturing the dynamics of violent behavior. With the features extracted by our DIFEM, we use various classification algorithms such as Random Forest, Decision tree, AdaBoost and k-Nearest Neighbor. Our approach has substantially lesser amount of parameter expense than the existing state-of-the-art (SOTA) methods employing deep learning techniques. We perform extensive experiments on three standard violence recognition datasets, showing promising performance in all three datasets. Our proposed method surpasses several SOTA violence recognition methods.

Title: CALICO: Conversational Agent Localization via Synthetic Data Generation

• Authors: Andy Rosenbaum, Pegah Kharazmi, Ershad Banijamali, Lu Zeng, Christopher DiPersio, Pan Wei, Gokmen Oz, Clement Chung, Karolina Owczarzak, Fabian Triefenbach, Wael Hamza
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05388
• Pdf URL: https://arxiv.org/pdf/2412.05388
• Copy Paste: [[2412.05388]] CALICO: Conversational Agent Localization via Synthetic Data Generation(https://arxiv.org/abs/2412.05388)
• Keywords: large language model
• Abstract: We present CALICO, a method to fine-tune Large Language Models (LLMs) to localize conversational agent training data from one language to another. For slots (named entities), CALICO supports three operations: verbatim copy, literal translation, and localization, i.e. generating slot values more appropriate in the target language, such as city and airport names located in countries where the language is spoken. Furthermore, we design an iterative filtering mechanism to discard noisy generated samples, which we show boosts the performance of the downstream conversational agent. To prove the effectiveness of CALICO, we build and release a new human-localized (HL) version of the MultiATIS++ travel information test set in 8 languages. Compared to the original human-translated (HT) version of the test set, we show that our new HL version is more challenging. We also show that CALICO out-performs state-of-the-art LINGUIST (which relies on literal slot translation out of context) both on the HT case, where CALICO generates more accurate slot translations, and on the HL case, where CALICO generates localized slots which are closer to the HL test set.

Title: Tabular data generation with tensor contraction layers and transformers

• Authors: Aníbal Silva, André Restivo, Moisés Santos, Carlos Soares
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.05390
• Pdf URL: https://arxiv.org/pdf/2412.05390
• Copy Paste: [[2412.05390]] Tabular data generation with tensor contraction layers and transformers(https://arxiv.org/abs/2412.05390)
• Keywords: transformer, generative
• Abstract: Generative modeling for tabular data has recently gained significant attention in the Deep Learning domain. Its objective is to estimate the underlying distribution of the data. However, estimating the underlying distribution of tabular data has its unique challenges. Specifically, this data modality is composed of mixed types of features, making it a non-trivial task for a model to learn intra-relationships between them. One approach to address mixture is to embed each feature into a continuous matrix via tokenization, while a solution to capture intra-relationships between variables is via the transformer architecture. In this work, we empirically investigate the potential of using embedding representations on tabular data generation, utilizing tensor contraction layers and transformers to model the underlying distribution of tabular data within Variational Autoencoders. Specifically, we compare four architectural approaches: a baseline VAE model, two variants that focus on tensor contraction layers and transformers respectively, and a hybrid model that integrates both techniques. Our empirical study, conducted across multiple datasets from the OpenML CC18 suite, compares models over density estimation and Machine Learning efficiency metrics. The main takeaway from our results is that leveraging embedding representations with the help of tensor contraction layers improves density estimation metrics, albeit maintaining competitive performance in terms of machine learning efficiency.

Title: HiVeGen -- Hierarchical LLM-based Verilog Generation for Scalable Chip Design

• Authors: Jinwei Tang, Jiayin Qin, Kiran Thorat, Chen Zhu-Tian, Yu Cao, Yang (Katie)Zhao, Caiwen Ding
• Subjects: cs.LG, cs.AI, cs.AR
• Abstract URL: https://arxiv.org/abs/2412.05393
• Pdf URL: https://arxiv.org/pdf/2412.05393
• Copy Paste: [[2412.05393]] HiVeGen -- Hierarchical LLM-based Verilog Generation for Scalable Chip Design(https://arxiv.org/abs/2412.05393)
• Keywords: large language model
• Abstract: With Large Language Models (LLMs) recently demonstrating impressive proficiency in code generation, it is promising to extend their abilities to Hardware Description Language (HDL). However, LLMs tend to generate single HDL code blocks rather than hierarchical structures for hardware designs, leading to hallucinations, particularly in complex designs like Domain-Specific Accelerators (DSAs). To address this, we propose HiVeGen, a hierarchical LLM-based Verilog generation framework that decomposes generation tasks into LLM-manageable hierarchical submodules. HiVeGen further harnesses the advantages of such hierarchical structures by integrating automatic Design Space Exploration (DSE) into hierarchy-aware prompt generation, introducing weight-based retrieval to enhance code reuse, and enabling real-time human-computer interaction to lower error-correction cost, significantly improving the quality of generated designs.

Title: YOLOv5-Based Object Detection for Emergency Response in Aerial Imagery

• Authors: Sindhu Boddu, Arindam Mukherjee, Arindrajit Seal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05394
• Pdf URL: https://arxiv.org/pdf/2412.05394
• Copy Paste: [[2412.05394]] YOLOv5-Based Object Detection for Emergency Response in Aerial Imagery(https://arxiv.org/abs/2412.05394)
• Keywords: robust
• Abstract: This paper presents a robust approach for object detection in aerial imagery using the YOLOv5 model. We focus on identifying critical objects such as ambulances, car crashes, police vehicles, tow trucks, fire engines, overturned cars, and vehicles on fire. By leveraging a custom dataset, we outline the complete pipeline from data collection and annotation to model training and evaluation. Our results demonstrate that YOLOv5 effectively balances speed and accuracy, making it suitable for real-time emergency response applications. This work addresses key challenges in aerial imagery, including small object detection and complex backgrounds, and provides insights for future research in automated emergency response systems.

Title: No Free Lunch From Random Feature Ensembles

• Authors: Benjamin S. Ruben, William L. Tong, Hamza Tahir Chaudhry, Cengiz Pehlevan
• Subjects: cs.LG, cond-mat.dis-nn, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.05418
• Pdf URL: https://arxiv.org/pdf/2412.05418
• Copy Paste: [[2412.05418]] No Free Lunch From Random Feature Ensembles(https://arxiv.org/abs/2412.05418)
• Keywords: transformer
• Abstract: Given a budget on total model size, one must decide whether to train a single, large neural network or to combine the predictions of many smaller networks. We study this trade-off for ensembles of random-feature ridge regression models. We prove that when a fixed number of trainable parameters are partitioned among $K$ independently trained models, $K=1$ achieves optimal performance, provided the ridge parameter is optimally tuned. We then derive scaling laws which describe how the test risk of an ensemble of regression models decays with its total size. We identify conditions on the kernel and task eigenstructure under which ensembles can achieve near-optimal scaling laws. Training ensembles of deep convolutional neural networks on CIFAR-10 and a transformer architecture on C4, we find that a single large network outperforms any ensemble of networks with the same total number of parameters, provided the weight decay and feature-learning strength are tuned to their optimal values.

Title: KEDformer:Knowledge Extraction Seasonal Trend Decomposition for Long-term Sequence Prediction

• Authors: Zhenkai Qin, Baozhong Wei, Caifeng Gao, Jianyuan Ni
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.05421
• Pdf URL: https://arxiv.org/pdf/2412.05421
• Copy Paste: [[2412.05421]] KEDformer:Knowledge Extraction Seasonal Trend Decomposition for Long-term Sequence Prediction(https://arxiv.org/abs/2412.05421)
• Keywords: extraction, transformer
• Abstract: Time series forecasting is a critical task in domains such as energy, finance, and meteorology, where accurate long-term predictions are essential. While Transformer-based models have shown promise in capturing temporal dependencies, their application to extended sequences is limited by computational inefficiencies and limited generalization. In this study, we propose KEDformer, a knowledge extraction-driven framework that integrates seasonal-trend decomposition to address these challenges. KEDformer leverages knowledge extraction methods that focus on the most informative weights within the self-attention mechanism to reduce computational overhead. Additionally, the proposed KEDformer framework decouples time series into seasonal and trend components. This decomposition enhances the model's ability to capture both short-term fluctuations and long-term patterns. Extensive experiments on five public datasets from energy, transportation, and weather domains demonstrate the effectiveness and competitiveness of KEDformer, providing an efficient solution for long-term time series forecasting.

Title: Swap Path Network for Robust Person Search Pre-training

• Authors: Lucas Jaffe, Avideh Zakhor
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05433
• Pdf URL: https://arxiv.org/pdf/2412.05433
• Copy Paste: [[2412.05433]] Swap Path Network for Robust Person Search Pre-training(https://arxiv.org/abs/2412.05433)
• Keywords: robust, extraction
• Abstract: In person search, we detect and rank matches to a query person image within a set of gallery scenes. Most person search models make use of a feature extraction backbone, followed by separate heads for detection and re-identification. While pre-training methods for vision backbones are well-established, pre-training additional modules for the person search task has not been previously examined. In this work, we present the first framework for end-to-end person search pre-training. Our framework splits person search into object-centric and query-centric methodologies, and we show that the query-centric framing is robust to label noise, and trainable using only weakly-labeled person bounding boxes. Further, we provide a novel model dubbed Swap Path Net (SPNet) which implements both query-centric and object-centric training objectives, and can swap between the two while using the same weights. Using SPNet, we show that query-centric pre-training, followed by object-centric fine-tuning, achieves state-of-the-art results on the standard PRW and CUHK-SYSU person search benchmarks, with 96.4% mAP on CUHK-SYSU and 61.2% mAP on PRW. In addition, we show that our method is more effective, efficient, and robust for person search pre-training than recent backbone-only pre-training alternatives.

Title: Granular Ball K-Class Twin Support Vector Classifier

• Authors: M. A. Ganaie, Vrushank Ahire, Anouck Girard
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05438
• Pdf URL: https://arxiv.org/pdf/2412.05438
• Copy Paste: [[2412.05438]] Granular Ball K-Class Twin Support Vector Classifier(https://arxiv.org/abs/2412.05438)
• Keywords: robust
• Abstract: This paper introduces the Granular Ball K-Class Twin Support Vector Classifier (GB-TWKSVC), a novel multi-class classification framework that combines Twin Support Vector Machines (TWSVM) with granular ball computing. The proposed method addresses key challenges in multi-class classification by utilizing granular ball representation for improved noise robustness and TWSVM's non-parallel hyperplane architecture solves two smaller quadratic programming problems, enhancing efficiency. Our approach introduces a novel formulation that effectively handles multi-class scenarios, advancing traditional binary classification methods. Experimental evaluation on diverse benchmark datasets shows that GB-TWKSVC significantly outperforms current state-of-the-art classifiers in both accuracy and computational performance. The method's effectiveness is validated through comprehensive statistical tests and complexity analysis. Our work advances classification algorithms by providing a mathematically sound framework that addresses the scalability and robustness needs of modern machine learning applications. The results demonstrate GB-TWKSVC's broad applicability across domains including pattern recognition, fault diagnosis, and large-scale data analytics, establishing it as a valuable addition to the classification algorithm landscape.

Title: A Graph-Based Approach for Conversational AI-Driven Personal Memory Capture and Retrieval in a Real-world Application

• Authors: Savini Kashmira, Jayanaka L. Dantanarayana, Joshua Brodsky, Ashish Mahendra, Yiping Kang, Krisztian Flautner, Lingjia Tang, Jason Mars
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.05447
• Pdf URL: https://arxiv.org/pdf/2412.05447
• Copy Paste: [[2412.05447]] A Graph-Based Approach for Conversational AI-Driven Personal Memory Capture and Retrieval in a Real-world Application(https://arxiv.org/abs/2412.05447)
• Keywords: large language model
• Abstract: TOBU is a novel mobile application that captures and retrieves `personal memories' (pictures/videos together with stories and context around those moments) in a user-engaging AI-guided conversational approach. Our initial prototype showed that existing retrieval techniques such as retrieval-augmented generation (RAG) systems fall short due to their limitations in understanding memory relationships, causing low recall, hallucination, and unsatisfactory user experience. We design TOBUGraph, a novel graph-based retrieval approach. During capturing, TOBUGraph leverages large language models (LLMs) to automatically create a dynamic knowledge graph of memories, establishing context and relationships of those memories. During retrieval, TOBUGraph combines LLMs with the memory graph to achieve comprehensive recall through graph traversal. Our evaluation using real user data demonstrates that TOBUGraph outperforms multiple RAG implementations in both precision and recall, significantly improving user experience through improved retrieval accuracy and reduced hallucination.

Title: Towards Effective GenAI Multi-Agent Collaboration: Design and Evaluation for Enterprise Applications

• Authors: Raphael Shu, Nilaksh Das, Michelle Yuan, Monica Sunkara, Yi Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05449
• Pdf URL: https://arxiv.org/pdf/2412.05449
• Copy Paste: [[2412.05449]] Towards Effective GenAI Multi-Agent Collaboration: Design and Evaluation for Enterprise Applications(https://arxiv.org/abs/2412.05449)
• Keywords: large language model
• Abstract: AI agents powered by large language models (LLMs) have shown strong capabilities in problem solving. Through combining many intelligent agents, multi-agent collaboration has emerged as a promising approach to tackle complex, multi-faceted problems that exceed the capabilities of single AI agents. However, designing the collaboration protocols and evaluating the effectiveness of these systems remains a significant challenge, especially for enterprise applications. This report addresses these challenges by presenting a comprehensive evaluation of coordination and routing capabilities in a novel multi-agent collaboration framework. We evaluate two key operational modes: (1) a coordination mode enabling complex task completion through parallel communication and payload referencing, and (2) a routing mode for efficient message forwarding between agents. We benchmark on a set of handcrafted scenarios from three enterprise domains, which are publicly released with the report. For coordination capabilities, we demonstrate the effectiveness of inter-agent communication and payload referencing mechanisms, achieving end-to-end goal success rates of 90%. Our analysis yields several key findings: multi-agent collaboration enhances goal success rates by up to 70% compared to single-agent approaches in our benchmarks; payload referencing improves performance on code-intensive tasks by 23%; latency can be substantially reduced with a routing mechanism that selectively bypasses agent orchestration. These findings offer valuable guidance for enterprise deployments of multi-agent systems and advance the development of scalable, efficient multi-agent collaboration frameworks.

Title: Knowledge Graphs are all you need: Leveraging KGs in Physics Question Answering

• Authors: Krishnasai Addala, Kabir Dev Paul Baghel, Dhruv Jain, Chhavi Kirtani, Avinash Anand, Rajiv Ratn Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05453
• Pdf URL: https://arxiv.org/pdf/2412.05453
• Copy Paste: [[2412.05453]] Knowledge Graphs are all you need: Leveraging KGs in Physics Question Answering(https://arxiv.org/abs/2412.05453)
• Keywords: large language model
• Abstract: This study explores the effectiveness of using knowledge graphs generated by large language models to decompose high school-level physics questions into sub-questions. We introduce a pipeline aimed at enhancing model response quality for Question Answering tasks. By employing LLMs to construct knowledge graphs that capture the internal logic of the questions, these graphs then guide the generation of subquestions. We hypothesize that this method yields sub-questions that are more logically consistent with the original questions compared to traditional decomposition techniques. Our results show that sub-questions derived from knowledge graphs exhibit significantly improved fidelity to the original question's logic. This approach not only enhances the learning experience by providing clearer and more contextually appropriate sub-questions but also highlights the potential of LLMs to transform educational methodologies. The findings indicate a promising direction for applying AI to improve the quality and effectiveness of educational content.

Title: CigTime: Corrective Instruction Generation Through Inverse Motion Editing

• Authors: Qihang Fang, Chengcheng Tang, Bugra Tekin, Yanchao Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05460
• Pdf URL: https://arxiv.org/pdf/2412.05460
• Copy Paste: [[2412.05460]] CigTime: Corrective Instruction Generation Through Inverse Motion Editing(https://arxiv.org/abs/2412.05460)
• Keywords: large language model
• Abstract: Recent advancements in models linking natural language with human motions have shown significant promise in motion generation and editing based on instructional text. Motivated by applications in sports coaching and motor skill learning, we investigate the inverse problem: generating corrective instructional text, leveraging motion editing and generation models. We introduce a novel approach that, given a user's current motion (source) and the desired motion (target), generates text instructions to guide the user towards achieving the target motion. We leverage large language models to generate corrective texts and utilize existing motion generation and editing frameworks to compile datasets of triplets (source motion, target motion, and corrective text). Using this data, we propose a new motion-language model for generating corrective instructions. We present both qualitative and quantitative results across a diverse range of applications that largely improve upon baselines. Our approach demonstrates its effectiveness in instructional scenarios, offering text-based guidance to correct and enhance user performance.

Title: Multi-Armed Bandit Approach for Optimizing Training on Synthetic Data

• Authors: Abdulrahman Kerim, Leandro Soriano Marcolino, Erickson R. Nascimento, Richard Jiang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05466
• Pdf URL: https://arxiv.org/pdf/2412.05466
• Copy Paste: [[2412.05466]] Multi-Armed Bandit Approach for Optimizing Training on Synthetic Data(https://arxiv.org/abs/2412.05466)
• Keywords: diffusion, large language model
• Abstract: Supervised machine learning methods require large-scale training datasets to perform well in practice. Synthetic data has been showing great progress recently and has been used as a complement to real data. However, there is yet a great urge to assess the usability of synthetically generated data. To this end, we propose a novel UCB-based training procedure combined with a dynamic usability metric. Our proposed metric integrates low-level and high-level information from synthetic images and their corresponding real and synthetic datasets, surpassing existing traditional metrics. By utilizing a UCB-based dynamic approach ensures continual enhancement of model learning. Unlike other approaches, our method effectively adapts to changes in the machine learning model's state and considers the evolving utility of training samples during the training process. We show that our metric is an effective way to rank synthetic images based on their usability. Furthermore, we propose a new attribute-aware bandit pipeline for generating synthetic data by integrating a Large Language Model with Stable Diffusion. Quantitative results show that our approach can boost the performance of a wide range of supervised classifiers. Notably, we observed an improvement of up to 10% in classification accuracy compared to traditional approaches, demonstrating the effectiveness of our approach. Our source code, datasets, and additional materials are publically available at this https URL.

Title: The BrowserGym Ecosystem for Web Agent Research

• Authors: Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Lacoste, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas Chapados
• Subjects: cs.LG, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.05467
• Pdf URL: https://arxiv.org/pdf/2412.05467
• Copy Paste: [[2412.05467]] The BrowserGym Ecosystem for Web Agent Research(https://arxiv.org/abs/2412.05467)
• Keywords: robust, large language model
• Abstract: The BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents, particularly those leveraging automation and Large Language Models (LLMs) for web interaction tasks. Many existing benchmarks suffer from fragmentation and inconsistent evaluation methodologies, making it challenging to achieve reliable comparisons and reproducible results. BrowserGym aims to solve this by providing a unified, gym-like environment with well-defined observation and action spaces, facilitating standardized evaluation across diverse benchmarks. Combined with AgentLab, a complementary framework that aids in agent creation, testing, and analysis, BrowserGym offers flexibility for integrating new benchmarks while ensuring consistent evaluation and comprehensive experiment management. This standardized approach seeks to reduce the time and complexity of developing web agents, supporting more reliable comparisons and facilitating in-depth analysis of agent behaviors, and could result in more adaptable, capable agents, ultimately accelerating innovation in LLM-driven automation. As a supporting evidence, we conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across all benchmarks currently available in BrowserGym. Among other findings, our results highlight a large discrepancy between OpenAI and Anthropic's latests models, with Claude-3.5-Sonnet leading the way on almost all benchmarks, except on vision-related tasks where GPT-4o is superior. Despite these advancements, our results emphasize that building robust and efficient web agents remains a significant challenge, due to the inherent complexity of real-world web environments and the limitations of current models.

Title: Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization

• Authors: Subhojyoti Mukherjee, Anusha Lalitha, Sailik Sengupta, Aniket Deshmukh, Branislav Kveton
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05469
• Pdf URL: https://arxiv.org/pdf/2412.05469
• Copy Paste: [[2412.05469]] Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization(https://arxiv.org/abs/2412.05469)
• Keywords: large language model
• Abstract: Multi-objective alignment from human feedback (MOAHF) in large language models (LLMs) is a challenging problem as human preferences are complex, multifaceted, and often conflicting. Recent works on MOAHF considered a-priori multi-objective optimization (MOO), where human preferences are known at training or inference time. In contrast, when human preferences are unknown or difficult to quantify, a natural approach is to cover the Pareto front by multiple diverse solutions. We propose an algorithm HaM for learning diverse LLM policies that maximizes their hypervolume. This is the first application of a-posteriori MOO to MOAHF. HaM is computationally and space efficient, and empirically superior across objectives such as harmlessness, helpfulness, humor, faithfulness, and hallucination, on various datasets.

Title: AI-powered Digital Twin of the Ocean: Reliable Uncertainty Quantification for Real-time Wave Height Prediction with Deep Ensemble

• Authors: Dongeon Lee, Sunwoong Yang, Jae-Won Oh, Su-Gil Cho, Sanghyuk Kim, Namwoo Kang
• Subjects: cs.LG, cs.CE, eess.SP, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2412.05475
• Pdf URL: https://arxiv.org/pdf/2412.05475
• Copy Paste: [[2412.05475]] AI-powered Digital Twin of the Ocean: Reliable Uncertainty Quantification for Real-time Wave Height Prediction with Deep Ensemble(https://arxiv.org/abs/2412.05475)
• Keywords: robust
• Abstract: Environmental pollution and the depletion of fossil fuels have prompted the need for eco-friendly power generation methods based on renewable energy. However, renewable energy sources often face challenges in providing stable power due to low energy density and non-stationary. Wave energy converters (WECs), in particular, need reliable real-time wave height prediction to address these issues caused by irregular wave patterns, which can lead to the inefficient and unstable operation of WECs. In this study, we propose an AI-powered reliable real-time wave height prediction model, aiming both high predictive accuracy and reliable uncertainty quantification (UQ). The proposed architecture LSTM-DE, integrates long short-term memory (LSTM) networks for temporal prediction with deep ensemble (DE) for robust UQ, achieving accuracy and reliability in wave height prediction. To further enhance the reliability of the predictive models, uncertainty calibration is applied, which has proven to significantly improve the quality of the quantified uncertainty. Based on the real operational data obtained from an oscillating water column-wave energy converter (OWC-WEC) system in Jeju, South Korea, we demonstrate that the proposed LSTM-DE model architecture achieves notable predictive accuracy (R2 > 0.9) while increasing the uncertainty quality by over 50% through simple calibration technique. Furthermore, a comprehensive parametric study is conducted to explore the effects of key model hyperparameters, offering valuable guidelines for diverse operational scenarios, characterized by differences in wavelength, amplitude, and period. The findings show that the proposed method provides robust and reliable real-time wave height predictions, facilitating digital twin of the ocean.

Title: Securing Social Media Against Deepfakes using Identity, Behavioral, and Geometric Signatures

• Authors: Muhammad Umar Farooq, Awais Khan, Ijaz Ul Haq, Khalid Mahmood Malik
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.05487
• Pdf URL: https://arxiv.org/pdf/2412.05487
• Copy Paste: [[2412.05487]] Securing Social Media Against Deepfakes using Identity, Behavioral, and Geometric Signatures(https://arxiv.org/abs/2412.05487)
• Keywords: fair
• Abstract: Trust in social media is a growing concern due to its ability to influence significant societal changes. However, this space is increasingly compromised by various types of deepfake multimedia, which undermine the authenticity of shared content. Although substantial efforts have been made to address the challenge of deepfake content, existing detection techniques face a major limitation in generalization: they tend to perform well only on specific types of deepfakes they were trained this http URL dependency on recognizing specific deepfake artifacts makes current methods vulnerable when applied to unseen or varied deepfakes, thereby compromising their performance in real-world applications such as social media platforms. To address the generalizability of deepfake detection, there is a need for a holistic approach that can capture a broader range of facial attributes and manipulations beyond isolated artifacts. To address this, we propose a novel deepfake detection framework featuring an effective feature descriptor that integrates Deep identity, Behavioral, and Geometric (DBaG) signatures, along with a classifier named DBaGNet. Specifically, the DBaGNet classifier utilizes the extracted DBaG signatures, leveraging a triplet loss objective to enhance generalized representation learning for improved classification. Specifically, the DBaGNet classifier utilizes the extracted DBaG signatures and applies a triplet loss objective to enhance generalized representation learning for improved classification. To test the effectiveness and generalizability of our proposed approach, we conduct extensive experiments using six benchmark deepfake datasets: WLDR, CelebDF, DFDC, FaceForensics++, DFD, and NVFAIR. Specifically, to ensure the effectiveness of our approach, we perform cross-dataset evaluations, and the results demonstrate significant performance gains over several state-of-the-art methods.

Title: Enhancing Sample Generation of Diffusion Models using Noise Level Correction

• Authors: Abulikemu Abuduweili, Chenyang Yuan, Changliu Liu, Frank Permenter
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.05488
• Pdf URL: https://arxiv.org/pdf/2412.05488
• Copy Paste: [[2412.05488]] Enhancing Sample Generation of Diffusion Models using Noise Level Correction(https://arxiv.org/abs/2412.05488)
• Keywords: diffusion
• Abstract: The denoising process of diffusion models can be interpreted as a projection of noisy samples onto the data manifold. Moreover, the noise level in these samples approximates their distance to the underlying manifold. Building on this insight, we propose a novel method to enhance sample generation by aligning the estimated noise level with the true distance of noisy samples to the manifold. Specifically, we introduce a noise level correction network, leveraging a pre-trained denoising network, to refine noise level estimates during the denoising process. Additionally, we extend this approach to various image restoration tasks by integrating task-specific constraints, including inpainting, deblurring, super-resolution, colorization, and compressed sensing. Experimental results demonstrate that our method significantly improves sample quality in both unconstrained and constrained generation scenarios. Notably, the proposed noise level correction framework is compatible with existing denoising schedulers (e.g., DDIM), offering additional performance improvements.

Title: Dynamic Digital Twins of Blockchain Systems: State Extraction and Mirroring

• Authors: Georgios Diamantopoulos, Nikos Tziritas, Rami Bahsoon, Nan Zhang, Georgios Theodoropoulos
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.05527
• Pdf URL: https://arxiv.org/pdf/2412.05527
• Copy Paste: [[2412.05527]] Dynamic Digital Twins of Blockchain Systems: State Extraction and Mirroring(https://arxiv.org/abs/2412.05527)
• Keywords: security, robust, extraction
• Abstract: Blockchain adoption is reaching an all-time high, with a plethora of blockchain architectures being developed to cover the needs of applications eager to integrate blockchain into their operations. However, blockchain systems suffer from the trilemma trade-off problem, which limits their ability to scale without sacrificing essential metrics such as decentralisation and security. The balance of the trilemma trade-off is primarily dictated by the consensus protocol used. Since consensus protocols are designed to function well under specific system conditions, and consequently, due to the blockchain's complex and dynamic nature, systems operating under a single consensus protocol are bound to face periods of inefficiency. The work presented in this paper constitutes part of an effort to design a Digital Twin-based blockchain management framework to balance the trilemma trade-off problem, which aims to adapt the consensus process to fit the conditions of the underlying system. Specifically, this work addresses the problems of extracting the blockchain system and mirroring it in its digital twin by proposing algorithms that overcome the challenges posed by blockchains' decentralised and asynchronous nature and the fundamental problems of global state and synchronisation in such systems. The robustness of the proposed algorithms is experimentally evaluated.

Title: Upcycling Noise for Federated Unlearning

• Authors: Jianan Chen, Qin Hu, Fangtian Zhong, Yan Zhuang, Minghui Xu
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.05529
• Pdf URL: https://arxiv.org/pdf/2412.05529
• Copy Paste: [[2412.05529]] Upcycling Noise for Federated Unlearning(https://arxiv.org/abs/2412.05529)
• Keywords: privacy, protect, attack, federate
• Abstract: In Federated Learning (FL), multiple clients collaboratively train a model without sharing raw data. This paradigm can be further enhanced by Differential Privacy (DP) to protect local data from information inference attacks and is thus termed DPFL. An emerging privacy requirement, ``the right to be forgotten'' for clients, poses new challenges to DPFL but remains largely unexplored. Despite numerous studies on federated unlearning (FU), they are inapplicable to DPFL because the noise introduced by the DP mechanism compromises their effectiveness and efficiency. In this paper, we propose Federated Unlearning with Indistinguishability (FUI) to unlearn the local data of a target client in DPFL for the first time. FUI consists of two main steps: local model retraction and global noise calibration, resulting in an unlearning model that is statistically indistinguishable from the retrained model. Specifically, we demonstrate that the noise added in DPFL can endow the unlearning model with a certain level of indistinguishability after local model retraction, and then fortify the degree of unlearning through global noise calibration. Additionally, for the efficient and consistent implementation of the proposed FUI, we formulate a two-stage Stackelberg game to derive optimal unlearning strategies for both the server and the target client. Privacy and convergence analyses confirm theoretical guarantees, while experimental results based on four real-world datasets illustrate that our proposed FUI achieves superior model performance and higher efficiency compared to mainstream FU schemes. Simulation results further verify the optimality of the derived unlearning strategies.

Title: CLIP-TNseg: A Multi-Modal Hybrid Framework for Thyroid Nodule Segmentation in Ultrasound Images

• Authors: Xinjie Sun, Boxiong Wei, Yalong Jiang, Liquan Mao, Qi Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05530
• Pdf URL: https://arxiv.org/pdf/2412.05530
• Copy Paste: [[2412.05530]] CLIP-TNseg: A Multi-Modal Hybrid Framework for Thyroid Nodule Segmentation in Ultrasound Images(https://arxiv.org/abs/2412.05530)
• Keywords: robust, interpretability, segmentation
• Abstract: Thyroid nodule segmentation in ultrasound images is crucial for accurate diagnosis and treatment planning. However, existing methods face challenges in segmentation accuracy, interpretability, and generalization, which hinder their performance. This letter proposes a novel framework, CLIP-TNseg, to address these issues by integrating a multimodal large model with a neural network architecture. CLIP-TNseg consists of two main branches: the Coarse-grained Branch, which extracts high-level semantic features from a frozen CLIP model, and the Fine-grained Branch, which captures fine-grained features using U-Net style residual blocks. These features are fused and processed by the prediction head to generate precise segmentation maps. CLIP-TNseg leverages the Coarse-grained Branch to enhance semantic understanding through textual and high-level visual features, while the Fine-grained Branch refines spatial details, enabling precise and robust segmentation. Extensive experiments on public and our newly collected datasets demonstrate its competitive performance. Our code and the original dataset are available at this https URL.

Title: Action Recognition based Industrial Safety Violation Detection

• Authors: Surya N Reddy, Vaibhav Kurrey, Mayank Nagar, Gagan Raj Gupta
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05531
• Pdf URL: https://arxiv.org/pdf/2412.05531
• Copy Paste: [[2412.05531]] Action Recognition based Industrial Safety Violation Detection(https://arxiv.org/abs/2412.05531)
• Keywords: protect
• Abstract: Proper use of personal protective equipment (PPE) can save the lives of industry workers and it is a widely used application of computer vision in the large manufacturing industries. However, most of the applications deployed generate a lot of false alarms (violations) because they tend to generalize the requirements of PPE across the industry and tasks. The key to resolving this issue is to understand the action being performed by the worker and customize the inference for the specific PPE requirements of that action. In this paper, we propose a system that employs activity recognition models to first understand the action being performed and then use object detection techniques to check for violations. This leads to a 23% improvement in the F1-score compared to the PPE-based approach on our test dataset of 109 videos.

Title: Enhancing Webshell Detection With Deep Learning-Powered Methods

• Authors: Ha L. Viet, On V. Phung, Hoa N. Nguyen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.05532
• Pdf URL: https://arxiv.org/pdf/2412.05532
• Copy Paste: [[2412.05532]] Enhancing Webshell Detection With Deep Learning-Powered Methods(https://arxiv.org/abs/2412.05532)
• Keywords: protect, attack, robust
• Abstract: Webshell attacks are becoming more common, requiring robust detection mechanisms to protect web applications. The dissertation clearly states two research directions: scanning web application source code and analyzing HTTP traffic to detect webshells. First, the dissertation proposes ASAF, an advanced DL-Powered Source-Code Scanning Framework that uses signature-based methods and deep learning algorithms to detect known and unknown webshells. We designed the framework to enable programming language-specific detection models. The dissertation used PHP for interpreted language and this http URL for compiled language to build a complete ASAF-based model for experimentation and comparison with other research results to prove its efficacy. Second, the dissertation introduces a deep neural network that detects webshells using real-time HTTP traffic analysis of web applications. The study proposes an algorithm to improve the deep learning model's loss function to address data imbalance. We tested and compared the model to other studies on the CSE-CIC-IDS2018 dataset to prove its efficacy. We integrated the model with NetIDPS to improve webshell identification. Automatically blacklist attack source IPs and block URIs querying webshells on the web server to prevent these attacks.

Title: Can large language models be privacy preserving and fair medical coders?

• Authors: Ali Dadsetan, Dorsa Soleymani, Xijie Zeng, Frank Rudzicz
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.05533
• Pdf URL: https://arxiv.org/pdf/2412.05533
• Copy Paste: [[2412.05533]] Can large language models be privacy preserving and fair medical coders?(https://arxiv.org/abs/2412.05533)
• Keywords: privacy, protect, fair, large language model
• Abstract: Protecting patient data privacy is a critical concern when deploying machine learning algorithms in healthcare. Differential privacy (DP) is a common method for preserving privacy in such settings and, in this work, we examine two key trade-offs in applying DP to the NLP task of medical coding (ICD classification). Regarding the privacy-utility trade-off, we observe a significant performance drop in the privacy preserving models, with more than a 40% reduction in micro F1 scores on the top 50 labels in the MIMIC-III dataset. From the perspective of the privacy-fairness trade-off, we also observe an increase of over 3% in the recall gap between male and female patients in the DP models. Further understanding these trade-offs will help towards the challenges of real-world deployment.

Title: Memory-enhanced Invariant Prompt Learning for Urban Flow Prediction under Distribution Shifts

• Authors: Haiyang Jiang, Tong Chen, Wentao Zhang, Nguyen Quoc Viet Hung, Yuan Yuan, Yong Li, Lizhen Cui
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.05534
• Pdf URL: https://arxiv.org/pdf/2412.05534
• Copy Paste: [[2412.05534]] Memory-enhanced Invariant Prompt Learning for Urban Flow Prediction under Distribution Shifts(https://arxiv.org/abs/2412.05534)
• Keywords: robust
• Abstract: Urban flow prediction is a classic spatial-temporal forecasting task that estimates the amount of future traffic flow for a given location. Though models represented by Spatial-Temporal Graph Neural Networks (STGNNs) have established themselves as capable predictors, they tend to suffer from distribution shifts that are common with the urban flow data due to the dynamics and unpredictability of spatial-temporal events. Unfortunately, in spatial-temporal applications, the dynamic environments can hardly be quantified via a fixed number of parameters, whereas learning time- and location-specific environments can quickly become computationally prohibitive. In this paper, we propose a novel framework named Memory-enhanced Invariant Prompt learning (MIP) for urban flow prediction under constant distribution shifts. Specifically, MIP is equipped with a learnable memory bank that is trained to memorize the causal features within the spatial-temporal graph. By querying a trainable memory bank that stores the causal features, we adaptively extract invariant and variant prompts (i.e., patterns) for a given location at every time step. Then, instead of intervening the raw data based on simulated environments, we directly perform intervention on variant prompts across space and time. With the intervened variant prompts in place, we use invariant learning to minimize the variance of predictions, so as to ensure that the predictions are only made with invariant features. With extensive comparative experiments on two public urban flow datasets, we thoroughly demonstrate the robustness of MIP against OOD data.

Title: Uncovering Vision Modality Threats in Image-to-Image Tasks

• Authors: Hao Cheng, Erjia Xiao, Jiayan Yang, Jiahang Cao, Qiang Zhang, Jize Zhang, Kaidi Xu, Jindong Gu, Renjing Xu
• Subjects: cs.CV, cs.PF
• Abstract URL: https://arxiv.org/abs/2412.05538
• Pdf URL: https://arxiv.org/pdf/2412.05538
• Copy Paste: [[2412.05538]] Uncovering Vision Modality Threats in Image-to-Image Tasks(https://arxiv.org/abs/2412.05538)
• Keywords: security, defense, attack
• Abstract: Current image generation models can effortlessly produce high-quality, highly realistic images, but this also increases the risk of misuse. In various Text-to-Image or Image-to-Image tasks, attackers can generate a series of images containing inappropriate content by simply editing the language modality input. Currently, to prevent this security threat, the various guard or defense methods that are proposed also focus on defending the language modality. However, in practical applications, threats in the visual modality, particularly in tasks involving the editing of real-world images, pose greater security risks as they can easily infringe upon the rights of the image owner. Therefore, this paper uses a method named typographic attack to reveal that various image generation models also commonly face threats in the vision modality. Furthermore, we also evaluate the defense performance of various existing methods when facing threats in the vision modality and uncover their ineffectiveness. Finally, we propose the Vision Modal Threats in Image Generation Models (VMT-IGMs) dataset, which would serve as a baseline for evaluating the vision modality vulnerability of various image generation models.

Title: Radiant: Large-scale 3D Gaussian Rendering based on Hierarchical Framework

• Authors: Haosong Peng, Tianyu Qi, Yufeng Zhan, Hao Li, Yalun Dai, Yuanqing Xia
• Subjects: cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.05546
• Pdf URL: https://arxiv.org/pdf/2412.05546
• Copy Paste: [[2412.05546]] Radiant: Large-scale 3D Gaussian Rendering based on Hierarchical Framework(https://arxiv.org/abs/2412.05546)
• Keywords: privacy
• Abstract: With the advancement of computer vision, the recently emerged 3D Gaussian Splatting (3DGS) has increasingly become a popular scene reconstruction algorithm due to its outstanding performance. Distributed 3DGS can efficiently utilize edge devices to directly train on the collected images, thereby offloading computational demands and enhancing efficiency. However, traditional distributed frameworks often overlook computational and communication challenges in real-world environments, hindering large-scale deployment and potentially posing privacy risks. In this paper, we propose Radiant, a hierarchical 3DGS algorithm designed for large-scale scene reconstruction that considers system heterogeneity, enhancing the model performance and training efficiency. Via extensive empirical study, we find that it is crucial to partition the regions for each edge appropriately and allocate varying camera positions to each device for image collection and training. The core of Radiant is partitioning regions based on heterogeneous environment information and allocating workloads to each device accordingly. Furthermore, we provide a 3DGS model aggregation algorithm that enhances the quality and ensures the continuity of models' boundaries. Finally, we develop a testbed, and experiments demonstrate that Radiant improved reconstruction quality by up to 25.7\% and reduced up to 79.6\% end-to-end latency.

Title: Street Gaussians without 3D Object Tracker

• Authors: Ruida Zhang, Chengxi Li, Chenyangguang Zhang, Xingyu Liu, Haili Yuan, Yanyan Li, Xiangyang Ji, Gim Hee Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05548
• Pdf URL: https://arxiv.org/pdf/2412.05548
• Copy Paste: [[2412.05548]] Street Gaussians without 3D Object Tracker(https://arxiv.org/abs/2412.05548)
• Keywords: robust
• Abstract: Realistic scene reconstruction in driving scenarios poses significant challenges due to fast-moving objects. Most existing methods rely on labor-intensive manual labeling of object poses to reconstruct dynamic objects in canonical space and move them based on these poses during rendering. While some approaches attempt to use 3D object trackers to replace manual annotations, the limited generalization of 3D trackers -- caused by the scarcity of large-scale 3D datasets -- results in inferior reconstructions in real-world settings. In contrast, 2D foundation models demonstrate strong generalization capabilities. To eliminate the reliance on 3D trackers and enhance robustness across diverse environments, we propose a stable object tracking module by leveraging associations from 2D deep trackers within a 3D object fusion strategy. We address inevitable tracking errors by further introducing a motion learning strategy in an implicit feature space that autonomously corrects trajectory errors and recovers missed detections. Experimental results on Waymo-NOTR datasets show we achieve state-of-the-art performance. Our code will be made publicly available.

Title: CoE: Deep Coupled Embedding for Non-Rigid Point Cloud Correspondences

• Authors: Huajian Zeng, Maolin Gao, Daniel Cremers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05557
• Pdf URL: https://arxiv.org/pdf/2412.05557
• Copy Paste: [[2412.05557]] CoE: Deep Coupled Embedding for Non-Rigid Point Cloud Correspondences(https://arxiv.org/abs/2412.05557)
• Keywords: robust, segmentation
• Abstract: The interest in matching non-rigidly deformed shapes represented as raw point clouds is rising due to the proliferation of low-cost 3D sensors. Yet, the task is challenging since point clouds are irregular and there is a lack of intrinsic shape information. We propose to tackle these challenges by learning a new shape representation -- a per-point high dimensional embedding, in an embedding space where semantically similar points share similar embeddings. The learned embedding has multiple beneficial properties: it is aware of the underlying shape geometry and is robust to shape deformations and various shape artefacts, such as noise and partiality. Consequently, this embedding can be directly employed to retrieve high-quality dense correspondences through a simple nearest neighbor search in the embedding space. Extensive experiments demonstrate new state-of-the-art results and robustness in numerous challenging non-rigid shape matching benchmarks and show its great potential in other shape analysis tasks, such as segmentation.

Title: Text-to-3D Gaussian Splatting with Physics-Grounded Motion Generation

• Authors: Wenqing Wang, Yun Fu
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.05560
• Pdf URL: https://arxiv.org/pdf/2412.05560
• Copy Paste: [[2412.05560]] Text-to-3D Gaussian Splatting with Physics-Grounded Motion Generation(https://arxiv.org/abs/2412.05560)
• Keywords: diffusion, large language model
• Abstract: Text-to-3D generation is a valuable technology in virtual reality and digital content creation. While recent works have pushed the boundaries of text-to-3D generation, producing high-fidelity 3D objects with inefficient prompts and simulating their physics-grounded motion accurately still remain unsolved challenges. To address these challenges, we present an innovative framework that utilizes the Large Language Model (LLM)-refined prompts and diffusion priors-guided Gaussian Splatting (GS) for generating 3D models with accurate appearances and geometric structures. We also incorporate a continuum mechanics-based deformation map and color regularization to synthesize vivid physics-grounded motion for the generated 3D Gaussians, adhering to the conservation of mass and momentum. By integrating text-to-3D generation with physics-grounded motion synthesis, our framework renders photo-realistic 3D objects that exhibit physics-aware motion, accurately reflecting the behaviors of the objects under various forces and constraints across different materials. Extensive experiments demonstrate that our approach achieves high-quality 3D generations with realistic physics-grounded motion.

Title: A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions

• Authors: Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z. Ren, Anirudha Majumdar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05563
• Pdf URL: https://arxiv.org/pdf/2412.05563
• Copy Paste: [[2412.05563]] A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions(https://arxiv.org/abs/2412.05563)
• Keywords: large language model
• Abstract: The remarkable performance of large language models (LLMs) in content generation, coding, and common-sense reasoning has spurred widespread integration into many facets of society. However, integration of LLMs raises valid questions on their reliability and trustworthiness, given their propensity to generate hallucinations: plausible, factually-incorrect responses, which are expressed with striking confidence. Previous work has shown that hallucinations and other non-factual responses generated by LLMs can be detected by examining the uncertainty of the LLM in its response to the pertinent prompt, driving significant research efforts devoted to quantifying the uncertainty of LLMs. This survey seeks to provide an extensive review of existing uncertainty quantification methods for LLMs, identifying their salient features, along with their strengths and weaknesses. We present existing methods within a relevant taxonomy, unifying ostensibly disparate methods to aid understanding of the state of the art. Furthermore, we highlight applications of uncertainty quantification methods for LLMs, spanning chatbot and textual applications to embodied artificial intelligence applications in robotics. We conclude with open research challenges in uncertainty quantification of LLMs, seeking to motivate future research.

Title: Dif4FF: Leveraging Multimodal Diffusion Models and Graph Neural Networks for Accurate New Fashion Product Performance Forecasting

• Authors: Andrea Avogaro, Luigi Capogrosso, Franco Fummi, Marco Cristani
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05566
• Pdf URL: https://arxiv.org/pdf/2412.05566
• Copy Paste: [[2412.05566]] Dif4FF: Leveraging Multimodal Diffusion Models and Graph Neural Networks for Accurate New Fashion Product Performance Forecasting(https://arxiv.org/abs/2412.05566)
• Keywords: diffusion
• Abstract: In the fast-fashion industry, overproduction and unsold inventory create significant environmental problems. Precise sales forecasts for unreleased items could drastically improve the efficiency and profits of industries. However, predicting the success of entirely new styles is difficult due to the absence of past data and ever-changing trends. Specifically, currently used deterministic models struggle with domain shifts when encountering items outside their training data. The recently proposed diffusion models address this issue using a continuous-time diffusion process. Specifically, these models enable us to predict the sales of new items, mitigating the domain shift challenges encountered by deterministic models. As a result, this paper proposes Dif4FF, a novel two-stage pipeline for New Fashion Product Performance Forecasting (NFPPF) that leverages the power of diffusion models conditioned on multimodal data related to specific clothes. Dif4FF first utilizes a multimodal score-based diffusion model to forecast multiple sales trajectories for various garments over time. The forecasts are refined using a powerful Graph Convolutional Network (GCN) architecture. By leveraging the GCN's capability to capture long-range dependencies within both the temporal and spatial data and seeking the optimal solution between these two dimensions, Dif4FF offers the most accurate and efficient forecasting system available in the literature for predicting the sales of new items. We tested Dif4FF on VISUELLE, the de facto standard for NFPPF, achieving new state-of-the-art results.

Title: A polar coordinate system represents syntax in large language models

• Authors: Pablo Diego-Simón, Stéphane D'Ascoli, Emmanuel Chemla, Yair Lakretz, Jean-Rémi King
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05571
• Pdf URL: https://arxiv.org/pdf/2412.05571
• Copy Paste: [[2412.05571]] A polar coordinate system represents syntax in large language models(https://arxiv.org/abs/2412.05571)
• Keywords: large language model
• Abstract: Originally formalized with symbolic representations, syntactic trees may also be effectively represented in the activations of large language models (LLMs). Indeed, a 'Structural Probe' can find a subspace of neural activations, where syntactically related words are relatively close to one-another. However, this syntactic code remains incomplete: the distance between the Structural Probe word embeddings can represent the existence but not the type and direction of syntactic relations. Here, we hypothesize that syntactic relations are, in fact, coded by the relative direction between nearby embeddings. To test this hypothesis, we introduce a 'Polar Probe' trained to read syntactic relations from both the distance and the direction between word embeddings. Our approach reveals three main findings. First, our Polar Probe successfully recovers the type and direction of syntactic relations, and substantially outperforms the Structural Probe by nearly two folds. Second, we confirm that this polar coordinate system exists in a low-dimensional subspace of the intermediate layers of many LLMs and becomes increasingly precise in the latest frontier models. Third, we demonstrate with a new benchmark that similar syntactic relations are coded similarly across the nested levels of syntactic trees. Overall, this work shows that LLMs spontaneously learn a geometry of neural activations that explicitly represents the main symbolic structures of linguistic theory.

Title: From Deterministic to Probabilistic: A Novel Perspective on Domain Generalization for Medical Image Segmentation

• Authors: Yuheng Xu, Taiping Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05572
• Pdf URL: https://arxiv.org/pdf/2412.05572
• Copy Paste: [[2412.05572]] From Deterministic to Probabilistic: A Novel Perspective on Domain Generalization for Medical Image Segmentation(https://arxiv.org/abs/2412.05572)
• Keywords: robust, segmentation
• Abstract: Traditional domain generalization methods often rely on domain alignment to reduce inter-domain distribution differences and learn domain-invariant representations. However, domain shifts are inherently difficult to eliminate, which limits model generalization. To address this, we propose an innovative framework that enhances data representation quality through probabilistic modeling and contrastive learning, reducing dependence on domain alignment and improving robustness under domain variations. Specifically, we combine deterministic features with uncertainty modeling to capture comprehensive feature distributions. Contrastive learning enforces distribution-level alignment by aligning the mean and covariance of feature distributions, enabling the model to dynamically adapt to domain variations and mitigate distribution shifts. Additionally, we design a frequency-domain-based structural enhancement strategy using discrete wavelet transforms to preserve critical structural details and reduce visual distortions caused by style variations. Experimental results demonstrate that the proposed framework significantly improves segmentation performance, providing a robust solution to domain generalization challenges in medical image segmentation.

Title: STONet: A novel neural operator for modeling solute transport in micro-cracked reservoirs

• Authors: Ehsan Haghighat, Mohammad Hesan Adeli, S Mohammad Mousavi, Ruben Juanes
• Subjects: cs.LG, cs.CE, cs.NE, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2412.05576
• Pdf URL: https://arxiv.org/pdf/2412.05576
• Copy Paste: [[2412.05576]] STONet: A novel neural operator for modeling solute transport in micro-cracked reservoirs(https://arxiv.org/abs/2412.05576)
• Keywords: transformer
• Abstract: In this work, we develop a novel neural operator, the Solute Transport Operator Network (STONet), to efficiently model contaminant transport in micro-cracked reservoirs. The model combines different networks to encode heterogeneous properties effectively. By predicting the concentration rate, we are able to accurately model the transport process. Numerical experiments demonstrate that our neural operator approach achieves accuracy comparable to that of the finite element method. The previously introduced Enriched DeepONet architecture has been revised, motivated by the architecture of the popular multi-head attention of transformers, to improve its performance without increasing the compute cost. The computational efficiency of the proposed model enables rapid and accurate predictions of solute transport, facilitating the optimization of reservoir management strategies and the assessment of environmental impacts. The data and code for the paper will be published at this https URL.

Title: LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods

• Authors: Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, Yiqun Liu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.05579
• Pdf URL: https://arxiv.org/pdf/2412.05579
• Copy Paste: [[2412.05579]] LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods(https://arxiv.org/abs/2412.05579)
• Keywords: interpretability, large language model
• Abstract: The rapid advancement of Large Language Models (LLMs) has driven their expanding application across various fields. One of the most promising applications is their role as evaluators based on natural language responses, referred to as ''LLMs-as-judges''. This framework has attracted growing attention from both academia and industry due to their excellent effectiveness, ability to generalize across tasks, and interpretability in the form of natural language. This paper presents a comprehensive survey of the LLMs-as-judges paradigm from five key perspectives: Functionality, Methodology, Applications, Meta-evaluation, and Limitations. We begin by providing a systematic definition of LLMs-as-Judges and introduce their functionality (Why use LLM judges?). Then we address methodology to construct an evaluation system with LLMs (How to use LLM judges?). Additionally, we investigate the potential domains for their application (Where to use LLM judges?) and discuss methods for evaluating them in various contexts (How to evaluate LLM judges?). Finally, we provide a detailed analysis of the limitations of LLM judges and discuss potential future directions. Through a structured and comprehensive analysis, we aim aims to provide insights on the development and application of LLMs-as-judges in both research and practice. We will continue to maintain the relevant resource list at this https URL.

Title: Electrocardiogram (ECG) Based Cardiac Arrhythmia Detection and Classification using Machine Learning Algorithms

• Authors: Atit Pokharel, Shashank Dahal, Pratik Sapkota, Bhupendra Bimal Chhetri
• Subjects: cs.LG, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.05583
• Pdf URL: https://arxiv.org/pdf/2412.05583
• Copy Paste: [[2412.05583]] Electrocardiogram (ECG) Based Cardiac Arrhythmia Detection and Classification using Machine Learning Algorithms(https://arxiv.org/abs/2412.05583)
• Keywords: robust
• Abstract: The rapid advancements in Artificial Intelligence, specifically Machine Learning (ML) and Deep Learning (DL), have opened new prospects in medical sciences for improved diagnosis, prognosis, and treatment of severe health conditions. This paper focuses on the development of an ML model with high predictive accuracy to classify arrhythmic electrocardiogram (ECG) signals. The ECG signals datasets utilized in this study were sourced from the PhysioNet and MIT-BIH databases. The research commenced with binary classification, where an optimized Bidirectional Long Short-Term Memory (Bi-LSTM) model yielded excellent results in differentiating normal and atrial fibrillation signals. A pivotal aspect of this research was a survey among medical professionals, which not only validated the practicality of AI-based ECG classifiers but also identified areas for improvement, including accuracy and the inclusion of more arrhythmia types. These insights drove the development of an advanced Convolutional Neural Network (CNN) system capable of classifying five different types of ECG signals with better accuracy and precision. The CNN model's robust performance was ensured through rigorous stratified 5-fold cross validation. A web portal was also developed to demonstrate real-world utility, offering access to the trained model for real-time classification. This study highlights the potential applications of such models in remote health monitoring, predictive healthcare, assistive diagnostic tools, and simulated environments for educational training and interdisciplinary collaboration between data scientists and medical personnel.

Title: UMSPU: Universal Multi-Size Phase Unwrapping via Mutual Self-Distillation and Adaptive Boosting Ensemble Segmenters

• Authors: Lintong Du, Huazhen Liu, Yijia Zhang, ShuXin Liu, Yuan Qu, Zenghui Zhang, Jiamiao Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05584
• Pdf URL: https://arxiv.org/pdf/2412.05584
• Copy Paste: [[2412.05584]] UMSPU: Universal Multi-Size Phase Unwrapping via Mutual Self-Distillation and Adaptive Boosting Ensemble Segmenters(https://arxiv.org/abs/2412.05584)
• Keywords: robust, segmentation
• Abstract: Spatial phase unwrapping is a key technique for extracting phase information to obtain 3D morphology and other features. Modern industrial measurement scenarios demand high precision, large image sizes, and high speed. However, conventional methods struggle with noise resistance and processing speed. Current deep learning methods are limited by the receptive field size and sparse semantic information, making them ineffective for large size images. To address this issue, we propose a mutual self-distillation (MSD) mechanism and adaptive boosting ensemble segmenters to construct a universal multi-size phase unwrapping network (UMSPU). MSD performs hierarchical attention refinement and achieves cross-layer collaborative learning through bidirectional distillation, ensuring fine-grained semantic representation across image sizes. The adaptive boosting ensemble segmenters combine weak segmenters with different receptive fields into a strong one, ensuring stable segmentation across spatial frequencies. Experimental results show that UMSPU overcomes image size limitations, achieving high precision across image sizes ranging from 256*256 to 2048*2048 (an 8 times increase). It also outperforms existing methods in speed, robustness, and generalization. Its practicality is further validated in structured light imaging and InSAR. We believe that UMSPU offers a universal solution for phase unwrapping, with broad potential for industrial applications.

Title: UNet++ and LSTM combined approach for Breast Ultrasound Image Segmentation

• Authors: Saba Hesaraki, Morteza Akbari, Ramin Mousa
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05585
• Pdf URL: https://arxiv.org/pdf/2412.05585
• Copy Paste: [[2412.05585]] UNet++ and LSTM combined approach for Breast Ultrasound Image Segmentation(https://arxiv.org/abs/2412.05585)
• Keywords: extraction, segmentation
• Abstract: Breast cancer stands as a prevalent cause of fatality among females on a global scale, with prompt detection playing a pivotal role in diminishing mortality rates. The utilization of ultrasound scans in the BUSI dataset for medical imagery pertaining to breast cancer has exhibited commendable segmentation outcomes through the application of UNet and UNet++ networks. Nevertheless, a notable drawback of these models resides in their inattention towards the temporal aspects embedded within the images. This research endeavors to enrich the UNet++ architecture by integrating LSTM layers and self-attention mechanisms to exploit temporal characteristics for segmentation purposes. Furthermore, the incorporation of a Multiscale Feature Extraction Module aims to grasp varied scale features within the UNet++. Through the amalgamation of our proposed methodology with data augmentation on the BUSI with GT dataset, an accuracy rate of 98.88%, specificity of 99.53%, precision of 95.34%, sensitivity of 91.20%, F1-score of 93.74, and Dice coefficient of 92.74% are achieved. These findings demonstrate competitiveness with cutting-edge techniques outlined in existing literature.

Title: TB-HSU: Hierarchical 3D Scene Understanding with Contextual Affordances

• Authors: Wenting Xu, Viorela Ila, Luping Zhou, Craig T. Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05596
• Pdf URL: https://arxiv.org/pdf/2412.05596
• Copy Paste: [[2412.05596]] TB-HSU: Hierarchical 3D Scene Understanding with Contextual Affordances(https://arxiv.org/abs/2412.05596)
• Keywords: transformer
• Abstract: The concept of function and affordance is a critical aspect of 3D scene understanding and supports task-oriented objectives. In this work, we develop a model that learns to structure and vary functional affordance across a 3D hierarchical scene graph representing the spatial organization of a scene. The varying functional affordance is designed to integrate with the varying spatial context of the graph. More specifically, we develop an algorithm that learns to construct a 3D hierarchical scene graph (3DHSG) that captures the spatial organization of the scene. Starting from segmented object point clouds and object semantic labels, we develop a 3DHSG with a top node that identifies the room label, child nodes that define local spatial regions inside the room with region-specific affordances, and grand-child nodes indicating object locations and object-specific affordances. To support this work, we create a custom 3DHSG dataset that provides ground truth data for local spatial regions with region-specific affordances and also object-specific affordances for each object. We employ a transformer-based model to learn the 3DHSG. We use a multi-task learning framework that learns both room classification and learns to define spatial regions within the room with region-specific affordances. Our work improves on the performance of state-of-the-art baseline models and shows one approach for applying transformer models to 3D scene understanding and the generation of 3DHSGs that capture the spatial organization of a room. The code and dataset are publicly available.

Title: RefSAM3D: Adapting SAM with Cross-modal Reference for 3D Medical Image Segmentation

• Authors: Xiang Gao, Kai Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05605
• Pdf URL: https://arxiv.org/pdf/2412.05605
• Copy Paste: [[2412.05605]] RefSAM3D: Adapting SAM with Cross-modal Reference for 3D Medical Image Segmentation(https://arxiv.org/abs/2412.05605)
• Keywords: transformer, segmentation
• Abstract: The Segment Anything Model (SAM), originally built on a 2D Vision Transformer (ViT), excels at capturing global patterns in 2D natural images but struggles with 3D medical imaging modalities like CT and MRI. These modalities require capturing spatial information in volumetric space for tasks such as organ segmentation and tumor quantification. To address this challenge, we introduce RefSAM3D, which adapts SAM for 3D medical imaging by incorporating a 3D image adapter and cross-modal reference prompt generation. Our approach modifies the visual encoder to handle 3D inputs and enhances the mask decoder for direct 3D mask generation. We also integrate textual prompts to improve segmentation accuracy and consistency in complex anatomical scenarios. By employing a hierarchical attention mechanism, our model effectively captures and integrates information across different scales. Extensive evaluations on multiple medical imaging datasets demonstrate the superior performance of RefSAM3D over state-of-the-art methods. Our contributions advance the application of SAM in accurately segmenting complex anatomical structures in medical imaging.

Title: Leveraging Security Observability to Strengthen Security of Digital Ecosystem Architecture

• Authors: Renjith Ramachandran
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.05617
• Pdf URL: https://arxiv.org/pdf/2412.05617
• Copy Paste: [[2412.05617]] Leveraging Security Observability to Strengthen Security of Digital Ecosystem Architecture(https://arxiv.org/abs/2412.05617)
• Keywords: security, protect, robust
• Abstract: In the current fast-paced digital environment, enterprises are striving to offer a seamless and integrated customer experience across multiple touchpoints. This improved experience often leads to higher conversion rates and increased customer loyalty. To deliver such an experience, enterprises must think beyond the traditional boundaries of their architecture. The architecture of the digital ecosystem is expanding and becoming more complex, achieved either by developing advanced features in-house or by integrating with third-party solutions, thus extending the boundaries of the enterprise architecture. This complexity poses significant challenges for both observability and security in a digital ecosystem, both of which are essential for maintaining robust and resilient systems. Observability entails monitoring and understanding the internal state of a system through logging, tracing, and metrics collection, allowing organizations to diagnose performance issues and detect anomalies in real time. Meanwhile, security is focused on protecting sensitive data and ensuring service integrity by defending against threats and vulnerabilities. The data collected through these observability practices can be analyzed to identify patterns and detect potential security threats or data leaks. This paper examines the interconnections between observability and security within digital ecosystem architectures, emphasizing how improved observability can strengthen security measures. The paper also discusses studies conducted in the AI/ML field aimed at enhancing security through the use of observability. These studies explore how advanced machine learning techniques can be applied to observability data to improve security measures and detect anomalies more effectively.

Title: Do We Need to Design Specific Diffusion Models for Different Tasks? Try ONE-PIC

• Authors: Ming Tao, Bing-Kun Bao, Yaowei Wang, Changsheng Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05619
• Pdf URL: https://arxiv.org/pdf/2412.05619
• Copy Paste: [[2412.05619]] Do We Need to Design Specific Diffusion Models for Different Tasks? Try ONE-PIC(https://arxiv.org/abs/2412.05619)
• Keywords: diffusion, generative, large language model
• Abstract: Large pretrained diffusion models have demonstrated impressive generation capabilities and have been adapted to various downstream tasks. However, unlike Large Language Models (LLMs) that can learn multiple tasks in a single model based on instructed data, diffusion models always require additional branches, task-specific training strategies, and losses for effective adaptation to different downstream tasks. This task-specific fine-tuning approach brings two drawbacks. 1) The task-specific additional networks create gaps between pretraining and fine-tuning which hinders the transfer of pretrained knowledge. 2) It necessitates careful additional network design, raising the barrier to learning and implementation, and making it less user-friendly. Thus, a question arises: Can we achieve a simple, efficient, and general approach to fine-tune diffusion models? To this end, we propose ONE-PIC. It enhances the inherited generative ability in the pretrained diffusion models without introducing additional modules. Specifically, we propose In-Visual-Context Tuning, which constructs task-specific training data by arranging source images and target images into a single image. This approach makes downstream fine-tuning closer to the pertaining, allowing our model to adapt more quickly to various downstream tasks. Moreover, we propose a Masking Strategy to unify different generative tasks. This strategy transforms various downstream fine-tuning tasks into predictions of the masked portions. The extensive experimental results demonstrate that our method is simple and efficient which streamlines the adaptation process and achieves excellent performance with lower costs. Code is available at this https URL.

Title: Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising

• Authors: Gongfan Fang, Xinyin Ma, Xinchao Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05628
• Pdf URL: https://arxiv.org/pdf/2412.05628
• Copy Paste: [[2412.05628]] Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising(https://arxiv.org/abs/2412.05628)
• Keywords: diffusion, transformer, generative
• Abstract: Transformer-based diffusion models have achieved significant advancements across a variety of generative tasks. However, producing high-quality outputs typically necessitates large transformer models, which result in substantial training and inference overhead. In this work, we investigate an alternative approach involving multiple experts for denoising, and introduce Remix-DiT, a novel method designed to enhance output quality at a low cost. The goal of Remix-DiT is to craft N diffusion experts for different denoising timesteps, yet without the need for expensive training of N independent models. To achieve this, Remix-DiT employs K basis models (where K < N) and utilizes learnable mixing coefficients to adaptively craft expert models. This design offers two significant advantages: first, although the total model size is increased, the model produced by the mixing operation shares the same architecture as a plain model, making the overall model as efficient as a standard diffusion transformer. Second, the learnable mixing adaptively allocates model capacity across timesteps, thereby effectively improving generation quality. Experiments conducted on the ImageNet dataset demonstrate that Remix-DiT achieves promising results compared to standard diffusion transformers and other multiple-expert methods. The code is available at this https URL.

Title: CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worlds

• Authors: Lei Wang, Jianxun Lian, Yi Huang, Yanqi Dai, Haoxuan Li, Xu Chen, Xing Xie, Ji-Rong Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05631
• Pdf URL: https://arxiv.org/pdf/2412.05631
• Copy Paste: [[2412.05631]] CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worlds(https://arxiv.org/abs/2412.05631)
• Keywords: large language model
• Abstract: Role-playing is a crucial capability of Large Language Models (LLMs), enabling a wide range of practical applications, including intelligent non-player characters, digital twins, and emotional companions. Evaluating this capability in LLMs is challenging due to the complex dynamics involved in role-playing, such as maintaining character fidelity throughout a storyline and navigating open-ended narratives without a definitive ground truth. Current evaluation methods, which primarily focus on question-answering or conversational snapshots, fall short of adequately capturing the nuanced character traits and behaviors essential for authentic role-playing. In this paper, we propose CharacterBox, which is a simulation sandbox designed to generate situational fine-grained character behavior trajectories. These behavior trajectories enable a more comprehensive and in-depth evaluation of role-playing capabilities. CharacterBox consists of two main components: the character agent and the narrator agent. The character agent, grounded in psychological and behavioral science, exhibits human-like behaviors, while the narrator agent coordinates interactions between character agents and environmental changes. Additionally, we introduce two trajectory-based methods that leverage CharacterBox to enhance LLM performance. To reduce costs and facilitate the adoption of CharacterBox by public communities, we fine-tune two smaller models, CharacterNR and CharacterRM, as substitutes for GPT API calls, and demonstrate their competitive performance compared to advanced GPT APIs.

Title: Biological Brain Age Estimation using Sex-Aware Adversarial Variational Autoencoder with Multimodal Neuroimages

• Authors: Abd Ur Rehman, Azka Rehman, Muhammad Usman, Abdullah Shahid, Sung-Min Gho, Aleum Lee, Tariq M. Khan, Imran Razzak
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05632
• Pdf URL: https://arxiv.org/pdf/2412.05632
• Copy Paste: [[2412.05632]] Biological Brain Age Estimation using Sex-Aware Adversarial Variational Autoencoder with Multimodal Neuroimages(https://arxiv.org/abs/2412.05632)
• Keywords: robust, generative
• Abstract: Brain aging involves structural and functional changes and therefore serves as a key biomarker for brain health. Combining structural magnetic resonance imaging (sMRI) and functional magnetic resonance imaging (fMRI) has the potential to improve brain age estimation by leveraging complementary data. However, fMRI data, being noisier than sMRI, complicates multimodal fusion. Traditional fusion methods often introduce more noise than useful information, which can reduce accuracy compared to using sMRI alone. In this paper, we propose a novel multimodal framework for biological brain age estimation, utilizing a sex-aware adversarial variational autoencoder (SA-AVAE). Our framework integrates adversarial and variational learning to effectively disentangle the latent features from both modalities. Specifically, we decompose the latent space into modality-specific codes and shared codes to represent complementary and common information across modalities, respectively. To enhance the disentanglement, we introduce cross-reconstruction and shared-distinct distance ratio loss as regularization terms. Importantly, we incorporate sex information into the learned latent code, enabling the model to capture sex-specific aging patterns for brain age estimation via an integrated regressor module. We evaluate our model using the publicly available OpenBHB dataset, a comprehensive multi-site dataset for brain age estimation. The results from ablation studies and comparisons with state-of-the-art methods demonstrate that our framework outperforms existing approaches and shows significant robustness across various age groups, highlighting its potential for real-time clinical applications in the early detection of neurodegenerative diseases.

Title: Efficient Continuous Video Flow Model for Video Prediction

• Authors: Gaurav Shrivastava, Abhinav Shrivastava
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05633
• Pdf URL: https://arxiv.org/pdf/2412.05633
• Copy Paste: [[2412.05633]] Efficient Continuous Video Flow Model for Video Prediction(https://arxiv.org/abs/2412.05633)
• Keywords: diffusion
• Abstract: Multi-step prediction models, such as diffusion and rectified flow models, have emerged as state-of-the-art solutions for generation tasks. However, these models exhibit higher latency in sampling new frames compared to single-step methods. This latency issue becomes a significant bottleneck when adapting such methods for video prediction tasks, given that a typical 60-second video comprises approximately 1.5K frames. In this paper, we propose a novel approach to modeling the multi-step process, aimed at alleviating latency constraints and facilitating the adaptation of such processes for video prediction tasks. Our approach not only reduces the number of sample steps required to predict the next frame but also minimizes computational demands by reducing the model size to one-third of the original size. We evaluate our method on standard video prediction datasets, including KTH, BAIR action robot, Human3.6M and UCF101, demonstrating its efficacy in achieving state-of-the-art performance on these benchmarks.

Title: Mixture of Hidden-Dimensions Transformer

• Authors: Yilong Chen, Junyuan Shang, Zhengyu Zhang, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05644
• Pdf URL: https://arxiv.org/pdf/2412.05644
• Copy Paste: [[2412.05644]] Mixture of Hidden-Dimensions Transformer(https://arxiv.org/abs/2412.05644)
• Keywords: transformer
• Abstract: Transformer models encounter challenges in scaling hidden dimensions efficiently, as uniformly increasing them inflates computational and memory costs while failing to emphasize the most relevant features for each token. For further understanding, we study hidden dimension sparsity and observe that trained Transformers utilize only a small fraction of token dimensions, revealing an "activation flow" pattern. Notably, there are shared sub-dimensions with sustained activation across multiple consecutive tokens and specialized sub-dimensions uniquely activated for each token. To better model token-relevant sub-dimensions, we propose MoHD (Mixture of Hidden Dimensions), a sparse conditional activation architecture. Particularly, MoHD employs shared sub-dimensions for common token features and a routing mechanism to dynamically activate specialized sub-dimensions. To mitigate potential information loss from sparsity, we design activation scaling and group fusion mechanisms to preserve activation flow. In this way, MoHD expands hidden dimensions with negligible increases in computation or parameters, efficient training and inference while maintaining performance. Evaluations across 10 NLP tasks show that MoHD surpasses Vanilla Transformers in parameter efficiency and task performance. It achieves 1.7% higher performance with 50% fewer activation parameters and 3.7% higher performance with a 3x parameter expansion at constant activation cost. MOHD offers a new perspective for scaling the model, showcasing the potential of hidden dimension sparsity to boost efficiency

Title: Shifting NER into High Gear: The Auto-AdvER Approach

• Authors: Filippos Ventirozos, Ioanna Nteka, Tania Nandy, Jozef Baca, Peter Appleby, Matthew Shardlow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05655
• Pdf URL: https://arxiv.org/pdf/2412.05655
• Copy Paste: [[2412.05655]] Shifting NER into High Gear: The Auto-AdvER Approach(https://arxiv.org/abs/2412.05655)
• Keywords: large language model
• Abstract: This paper presents a case study on the development of Auto-AdvER, a specialised named entity recognition schema and dataset for text in the car advertisement genre. Developed with industry needs in mind, Auto-AdvER is designed to enhance text mining analytics in this domain and contributes a linguistically unique NER dataset. We present a schema consisting of three labels: "Condition", "Historic" and "Sales Options". We outline the guiding principles for annotation, describe the methodology for schema development, and show the results of an annotation study demonstrating inter-annotator agreement of 92% F1-Score. Furthermore, we compare the performance by using encoder-only models: BERT, DeBERTaV3 and decoder-only open and closed source Large Language Models (LLMs): Llama, Qwen, GPT-4 and Gemini. Our results show that the class of LLMs outperforms the smaller encoder-only models. However, the LLMs are costly and far from perfect for this task. We present this work as a stepping stone toward more fine-grained analysis and discuss Auto-AdvER's potential impact on advertisement analytics and customer insights, including applications such as the analysis of market dynamics and data-driven predictive maintenance. Our schema, as well as our associated findings, are suitable for both private and public entities considering named entity recognition in the automotive domain, or other specialist domains.

Title: Towards Robust Spatio-Temporal Auto-Regressive Prediction: Adams-Bashforth Time Integration with Adaptive Multi-Step Rollout

• Authors: Sunwoong Yang, Ricardo Vinuesa, Namwoo Kang
• Subjects: cs.LG, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2412.05657
• Pdf URL: https://arxiv.org/pdf/2412.05657
• Copy Paste: [[2412.05657]] Towards Robust Spatio-Temporal Auto-Regressive Prediction: Adams-Bashforth Time Integration with Adaptive Multi-Step Rollout(https://arxiv.org/abs/2412.05657)
• Keywords: robust
• Abstract: This study addresses the critical challenge of error accumulation in spatio-temporal auto-regressive predictions within scientific machine learning models by introducing innovative temporal integration schemes and adaptive multi-step rollout strategies. We present a comprehensive analysis of time integration methods, highlighting the adaptation of the two-step Adams-Bashforth scheme to enhance long-term prediction robustness in auto-regressive models. Additionally, we improve temporal prediction accuracy through a multi-step rollout strategy that incorporates multiple future time steps during training, supported by three newly proposed approaches that dynamically adjust the importance of each future step. By integrating the Adams-Bashforth scheme with adaptive multi-step strategies, our graph neural network-based auto-regressive model accurately predicts 350 future time steps, even under practical constraints such as limited training data and minimal model capacity -- achieving an error of only 1.6% compared to the vanilla auto-regressive approach. Moreover, our framework demonstrates an 83% improvement in rollout performance over the standard noise injection method, a standard technique for enhancing long-term rollout performance. Its effectiveness is further validated in more challenging scenarios with truncated meshes, showcasing its adaptability and robustness in practical applications. This work introduces a versatile framework for robust long-term spatio-temporal auto-regressive predictions, effectively mitigating error accumulation across various model types and engineering discipline.

Title: Multimodal Biometric Authentication Using Camera-Based PPG and Fingerprint Fusion

• Authors: Xue Xian Zheng, M. M. Ur Rahma, Bilal Taha, Mudassir Masood, Dimitrios Hatzinakos, Tareq Al-Naffouri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05660
• Pdf URL: https://arxiv.org/pdf/2412.05660
• Copy Paste: [[2412.05660]] Multimodal Biometric Authentication Using Camera-Based PPG and Fingerprint Fusion(https://arxiv.org/abs/2412.05660)
• Keywords: secure, biometric
• Abstract: Camera-based photoplethysmography (PPG) obtained from smartphones has shown great promise for personalized healthcare and secure authentication. This paper presents a multimodal biometric system that integrates PPG signals extracted from videos with fingerprint data to enhance the accuracy of user verification. The system requires users to place their fingertip on the camera lens for a few seconds, allowing the capture and processing of unique biometric characteristics. Our approach employs a neural network with two structured state-space model (SSM) encoders to manage the distinct modalities. Fingerprint images are transformed into pixel sequences, and along with segmented PPG waveforms, they are input into the encoders. A cross-modal attention mechanism then extracts refined feature representations, and a distribution-oriented contrastive loss function aligns these features within a unified latent space. Experimental results demonstrate the system's superior performance across various evaluation metrics in both single-session and dual-session authentication scenarios.

Title: Detecting outliers by clustering algorithms

• Authors: Qi Li, Shuliang Wang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.05669
• Pdf URL: https://arxiv.org/pdf/2412.05669
• Copy Paste: [[2412.05669]] Detecting outliers by clustering algorithms(https://arxiv.org/abs/2412.05669)
• Keywords: robust
• Abstract: Clustering and outlier detection are two important tasks in data mining. Outliers frequently interfere with clustering algorithms to determine the similarity between objects, resulting in unreliable clustering results. Currently, only a few clustering algorithms (e.g., DBSCAN) have the ability to detect outliers to eliminate interference. For other clustering algorithms, it is tedious to introduce another outlier detection task to eliminate outliers before each clustering process. Obviously, how to equip more clustering algorithms with outlier detection ability is very meaningful. Although a common strategy allows clustering algorithms to detect outliers based on the distance between objects and clusters, it is contradictory to improving the performance of clustering algorithms on the datasets with outliers. In this paper, we propose a novel outlier detection approach, called ODAR, for clustering. ODAR maps outliers and normal objects into two separated clusters by feature transformation. As a result, any clustering algorithm can detect outliers by identifying clusters. Experiments show that ODAR is robust to diverse datasets. Compared with baseline methods, the clustering algorithms achieve the best on 7 out of 10 datasets with the help of ODAR, with at least 5% improvement in accuracy.

Title: M$^3$PC: Test-time Model Predictive Control for Pretrained Masked Trajectory Model

• Authors: Kehan Wen, Yutong Hu, Yao Mu, Lei Ke
• Subjects: cs.LG, cs.RO, eess.SY
• Abstract URL: https://arxiv.org/abs/2412.05675
• Pdf URL: https://arxiv.org/pdf/2412.05675
• Copy Paste: [[2412.05675]] M$^3$PC: Test-time Model Predictive Control for Pretrained Masked Trajectory Model(https://arxiv.org/abs/2412.05675)
• Keywords: transformer
• Abstract: Recent work in Offline Reinforcement Learning (RL) has shown that a unified Transformer trained under a masked auto-encoding objective can effectively capture the relationships between different modalities (e.g., states, actions, rewards) within given trajectory datasets. However, this information has not been fully exploited during the inference phase, where the agent needs to generate an optimal policy instead of just reconstructing masked components from unmasked ones. Given that a pretrained trajectory model can act as both a Policy Model and a World Model with appropriate mask patterns, we propose using Model Predictive Control (MPC) at test time to leverage the model's own predictive capability to guide its action selection. Empirical results on D4RL and RoboMimic show that our inference-phase MPC significantly improves the decision-making performance of a pretrained trajectory model without any additional parameter training. Furthermore, our framework can be adapted to Offline to Online (O2O) RL and Goal Reaching RL, resulting in more substantial performance gains when an additional online interaction budget is provided, and better generalization capabilities when different task targets are specified. Code is available: this https URL.

Title: Nearly Solved? Robust Deepfake Detection Requires More than Visual Forensics

• Authors: Guy Levy, Nathan Liebmann
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05676
• Pdf URL: https://arxiv.org/pdf/2412.05676
• Copy Paste: [[2412.05676]] Nearly Solved? Robust Deepfake Detection Requires More than Visual Forensics(https://arxiv.org/abs/2412.05676)
• Keywords: attack, robust
• Abstract: Deepfakes are on the rise, with increased sophistication and prevalence allowing for high-profile social engineering attacks. Detecting them in the wild is therefore important as ever, giving rise to new approaches breaking benchmark records in this task. In line with previous work, we show that recently developed state-of-the-art detectors are susceptible to classical adversarial attacks, even in a highly-realistic black-box setting, putting their usability in question. We argue that crucial 'robust features' of deepfakes are in their higher semantics, and follow that with evidence that a detector based on a semantic embedding model is less susceptible to black-box perturbation attacks. We show that large visuo-lingual models like GPT-4o can perform zero-shot deepfake detection better than current state-of-the-art methods, and introduce a novel attack based on high-level semantic manipulation. Finally, we argue that hybridising low- and high-level detectors can improve adversarial robustness, based on their complementary strengths and weaknesses.

Title: RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Experts

• Authors: Xu Liu, Zhouhui Lian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05679
• Pdf URL: https://arxiv.org/pdf/2412.05679
• Copy Paste: [[2412.05679]] RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Experts(https://arxiv.org/abs/2412.05679)
• Keywords: segmentation
• Abstract: Remote Sensing Vision-Language Models (RS VLMs) have made much progress in the tasks of remote sensing (RS) image comprehension. While performing well in multi-modal reasoning and multi-turn conversations, the existing models lack pixel-level understanding and struggle with multi-image inputs. In this work, we propose RSUniVLM, a unified, end-to-end RS VLM designed for comprehensive vision understanding across multiple granularity, including image-level, region-level, and pixel-level tasks. RSUniVLM also performs effectively in multi-image analysis, with instances of change detection and change captioning. To enhance the model's ability to capture visual information at different levels without increasing model size, we design a novel architecture called Granularity-oriented Mixture of Experts to constraint the model to about 1 billion parameters. We also construct a large-scale RS instruction-following dataset based on a variety of existing datasets in both RS and general domain, encompassing various tasks such as object localization, visual question answering, and semantic segmentation. Substantial experiments have been conducted to validate the superiority of the proposed RSUniVLM up to state-of-the-art across various RS tasks. Code and model will be available at \href{this https URL}{here}.

Title: WATER-GS: Toward Copyright Protection for 3D Gaussian Splatting via Universal Watermarking

• Authors: Yuqi Tan, Xiang Liu, Shuzhao Xie, Bin Chen, Shu-Tao Xia, Zhi Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.05695
• Pdf URL: https://arxiv.org/pdf/2412.05695
• Copy Paste: [[2412.05695]] WATER-GS: Toward Copyright Protection for 3D Gaussian Splatting via Universal Watermarking(https://arxiv.org/abs/2412.05695)
• Keywords: protect, robust, watermark, generative
• Abstract: 3D Gaussian Splatting (3DGS) has emerged as a pivotal technique for 3D scene representation, providing rapid rendering speeds and high fidelity. As 3DGS gains prominence, safeguarding its intellectual property becomes increasingly crucial since 3DGS could be used to imitate unauthorized scene creations and raise copyright issues. Existing watermarking methods for implicit NeRFs cannot be directly applied to 3DGS due to its explicit representation and real-time rendering process, leaving watermarking for 3DGS largely unexplored. In response, we propose WATER-GS, a novel method designed to protect 3DGS copyrights through a universal watermarking strategy. First, we introduce a pre-trained watermark decoder, treating raw 3DGS generative modules as potential watermark encoders to ensure imperceptibility. Additionally, we implement novel 3D distortion layers to enhance the robustness of the embedded watermark against common real-world distortions of point cloud data. Comprehensive experiments and ablation studies demonstrate that WATER-GS effectively embeds imperceptible and robust watermarks into 3DGS without compromising rendering efficiency and quality. Our experiments indicate that the 3D distortion layers can yield up to a 20% improvement in accuracy rate. Notably, our method is adaptable to different 3DGS variants, including 3DGS compression frameworks and 2D Gaussian splatting.

Title: Segment-Level Road Obstacle Detection Using Visual Foundation Model Priors and Likelihood Ratios

• Authors: Youssef Shoeb, Nazir Nayal, Azarm Nowzard, Fatma Güney, Hanno Gottschalk
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05707
• Pdf URL: https://arxiv.org/pdf/2412.05707
• Copy Paste: [[2412.05707]] Segment-Level Road Obstacle Detection Using Visual Foundation Model Priors and Likelihood Ratios(https://arxiv.org/abs/2412.05707)
• Keywords: robust
• Abstract: Detecting road obstacles is essential for autonomous vehicles to navigate dynamic and complex traffic environments safely. Current road obstacle detection methods typically assign a score to each pixel and apply a threshold to generate final predictions. However, selecting an appropriate threshold is challenging, and the per-pixel classification approach often leads to fragmented predictions with numerous false positives. In this work, we propose a novel method that leverages segment-level features from visual foundation models and likelihood ratios to predict road obstacles directly. By focusing on segments rather than individual pixels, our approach enhances detection accuracy, reduces false positives, and offers increased robustness to scene variability. We benchmark our approach against existing methods on the RoadObstacle and LostAndFound datasets, achieving state-of-the-art performance without needing a predefined threshold.

Title: On the effective transfer of knowledge from English to Hindi Wikipedia

• Authors: Paramita Das, Amartya Roy, Ritabrata Chakraborty, Animesh Mukherjee
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05708
• Pdf URL: https://arxiv.org/pdf/2412.05708
• Copy Paste: [[2412.05708]] On the effective transfer of knowledge from English to Hindi Wikipedia(https://arxiv.org/abs/2412.05708)
• Keywords: large language model
• Abstract: Although Wikipedia is the largest multilingual encyclopedia, it remains inherently incomplete. There is a significant disparity in the quality of content between high-resource languages (HRLs, e.g., English) and low-resource languages (LRLs, e.g., Hindi), with many LRL articles lacking adequate information. To bridge these content gaps, we propose a lightweight framework to enhance knowledge equity between English and Hindi. In case the English Wikipedia page is not up-to-date, our framework extracts relevant information from external resources readily available (such as English books) and adapts it to align with Wikipedia's distinctive style, including its \textit{neutral point of view} (NPOV) policy, using in-context learning capabilities of large language models. The adapted content is then machine-translated into Hindi for integration into the corresponding Wikipedia articles. On the other hand, if the English version is comprehensive and up-to-date, the framework directly transfers knowledge from English to Hindi. Our framework effectively generates new content for Hindi Wikipedia sections, enhancing Hindi Wikipedia articles respectively by 65% and 62% according to automatic and human judgment-based evaluations.

Title: PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks

• Authors: Soumya Suvra Ghosal, Soumyabrata Pal, Koyel Mukherjee, Dinesh Manocha
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.05710
• Pdf URL: https://arxiv.org/pdf/2412.05710
• Copy Paste: [[2412.05710]] PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks(https://arxiv.org/abs/2412.05710)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have recently demonstrated impressive few-shot learning capabilities through in-context learning (ICL). However, ICL performance is highly dependent on the choice of few-shot demonstrations, making the selection of the most optimal examples a persistent research challenge. This issue is further amplified in low-resource Indic languages, where the scarcity of ground-truth data complicates the selection process. In this work, we propose PromptRefine, a novel Alternating Minimization approach for example selection that improves ICL performance on low-resource Indic languages. PromptRefine leverages auxiliary example banks from related high-resource Indic languages and employs multi-task learning techniques to align language-specific retrievers, enabling effective cross-language retrieval. Additionally, we incorporate diversity in the selected examples to enhance generalization and reduce bias. Through comprehensive evaluations on four text generation tasks -- Cross-Lingual Question Answering, Multilingual Question Answering, Machine Translation, and Cross-Lingual Summarization using state-of-the-art LLMs such as LLAMA-3.1-8B, LLAMA-2-7B, Qwen-2-7B, and Qwen-2.5-7B, we demonstrate that PromptRefine significantly outperforms existing frameworks for retrieving examples.

Title: Evaluating Hallucination in Text-to-Image Diffusion Models with Scene-Graph based Question-Answering Agent

• Authors: Ziyuan Qin, Dongjie Cheng, Haoyu Wang, Huahui Yi, Yuting Shao, Zhiyuan Fan, Kang Li, Qicheng Lao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05722
• Pdf URL: https://arxiv.org/pdf/2412.05722
• Copy Paste: [[2412.05722]] Evaluating Hallucination in Text-to-Image Diffusion Models with Scene-Graph based Question-Answering Agent(https://arxiv.org/abs/2412.05722)
• Keywords: diffusion, large language model
• Abstract: Contemporary Text-to-Image (T2I) models frequently depend on qualitative human evaluations to assess the consistency between synthesized images and the text prompts. There is a demand for quantitative and automatic evaluation tools, given that human evaluation lacks reproducibility. We believe that an effective T2I evaluation metric should accomplish the following: detect instances where the generated images do not align with the textual prompts, a discrepancy we define as the `hallucination problem' in T2I tasks; record the types and frequency of hallucination issues, aiding users in understanding the causes of errors; and provide a comprehensive and intuitive scoring that close to human standard. To achieve these objectives, we propose a method based on large language models (LLMs) for conducting question-answering with an extracted scene-graph and created a dataset with human-rated scores for generated images. From the methodology perspective, we combine knowledge-enhanced question-answering tasks with image evaluation tasks, making the evaluation metrics more controllable and easier to interpret. For the contribution on the dataset side, we generated 12,000 synthesized images based on 1,000 composited prompts using three advanced T2I models. Subsequently, we conduct human scoring on all synthesized images and prompt pairs to validate the accuracy and effectiveness of our method as an evaluation metric. All generated images and the human-labeled scores will be made publicly available in the future to facilitate ongoing research on this crucial issue. Extensive experiments show that our method aligns more closely with human scoring patterns than other evaluation metrics.

Title: A Tiered GAN Approach for Monet-Style Image Generation

• Authors: FNU Neha, Deepshikha Bhati, Deepak Kumar Shukla, Md Amiruzzaman
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05724
• Pdf URL: https://arxiv.org/pdf/2412.05724
• Copy Paste: [[2412.05724]] A Tiered GAN Approach for Monet-Style Image Generation(https://arxiv.org/abs/2412.05724)
• Keywords: generative
• Abstract: Generative Adversarial Networks (GANs) have proven to be a powerful tool in generating artistic images, capable of mimicking the styles of renowned painters, such as Claude Monet. This paper introduces a tiered GAN model to progressively refine image quality through a multi-stage process, enhancing the generated images at each step. The model transforms random noise into detailed artistic representations, addressing common challenges such as instability in training, mode collapse, and output quality. This approach combines downsampling and convolutional techniques, enabling the generation of high-quality Monet-style artwork while optimizing computational efficiency. Experimental results demonstrate the architecture's ability to produce foundational artistic structures, though further refinements are necessary for achieving higher levels of realism and fidelity to Monet's style. Future work focuses on improving training methodologies and model complexity to bridge the gap between generated and true artistic images. Additionally, the limitations of traditional GANs in artistic generation are analyzed, and strategies to overcome these shortcomings are proposed.

Title: Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events

• Authors: Aditya Chinchure, Sahithya Ravi, Raymond Ng, Vered Shwartz, Boyang Li, Leonid Sigal
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05725
• Pdf URL: https://arxiv.org/pdf/2412.05725
• Copy Paste: [[2412.05725]] Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events(https://arxiv.org/abs/2412.05725)
• Keywords: generative
• Abstract: The commonsense reasoning capabilities of vision-language models (VLMs), especially in abductive reasoning and defeasible reasoning, remain poorly understood. Most benchmarks focus on typical visual scenarios, making it difficult to discern whether model performance stems from keen perception and reasoning skills, or reliance on pure statistical recall. We argue that by focusing on atypical events in videos, clearer insights can be gained on the core capabilities of VLMs. Explaining and understanding such out-of-distribution events requires models to extend beyond basic pattern recognition and regurgitation of their prior knowledge. To this end, we introduce BlackSwanSuite, a benchmark for evaluating VLMs' ability to reason about unexpected events through abductive and defeasible tasks. Our tasks artificially limit the amount of visual information provided to models while questioning them about hidden unexpected events, or provide new visual information that could change an existing hypothesis about the event. We curate a comprehensive benchmark suite comprising over 3,800 MCQ, 4,900 generative and 6,700 yes/no tasks, spanning 1,655 videos. After extensively evaluating various state-of-the-art VLMs, including GPT-4o and Gemini 1.5 Pro, as well as open-source VLMs such as LLaVA-Video, we find significant performance gaps of up to 32% from humans on these tasks. Our findings reveal key limitations in current VLMs, emphasizing the need for enhanced model architectures and training strategies.

Title: Integrating YOLO11 and Convolution Block Attention Module for Multi-Season Segmentation of Tree Trunks and Branches in Commercial Apple Orchards

• Authors: Ranjan Sapkota, Manoj Karkee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05728
• Pdf URL: https://arxiv.org/pdf/2412.05728
• Copy Paste: [[2412.05728]] Integrating YOLO11 and Convolution Block Attention Module for Multi-Season Segmentation of Tree Trunks and Branches in Commercial Apple Orchards(https://arxiv.org/abs/2412.05728)
• Keywords: segmentation
• Abstract: In this study, we developed a customized instance segmentation model by integrating the Convolutional Block Attention Module (CBAM) with the YOLO11 architecture. This model, trained on a mixed dataset of dormant and canopy season apple orchard images, aimed to enhance the segmentation of tree trunks and branches under varying seasonal conditions throughout the year. The model was individually validated across dormant and canopy season images after training the YOLO11-CBAM on the mixed dataset collected over the two seasons. Additional testing of the model during pre-bloom, flower bloom, fruit thinning, and harvest season was performed. The highest recall and precision metrics were observed in the YOLO11x-seg-CBAM and YOLO11m-seg-CBAM respectively. Particularly, YOLO11m-seg with CBAM showed the highest precision of 0.83 as performed for the Trunk class in training, while without the CBAM, YOLO11m-seg achieved 0.80 precision score for the Trunk class. Likewise, for branch class, YOLO11m-seg with CBAM achieved the highest precision score value of 0.75 while without the CBAM, the YOLO11m-seg achieved a precision of 0.73. For dormant season validation, YOLO11x-seg exhibited the highest precision at 0.91. Canopy season validation highlighted YOLO11s-seg with superior precision across all classes, achieving 0.516 for Branch, and 0.64 for Trunk. The modeling approach, trained on two season datasets as dormant and canopy season images, demonstrated the potential of the YOLO11-CBAM integration to effectively detect and segment tree trunks and branches year-round across all seasonal variations. Keywords: YOLOv11, YOLOv11 Tree Detection, YOLOv11 Branch Detection and Segmentation, Machine Vision, Deep Learning, Machine Learning

Title: PrivAgent: Agentic-based Red-teaming for LLM Privacy Leakage

• Authors: Yuzhou Nie, Zhun Wang, Ye Yu, Xian Wu, Xuandong Zhao, Wenbo Guo, Dawn Song
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05734
• Pdf URL: https://arxiv.org/pdf/2412.05734
• Copy Paste: [[2412.05734]] PrivAgent: Agentic-based Red-teaming for LLM Privacy Leakage(https://arxiv.org/abs/2412.05734)
• Keywords: privacy, defense, attack, extraction
• Abstract: Recent studies have discovered that LLMs have serious privacy leakage concerns, where an LLM may be fooled into outputting private information under carefully crafted adversarial prompts. These risks include leaking system prompts, personally identifiable information, training data, and model parameters. Most existing red-teaming approaches for privacy leakage rely on humans to craft the adversarial prompts. A few automated methods are proposed for system prompt extraction, but they cannot be applied to more severe risks (e.g., training data extraction) and have limited effectiveness even for system prompt extraction. In this paper, we propose PrivAgent, a novel black-box red-teaming framework for LLM privacy leakage. We formulate different risks as a search problem with a unified attack goal. Our framework trains an open-source LLM through reinforcement learning as the attack agent to generate adversarial prompts for different target models under different risks. We propose a novel reward function to provide effective and fine-grained rewards for the attack agent. Finally, we introduce customizations to better fit our general framework to system prompt extraction and training data extraction. Through extensive evaluations, we first show that PrivAgent outperforms existing automated methods in system prompt leakage against six popular LLMs. Notably, our approach achieves a 100% success rate in extracting system prompts from real-world applications in OpenAI's GPT Store. We also show PrivAgent's effectiveness in extracting training data from an open-source LLM with a success rate of 5.9%. We further demonstrate PrivAgent's effectiveness in evading the existing guardrail defense and its helpfulness in enabling better safety alignment. Finally, we validate our customized designs through a detailed ablation study. We release our code here this https URL.

Title: REGE: A Method for Incorporating Uncertainty in Graph Embeddings

• Authors: Zohair Shafi, Germans Savcisens, Tina Eliassi-Rad
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05735
• Pdf URL: https://arxiv.org/pdf/2412.05735
• Copy Paste: [[2412.05735]] REGE: A Method for Incorporating Uncertainty in Graph Embeddings(https://arxiv.org/abs/2412.05735)
• Keywords: attack
• Abstract: Machine learning models for graphs in real-world applications are prone to two primary types of uncertainty: (1) those that arise from incomplete and noisy data and (2) those that arise from uncertainty of the model in its output. These sources of uncertainty are not mutually exclusive. Additionally, models are susceptible to targeted adversarial attacks, which exacerbate both of these uncertainties. In this work, we introduce Radius Enhanced Graph Embeddings (REGE), an approach that measures and incorporates uncertainty in data to produce graph embeddings with radius values that represent the uncertainty of the model's output. REGE employs curriculum learning to incorporate data uncertainty and conformal learning to address the uncertainty in the model's output. In our experiments, we show that REGE's graph embeddings perform better under adversarial attacks by an average of 1.5% (accuracy) against state-of-the-art methods.

Title: Balancing Confidentiality and Transparency for Blockchain-based Process-Aware Information Systems

• Authors: Alessandro Marcelletti, Edoardo Marangone, Claudio Di Ciccio
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.05737
• Pdf URL: https://arxiv.org/pdf/2412.05737
• Copy Paste: [[2412.05737]] Balancing Confidentiality and Transparency for Blockchain-based Process-Aware Information Systems(https://arxiv.org/abs/2412.05737)
• Keywords: security, robust
• Abstract: Blockchain enables novel, trustworthy Process-Aware Information Systems (PAISs) by enforcing the security, robustness, and traceability of operations. In particular, transparency ensures that all information exchanges are openly accessible, fostering trust within the system. Although this is a desirable property to enable notarization and auditing activities, it also represents a limitation for such cases where confidentiality is a requirement since interactions involve sensible data. Current solutions rely on obfuscation techniques or private infrastructures, hindering the enforcing capabilities of smart contracts and the public verifiability of transactions. Against this background, we propose CONFETTY, an architecture for blockchain-based PAISs aimed at preserving both confidentiality and transparency. Smart contracts enact, enforce and store public interactions, while attribute-based encryption techniques are adopted to specify access grants to confidential information. We assess the security of our solution through a systematic threat model analysis and assess its practical feasibility by gauging the performance of our implemented prototype in different scenarios from the literature.

Title: A Comparative Study on Code Generation with Transformers

• Authors: Namrata Das, Rakshya Panta, Neelam Karki, Ruchi Manandhar, Dinesh Baniya Kshatri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05749
• Pdf URL: https://arxiv.org/pdf/2412.05749
• Copy Paste: [[2412.05749]] A Comparative Study on Code Generation with Transformers(https://arxiv.org/abs/2412.05749)
• Keywords: robust, transformer, large language model
• Abstract: In an era of widespread influence of Natural Language Processing (NLP), there have been multiple research efforts to supplant traditional manual coding techniques with automated systems capable of generating solutions autonomously. With rapid research for code generation and a sole focus on large language models, there emerges a need to compare and evaluate the performance of transformer architectures based on several complexities of the model. This paper introduces the concept of a "A Comparative Study on Code Generation with Transformers," a model based on Transformer architecture, and NLP methodologies to automatically generate C++ source code for different varieties of problems. Here, a comparative study is performed to evaluate the robustness of transformer-based models on the basis of their architecture complexities and their capability to handle diverse problem sets, from basic arithmetic to complex computations.

Title: Compositional Image Retrieval via Instruction-Aware Contrastive Learning

• Authors: Wenliang Zhong, Weizhi An, Feng Jiang, Hehuan Ma, Yuzhi Guo, Junzhou Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05756
• Pdf URL: https://arxiv.org/pdf/2412.05756
• Copy Paste: [[2412.05756]] Compositional Image Retrieval via Instruction-Aware Contrastive Learning(https://arxiv.org/abs/2412.05756)
• Keywords: extraction, large language model
• Abstract: Composed Image Retrieval (CIR) involves retrieving a target image based on a composed query of an image paired with text that specifies modifications or changes to the visual reference. CIR is inherently an instruction-following task, as the model needs to interpret and apply modifications to the image. In practice, due to the scarcity of annotated data in downstream tasks, Zero-Shot CIR (ZS-CIR) is desirable. While existing ZS-CIR models based on CLIP have shown promising results, their capability in interpreting and following modification instructions remains limited. Some research attempts to address this by incorporating Large Language Models (LLMs). However, these approaches still face challenges in effectively integrating multimodal information and instruction understanding. To tackle above challenges, we propose a novel embedding method utilizing an instruction-tuned Multimodal LLM (MLLM) to generate composed representation, which significantly enhance the instruction following capability for a comprehensive integration between images and instructions. Nevertheless, directly applying MLLMs introduces a new challenge since MLLMs are primarily designed for text generation rather than embedding extraction as required in CIR. To address this, we introduce a two-stage training strategy to efficiently learn a joint multimodal embedding space and further refining the ability to follow modification instructions by tuning the model in a triplet dataset similar to the CIR format. Extensive experiments on four public datasets: FashionIQ, CIRR, GeneCIS, and CIRCO demonstrates the superior performance of our model, outperforming state-of-the-art baselines by a significant margin. Codes are available at the GitHub repository.

Title: Policy-shaped prediction: avoiding distractions in model-based reinforcement learning

• Authors: Miles Hutson, Isaac Kauvar, Nick Haber
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05766
• Pdf URL: https://arxiv.org/pdf/2412.05766
• Copy Paste: [[2412.05766]] Policy-shaped prediction: avoiding distractions in model-based reinforcement learning(https://arxiv.org/abs/2412.05766)
• Keywords: robust, segmentation
• Abstract: Model-based reinforcement learning (MBRL) is a promising route to sample-efficient policy optimization. However, a known vulnerability of reconstruction-based MBRL consists of scenarios in which detailed aspects of the world are highly predictable, but irrelevant to learning a good policy. Such scenarios can lead the model to exhaust its capacity on meaningless content, at the cost of neglecting important environment dynamics. While existing approaches attempt to solve this problem, we highlight its continuing impact on leading MBRL methods -- including DreamerV3 and DreamerPro -- with a novel environment where background distractions are intricate, predictable, and useless for planning future actions. To address this challenge we develop a method for focusing the capacity of the world model through synergy of a pretrained segmentation model, a task-aware reconstruction loss, and adversarial learning. Our method outperforms a variety of other approaches designed to reduce the impact of distractors, and is an advance towards robust model-based reinforcement learning.

Title: DeMem: Privacy-Enhanced Robust Adversarial Learning via De-Memorization

• Authors: Xiaoyu Luo, Qiongxiu Li
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.05767
• Pdf URL: https://arxiv.org/pdf/2412.05767
• Copy Paste: [[2412.05767]] DeMem: Privacy-Enhanced Robust Adversarial Learning via De-Memorization(https://arxiv.org/abs/2412.05767)
• Keywords: privacy, protect, attack, robust
• Abstract: Adversarial robustness, the ability of a model to withstand manipulated inputs that cause errors, is essential for ensuring the trustworthiness of machine learning models in real-world applications. However, previous studies have shown that enhancing adversarial robustness through adversarial training increases vulnerability to privacy attacks. While differential privacy can mitigate these attacks, it often compromises robustness against both natural and adversarial samples. Our analysis reveals that differential privacy disproportionately impacts low-risk samples, causing an unintended performance drop. To address this, we propose DeMem, which selectively targets high-risk samples, achieving a better balance between privacy protection and model robustness. DeMem is versatile and can be seamlessly integrated into various adversarial training techniques. Extensive evaluations across multiple training methods and datasets demonstrate that DeMem significantly reduces privacy leakage while maintaining robustness against both natural and adversarial samples. These results confirm DeMem's effectiveness and broad applicability in enhancing privacy without compromising robustness.

Title: Uncovering Uncertainty in Transformer Inference

• Authors: Greyson Brothers, Willa Mannering, Amber Tien, John Winder
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05768
• Pdf URL: https://arxiv.org/pdf/2412.05768
• Copy Paste: [[2412.05768]] Uncovering Uncertainty in Transformer Inference(https://arxiv.org/abs/2412.05768)
• Keywords: transformer
• Abstract: We explore the Iterative Inference Hypothesis (IIH) within the context of transformer-based language models, aiming to understand how a model's latent representations are progressively refined and whether observable differences are present between correct and incorrect generations. Our findings provide empirical support for the IIH, showing that the nth token embedding in the residual stream follows a trajectory of decreasing loss. Additionally, we observe that the rate at which residual embeddings converge to a stable output representation reflects uncertainty in the token generation process. Finally, we introduce a method utilizing cross-entropy to detect this uncertainty and demonstrate its potential to distinguish between correct and incorrect token generations on a dataset of idioms.

Title: KITE-DDI: A Knowledge graph Integrated Transformer Model for accurately predicting Drug-Drug Interaction Events from Drug SMILES and Biomedical Knowledge Graph

• Authors: Azwad Tamir, Jiann-Shiun Yuan
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2412.05770
• Pdf URL: https://arxiv.org/pdf/2412.05770
• Copy Paste: [[2412.05770]] KITE-DDI: A Knowledge graph Integrated Transformer Model for accurately predicting Drug-Drug Interaction Events from Drug SMILES and Biomedical Knowledge Graph(https://arxiv.org/abs/2412.05770)
• Keywords: transformer
• Abstract: It is a common practice in modern medicine to prescribe multiple medications simultaneously to treat diseases. However, these medications could have adverse reactions between them, known as Drug-Drug Interactions (DDI), which have the potential to cause significant bodily injury and could even be fatal. Hence, it is essential to identify all the DDI events before prescribing multiple drugs to a patient. Most contemporary research for predicting DDI events relies on either information from Biomedical Knowledge graphs (KG) or drug SMILES, with very few managing to merge data from both to make predictions. While others use heuristic algorithms to extract features from SMILES and KGs, which are then fed into a Deep Learning framework to generate output. In this study, we propose a KG-integrated Transformer architecture to generate an end-to-end fully automated Machine Learning pipeline for predicting DDI events with high accuracy. The algorithm takes full-scale molecular SMILES sequences of a pair of drugs and a biomedical KG as input and predicts the interaction between the two drugs with high precision. The results show superior performance in two different benchmark datasets compared to existing state-of-the-art models especially when the test and training sets contain distinct sets of drug molecules. This demonstrates the strong generalization of the proposed model, indicating its potential for DDI event prediction for newly developed drugs. The model does not depend on heuristic models for generating embeddings and has a minimal number of hyperparameters, making it easy to use while demonstrating outstanding performance in low-data scenarios.

Title: ProtGO: A Transformer based Fusion Model for accurately predicting Gene Ontology (GO) Terms from full scale Protein Sequences

• Authors: Azwad Tamir, Jiann-Shiun Yuan
• Subjects: cs.LG, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2412.05776
• Pdf URL: https://arxiv.org/pdf/2412.05776
• Copy Paste: [[2412.05776]] ProtGO: A Transformer based Fusion Model for accurately predicting Gene Ontology (GO) Terms from full scale Protein Sequences(https://arxiv.org/abs/2412.05776)
• Keywords: transformer
• Abstract: Recent developments in next generation sequencing technology have led to the creation of extensive, open-source protein databases consisting of hundreds of millions of sequences. To render these sequences applicable in biomedical applications, they must be meticulously annotated by wet lab testing or extracting them from existing literature. Over the last few years, researchers have developed numerous automatic annotation systems, particularly deep learning models based on machine learning and artificial intelligence, to address this issue. In this work, we propose a transformer-based fusion model capable of predicting Gene Ontology (GO) terms from full-scale protein sequences, achieving state-of-the-art accuracy compared to other contemporary machine learning annotation systems. The approach performs particularly well on clustered split datasets, which comprise training and testing samples originating from distinct distributions that are structurally diverse. This demonstrates that the model is able to understand both short and long term dependencies within the enzyme's structure and can precisely identify the motifs associated with the various GO terms. Furthermore, the technique is lightweight and less computationally expensive compared to the benchmark methods, while at the same time not unaffected by sequence length, rendering it appropriate for diverse applications with varying sequence lengths.

Title: BudgetFusion: Perceptually-Guided Adaptive Diffusion Models

• Authors: Qinchan (Wing)Li, Kenneth Chen, Changyue (Tina)Su, Qi Sun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05780
• Pdf URL: https://arxiv.org/pdf/2412.05780
• Copy Paste: [[2412.05780]] BudgetFusion: Perceptually-Guided Adaptive Diffusion Models(https://arxiv.org/abs/2412.05780)
• Keywords: diffusion, generative
• Abstract: Diffusion models have shown unprecedented success in the task of text-to-image generation. While these models are capable of generating high-quality and realistic images, the complexity of sequential denoising has raised societal concerns regarding high computational demands and energy consumption. In response, various efforts have been made to improve inference efficiency. However, most of the existing efforts have taken a fixed approach with neural network simplification or text prompt optimization. Are the quality improvements from all denoising computations equally perceivable to humans? We observed that images from different text prompts may require different computational efforts given the desired content. The observation motivates us to present BudgetFusion, a novel model that suggests the most perceptually efficient number of diffusion steps before a diffusion model starts to generate an image. This is achieved by predicting multi-level perceptual metrics relative to diffusion steps. With the popular Stable Diffusion as an example, we conduct both numerical analyses and user studies. Our experiments show that BudgetFusion saves up to five seconds per prompt without compromising perceptual similarity. We hope this work can initiate efforts toward answering a core question: how much do humans perceptually gain from images created by a generative model, per watt of energy?

Title: Open-Source Acceleration of Stable-Diffusion.cpp

• Authors: Jingxu Ng, Cheng Lv, Pu Zhao, Wei Niu, Juyi Lin, Yanzhi Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05781
• Pdf URL: https://arxiv.org/pdf/2412.05781
• Copy Paste: [[2412.05781]] Open-Source Acceleration of Stable-Diffusion.cpp(https://arxiv.org/abs/2412.05781)
• Keywords: diffusion
• Abstract: Stable diffusion plays a crucial role in generating high-quality images. However, image generation is time-consuming and memory-intensive. To address this, this http URL (Sdcpp) emerges as an efficient inference framework to accelerate the diffusion models. Although it is lightweight, the current implementation of ggml_conv_2d operator in Sdcpp is suboptimal, exhibiting both high inference latency and massive memory usage. To address this, in this work, we present an optimized version of Sdcpp leveraging the Winograd algorithm to accelerate 2D convolution operations, which is the primary bottleneck in the pipeline. By analyzing both dependent and independent computation graphs, we exploit the device's locality and parallelism to achieve substantial performance improvements. Our framework delivers correct end-to-end results across various stable diffusion models, including SDv1.4, v1.5, v2.1, SDXL, and SDXL-Turbo. Our evaluation results demonstrate a speedup up to 2.76x for individual convolutional layers and an inference speedup up to 4.79x for the overall image generation process, compared with the original Sdcpp. Homepage: this https URL

Title: Language-Guided Image Tokenization for Generation

• Authors: Kaiwen Zha, Lijun Yu, Alireza Fathi, David A. Ross, Cordelia Schmid, Dina Katabi, Xiuye Gu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05796
• Pdf URL: https://arxiv.org/pdf/2412.05796
• Copy Paste: [[2412.05796]] Language-Guided Image Tokenization for Generation(https://arxiv.org/abs/2412.05796)
• Keywords: diffusion, transformer
• Abstract: Image tokenization, the process of transforming raw image pixels into a compact low-dimensional latent representation, has proven crucial for scalable and efficient image generation. However, mainstream image tokenization methods generally have limited compression rates, making high-resolution image generation computationally expensive. To address this challenge, we propose to leverage language for efficient image tokenization, and we call our method Text-Conditioned Image Tokenization (TexTok). TexTok is a simple yet effective tokenization framework that leverages language to provide high-level semantics. By conditioning the tokenization process on descriptive text captions, TexTok allows the tokenization process to focus on encoding fine-grained visual details into latent tokens, leading to enhanced reconstruction quality and higher compression rates. Compared to the conventional tokenizer without text conditioning, TexTok achieves average reconstruction FID improvements of 29.2% and 48.1% on ImageNet-256 and -512 benchmarks respectively, across varying numbers of tokens. These tokenization improvements consistently translate to 16.3% and 34.3% average improvements in generation FID. By simply replacing the tokenizer in Diffusion Transformer (DiT) with TexTok, our system can achieve a 93.5x inference speedup while still outperforming the original DiT using only 32 tokens on ImageNet-512. TexTok with a vanilla DiT generator achieves state-of-the-art FID scores of 1.46 and 1.62 on ImageNet-256 and -512 respectively. Furthermore, we demonstrate TexTok's superiority on the text-to-image generation task, effectively utilizing the off-the-shelf text captions in tokenization.

Title: [CLS] Token Tells Everything Needed for Training-free Efficient MLLMs

• Authors: Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05819
• Pdf URL: https://arxiv.org/pdf/2412.05819
• Copy Paste: [[2412.05819]] [CLS] Token Tells Everything Needed for Training-free Efficient MLLMs(https://arxiv.org/abs/2412.05819)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across a wide range of vision-language tasks, garnering significant attention in the computer vision. However, their efficient deployment remains a substantial challenge due to high computational costs and memory requirements. Recognizing the redundancy of information within the vision modality, recent studies have explored methods for compressing visual tokens in MLLMs to enhance efficiency in a training-free manner. Despite their effectiveness, existing methods like Fast rely on the attention between visual tokens and prompt text tokens as the importance indicator, overlooking the relevance to response text and thus introducing perception bias. In this paper, we demonstrate that in MLLMs, the [CLS] token in the visual encoder inherently knows which visual tokens are important for MLLMs. Building on this prior, we introduce a simple yet effective method for train-free visual token compression, called VTC-CLS. Firstly, it leverages the attention score of the [CLS] token on visual tokens as an importance indicator for pruning visual tokens. Besides, we also explore ensembling the importance scores derived by the [CLS] token from different layers to capture the key visual information more comprehensively. Extensive experiments demonstrate that our VTC-CLS achieves the state-of-the-art performance across various tasks compared with baseline methods. It also brings notably less computational costs in a training-free manner, highlighting its effectiveness and superiority. Code and models are available at \url{this https URL}.

Title: An Entailment Tree Generation Approach for Multimodal Multi-Hop Question Answering with Mixture-of-Experts and Iterative Feedback Mechanism

• Authors: Qing Zhang, Haocheng Lv, Jie Liu, Zhiyun Chen, Jianyong Duan, Hao Wang, Li He, Mingying Xv
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05821
• Pdf URL: https://arxiv.org/pdf/2412.05821
• Copy Paste: [[2412.05821]] An Entailment Tree Generation Approach for Multimodal Multi-Hop Question Answering with Mixture-of-Experts and Iterative Feedback Mechanism(https://arxiv.org/abs/2412.05821)
• Keywords: interpretability
• Abstract: With the rise of large-scale language models (LLMs), it is currently popular and effective to convert multimodal information into text descriptions for multimodal multi-hop question answering. However, we argue that the current methods of multi-modal multi-hop question answering still mainly face two challenges: 1) The retrieved evidence containing a large amount of redundant information, inevitably leads to a significant drop in performance due to irrelevant information misleading the prediction. 2) The reasoning process without interpretable reasoning steps makes the model difficult to discover the logical errors for handling complex questions. To solve these problems, we propose a unified LLMs-based approach but without heavily relying on them due to the LLM's potential errors, and innovatively treat multimodal multi-hop question answering as a joint entailment tree generation and question answering problem. Specifically, we design a multi-task learning framework with a focus on facilitating common knowledge sharing across interpretability and prediction tasks while preventing task-specific errors from interfering with each other via mixture of experts. Afterward, we design an iterative feedback mechanism to further enhance both tasks by feeding back the results of the joint training to the LLM for regenerating entailment trees, aiming to iteratively refine the potential answer. Notably, our method has won the first place in the official leaderboard of WebQA (since April 10, 2024), and achieves competitive results on MultimodalQA.

Title: DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices

• Authors: Yongzhe Jia, Xuyun Zhang, Hongsheng Hu, Kim-Kwang Raymond Choo, Lianyong Qi, Xiaolong Xu, Amin Beheshti, Wanchun Dou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05823
• Pdf URL: https://arxiv.org/pdf/2412.05823
• Copy Paste: [[2412.05823]] DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices(https://arxiv.org/abs/2412.05823)
• Keywords: robust, federate
• Abstract: Federated learning (FL) has emerged as a prominent machine learning paradigm in edge computing environments, enabling edge devices to collaboratively optimize a global model without sharing their private data. However, existing FL frameworks suffer from efficacy deterioration due to the system heterogeneity inherent in edge computing, especially in the presence of domain shifts across local data. In this paper, we propose a heterogeneous FL framework DapperFL, to enhance model performance across multiple domains. In DapperFL, we introduce a dedicated Model Fusion Pruning (MFP) module to produce personalized compact local models for clients to address the system heterogeneity challenges. The MFP module prunes local models with fused knowledge obtained from both local and remaining domains, ensuring robustness to domain shifts. Additionally, we design a Domain Adaptive Regularization (DAR) module to further improve the overall performance of DapperFL. The DAR module employs regularization generated by the pruned model, aiming to learn robust representations across domains. Furthermore, we introduce a specific aggregation algorithm for aggregating heterogeneous local models with tailored architectures and weights. We implement DapperFL on a realworld FL platform with heterogeneous clients. Experimental results on benchmark datasets with multiple domains demonstrate that DapperFL outperforms several state-of-the-art FL frameworks by up to 2.28%, while significantly achieving model volume reductions ranging from 20% to 80%. Our code is available at: this https URL.

Title: Self-Supervised Learning with Probabilistic Density Labeling for Rainfall Probability Estimation

• Authors: Junha Lee, Sojung An, Sujeong You, Namik Cho
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05825
• Pdf URL: https://arxiv.org/pdf/2412.05825
• Copy Paste: [[2412.05825]] Self-Supervised Learning with Probabilistic Density Labeling for Rainfall Probability Estimation(https://arxiv.org/abs/2412.05825)
• Keywords: segmentation
• Abstract: Numerical weather prediction (NWP) models are fundamental in meteorology for simulating and forecasting the behavior of various atmospheric variables. The accuracy of precipitation forecasts and the acquisition of sufficient lead time are crucial for preventing hazardous weather events. However, the performance of NWP models is limited by the nonlinear and unpredictable patterns of extreme weather phenomena driven by temporal dynamics. In this regard, we propose a \textbf{S}elf-\textbf{S}upervised \textbf{L}earning with \textbf{P}robabilistic \textbf{D}ensity \textbf{L}abeling (SSLPDL) for estimating rainfall probability by post-processing NWP forecasts. Our post-processing method uses self-supervised learning (SSL) with masked modeling for reconstructing atmospheric physics variables, enabling the model to learn the dependency between variables. The pre-trained encoder is then utilized in transfer learning to a precipitation segmentation task. Furthermore, we introduce a straightforward labeling approach based on probability density to address the class imbalance in extreme weather phenomena like heavy rain events. Experimental results show that SSLPDL surpasses other precipitation forecasting models in regional precipitation post-processing and demonstrates competitive performance in extending forecast lead times. Our code is available at this https URL

Title: Doppelgangers++: Improved Visual Disambiguation with Geometric 3D Features

• Authors: Yuanbo Xiangli, Ruojin Cai, Hanyu Chen, Jeffrey Byrne, Noah Snavely
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05826
• Pdf URL: https://arxiv.org/pdf/2412.05826
• Copy Paste: [[2412.05826]] Doppelgangers++: Improved Visual Disambiguation with Geometric 3D Features(https://arxiv.org/abs/2412.05826)
• Keywords: robust, transformer
• Abstract: Accurate 3D reconstruction is frequently hindered by visual aliasing, where visually similar but distinct surfaces (aka, doppelgangers), are incorrectly matched. These spurious matches distort the structure-from-motion (SfM) process, leading to misplaced model elements and reduced accuracy. Prior efforts addressed this with CNN classifiers trained on curated datasets, but these approaches struggle to generalize across diverse real-world scenes and can require extensive parameter tuning. In this work, we present Doppelgangers++, a method to enhance doppelganger detection and improve 3D reconstruction accuracy. Our contributions include a diversified training dataset that incorporates geo-tagged images from everyday scenes to expand robustness beyond landmark-based datasets. We further propose a Transformer-based classifier that leverages 3D-aware features from the MASt3R model, achieving superior precision and recall across both in-domain and out-of-domain tests. Doppelgangers++ integrates seamlessly into standard SfM and MASt3R-SfM pipelines, offering efficiency and adaptability across varied scenes. To evaluate SfM accuracy, we introduce an automated, geotag-based method for validating reconstructed models, eliminating the need for manual inspection. Through extensive experiments, we demonstrate that Doppelgangers++ significantly enhances pairwise visual disambiguation and improves 3D reconstruction quality in complex and diverse scenarios.

Title: Self-Guidance: Boosting Flow and Diffusion Generation on Their Own

• Authors: Tiancheng Li, Weijian Luo, Zhiyang Chen, Liyuan Ma, Guo-Jun Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05827
• Pdf URL: https://arxiv.org/pdf/2412.05827
• Copy Paste: [[2412.05827]] Self-Guidance: Boosting Flow and Diffusion Generation on Their Own(https://arxiv.org/abs/2412.05827)
• Keywords: diffusion, transformer
• Abstract: Proper guidance strategies are essential to get optimal generation results without re-training diffusion and flow-based text-to-image models. However, existing guidances either require specific training or strong inductive biases of neural network architectures, potentially limiting their applications. To address these issues, in this paper, we introduce Self-Guidance (SG), a strong diffusion guidance that neither needs specific training nor requires certain forms of neural network architectures. Different from previous approaches, the Self-Guidance calculates the guidance vectors by measuring the difference between the velocities of two successive diffusion timesteps. Therefore, SG can be readily applied for both conditional and unconditional models with flexible network architectures. We conduct intensive experiments on both text-to-image generation and text-to-video generations across flexible architectures including UNet-based models and diffusion transformer-based models. On current state-of-the-art diffusion models such as Stable Diffusion 3.5 and FLUX, SG significantly boosts the image generation performance in terms of FID, and Human Preference Scores. Moreover, we find that SG has a surprisingly positive effect on the generation of high-quality human bodies such as hands, faces, and arms, showing strong potential to overcome traditional challenges on human body generations with minimal effort. We will release our implementation of SG on SD 3.5 and FLUX models along with this paper.

Title: Large Language Models Merging for Enhancing the Link Stealing Attack on Graph Neural Networks

• Authors: Faqian Guan, Tianqing Zhu, Wenhan Chang, Wei Ren, Wanlei Zhou
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05830
• Pdf URL: https://arxiv.org/pdf/2412.05830
• Copy Paste: [[2412.05830]] Large Language Models Merging for Enhancing the Link Stealing Attack on Graph Neural Networks(https://arxiv.org/abs/2412.05830)
• Keywords: privacy, attack, steal, large language model
• Abstract: Graph Neural Networks (GNNs), specifically designed to process the graph data, have achieved remarkable success in various applications. Link stealing attacks on graph data pose a significant privacy threat, as attackers aim to extract sensitive relationships between nodes (entities), potentially leading to academic misconduct, fraudulent transactions, or other malicious activities. Previous studies have primarily focused on single datasets and did not explore cross-dataset attacks, let alone attacks that leverage the combined knowledge of multiple attackers. However, we find that an attacker can combine the data knowledge of multiple attackers to create a more effective attack model, which can be referred to cross-dataset attacks. Moreover, if knowledge can be extracted with the help of Large Language Models (LLMs), the attack capability will be more significant. In this paper, we propose a novel link stealing attack method that takes advantage of cross-dataset and Large Language Models (LLMs). The LLM is applied to process datasets with different data structures in cross-dataset attacks. Each attacker fine-tunes the LLM on their specific dataset to generate a tailored attack model. We then introduce a novel model merging method to integrate the parameters of these attacker-specific models effectively. The result is a merged attack model with superior generalization capabilities, enabling effective attacks not only on the attackers' datasets but also on previously unseen (out-of-domain) datasets. We conducted extensive experiments in four datasets to demonstrate the effectiveness of our method. Additional experiments with three different GNN and LLM architectures further illustrate the generality of our approach.

Title: CSG: A Context-Semantic Guided Diffusion Approach in De Novo Musculoskeletal Ultrasound Image Generation

• Authors: Elay Dahan, Hedda Cohen Indelman, Angeles M. Perez-Agosto, Carmit Shiran, Gopal Avinash, Doron Shaked, Nati Daniel
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.05833
• Pdf URL: https://arxiv.org/pdf/2412.05833
• Copy Paste: [[2412.05833]] CSG: A Context-Semantic Guided Diffusion Approach in De Novo Musculoskeletal Ultrasound Image Generation(https://arxiv.org/abs/2412.05833)
• Keywords: diffusion, generative, segmentation
• Abstract: The use of synthetic images in medical imaging Artificial Intelligence (AI) solutions has been shown to be beneficial in addressing the limited availability of diverse, unbiased, and representative data. Despite the extensive use of synthetic image generation methods, controlling the semantics variability and context details remains challenging, limiting their effectiveness in producing diverse and representative medical image datasets. In this work, we introduce a scalable semantic and context-conditioned generative model, coined CSG (Context-Semantic Guidance). This dual conditioning approach allows for comprehensive control over both structure and appearance, advancing the synthesis of realistic and diverse ultrasound images. We demonstrate the ability of CSG to generate findings (pathological anomalies) in musculoskeletal (MSK) ultrasound images. Moreover, we test the quality of the synthetic images using a three-fold validation protocol. The results show that the synthetic images generated by CSG improve the performance of semantic segmentation models, exhibit enhanced similarity to real images compared to the baseline methods, and are undistinguishable from real images according to a Turing test. Furthermore, we demonstrate an extension of the CSG that allows enhancing the variability space of images by synthetically generating augmentations of anatomical geometries and textures.

Title: Tiny Object Detection with Single Point Supervision

• Authors: Haoran Zhu, Chang Xu, Ruixiang Zhang, Fang Xu, Wen Yang, Haijian Zhang, Gui-Song Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05837
• Pdf URL: https://arxiv.org/pdf/2412.05837
• Copy Paste: [[2412.05837]] Tiny Object Detection with Single Point Supervision(https://arxiv.org/abs/2412.05837)
• Keywords: robust
• Abstract: Tiny objects, with their limited spatial resolution, often resemble point-like distributions. As a result, bounding box prediction using point-level supervision emerges as a natural and cost-effective alternative to traditional box-level supervision. However, the small scale and lack of distinctive features of tiny objects make point annotations prone to noise, posing significant hurdles for model robustness. To tackle these challenges, we propose Point Teacher--the first end-to-end point-supervised method for robust tiny object detection in aerial images. To handle label noise from scale ambiguity and location shifts in point annotations, Point Teacher employs the teacher-student architecture and decouples the learning into a two-phase denoising process. In this framework, the teacher network progressively denoises the pseudo boxes derived from noisy point annotations, guiding the student network's learning. Specifically, in the first phase, random masking of image regions facilitates regression learning, enabling the teacher to transform noisy point annotations into coarse pseudo boxes. In the second phase, these coarse pseudo boxes are refined using dynamic multiple instance learning, which adaptively selects the most reliable instance from dynamically constructed proposal bags around the coarse pseudo boxes. Extensive experiments on three tiny object datasets (i.e., AI-TOD-v2, SODA-A, and TinyPerson) validate the proposed method's effectiveness and robustness against point location shifts. Notably, relying solely on point supervision, our Point Teacher already shows comparable performance with box-supervised learning methods. Codes and models will be made publicly available.

Title: DREAM: Domain-agnostic Reverse Engineering Attributes of Black-box Model

• Authors: Rongqing Li, Jiaqi Yu, Changsheng Li, Wenhan Luo, Ye Yuan, Guoren Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05842
• Pdf URL: https://arxiv.org/pdf/2412.05842
• Copy Paste: [[2412.05842]] DREAM: Domain-agnostic Reverse Engineering Attributes of Black-box Model(https://arxiv.org/abs/2412.05842)
• Keywords: attack
• Abstract: Deep learning models are usually black boxes when deployed on machine learning platforms. Prior works have shown that the attributes (e.g., the number of convolutional layers) of a target black-box model can be exposed through a sequence of queries. There is a crucial limitation: these works assume the training dataset of the target model is known beforehand and leverage this dataset for model attribute attack. However, it is difficult to access the training dataset of the target black-box model in reality. Therefore, whether the attributes of a target black-box model could be still revealed in this case is doubtful. In this paper, we investigate a new problem of black-box reverse engineering, without requiring the availability of the target model's training dataset. We put forward a general and principled framework DREAM, by casting this problem as out-of-distribution (OOD) generalization. In this way, we can learn a domain-agnostic meta-model to infer the attributes of the target black-box model with unknown training data. This makes our method one of the kinds that can gracefully apply to an arbitrary domain for model attribute reverse engineering with strong generalization ability. Extensive experimental results demonstrate the superiority of our proposed method over the baselines.

Title: A Self-Learning Multimodal Approach for Fake News Detection

• Authors: Hao Chen, Hui Guo, Baochen Hu, Shu Hu, Jinrong Hu, Siwei Lyu, Xi Wu, Xin Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05843
• Pdf URL: https://arxiv.org/pdf/2412.05843
• Copy Paste: [[2412.05843]] A Self-Learning Multimodal Approach for Fake News Detection(https://arxiv.org/abs/2412.05843)
• Keywords: robust, extraction, large language model
• Abstract: The rapid growth of social media has resulted in an explosion of online news content, leading to a significant increase in the spread of misleading or false information. While machine learning techniques have been widely applied to detect fake news, the scarcity of labeled datasets remains a critical challenge. Misinformation frequently appears as paired text and images, where a news article or headline is accompanied by a related visuals. In this paper, we introduce a self-learning multimodal model for fake news classification. The model leverages contrastive learning, a robust method for feature extraction that operates without requiring labeled data, and integrates the strengths of Large Language Models (LLMs) to jointly analyze both text and image features. LLMs are excel at this task due to their ability to process diverse linguistic data drawn from extensive training corpora. Our experimental results on a public dataset demonstrate that the proposed model outperforms several state-of-the-art classification approaches, achieving over 85% accuracy, precision, recall, and F1-score. These findings highlight the model's effectiveness in tackling the challenges of multimodal fake news detection.

Title: Are Clinical T5 Models Better for Clinical Text?

• Authors: Yahan Li, Keith Harrigian, Ayah Zirikly, Mark Dredze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05845
• Pdf URL: https://arxiv.org/pdf/2412.05845
• Copy Paste: [[2412.05845]] Are Clinical T5 Models Better for Clinical Text?(https://arxiv.org/abs/2412.05845)
• Keywords: transformer, large language model
• Abstract: Large language models with a transformer-based encoder/decoder architecture, such as T5, have become standard platforms for supervised tasks. To bring these technologies to the clinical domain, recent work has trained new or adapted existing models to clinical data. However, the evaluation of these clinical T5 models and comparison to other models has been limited. Are the clinical T5 models better choices than FLAN-tuned generic T5 models? Do they generalize better to new clinical domains that differ from the training sets? We comprehensively evaluate these models across several clinical tasks and domains. We find that clinical T5 models provide marginal improvements over existing models, and perform worse when evaluated on different domains. Our results inform future choices in developing clinical LLMs.

Title: Kernel Stochastic Configuration Networks for Nonlinear Regression

• Authors: Yongxuan Chen, Dianhui Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05846
• Pdf URL: https://arxiv.org/pdf/2412.05846
• Copy Paste: [[2412.05846]] Kernel Stochastic Configuration Networks for Nonlinear Regression(https://arxiv.org/abs/2412.05846)
• Keywords: robust
• Abstract: Stochastic configuration networks (SCNs), as a class of randomized learner models, are featured by its way of random parameters assignment in the light of a supervisory mechanism, resulting in the universal approximation property at algorithmic level. This paper presents a kernel version of SCNs, termed KSCNs, aiming to enhance model's representation learning capability and performance stability. The random bases of a built SCN model can be used to span a reproducing kernel Hilbert space (RKHS), followed by our proposed algorithm for constructing KSCNs. It is shown that the data distribution in the reconstructive space is favorable for regression solving and the proposed KSCN learner models hold the universal approximation property. Three benchmark datasets including two industrial datasets are used in this study for performance evaluation. Experimental results with comparisons against existing solutions clearly demonstrate that the proposed KSCN remarkably outperforms the original SCNs and some typical kernel methods for resolving nonlinear regression problems in terms of the learning performance, the model's stability and robustness with respect to the kernel parameter settings.

Title: MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generation

• Authors: Shuwei Shi, Biao Gong, Xi Chen, Dandan Zheng, Shuai Tan, Zizheng Yang, Yuyuan Li, Jingwen He, Kecheng Zheng, Jingdong Chen, Ming Yang, Yinqiang Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05848
• Pdf URL: https://arxiv.org/pdf/2412.05848
• Copy Paste: [[2412.05848]] MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generation(https://arxiv.org/abs/2412.05848)
• Keywords: diffusion, transformer
• Abstract: The image-to-video (I2V) generation is conditioned on the static image, which has been enhanced recently by the motion intensity as an additional control signal. These motion-aware models are appealing to generate diverse motion patterns, yet there lacks a reliable motion estimator for training such models on large-scale video set in the wild. Traditional metrics, e.g., SSIM or optical flow, are hard to generalize to arbitrary videos, while, it is very tough for human annotators to label the abstract motion intensity neither. Furthermore, the motion intensity shall reveal both local object motion and global camera movement, which has not been studied before. This paper addresses the challenge with a new motion estimator, capable of measuring the decoupled motion intensities of objects and cameras in video. We leverage the contrastive learning on randomly paired videos and distinguish the video with greater motion intensity. Such a paradigm is friendly for annotation and easy to scale up to achieve stable performance on motion estimation. We then present a new I2V model, named MotionStone, developed with the decoupled motion estimator. Experimental results demonstrate the stability of the proposed motion estimator and the state-of-the-art performance of MotionStone on I2V generation. These advantages warrant the decoupled motion estimator to serve as a general plug-in enhancer for both data processing and video generation training.

Title: Cooperative SQL Generation for Segmented Databases By Using Multi-functional LLM Agents

• Authors: Zhiguang Wu, Fengbin Zhu, Xuequn Shang, Yupei Zhang, Pan Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05850
• Pdf URL: https://arxiv.org/pdf/2412.05850
• Copy Paste: [[2412.05850]] Cooperative SQL Generation for Segmented Databases By Using Multi-functional LLM Agents(https://arxiv.org/abs/2412.05850)
• Keywords: large language model
• Abstract: Text-to-SQL task aims to automatically yield SQL queries according to user text questions. To address this problem, we propose a Cooperative SQL Generation framework based on Multi-functional Agents (CSMA) through information interaction among large language model (LLM) based agents who own part of the database schema seperately. Inspired by the collaboration in human teamwork, CSMA consists of three stages: 1) Question-related schema collection, 2) Question-corresponding SQL query generation, and 3) SQL query correctness check. In the first stage, agents analyze their respective schema and communicate with each other to collect the schema information relevant to the question. In the second stage, agents try to generate the corresponding SQL query for the question using the collected information. In the third stage, agents check if the SQL query is created correctly according to their known information. This interaction-based method makes the question-relevant part of database schema from each agent to be used for SQL generation and check. Experiments on the Spider and Bird benckmark demonstrate that CSMA achieves a high performance level comparable to the state-of-the-arts, meanwhile holding the private data in these individual agents.

Title: Domain-Specific Translation with Open-Source Large Language Models: Resource-Oriented Analysis

• Authors: Aman Kassahun Wassie, Mahdi Molaei, Yasmin Moslem
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05862
• Pdf URL: https://arxiv.org/pdf/2412.05862
• Copy Paste: [[2412.05862]] Domain-Specific Translation with Open-Source Large Language Models: Resource-Oriented Analysis(https://arxiv.org/abs/2412.05862)
• Keywords: large language model
• Abstract: In this work, we compare the domain-specific translation performance of open-source autoregressive decoder-only large language models (LLMs) with task-oriented machine translation (MT) models. Our experiments focus on the medical domain and cover four language pairs with varied resource availability: English-to-French, English-to-Portuguese, English-to-Swahili, and Swahili-to-English. Despite recent advancements, LLMs exhibit a clear gap in specialized translation quality compared to multilingual encoder-decoder MT models such as NLLB-200. In three out of four language directions in our study, NLLB-200 3.3B outperforms all LLMs in the size range of 8B parameters in medical translation. While fine-tuning LLMs such as Mistral and Llama improves their performance at medical translation, these models still fall short compared to fine-tuned NLLB-200 3.3B models. Our findings highlight the ongoing need for specialized MT models to achieve higher-quality domain-specific translation, especially in medium-resource and low-resource settings. As larger LLMs outperform their 8B variants, this also encourages pre-training domain-specific medium-sized LMs to improve quality and efficiency in specialized translation tasks.

Title: 3D-Consistent Image Inpainting with Diffusion Models

• Authors: Leonid Antsfeld, Boris Chidlovskii
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05881
• Pdf URL: https://arxiv.org/pdf/2412.05881
• Copy Paste: [[2412.05881]] 3D-Consistent Image Inpainting with Diffusion Models(https://arxiv.org/abs/2412.05881)
• Keywords: diffusion, generative
• Abstract: We address the problem of 3D inconsistency of image inpainting based on diffusion models. We propose a generative model using image pairs that belong to the same scene. To achieve the 3D-consistent and semantically coherent inpainting, we modify the generative diffusion model by incorporating an alternative point of view of the scene into the denoising process. This creates an inductive bias that allows to recover 3D priors while training to denoise in 2D, without explicit 3D supervision. Training unconditional diffusion models with additional images as in-context guidance allows to harmonize the masked and non-masked regions while repainting and ensures the 3D consistency. We evaluate our method on one synthetic and three real-world datasets and show that it generates semantically coherent and 3D-consistent inpaintings and outperforms the state-of-art methods.

Title: Understanding the Impact of Graph Reduction on Adversarial Robustness in Graph Neural Networks

• Authors: Kerui Wu, Ka-Ho Chow, Wenqi Wei, Lei Yu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.05883
• Pdf URL: https://arxiv.org/pdf/2412.05883
• Copy Paste: [[2412.05883]] Understanding the Impact of Graph Reduction on Adversarial Robustness in Graph Neural Networks(https://arxiv.org/abs/2412.05883)
• Keywords: attack, robust
• Abstract: As Graph Neural Networks (GNNs) become increasingly popular for learning from large-scale graph data across various domains, their susceptibility to adversarial attacks when using graph reduction techniques for scalability remains underexplored. In this paper, we present an extensive empirical study to investigate the impact of graph reduction techniques, specifically graph coarsening and sparsification, on the robustness of GNNs against adversarial attacks. Through extensive experiments involving multiple datasets and GNN architectures, we examine the effects of four sparsification and six coarsening methods on the poisoning attacks. Our results indicate that, while graph sparsification can mitigate the effectiveness of certain poisoning attacks, such as Mettack, it has limited impact on others, like PGD. Conversely, graph coarsening tends to amplify the adversarial impact, significantly reducing classification accuracy as the reduction ratio decreases. Additionally, we provide a novel analysis of the causes driving these effects and examine how defensive GNN models perform under graph reduction, offering practical insights for designing robust GNNs within graph acceleration systems.

Title: An Overview of Cyber Security Funding for Open Source Software

• Authors: Jukka Ruohonen, Gaurav Choudhary, Adam Alami
• Subjects: cs.CR, cs.CY, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.05887
• Pdf URL: https://arxiv.org/pdf/2412.05887
• Copy Paste: [[2412.05887]] An Overview of Cyber Security Funding for Open Source Software(https://arxiv.org/abs/2412.05887)
• Keywords: security
• Abstract: Many open source software (OSS) projects need more human resources for maintenance, improvements, and sometimes even their survival. This need allegedly applies even to vital OSS projects that can be seen as being a part of the world's critical infrastructures. To address this resourcing problem, new funding instruments for OSS projects have been established in recent years. The paper examines two such funding bodies for OSS and the projects they have funded. The focus of both funding bodies is on software security and cyber security in general. Based on a qualitative analysis, particularly OSS supply chains, network and cryptography libraries, programming languages, and operating systems and their low-level components have been funded and thus seen as critical in terms of cyber security by the two funding bodies. In addition to this and other results, the paper makes a contribution by connecting the research branches of critical infrastructure and sustainability of OSS projects. A further contribution is made by connecting the topic examined to recent cyber security regulations. Furthermore, an important argument is raised that neither cyber security nor sustainability alone can entirely explain the rationales behind the funding decisions made by the two bodies.

Title: MCP-MedSAM: A Powerful Lightweight Medical Segment Anything Model Trained with a Single GPU in Just One Day

• Authors: Donghang Lyu, Ruochen Gao, Marius Staring
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05888
• Pdf URL: https://arxiv.org/pdf/2412.05888
• Copy Paste: [[2412.05888]] MCP-MedSAM: A Powerful Lightweight Medical Segment Anything Model Trained with a Single GPU in Just One Day(https://arxiv.org/abs/2412.05888)
• Keywords: segmentation
• Abstract: Medical image segmentation involves partitioning medical images into meaningful regions, with a focus on identifying anatomical structures or abnormalities. It has broad applications in healthcare, and deep learning methods have enabled significant advancements in automating this process. Recently, the introduction of the Segmentation Anything Model (SAM), the first foundation model for segmentation task, has prompted researchers to adapt it for the medical domain to improve performance across various tasks. However, SAM's large model size and high GPU requirements hinder its scalability and development in the medical domain. To address these challenges, research has increasingly focused on lightweight adaptations of SAM to reduce its parameter count, enabling training with limited GPU resources while maintaining competitive segmentation performance. In this work, we propose MCP-MedSAM, a powerful and lightweight medical SAM model designed to be trainable on a single GPU within one day while delivering superior segmentation performance. Our method was trained and evaluated using a large-scale challenge dataset\footnote{\url{this https URL}\label{comp}}, compared to top-ranking methods on the challenge leaderboard, MCP-MedSAM achieved superior performance while requiring only one day of training on a single GPU. The code is publicly available at \url{this https URL}.

Title: BAMBA: A Bimodal Adversarial Multi-Round Black-Box Jailbreak Attacker for LVLMs

• Authors: Ruoxi Cheng, Yizhong Ding, Shuirong Cao, Shaowei Yuan, Zhiqiang Wang, Xiaojun Jia
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05892
• Pdf URL: https://arxiv.org/pdf/2412.05892
• Copy Paste: [[2412.05892]] BAMBA: A Bimodal Adversarial Multi-Round Black-Box Jailbreak Attacker for LVLMs(https://arxiv.org/abs/2412.05892)
• Keywords: attack
• Abstract: LVLMs are widely used but vulnerable to illegal or unethical responses under jailbreak attacks. To ensure their responsible deployment in real-world applications, it is essential to understand their vulnerabilities. There are four main issues in current work: single-round attack limitation, insufficient dual-modal synergy, poor transferability to black-box models, and reliance on prompt engineering. To address these limitations, we propose BAMBA, a bimodal adversarial multi-round black-box jailbreak attacker for LVLMs. We first use an image optimizer to learn malicious features from a harmful corpus, then deepen these features through a bimodal optimizer through text-image interaction, generating adversarial text and image for jailbreak. Experiments on various LVLMs and datasets demonstrate that BAMBA outperforms other baselines.

Title: XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference

• Authors: Weizhuo Li, Zhigang Wang, Yu Gu, Ge Yu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05896
• Pdf URL: https://arxiv.org/pdf/2412.05896
• Copy Paste: [[2412.05896]] XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference(https://arxiv.org/abs/2412.05896)
• Keywords: generative, large language model
• Abstract: Recently the generative Large Language Model (LLM) has achieved remarkable success in numerous applications. Notably its inference generates output tokens one-by-one, leading to many redundant computations. The widely-used KV-Cache framework makes a compromise between time and space complexities. However, caching data generates the increasingly growing memory demand, that can quickly exhaust the limited memory capacity of the modern accelerator like GPUs, particularly in long-context inference tasks. Existing studies reduce memory consumption by evicting some of cached data that have less important impact on inference accuracy. But the benefit in practice is far from ideal due to the static cache allocation across different LLM network layers. This paper observes that the layer-specific cached data have very different impacts on accuracy. We quantify this difference, and give experimental and theoretical validation. We accordingly make a formal analysis and shows that customizing the cache size for each layer in a personalized manner can yield a significant memory reduction, while still providing comparable accuracy. We simulate the cache allocation as a combinatorial optimization problem and give a global optimal solution. In particular, we devise a mini- and sampling-based inference over a lightweight variant of the LLM model, so as to quickly capture the difference and then feed it into the personalized algorithms. Extensive experiments on real-world datasets demonstrate that our proposals can reduce KV cache memory consumption by 61.6% on average, improve computational efficiency by 2.1x and then increase the throughput by up to 5.5x.

Title: Accelerating Video Diffusion Models via Distribution Matching

• Authors: Yuanzhi Zhu, Hanshu Yan, Huan Yang, Kai Zhang, Junnan Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05899
• Pdf URL: https://arxiv.org/pdf/2412.05899
• Copy Paste: [[2412.05899]] Accelerating Video Diffusion Models via Distribution Matching(https://arxiv.org/abs/2412.05899)
• Keywords: diffusion, generative
• Abstract: Generative models, particularly diffusion models, have made significant success in data synthesis across various modalities, including images, videos, and 3D assets. However, current diffusion models are computationally intensive, often requiring numerous sampling steps that limit their practical application, especially in video generation. This work introduces a novel framework for diffusion distillation and distribution matching that dramatically reduces the number of inference steps while maintaining-and potentially improving-generation quality. Our approach focuses on distilling pre-trained diffusion models into a more efficient few-step generator, specifically targeting video generation. By leveraging a combination of video GAN loss and a novel 2D score distribution matching loss, we demonstrate the potential to generate high-quality video frames with substantially fewer sampling steps. To be specific, the proposed method incorporates a denoising GAN discriminator to distil from the real data and a pre-trained image diffusion model to enhance the frame quality and the prompt-following capabilities. Experimental results using AnimateDiff as the teacher model showcase the method's effectiveness, achieving superior performance in just four sampling steps compared to existing techniques.

Title: Quantum Threat in Healthcare IoT: Challenges and Mitigation Strategies

• Authors: Asif Alif, Khondokar Fida Hasan, Jesse Laeuchli, Mohammad Jabed Morshed Chowdhury
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.05904
• Pdf URL: https://arxiv.org/pdf/2412.05904
• Copy Paste: [[2412.05904]] Quantum Threat in Healthcare IoT: Challenges and Mitigation Strategies(https://arxiv.org/abs/2412.05904)
• Keywords: security, privacy, protect
• Abstract: The Internet of Things (IoT) has transformed healthcare, facilitating remote patient monitoring, enhanced medication adherence, and chronic disease management. However, this interconnected ecosystem faces significant vulnerabilities with the advent of quantum computing, which threatens to break existing encryption standards protecting sensitive patient data in IoT-enabled medical devices. This chapter examines the quantum threat to healthcare IoT security, highlighting the potential impacts of compromised encryption, including privacy breaches, device failures, and manipulated medical records. It introduces post-quantum cryptography (PQC) and quantum-resistant techniques like quantum key distribution (QKD), addressing their application in resource-constrained healthcare IoT devices such as pacemakers, monitoring tools, and telemedicine systems. The chapter further explores the challenges of integrating these solutions and reviews global efforts in mitigating quantum risks, offering insights into suitable PQC primitives for various healthcare use cases.

Title: GBR: Generative Bundle Refinement for High-fidelity Gaussian Splatting and Meshing

• Authors: Jianing Zhang, Yuchao Zheng, Ziwei Li, Qionghai Dai, Xiaoyun Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05908
• Pdf URL: https://arxiv.org/pdf/2412.05908
• Copy Paste: [[2412.05908]] GBR: Generative Bundle Refinement for High-fidelity Gaussian Splatting and Meshing(https://arxiv.org/abs/2412.05908)
• Keywords: robust, diffusion, generative
• Abstract: Gaussian splatting has gained attention for its efficient representation and rendering of 3D scenes using continuous Gaussian primitives. However, it struggles with sparse-view inputs due to limited geometric and photometric information, causing ambiguities in depth, shape, and texture. we propose GBR: Generative Bundle Refinement, a method for high-fidelity Gaussian splatting and meshing using only 4-6 input views. GBR integrates a neural bundle adjustment module to enhance geometry accuracy and a generative depth refinement module to improve geometry fidelity. More specifically, the neural bundle adjustment module integrates a foundation network to produce initial 3D point maps and point matches from unposed images, followed by bundle adjustment optimization to improve multiview consistency and point cloud accuracy. The generative depth refinement module employs a diffusion-based strategy to enhance geometric details and fidelity while preserving the scale. Finally, for Gaussian splatting optimization, we propose a multimodal loss function incorporating depth and normal consistency, geometric regularization, and pseudo-view supervision, providing robust guidance under sparse-view conditions. Experiments on widely used datasets show that GBR significantly outperforms existing methods under sparse-view inputs. Additionally, GBR demonstrates the ability to reconstruct and render large-scale real-world scenes, such as the Pavilion of Prince Teng and the Great Wall, with remarkable details using only 6 views.

Title: Paraphrase-Aligned Machine Translation

• Authors: Ke-Ching Chang, Chung-Chi Chen, An-Zi Yen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05916
• Pdf URL: https://arxiv.org/pdf/2412.05916
• Copy Paste: [[2412.05916]] Paraphrase-Aligned Machine Translation(https://arxiv.org/abs/2412.05916)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated significant capabilities in machine translation. However, their translation quality is sometimes questioned, as the generated outputs may deviate from expressions typically used by native speakers. These deviations often arise from differences in sentence structure between language systems. To address this issue, we propose ParaAlign Translator, a method that fine-tunes LLMs to paraphrase sentences, aligning their structures with those of the target language systems. This approach improves the performance of subsequent translations. Experimental results demonstrate that the proposed method enhances the LLaMA-3-8B model's performance in both resource-rich and low-resource scenarios and achieves parity with or surpassing the much larger LLaMA-3-70B model.

Title: BiDM: Pushing the Limit of Quantization for Diffusion Models

• Authors: Xingyu Zheng, Xianglong Liu, Yichen Bian, Xudong Ma, Yulun Zhang, Jiakai Wang, Jinyang Guo, Haotong Qin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05926
• Pdf URL: https://arxiv.org/pdf/2412.05926
• Copy Paste: [[2412.05926]] BiDM: Pushing the Limit of Quantization for Diffusion Models(https://arxiv.org/abs/2412.05926)
• Keywords: diffusion, generative
• Abstract: Diffusion models (DMs) have been significantly developed and widely used in various applications due to their excellent generative qualities. However, the expensive computation and massive parameters of DMs hinder their practical use in resource-constrained scenarios. As one of the effective compression approaches, quantization allows DMs to achieve storage saving and inference acceleration by reducing bit-width while maintaining generation performance. However, as the most extreme quantization form, 1-bit binarization causes the generation performance of DMs to face severe degradation or even collapse. This paper proposes a novel method, namely BiDM, for fully binarizing weights and activations of DMs, pushing quantization to the 1-bit limit. From a temporal perspective, we introduce the Timestep-friendly Binary Structure (TBS), which uses learnable activation binarizers and cross-timestep feature connections to address the highly timestep-correlated activation features of DMs. From a spatial perspective, we propose Space Patched Distillation (SPD) to address the difficulty of matching binary features during distillation, focusing on the spatial locality of image generation tasks and noise estimation networks. As the first work to fully binarize DMs, the W1A1 BiDM on the LDM-4 model for LSUN-Bedrooms 256$\times$256 achieves a remarkable FID of 22.74, significantly outperforming the current state-of-the-art general binarization methods with an FID of 59.44 and invalid generative samples, and achieves up to excellent 28.0 times storage and 52.7 times OPs savings. The code is available at this https URL .

Title: Enhanced 3D Generation by 2D Editing

• Authors: Haoran Li, Yuli Tian, Yong Liao, Lin Wang, Yuyang Wang, Peng Yuan Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05929
• Pdf URL: https://arxiv.org/pdf/2412.05929
• Copy Paste: [[2412.05929]] Enhanced 3D Generation by 2D Editing(https://arxiv.org/abs/2412.05929)
• Keywords: diffusion
• Abstract: Distilling 3D representations from pretrained 2D diffusion models is essential for 3D creative applications across gaming, film, and interior design. Current SDS-based methods are hindered by inefficient information distillation from diffusion models, which prevents the creation of photorealistic 3D contents. Our research reevaluates the SDS approach by analyzing its fundamental nature as a basic image editing process that commonly results in over-saturation, over-smoothing and lack of rich content due to the poor-quality single-step denoising. To address these limitations, we propose GE3D (3D Generation by Editing). Each iteration of GE3D utilizes a 2D editing framework that combines a noising trajectory to preserve the information of the input image, alongside a text-guided denoising trajectory. We optimize the process by aligning the latents across both trajectories. This approach fully exploits pretrained diffusion models to distill multi-granularity information through multiple denoising steps, resulting in photorealistic 3D outputs. Both theoretical and experimental results confirm the effectiveness of our approach, which not only advances 3D generation technology but also establishes a novel connection between 3D generation and 2D editing. This could potentially inspire further research in the field. Code and demos are released at this https URL.

Title: Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models

• Authors: Ma Teng, Jia Xiaojun, Duan Ranjie, Li Xinfeng, Huang Yihao, Chu Zhixuan, Liu Yang, Ren Wenqi
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05934
• Pdf URL: https://arxiv.org/pdf/2412.05934
• Copy Paste: [[2412.05934]] Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models(https://arxiv.org/abs/2412.05934)
• Keywords: security, protect, attack, large language model
• Abstract: With the rapid advancement of multimodal large language models (MLLMs), concerns regarding their security have increasingly captured the attention of both academia and industry. Although MLLMs are vulnerable to jailbreak attacks, designing effective multimodal jailbreak attacks poses unique challenges, especially given the distinct protective measures implemented across various modalities in commercial models. Previous works concentrate risks into a single modality, resulting in limited jailbreak performance. In this paper, we propose a heuristic-induced multimodal risk distribution jailbreak attack method, called HIMRD, which consists of two elements: multimodal risk distribution strategy and heuristic-induced search strategy. The multimodal risk distribution strategy is used to segment harmful instructions across multiple modalities to effectively circumvent MLLMs' security protection. The heuristic-induced search strategy identifies two types of prompts: the understanding-enhancing prompt, which helps the MLLM reconstruct the malicious prompt, and the inducing prompt, which increases the likelihood of affirmative outputs over refusals, enabling a successful jailbreak attack. Extensive experiments demonstrate that this approach effectively uncovers vulnerabilities in MLLMs, achieving an average attack success rate of 90% across seven popular open-source MLLMs and an average attack success rate of around 68% in three popular closed-source MLLMs. Our code will coming soon. Warning: This paper contains offensive and harmful examples, reader discretion is advised.

Title: Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models

• Authors: Xiao Xu, Tianhao Niu, Yuxi Xie, Libo Qin, Wanxiang Che, Min-Yen Kan
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05939
• Pdf URL: https://arxiv.org/pdf/2412.05939
• Copy Paste: [[2412.05939]] Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models(https://arxiv.org/abs/2412.05939)
• Keywords: fair, large language model
• Abstract: Multimodal Large Language Models (MLLMs) excel in vision--language tasks by pre-training solely on coarse-grained concept annotations (e.g., image captions). We hypothesize that integrating fine-grained concept annotations (e.g., object labels and object regions) will further improve performance, as both data granularities complement each other in terms of breadth and depth in concept representation. We introduce a new dataset featuring Multimodal Multi-Grained Concept annotations (MMGiC) for MLLMs. In constructing MMGiC, we explore the impact of different data recipes on multimodal comprehension and generation. Our analyses reveal that multi-grained concept annotations integrate and complement each other, under our structured template and a general MLLM framework. We clearly explore and demonstrate the potential of MMGiC to help MLLMs better locate and learn concepts, aligning vision and language at multiple granularities. We further validate our hypothesis by investigating the fair comparison and effective collaboration between MMGiC and image--caption data on 12 multimodal comprehension and generation benchmarks, e.g., their appropriate combination achieve 3.95% and 2.34% absolute improvements over image--caption data alone on POPE and SEED-Bench. Code, data and models will be available at this https URL.

Title: Adversarial Transferability in Deep Denoising Models: Theoretical Insights and Robustness Enhancement via Out-of-Distribution Typical Set Sampling

• Authors: Jie Ning, Jiebao Sun, Shengzhu Shi, Zhichang Guo, Yao Li, Hongwei Li, Boying Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05943
• Pdf URL: https://arxiv.org/pdf/2412.05943
• Copy Paste: [[2412.05943]] Adversarial Transferability in Deep Denoising Models: Theoretical Insights and Robustness Enhancement via Out-of-Distribution Typical Set Sampling(https://arxiv.org/abs/2412.05943)
• Keywords: defense, attack, robust, transformer
• Abstract: Deep learning-based image denoising models demonstrate remarkable performance, but their lack of robustness analysis remains a significant concern. A major issue is that these models are susceptible to adversarial attacks, where small, carefully crafted perturbations to input data can cause them to fail. Surprisingly, perturbations specifically crafted for one model can easily transfer across various models, including CNNs, Transformers, unfolding models, and plug-and-play models, leading to failures in those models as well. Such high adversarial transferability is not observed in classification models. We analyze the possible underlying reasons behind the high adversarial transferability through a series of hypotheses and validation experiments. By characterizing the manifolds of Gaussian noise and adversarial perturbations using the concept of typical set and the asymptotic equipartition property, we prove that adversarial samples deviate slightly from the typical set of the original input distribution, causing the models to fail. Based on these insights, we propose a novel adversarial defense method: the Out-of-Distribution Typical Set Sampling Training strategy (TS). TS not only significantly enhances the model's robustness but also marginally improves denoising performance compared to the original model.

Title: FOF-X: Towards Real-time Detailed Human Reconstruction from a Single Image

• Authors: Qiao Feng, Yebin Liu, Yu-Kun Lai, Jingyu Yang, Kun Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05961
• Pdf URL: https://arxiv.org/pdf/2412.05961
• Copy Paste: [[2412.05961]] FOF-X: Towards Real-time Detailed Human Reconstruction from a Single Image(https://arxiv.org/abs/2412.05961)
• Keywords: robust
• Abstract: We introduce FOF-X for real-time reconstruction of detailed human geometry from a single image. Balancing real-time speed against high-quality results is a persistent challenge, mainly due to the high computational demands of existing 3D representations. To address this, we propose Fourier Occupancy Field (FOF), an efficient 3D representation by learning the Fourier series. The core of FOF is to factorize a 3D occupancy field into a 2D vector field, retaining topology and spatial relationships within the 3D domain while facilitating compatibility with 2D convolutional neural networks. Such a representation bridges the gap between 3D and 2D domains, enabling the integration of human parametric models as priors and enhancing the reconstruction robustness. Based on FOF, we design a new reconstruction framework, FOF-X, to avoid the performance degradation caused by texture and lighting. This enables our real-time reconstruction system to better handle the domain gap between training images and real images. Additionally, in FOF-X, we enhance the inter-conversion algorithms between FOF and mesh representations with a Laplacian constraint and an automaton-based discontinuity matcher, improving both quality and robustness. We validate the strengths of our approach on different datasets and real-captured data, where FOF-X achieves new state-of-the-art results. The code will be released for research purposes.

Title: Efficient Semantic Splatting for Remote Sensing Multi-view Segmentation

• Authors: Zipeng Qi, Hao Chen, Haotian Zhang, Zhengxia Zou, Zhenwei Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05969
• Pdf URL: https://arxiv.org/pdf/2412.05969
• Copy Paste: [[2412.05969]] Efficient Semantic Splatting for Remote Sensing Multi-view Segmentation(https://arxiv.org/abs/2412.05969)
• Keywords: segmentation
• Abstract: In this paper, we propose a novel semantic splatting approach based on Gaussian Splatting to achieve efficient and low-latency. Our method projects the RGB attributes and semantic features of point clouds onto the image plane, simultaneously rendering RGB images and semantic segmentation results. Leveraging the explicit structure of point clouds and a one-time rendering strategy, our approach significantly enhances efficiency during optimization and rendering. Additionally, we employ SAM2 to generate pseudo-labels for boundary regions, which often lack sufficient supervision, and introduce two-level aggregation losses at the 2D feature map and 3D spatial levels to improve the view-consistent and spatial continuity.

Title: Anti-Reference: Universal and Immediate Defense Against Reference-Based Generation

• Authors: Yiren Song, Shengtao Lou, Xiaokang Liu, Hai Ci, Pei Yang, Jiaming Liu, Mike Zheng Shou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05980
• Pdf URL: https://arxiv.org/pdf/2412.05980
• Copy Paste: [[2412.05980]] Anti-Reference: Universal and Immediate Defense Against Reference-Based Generation(https://arxiv.org/abs/2412.05980)
• Keywords: security, protect, defense, attack, diffusion, generative
• Abstract: Diffusion models have revolutionized generative modeling with their exceptional ability to produce high-fidelity images. However, misuse of such potent tools can lead to the creation of fake news or disturbing content targeting individuals, resulting in significant social harm. In this paper, we introduce Anti-Reference, a novel method that protects images from the threats posed by reference-based generation techniques by adding imperceptible adversarial noise to the images. We propose a unified loss function that enables joint attacks on fine-tuning-based customization methods, non-fine-tuning customization methods, and human-centric driving methods. Based on this loss, we train a Adversarial Noise Encoder to predict the noise or directly optimize the noise using the PGD method. Our method shows certain transfer attack capabilities, effectively challenging both gray-box models and some commercial APIs. Extensive experiments validate the performance of Anti-Reference, establishing a new benchmark in image security.

Title: Chimera: Improving Generalist Model with Domain-Specific Experts

• Authors: Tianshuo Peng, Mingsheng Li, Hongbin Zhou, Renqiu Xia, Renrui Zhang, Lei Bai, Song Mao, Bin Wang, Conghui He, Aojun Zhou, Botian Shi, Tao Chen, Bo Zhang, Xiangyu Yue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05983
• Pdf URL: https://arxiv.org/pdf/2412.05983
• Copy Paste: [[2412.05983]] Chimera: Improving Generalist Model with Domain-Specific Experts(https://arxiv.org/abs/2412.05983)
• Keywords: extraction
• Abstract: Recent advancements in Large Multi-modal Models (LMMs) underscore the importance of scaling by increasing image-text paired data, achieving impressive performance on general tasks. Despite their effectiveness in broad applications, generalist models are primarily trained on web-scale datasets dominated by natural images, resulting in the sacrifice of specialized capabilities for domain-specific tasks that require extensive domain prior knowledge. Moreover, directly integrating expert models tailored for specific domains is challenging due to the representational gap and imbalanced optimization between the generalist model and experts. To address these challenges, we introduce Chimera, a scalable and low-cost multi-modal pipeline designed to boost the ability of existing LMMs with domain-specific experts. Specifically, we design a progressive training strategy to integrate features from expert models into the input of a generalist LMM. To address the imbalanced optimization caused by the well-aligned general visual encoder, we introduce a novel Generalist-Specialist Collaboration Masking (GSCM) mechanism. This results in a versatile model that excels across the chart, table, math, and document domains, achieving state-of-the-art performance on multi-modal reasoning and visual content extraction tasks, both of which are challenging tasks for assessing existing LMMs.

Title: Nested Diffusion Models Using Hierarchical Latent Priors

• Authors: Xiao Zhang, Ruoxi Jiang, Rebecca Willett, Michael Maire
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05984
• Pdf URL: https://arxiv.org/pdf/2412.05984
• Copy Paste: [[2412.05984]] Nested Diffusion Models Using Hierarchical Latent Priors(https://arxiv.org/abs/2412.05984)
• Keywords: diffusion, generative
• Abstract: We introduce nested diffusion models, an efficient and powerful hierarchical generative framework that substantially enhances the generation quality of diffusion models, particularly for images of complex scenes. Our approach employs a series of diffusion models to progressively generate latent variables at different semantic levels. Each model in this series is conditioned on the output of the preceding higher-level models, culminating in image generation. Hierarchical latent variables guide the generation process along predefined semantic pathways, allowing our approach to capture intricate structural details while significantly improving image quality. To construct these latent variables, we leverage a pre-trained visual encoder, which learns strong semantic visual representations, and modulate its capacity via dimensionality reduction and noise injection. Across multiple datasets, our system demonstrates significant enhancements in image quality for both unconditional and class/text conditional generation. Moreover, our unconditional generation system substantially outperforms the baseline conditional system. These advancements incur minimal computational overhead as the more abstract levels of our hierarchy work with lower-dimensional representations.

Title: PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representations

• Authors: Namgyu Kang, Jaemin Oh, Youngjoon Hong, Eunbyung Park
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05994
• Pdf URL: https://arxiv.org/pdf/2412.05994
• Copy Paste: [[2412.05994]] PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representations(https://arxiv.org/abs/2412.05994)
• Keywords: robust
• Abstract: The approximation of Partial Differential Equations (PDEs) using neural networks has seen significant advancements through Physics-Informed Neural Networks (PINNs). Despite their straightforward optimization framework and flexibility in implementing various PDEs, PINNs often suffer from limited accuracy due to the spectral bias of Multi-Layer Perceptrons (MLPs), which struggle to effectively learn high-frequency and non-linear components. Recently, parametric mesh representations in combination with neural networks have been investigated as a promising approach to eliminate the inductive biases of neural networks. However, they usually require very high-resolution grids and a large number of collocation points to achieve high accuracy while avoiding overfitting issues. In addition, the fixed positions of the mesh parameters restrict their flexibility, making it challenging to accurately approximate complex PDEs. To overcome these limitations, we propose Physics-Informed Gaussians (PIGs), which combine feature embeddings using Gaussian functions with a lightweight neural network. Our approach uses trainable parameters for the mean and variance of each Gaussian, allowing for dynamic adjustment of their positions and shapes during training. This adaptability enables our model to optimally approximate PDE solutions, unlike models with fixed parameter positions. Furthermore, the proposed approach maintains the same optimization framework used in PINNs, allowing us to benefit from their excellent properties. Experimental results show the competitive performance of our model across various PDEs, demonstrating its potential as a robust tool for solving complex PDEs. Our project page is available at this https URL

Title: Paddy Disease Detection and Classification Using Computer Vision Techniques: A Mobile Application to Detect Paddy Disease

• Authors: Bimarsha Khanal, Paras Poudel, Anish Chapagai, Bijan Regmi, Sitaram Pokhrel, Salik Ram Khanal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05996
• Pdf URL: https://arxiv.org/pdf/2412.05996
• Copy Paste: [[2412.05996]] Paddy Disease Detection and Classification Using Computer Vision Techniques: A Mobile Application to Detect Paddy Disease(https://arxiv.org/abs/2412.05996)
• Keywords: security, transformer
• Abstract: Plant diseases significantly impact our food supply, causing problems for farmers, economies reliant on agriculture, and global food security. Accurate and timely plant disease diagnosis is crucial for effective treatment and minimizing yield losses. Despite advancements in agricultural technology, a precise and early diagnosis remains a challenge, especially in underdeveloped regions where agriculture is crucial and agricultural experts are scarce. However, adopting Deep Learning applications can assist in accurately identifying diseases without needing plant pathologists. In this study, the effectiveness of various computer vision models for detecting paddy diseases is evaluated and proposed the best deep learning-based disease detection system. Both classification and detection using the Paddy Doctor dataset, which contains over 20,000 annotated images of paddy leaves for disease diagnosis are tested and evaluated. For detection, we utilized the YOLOv8 model-based model were used for paddy disease detection and CNN models and the Vision Transformer were used for disease classification. The average mAP50 of 69% for detection tasks was achieved and the Vision Transformer classification accuracy was 99.38%. It was found that detection models are effective at identifying multiple diseases simultaneously with less computing power, whereas classification models, though computationally expensive, exhibit better performance for classifying single diseases. Additionally, a mobile application was developed to enable farmers to identify paddy diseases instantly. Experiments with the app showed encouraging results in utilizing the trained models for both disease classification and treatment guidance.

Title: Does RLHF Scale? Exploring the Impacts From Data, Model, and Method

• Authors: Zhenyu Hou, Pengfan Du, Yilin Niu, Zhengxiao Du, Aohan Zeng, Xiao Liu, Minlie Huang, Hongning Wang, Jie Tang, Yuxiao Dong
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06000
• Pdf URL: https://arxiv.org/pdf/2412.06000
• Copy Paste: [[2412.06000]] Does RLHF Scale? Exploring the Impacts From Data, Model, and Method(https://arxiv.org/abs/2412.06000)
• Keywords: large language model
• Abstract: This study explores the scaling properties of Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs). Although RLHF is considered an important step in post-training of LLMs, its scaling potential is still largely unknown. We systematically analyze key components in the RLHF framework--model size, data composition, and inference budget--and their impacts on performance. Our findings show that increasing data diversity and volume improves reward model performance, helping process-supervision models scale better. For policy training, more response samples per prompt boost performance initially but quickly plateau. And larger reward models offer modest gains in policy training. In addition, larger policy models benefit less from RLHF with a fixed reward model. Overall, RLHF scales less efficiently than pretraining, with diminishing returns from additional computational resources. Based on these observations, we propose strategies to optimize RLHF performance within computational limits.

Title: Enhancing Content Representation for AR Image Quality Assessment Using Knowledge Distillation

• Authors: Aymen Sekhri, Seyed Ali Amirshahi, Mohamed-Chaker Larabi
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.06003
• Pdf URL: https://arxiv.org/pdf/2412.06003
• Copy Paste: [[2412.06003]] Enhancing Content Representation for AR Image Quality Assessment Using Knowledge Distillation(https://arxiv.org/abs/2412.06003)
• Keywords: robust, transformer
• Abstract: Augmented Reality (AR) is a major immersive media technology that enriches our perception of reality by overlaying digital content (the foreground) onto physical environments (the background). It has far-reaching applications, from entertainment and gaming to education, healthcare, and industrial training. Nevertheless, challenges such as visual confusion and classical distortions can result in user discomfort when using the technology. Evaluating AR quality of experience becomes essential to measure user satisfaction and engagement, facilitating the refinement necessary for creating immersive and robust experiences. Though, the scarcity of data and the distinctive characteristics of AR technology render the development of effective quality assessment metrics challenging. This paper presents a deep learning-based objective metric designed specifically for assessing image quality for AR scenarios. The approach entails four key steps, (1) fine-tuning a self-supervised pre-trained vision transformer to extract prominent features from reference images and distilling this knowledge to improve representations of distorted images, (2) quantifying distortions by computing shift representations, (3) employing cross-attention-based decoders to capture perceptual quality features, and (4) integrating regularization techniques and label smoothing to address the overfitting problem. To validate the proposed approach, we conduct extensive experiments on the ARIQA dataset. The results showcase the superior performance of our proposed approach across all model variants, namely TransformAR, TransformAR-KD, and TransformAR-KD+ in comparison to existing state-of-the-art methods.

Title: 1-800-SHARED-TASKS at RegNLP: Lexical Reranking of Semantic Retrieval (LeSeR) for Regulatory Question Answering

• Authors: Jebish Purbey, Drishti Sharma, Siddhant Gupta, Khawaja Murad, Siddartha Pullakhandam, Ram Mohan Rao Kadiyala
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06009
• Pdf URL: https://arxiv.org/pdf/2412.06009
• Copy Paste: [[2412.06009]] 1-800-SHARED-TASKS at RegNLP: Lexical Reranking of Semantic Retrieval (LeSeR) for Regulatory Question Answering(https://arxiv.org/abs/2412.06009)
• Keywords: robust
• Abstract: This paper presents the system description of our entry for the COLING 2025 RegNLP RIRAG (Regulatory Information Retrieval and Answer Generation) challenge, focusing on leveraging advanced information retrieval and answer generation techniques in regulatory domains. We experimented with a combination of embedding models, including Stella, BGE, CDE, and Mpnet, and leveraged fine-tuning and reranking for retrieving relevant documents in top ranks. We utilized a novel approach, LeSeR, which achieved competitive results with a recall@10 of 0.8201 and map@10 of 0.6655 for retrievals. This work highlights the transformative potential of natural language processing techniques in regulatory applications, offering insights into their capabilities for implementing a retrieval augmented generation system while identifying areas for future improvement in robustness and domain adaptation.

Title: Post-hoc Probabilistic Vision-Language Models

• Authors: Anton Baumann, Rui Li, Marcus Klasson, Santeri Mentu, Shyamgopal Karthik, Zeynep Akata, Arno Solin, Martin Trapp
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06014
• Pdf URL: https://arxiv.org/pdf/2412.06014
• Copy Paste: [[2412.06014]] Post-hoc Probabilistic Vision-Language Models(https://arxiv.org/abs/2412.06014)
• Keywords: generative
• Abstract: Vision-language models (VLMs), such as CLIP and SigLIP, have found remarkable success in classification, retrieval, and generative tasks. For this, VLMs deterministically map images and text descriptions to a joint latent space in which their similarity is assessed using the cosine similarity. However, a deterministic mapping of inputs fails to capture uncertainties over concepts arising from domain shifts when used in downstream tasks. In this work, we propose post-hoc uncertainty estimation in VLMs that does not require additional training. Our method leverages a Bayesian posterior approximation over the last layers in VLMs and analytically quantifies uncertainties over cosine similarities. We demonstrate its effectiveness for uncertainty quantification and support set selection in active learning. Compared to baselines, we obtain improved and well-calibrated predictive uncertainties, interpretable uncertainty estimates, and sample-efficient active learning. Our results show promise for safety-critical applications of large-scale models.

Title: siForest: Detecting Network Anomalies with Set-Structured Isolation Forest

• Authors: Christie Djidjev
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06015
• Pdf URL: https://arxiv.org/pdf/2412.06015
• Copy Paste: [[2412.06015]] siForest: Detecting Network Anomalies with Set-Structured Isolation Forest(https://arxiv.org/abs/2412.06015)
• Keywords: security, defense, robust
• Abstract: As cyber threats continue to evolve in sophistication and scale, the ability to detect anomalous network behavior has become critical for maintaining robust cybersecurity defenses. Modern cybersecurity systems face the overwhelming challenge of analyzing billions of daily network interactions to identify potential threats, making efficient and accurate anomaly detection algorithms crucial for network defense. This paper investigates the use of variations of the Isolation Forest (iForest) machine learning algorithm for detecting anomalies in internet scan data. In particular, it presents the Set-Partitioned Isolation Forest (siForest), a novel extension of the iForest method designed to detect anomalies in set-structured data. By treating instances such as sets of multiple network scans with the same IP address as cohesive units, siForest effectively addresses some challenges of analyzing complex, multidimensional datasets. Extensive experiments on synthetic datasets simulating diverse anomaly scenarios in network traffic demonstrate that siForest has the potential to outperform traditional approaches on some types of internet scan data.

Title: Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation

• Authors: Hyeonho Jeong, Chun-Hao Paul Huang, Jong Chul Ye, Niloy Mitra, Duygu Ceylan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06016
• Pdf URL: https://arxiv.org/pdf/2412.06016
• Copy Paste: [[2412.06016]] Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation(https://arxiv.org/abs/2412.06016)
• Keywords: diffusion
• Abstract: While recent foundational video generators produce visually rich output, they still struggle with appearance drift, where objects gradually degrade or change inconsistently across frames, breaking visual coherence. We hypothesize that this is because there is no explicit supervision in terms of spatial tracking at the feature level. We propose Track4Gen, a spatially aware video generator that combines video diffusion loss with point tracking across frames, providing enhanced spatial supervision on the diffusion features. Track4Gen merges the video generation and point tracking tasks into a single network by making minimal changes to existing video generation architectures. Using Stable Video Diffusion as a backbone, Track4Gen demonstrates that it is possible to unify video generation and point tracking, which are typically handled as separate tasks. Our extensive evaluations show that Track4Gen effectively reduces appearance drift, resulting in temporally stable and visually coherent video generation. Project page: this http URL

Title: A Dynamic Tree Structure for Hierarchical On-Chain Asset Management

• Authors: Mojtaba Eshghie, Gustav Andersson Kasche
• Subjects: cs.CR, cs.CY, cs.DS, cs.ET
• Abstract URL: https://arxiv.org/abs/2412.06026
• Pdf URL: https://arxiv.org/pdf/2412.06026
• Copy Paste: [[2412.06026]] A Dynamic Tree Structure for Hierarchical On-Chain Asset Management(https://arxiv.org/abs/2412.06026)
• Keywords: secure, security
• Abstract: In this paper, we introduce the Sarv, a novel non-monolithic blockchain-based data structure designed to represent hierarchical relationships between digitally representable components. Sarv serves as an underlying infrastructure for a wide range of applications requiring hierarchical data management, such as supply chain tracking, asset management, and circular economy implementations. Our approach leverages a tree-based data structure to accurately reflect products and their sub-components, enabling functionalities such as modification, disassembly, borrowing, and refurbishment, mirroring real-world operations. The hierarchy within Sarv is embedded in the on-chain data structure through a smart contract-based design, utilizing Algorand Standard Assets (ASAs). The uniqueness of Sarv lies in its compact and non-monolithic architecture, its mutability, and a two-layer action authorization scheme that enhances security and delegation of asset management. We demonstrate that Sarv addresses real-world requirements by providing a scalable, mutable, and secure solution for managing hierarchical data on the blockchain.

Title: FlexDiT: Dynamic Token Density Control for Diffusion Transformer

• Authors: Shuning Chang, Pichao Wang, Jiasheng Tang, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06028
• Pdf URL: https://arxiv.org/pdf/2412.06028
• Copy Paste: [[2412.06028]] FlexDiT: Dynamic Token Density Control for Diffusion Transformer(https://arxiv.org/abs/2412.06028)
• Keywords: extraction, diffusion, transformer, generative
• Abstract: Diffusion Transformers (DiT) deliver impressive generative performance but face prohibitive computational demands due to both the quadratic complexity of token-based self-attention and the need for extensive sampling steps. While recent research has focused on accelerating sampling, the structural inefficiencies of DiT remain underexplored. We propose FlexDiT, a framework that dynamically adapts token density across both spatial and temporal dimensions to achieve computational efficiency without compromising generation quality. Spatially, FlexDiT employs a three-segment architecture that allocates token density based on feature requirements at each layer: Poolingformer in the bottom layers for efficient global feature extraction, Sparse-Dense Token Modules (SDTM) in the middle layers to balance global context with local detail, and dense tokens in the top layers to refine high-frequency details. Temporally, FlexDiT dynamically modulates token density across denoising stages, progressively increasing token count as finer details emerge in later timesteps. This synergy between FlexDiT's spatially adaptive architecture and its temporal pruning strategy enables a unified framework that balances efficiency and fidelity throughout the generation process. Our experiments demonstrate FlexDiT's effectiveness, achieving a 55% reduction in FLOPs and a 175% improvement in inference speed on DiT-XL with only a 0.09 increase in FID score on 512$\times$512 ImageNet images, a 56% reduction in FLOPs across video generation datasets including FaceForensics, SkyTimelapse, UCF101, and Taichi-HD, and a 69% improvement in inference speed on PixArt-$\alpha$ on text-to-image generation task with a 0.24 FID score decrease. FlexDiT provides a scalable solution for high-quality diffusion-based generation compatible with further sampling optimization techniques.

Title: Latent-Reframe: Enabling Camera Control for Video Diffusion Model without Training

• Authors: Zhenghong Zhou, Jie An, Jiebo Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06029
• Pdf URL: https://arxiv.org/pdf/2412.06029
• Copy Paste: [[2412.06029]] Latent-Reframe: Enabling Camera Control for Video Diffusion Model without Training(https://arxiv.org/abs/2412.06029)
• Keywords: diffusion
• Abstract: Precise camera pose control is crucial for video generation with diffusion models. Existing methods require fine-tuning with additional datasets containing paired videos and camera pose annotations, which are both data-intensive and computationally costly, and can disrupt the pre-trained model distribution. We introduce Latent-Reframe, which enables camera control in a pre-trained video diffusion model without fine-tuning. Unlike existing methods, Latent-Reframe operates during the sampling stage, maintaining efficiency while preserving the original model distribution. Our approach reframes the latent code of video frames to align with the input camera trajectory through time-aware point clouds. Latent code inpainting and harmonization then refine the model latent space, ensuring high-quality video generation. Experimental results demonstrate that Latent-Reframe achieves comparable or superior camera control precision and video quality to training-based methods, without the need for fine-tuning on additional datasets.

Title: Perceptual Hash Inversion Attacks on Image-Based Sexual Abuse Removal Tools

• Authors: Sophie Hawkes, Christian Weinert, Teresa Almeida, Maryam Mehrnezhad
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06056
• Pdf URL: https://arxiv.org/pdf/2412.06056
• Copy Paste: [[2412.06056]] Perceptual Hash Inversion Attacks on Image-Based Sexual Abuse Removal Tools(https://arxiv.org/abs/2412.06056)
• Keywords: secure, privacy, attack, generative
• Abstract: We show that perceptual hashing, crucial for detecting and removing image-based sexual abuse (IBSA) online, faces vulnerabilities from low-budget inversion attacks based on generative AI. This jeopardizes the privacy of users, especially vulnerable groups. We advocate to implement secure hash matching in IBSA removal tools to mitigate potentially fatal consequences.

Title: Steering Large Language Models to Evaluate and Amplify Creativity

• Authors: Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck, Shao-yen Tseng, Vasudev Lal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06060
• Pdf URL: https://arxiv.org/pdf/2412.06060
• Copy Paste: [[2412.06060]] Steering Large Language Models to Evaluate and Amplify Creativity(https://arxiv.org/abs/2412.06060)
• Keywords: robust, large language model
• Abstract: Although capable of generating creative text, Large Language Models (LLMs) are poor judges of what constitutes "creativity". In this work, we show that we can leverage this knowledge of how to write creatively in order to better judge what is creative. We take a mechanistic approach that extracts differences in the internal states of an LLM when prompted to respond "boringly" or "creatively" to provide a robust measure of creativity that corresponds strongly with human judgment. We also show these internal state differences can be applied to enhance the creativity of generated text at inference time.

Title: Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond

• Authors: Yekun Ke, Yingyu Liang, Zhenmei Shi, Zhao Song, Chiwun Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06061
• Pdf URL: https://arxiv.org/pdf/2412.06061
• Copy Paste: [[2412.06061]] Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond(https://arxiv.org/abs/2412.06061)
• Keywords: transformer
• Abstract: The application of transformer-based models on time series forecasting (TSF) tasks has long been popular to study. However, many of these works fail to beat the simple linear residual model, and the theoretical understanding of this issue is still limited. In this work, we propose the first theoretical explanation of the inefficiency of transformers on TSF tasks. We attribute the mechanism behind it to {\bf Asymmetric Learning} in training attention networks. When the sign of the previous step is inconsistent with the sign of the current step in the next-step-prediction time series, attention fails to learn the residual features. This makes it difficult to generalize on out-of-distribution (OOD) data, especially on the sign-inconsistent next-step-prediction data, with the same representation pattern, whereas a linear residual network could easily accomplish it. We hope our theoretical insights provide important necessary conditions for designing the expressive and efficient transformer-based architecture for practitioners.

Title: On Socially Fair Low-Rank Approximation and Column Subset Selection

• Authors: Zhao Song, Ali Vakilian, David P. Woodruff, Samson Zhou
• Subjects: cs.LG, cs.DS, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.06063
• Pdf URL: https://arxiv.org/pdf/2412.06063
• Copy Paste: [[2412.06063]] On Socially Fair Low-Rank Approximation and Column Subset Selection(https://arxiv.org/abs/2412.06063)
• Keywords: fair
• Abstract: Low-rank approximation and column subset selection are two fundamental and related problems that are applied across a wealth of machine learning applications. In this paper, we study the question of socially fair low-rank approximation and socially fair column subset selection, where the goal is to minimize the loss over all sub-populations of the data. We show that surprisingly, even constant-factor approximation to fair low-rank approximation requires exponential time under certain standard complexity hypotheses. On the positive side, we give an algorithm for fair low-rank approximation that, for a constant number of groups and constant-factor accuracy, runs in $2^{\text{poly}(k)}$ time rather than the naïve $n^{\text{poly}(k)}$, which is a substantial improvement when the dataset has a large number $n$ of observations. We then show that there exist bicriteria approximation algorithms for fair low-rank approximation and fair column subset selection that run in polynomial time.

Title: KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models

• Authors: Fan Wang, Juyong Jiang, Chansung Park, Sunghun Kim, Jing Tang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06071
• Pdf URL: https://arxiv.org/pdf/2412.06071
• Copy Paste: [[2412.06071]] KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models(https://arxiv.org/abs/2412.06071)
• Keywords: large language model
• Abstract: The increasing sizes of large language models (LLMs) result in significant computational overhead and memory usage when adapting these models to specific tasks or domains. Various parameter-efficient fine-tuning (PEFT) methods have been devised to mitigate these challenges by training a small set of parameters for the task-specific updates of the model weights. Among PEFT methods, LoRA stands out for its simplicity and efficiency, inspiring the development of a series of variants. However, LoRA and its successors disregard the knowledge that is noisy or irrelevant to the targeted task, detrimentally impacting model performance and leading to suboptimality. To address this limitation, we introduce Knowledge-aware Singular-value Adaptation (KaSA), a PEFT method that leverages singular value decomposition (SVD) with knowledge-aware singular values to dynamically activate knowledge based on its relevance to the task at hand. We conduct extensive experiments across a range of LLMs on tasks spanning natural language understanding (NLU), generation (NLG), instruction following, and commonsense reasoning. The experimental results demonstrate that KaSA consistently outperforms FFT and 14 popular PEFT baselines across 16 benchmarks and 4 synthetic datasets, underscoring our method's efficacy and adaptability. The source code of our method is available at this https URL.

Title: Hyperspectral Image Spectral-Spatial Feature Extraction via Tensor Principal Component Analysis

• Authors: Yuemei Ren, Liang Liao, Stephen John Maybank, Yanning Zhang, Xin Liu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06075
• Pdf URL: https://arxiv.org/pdf/2412.06075
• Copy Paste: [[2412.06075]] Hyperspectral Image Spectral-Spatial Feature Extraction via Tensor Principal Component Analysis(https://arxiv.org/abs/2412.06075)
• Keywords: extraction
• Abstract: This paper addresses the challenge of spectral-spatial feature extraction for hyperspectral image classification by introducing a novel tensor-based framework. The proposed approach incorporates circular convolution into a tensor structure to effectively capture and integrate both spectral and spatial information. Building upon this framework, the traditional Principal Component Analysis (PCA) technique is extended to its tensor-based counterpart, referred to as Tensor Principal Component Analysis (TPCA). The proposed TPCA method leverages the inherent multi-dimensional structure of hyperspectral data, thereby enabling more effective feature representation. Experimental results on benchmark hyperspectral datasets demonstrate that classification models using TPCA features consistently outperform those using traditional PCA and other state-of-the-art techniques. These findings highlight the potential of the tensor-based framework in advancing hyperspectral image analysis.

Title: Are foundation models for computer vision good conformal predictors?

• Authors: Leo Fillioux, Julio Silva-Rodríguez, Ismail Ben Ayed, Paul-Henry Cournède, Maria Vakalopoulou, Stergios Christodoulidis, Jose Dolz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06082
• Pdf URL: https://arxiv.org/pdf/2412.06082
• Copy Paste: [[2412.06082]] Are foundation models for computer vision good conformal predictors?(https://arxiv.org/abs/2412.06082)
• Keywords: transformer
• Abstract: Recent advances in self-supervision and constrastive learning have brought the performance of foundation models to unprecedented levels in a variety of tasks. Fueled by this progress, these models are becoming the prevailing approach for a wide array of real-world vision problems, including risk-sensitive and high-stakes applications. However, ensuring safe deployment in these scenarios requires a more comprehensive understanding of their uncertainty modeling capabilities, which has been barely explored. In this work, we delve into the behavior of vision and vision-language foundation models under Conformal Prediction (CP), a statistical framework that provides theoretical guarantees of marginal coverage of the true class. Across extensive experiments including popular vision classification benchmarks, well-known foundation vision models, and three CP methods, our findings reveal that foundation models are well-suited for conformalization procedures, particularly those integrating Vision Transformers. Furthermore, we show that calibrating the confidence predictions of these models leads to efficiency degradation of the conformal set on adaptive CP methods. In contrast, few-shot adaptation to downstream tasks generally enhances conformal scores, where we identify Adapters as a better conformable alternative compared to Prompt Learning strategies. Our empirical study identifies APS as particularly promising in the context of vision foundation models, as it does not violate the marginal coverage property across multiple challenging, yet realistic scenarios.

Title: A4-Unet: Deformable Multi-Scale Attention Network for Brain Tumor Segmentation

• Authors: Ruoxin Wang, Tianyi Tang, Haiming Du, Yuxuan Cheng, Yu Wang, Lingjie Yang, Xiaohui Duan, Yunfang Yu, Yu Zhou, Donglong Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06088
• Pdf URL: https://arxiv.org/pdf/2412.06088
• Copy Paste: [[2412.06088]] A4-Unet: Deformable Multi-Scale Attention Network for Brain Tumor Segmentation(https://arxiv.org/abs/2412.06088)
• Keywords: segmentation
• Abstract: Brain tumor segmentation models have aided diagnosis in recent years. However, they face MRI complexity and variability challenges, including irregular shapes and unclear boundaries, leading to noise, misclassification, and incomplete segmentation, thereby limiting accuracy. To address these issues, we adhere to an outstanding Convolutional Neural Networks (CNNs) design paradigm and propose a novel network named A4-Unet. In A4-Unet, Deformable Large Kernel Attention (DLKA) is incorporated in the encoder, allowing for improved capture of multi-scale tumors. Swin Spatial Pyramid Pooling (SSPP) with cross-channel attention is employed in a bottleneck further to study long-distance dependencies within images and channel relationships. To enhance accuracy, a Combined Attention Module (CAM) with Discrete Cosine Transform (DCT) orthogonality for channel weighting and convolutional element-wise multiplication is introduced for spatial weighting in the decoder. Attention gates (AG) are added in the skip connection to highlight the foreground while suppressing irrelevant background information. The proposed network is evaluated on three authoritative MRI brain tumor benchmarks and a proprietary dataset, and it achieves a 94.4% Dice score on the BraTS 2020 dataset, thereby establishing multiple new state-of-the-art benchmarks. The code is available here: this https URL.

Title: GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis

• Authors: Ashish Goswami, Satyam Kumar Modi, Santhosh Rishi Deshineni, Harman Singh, Prathosh A. P, Parag Singla
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06089
• Pdf URL: https://arxiv.org/pdf/2412.06089
• Copy Paste: [[2412.06089]] GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis(https://arxiv.org/abs/2412.06089)
• Keywords: diffusion
• Abstract: Text-to-image (T2I) generation has seen significant progress with diffusion models, enabling generation of photo-realistic images from text prompts. Despite this progress, existing methods still face challenges in following complex text prompts, especially those requiring compositional and multi-step reasoning. Given such complex instructions, SOTA models often make mistakes in faithfully modeling object attributes, and relationships among them. In this work, we present an alternate paradigm for T2I synthesis, decomposing the task of complex multi-step generation into three steps, (a) Generate: we first generate an image using existing diffusion models (b) Plan: we make use of Multi-Modal LLMs (MLLMs) to identify the mistakes in the generated image expressed in terms of individual objects and their properties, and produce a sequence of corrective steps required in the form of an edit-plan. (c) Edit: we make use of an existing text-guided image editing models to sequentially execute our edit-plan over the generated image to get the desired image which is faithful to the original instruction. Our approach derives its strength from the fact that it is modular in nature, is training free, and can be applied over any combination of image generation and editing models. As an added contribution, we also develop a model capable of compositional editing, which further helps improve the overall accuracy of our proposed approach. Our method flexibly trades inference time compute with performance on compositional text prompts. We perform extensive experimental evaluation across 3 benchmarks and 10 T2I models including DALLE-3 and the latest -- SD-3.5-Large. Our approach not only improves the performance of the SOTA models, by upto 3 points, it also reduces the performance gap between weaker and stronger models. $\href{this https URL}{this https URL}$

Title: Trust No AI: Prompt Injection Along The CIA Security Triad

• Authors: Johann Rehberger (Independent Researcher, Embrace The Red)
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06090
• Pdf URL: https://arxiv.org/pdf/2412.06090
• Copy Paste: [[2412.06090]] Trust No AI: Prompt Injection Along The CIA Security Triad(https://arxiv.org/abs/2412.06090)
• Keywords: security, large language model
• Abstract: The CIA security triad - Confidentiality, Integrity, and Availability - is a cornerstone of data and cybersecurity. With the emergence of large language model (LLM) applications, a new class of threat, known as prompt injection, was first identified in 2022. Since then, numerous real-world vulnerabilities and exploits have been documented in production LLM systems, including those from leading vendors like OpenAI, Microsoft, Anthropic and Google. This paper compiles real-world exploits and proof-of concept examples, based on the research conducted and publicly documented by the author, demonstrating how prompt injection undermines the CIA triad and poses ongoing risks to cybersecurity and AI systems at large.

Title: A Linear-Time Algorithm for the Closest Vector Problem of Triangular Lattices

• Authors: Kenta Takahashi, Wataru Nakamura
• Subjects: cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2412.06091
• Pdf URL: https://arxiv.org/pdf/2412.06091
• Copy Paste: [[2412.06091]] A Linear-Time Algorithm for the Closest Vector Problem of Triangular Lattices(https://arxiv.org/abs/2412.06091)
• Keywords: biometric
• Abstract: Fuzzy Extractor (FE) and Fuzzy Signature (FS) are useful schemes for generating cryptographic keys from fuzzy data such as biometric features. Several techniques have been proposed to implement FE and FS for fuzzy data in an Euclidean space, such as facial feature vectors, that use triangular lattice-based error correction. In these techniques, solving the closest vector problem (CVP) in a high dimensional (e.g., 128--512 dim.) lattice is required at the time of key reproduction or signing. However, solving CVP becomes computationally hard as the dimension $n$ increases. In this paper, we first propose a CVP algorithm in triangular lattices with $O(n \log n)$-time whereas the conventional one requires $O(n^2)$-time. Then we further improve it and construct an $O(n)$-time algorithm.

Title: Enhanced Computationally Efficient Long LoRA Inspired Perceiver Architectures for Auto-Regressive Language Modeling

• Authors: Kaleel Mahmood, Shaoyi Huang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06106
• Pdf URL: https://arxiv.org/pdf/2412.06106
• Copy Paste: [[2412.06106]] Enhanced Computationally Efficient Long LoRA Inspired Perceiver Architectures for Auto-Regressive Language Modeling(https://arxiv.org/abs/2412.06106)
• Keywords: transformer, large language model
• Abstract: The Transformer architecture has revolutionized the Natural Language Processing field and is the backbone of Large Language Models (LLMs). The Transformer uses the attention mechanism that computes the pair-wise similarity between its input tokens to produce latent vectors that are able to understand the semantic meaning of the input text. One of the challenges in the Transformer architecture is the quadratic complexity of the attention mechanism that prohibits the efficient processing of long sequence lengths. While many recent research works have attempted to provide a reduction from $O(n^2)$ time complexity of attention to semi-linear complexity, it remains an unsolved problem in the sense of maintaining a high performance when such complexity is reduced. One of the important works in this respect is the Perceiver class of architectures that have demonstrated excellent performance while reducing the computation complexity. In this paper, we use the PerceiverAR that was proposed for Auto-Regressive modeling as a baseline, and provide three different architectural enhancements to it with varying computation overhead tradeoffs. Inspired by the recently proposed efficient attention computation approach of Long-LoRA, we then present an equally efficient Perceiver-based architecture (termed as Long LoRA Pereceiver - LLP) that can be used as the base architecture in LLMs instead of just a fine-tuning add-on. Our results on different benchmarks indicate impressive improvements compared to recent Transformer based models.

Title: Infusing Prompts with Syntax and Semantics

• Authors: Anton Bulle Labate, Fabio Gagliardi Cozman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06107
• Pdf URL: https://arxiv.org/pdf/2412.06107
• Copy Paste: [[2412.06107]] Infusing Prompts with Syntax and Semantics(https://arxiv.org/abs/2412.06107)
• Keywords: large language model
• Abstract: Despite impressive success, language models often generate outputs with flawed linguistic structure. We analyze the effect of directly infusing various kinds of syntactic and semantic information into large language models. To demonstrate the value of our proposals, we focus on the translation of natural language queries to SQL, in particular dealing with languages with less resources than English, to better investigate how much help we can get from low cost syntactic and semantic information. We show that linguistic analysis can significantly boost language models, to the point that we have surpassed previous best systems.

Title: Privacy-Preserving Large Language Models: Mechanisms, Applications, and Future Directions

• Authors: Guoshenghui Zhao, Eric Song
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06113
• Pdf URL: https://arxiv.org/pdf/2412.06113
• Copy Paste: [[2412.06113]] Privacy-Preserving Large Language Models: Mechanisms, Applications, and Future Directions(https://arxiv.org/abs/2412.06113)
• Keywords: privacy, attack, robust, membership infer, federate, large language model
• Abstract: The rapid advancement of large language models (LLMs) has revolutionized natural language processing, enabling applications in diverse domains such as healthcare, finance and education. However, the growing reliance on extensive data for training and inference has raised significant privacy concerns, ranging from data leakage to adversarial attacks. This survey comprehensively explores the landscape of privacy-preserving mechanisms tailored for LLMs, including differential privacy, federated learning, cryptographic protocols, and trusted execution environments. We examine their efficacy in addressing key privacy challenges, such as membership inference and model inversion attacks, while balancing trade-offs between privacy and model utility. Furthermore, we analyze privacy-preserving applications of LLMs in privacy-sensitive domains, highlighting successful implementations and inherent limitations. Finally, this survey identifies emerging research directions, emphasizing the need for novel frameworks that integrate privacy by design into the lifecycle of LLMs. By synthesizing state-of-the-art approaches and future trends, this paper provides a foundation for developing robust, privacy-preserving large language models that safeguard sensitive information without compromising performance.

Title: Lightweight Federated Learning with Differential Privacy and Straggler Resilience

• Authors: Shu Hong, Xiaojun Lin, Lingjie Duan
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.06120
• Pdf URL: https://arxiv.org/pdf/2412.06120
• Copy Paste: [[2412.06120]] Lightweight Federated Learning with Differential Privacy and Straggler Resilience(https://arxiv.org/abs/2412.06120)
• Keywords: secure, privacy, attack, federate
• Abstract: Federated learning (FL) enables collaborative model training through model parameter exchanges instead of raw data. To avoid potential inference attacks from exchanged parameters, differential privacy (DP) offers rigorous guarantee against various attacks. However, conventional methods of ensuring DP by adding local noise alone often result in low training accuracy. Combining secure multi-party computation (SMPC) with DP, while improving the accuracy, incurs high communication and computation overheads and straggler vulnerability, in either client-to-server or client-to-client links. In this paper, we propose LightDP-FL, a novel lightweight scheme that ensures provable DP against untrusted peers and server, while maintaining straggler-resilience, low overheads and high training accuracy. Our approach incorporates both individual and pairwise noise into each client's parameter, which can be implemented with minimal overheads. Given the uncertain straggler and colluder sets, we utilize the upper bound on the numbers of stragglers and colluders to prove sufficient noise variance conditions to ensure DP in the worst case. Moreover, we optimize the expected convergence bound to ensure accuracy performance by flexibly controlling the noise variances. Using the CIFAR-10 dataset, our experimental results demonstrate that LightDP-FL achieves faster convergence and stronger straggler resilience of our scheme compared to baseline methods of the same DP level.

Title: HSDA: High-frequency Shuffle Data Augmentation for Bird's-Eye-View Map Segmentation

• Authors: Calvin Glisson, Qiuxiao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06127
• Pdf URL: https://arxiv.org/pdf/2412.06127
• Copy Paste: [[2412.06127]] HSDA: High-frequency Shuffle Data Augmentation for Bird's-Eye-View Map Segmentation(https://arxiv.org/abs/2412.06127)
• Keywords: segmentation
• Abstract: Autonomous driving has garnered significant attention in recent research, and Bird's-Eye-View (BEV) map segmentation plays a vital role in the field, providing the basis for safe and reliable operation. While data augmentation is a commonly used technique for improving BEV map segmentation networks, existing approaches predominantly focus on manipulating spatial domain representations. In this work, we investigate the potential of frequency domain data augmentation for camera-based BEV map segmentation. We observe that high-frequency information in camera images is particularly crucial for accurate segmentation. Based on this insight, we propose High-frequency Shuffle Data Augmentation (HSDA), a novel data augmentation strategy that enhances a network's ability to interpret high-frequency image content. This approach encourages the network to distinguish relevant high-frequency information from noise, leading to improved segmentation results for small and intricate image regions, as well as sharper edge and detail perception. Evaluated on the nuScenes dataset, our method demonstrates broad applicability across various BEV map segmentation networks, achieving a new state-of-the-art mean Intersection over Union (mIoU) of 61.3% for camera-only systems. This significant improvement underscores the potential of frequency domain data augmentation for advancing the field of autonomous driving perception. Code has been released: this https URL

Title: GCUNet: A GNN-Based Contextual Learning Network for Tertiary Lymphoid Structure Semantic Segmentation in Whole Slide Image

• Authors: Lei Su, Yang Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06129
• Pdf URL: https://arxiv.org/pdf/2412.06129
• Copy Paste: [[2412.06129]] GCUNet: A GNN-Based Contextual Learning Network for Tertiary Lymphoid Structure Semantic Segmentation in Whole Slide Image(https://arxiv.org/abs/2412.06129)
• Keywords: segmentation
• Abstract: We focus on tertiary lymphoid structure (TLS) semantic segmentation in whole slide image (WSI). Unlike TLS binary segmentation, TLS semantic segmentation identifies boundaries and maturity, which requires integrating contextual information to discover discriminative features. Due to the extensive scale of WSI (e.g., 100,000 \times 100,000 pixels), the segmentation of TLS is usually carried out through a patch-based strategy. However, this prevents the model from accessing information outside of the patches, limiting the performance. To address this issue, we propose GCUNet, a GNN-based contextual learning network for TLS semantic segmentation. Given an image patch (target) to be segmented, GCUNet first progressively aggregates long-range and fine-grained context outside the target. Then, a Detail and Context Fusion block (DCFusion) is designed to integrate the context and detail of the target to predict the segmentation mask. We build four TLS semantic segmentation datasets, called TCGA-COAD, TCGA-LUSC, TCGA-BLCA and INHOUSE-PAAD, and make the former three datasets (comprising 826 WSIs and 15,276 TLSs) publicly available to promote the TLS semantic segmentation. Experiments on these datasets demonstrate the superiority of GCUNet, achieving at least 7.41% improvement in mF1 compared with SOTA.

Title: Evaluating and Mitigating Social Bias for Large Language Models in Open-ended Settings

• Authors: Zhao Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06134
• Pdf URL: https://arxiv.org/pdf/2412.06134
• Copy Paste: [[2412.06134]] Evaluating and Mitigating Social Bias for Large Language Models in Open-ended Settings(https://arxiv.org/abs/2412.06134)
• Keywords: protect, large language model
• Abstract: Current social bias benchmarks for Large Language Models (LLMs) primarily rely on pre-defined question formats like multiple-choice, limiting their ability to reflect the complexity and open-ended nature of real-world interactions. To address this gap, we extend an existing BBQ dataset introduced by incorporating fill-in-the-blank and short-answer question types, designed to evaluate biases in an open-ended setting. Our finding reveals that LLMs tend to produce responses that are more biased against certain protected attributes, like age and socio-economic status. On the other hand, these biased outputs produced by LLMs can serve as valuable contexts and chains of thought for debiasing. Our debiasing approach combined zero-shot, few-shot, and chain-of-thought could significantly reduce the level of bias to almost 0. We open-source our evaluation and debiasing code hoping to encourage further measurements and mitigation of bias and stereotype in LLMs.

Title: AIDE: Task-Specific Fine Tuning with Attribute Guided Multi-Hop Data Expansion

• Authors: Jiayu Li, Xuan Zhu, Fang Liu, Yanjun Qi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06136
• Pdf URL: https://arxiv.org/pdf/2412.06136
• Copy Paste: [[2412.06136]] AIDE: Task-Specific Fine Tuning with Attribute Guided Multi-Hop Data Expansion(https://arxiv.org/abs/2412.06136)
• Keywords: large language model
• Abstract: Fine-tuning large language models (LLMs) for specific tasks requires high-quality, diverse training data relevant to the task. Recent research has leveraged LLMs to synthesize training data, but existing approaches either depend on large seed datasets or struggle to ensure both task relevance and data diversity in the generated outputs. To address these challenges, we propose AIDE, a novel data synthesis framework that uses a multi-hop process to expand 10 seed data points while ensuring diversity and task relevance. AIDE extracts the main topic and key knowledge attributes from the seed data to guide the synthesis process. In each subsequent hop, it extracts the topic and attributes from the newly generated data and continues guided synthesis. This process repeats for a total of K hops. To prevent irrelevant data generation as the hop depth increases, AIDE incorporates a residual connection mechanism and uses self-reflection to improve data quality. Our empirical results demonstrate that fine-tuning Mistral-7B, Llama-3.1-8B and Llama-3.2-3B with AIDE achieves more than 10% accuracy improvements over the base models across 13 tasks from 5 different benchmarks, while outperforming the models fine-tuned with state-of-the-art data synthesis methods like Evol-Instruct, DataTune and Prompt2Model.

Title: SGIA: Enhancing Fine-Grained Visual Classification with Sequence Generative Image Augmentation

• Authors: Qiyu Liao, Xin Yuan, Min Xu, Dadong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06138
• Pdf URL: https://arxiv.org/pdf/2412.06138
• Copy Paste: [[2412.06138]] SGIA: Enhancing Fine-Grained Visual Classification with Sequence Generative Image Augmentation(https://arxiv.org/abs/2412.06138)
• Keywords: diffusion, generative
• Abstract: In Fine-Grained Visual Classification (FGVC), distinguishing highly similar subcategories remains a formidable challenge, often necessitating datasets with extensive variability. The acquisition and annotation of such FGVC datasets are notably difficult and costly, demanding specialized knowledge to identify subtle distinctions among closely related categories. Our study introduces a novel approach employing the Sequence Latent Diffusion Model (SLDM) for augmenting FGVC datasets, called Sequence Generative Image Augmentation (SGIA). Our method features a unique Bridging Transfer Learning (BTL) process, designed to minimize the domain gap between real and synthetically augmented data. This approach notably surpasses existing methods in generating more realistic image samples, providing a diverse range of pose transformations that extend beyond the traditional rigid transformations and style changes in generative augmentation. We demonstrate the effectiveness of our augmented dataset with substantial improvements in FGVC tasks on various datasets, models, and training strategies, especially in few-shot learning scenarios. Our method outperforms conventional image augmentation techniques in benchmark tests on three FGVC datasets, showcasing superior realism, variability, and representational quality. Our work sets a new benchmark and outperforms the previous state-of-the-art models in classification accuracy by 0.5% for the CUB-200-2011 dataset and advances the application of generative models in FGVC data augmentation.

Title: AgentAlign: Misalignment-Adapted Multi-Agent Perception for Resilient Inter-Agent Sensor Correlations

• Authors: Zonglin Meng, Yun Zhang, Zhaoliang Zheng, Zhihao Zhao, Jiaqi Ma
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.06142
• Pdf URL: https://arxiv.org/pdf/2412.06142
• Copy Paste: [[2412.06142]] AgentAlign: Misalignment-Adapted Multi-Agent Perception for Resilient Inter-Agent Sensor Correlations(https://arxiv.org/abs/2412.06142)
• Keywords: robust
• Abstract: Cooperative perception has attracted wide attention given its capability to leverage shared information across connected automated vehicles (CAVs) and smart infrastructures to address sensing occlusion and range limitation issues. However, existing research overlooks the fragile multi-sensor correlations in multi-agent settings, as the heterogeneous agent sensor measurements are highly susceptible to environmental factors, leading to weakened inter-agent sensor interactions. The varying operational conditions and other real-world factors inevitably introduce multifactorial noise and consequentially lead to multi-sensor misalignment, making the deployment of multi-agent multi-modality perception particularly challenging in the real world. In this paper, we propose AgentAlign, a real-world heterogeneous agent cross-modality feature alignment framework, to effectively address these multi-modality misalignment issues. Our method introduces a cross-modality feature alignment space (CFAS) and heterogeneous agent feature alignment (HAFA) mechanism to harmonize multi-modality features across various agents dynamically. Additionally, we present a novel V2XSet-noise dataset that simulates realistic sensor imperfections under diverse environmental conditions, facilitating a systematic evaluation of our approach's robustness. Extensive experiments on the V2X-Real and V2XSet-Noise benchmarks demonstrate that our framework achieves state-of-the-art performance, underscoring its potential for real-world applications in cooperative autonomous driving. The controllable V2XSet-Noise dataset and generation pipeline will be released in the future.

Title: Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matters

• Authors: Yuan Wang, Ouxiang Li, Tingting Mu, Yanbin Hao, Kuien Liu, Xiang Wang, Xiangnan He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06143
• Pdf URL: https://arxiv.org/pdf/2412.06143
• Copy Paste: [[2412.06143]] Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matters(https://arxiv.org/abs/2412.06143)
• Keywords: diffusion
• Abstract: The success of text-to-image generation enabled by diffuion models has imposed an urgent need to erase unwanted concepts, e.g., copyrighted, offensive, and unsafe ones, from the pre-trained models in a precise, timely, and low-cost manner. The twofold demand of concept erasure requires a precise removal of the target concept during generation (i.e., erasure efficacy), while a minimal impact on non-target content generation (i.e., prior preservation). Existing methods are either computationally costly or face challenges in maintaining an effective balance between erasure efficacy and prior preservation. To improve, we propose a precise, fast, and low-cost concept erasure method, called Adaptive Vaule Decomposer (AdaVD), which is training-free. This method is grounded in a classical linear algebraic orthogonal complement operation, implemented in the value space of each cross-attention layer within the UNet of diffusion models. An effective shift factor is designed to adaptively navigate the erasure strength, enhancing prior preservation without sacrificing erasure efficacy. Extensive experimental results show that the proposed AdaVD is effective at both single and multiple concept erasure, showing a 2- to 10-fold improvement in prior preservation as compared to the second best, meanwhile achieving the best or near best erasure efficacy, when comparing with both training-based and training-free state of the arts. AdaVD supports a series of diffusion models and downstream image generation tasks, the code is available on the project page: this https URL

Title: Hate Speech According to the Law: An Analysis for Effective Detection

• Authors: Katerina Korre, John Pavlopoulos, Paolo Gajo, Alberto Barrón-Cedeño
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06144
• Pdf URL: https://arxiv.org/pdf/2412.06144
• Copy Paste: [[2412.06144]] Hate Speech According to the Law: An Analysis for Effective Detection(https://arxiv.org/abs/2412.06144)
• Keywords: robust, large language model
• Abstract: The issue of hate speech extends beyond the confines of the online realm. It is a problem with real-life repercussions, prompting most nations to formulate legal frameworks that classify hate speech as a punishable offence. These legal frameworks differ from one country to another, contributing to the big chaos that online platforms have to face when addressing reported instances of hate speech. With the definitions of hate speech falling short in introducing a robust framework, we turn our gaze onto hate speech laws. We consult the opinion of legal experts on a hate speech dataset and we experiment by employing various approaches such as pretrained models both on hate speech and legal data, as well as exploiting two large language models (Qwen2-7B-Instruct and Meta-Llama-3-70B). Due to the time-consuming nature of data acquisition for prosecutable hate speech, we use pseudo-labeling to improve our pretrained models. This study highlights the importance of amplifying research on prosecutable hate speech and provides insights into effective strategies for combating hate speech within the parameters of legal frameworks. Our findings show that legal knowledge in the form of annotations can be useful when classifying prosecutable hate speech, yet more focus should be paid on the differences between the laws.

Title: An Effective and Resilient Backdoor Attack Framework against Deep Neural Networks and Vision Transformers

• Authors: Xueluan Gong, Bowei Tian, Meng Xue, Yuan Wu, Yanjiao Chen, Qian Wang
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06149
• Pdf URL: https://arxiv.org/pdf/2412.06149
• Copy Paste: [[2412.06149]] An Effective and Resilient Backdoor Attack Framework against Deep Neural Networks and Vision Transformers(https://arxiv.org/abs/2412.06149)
• Keywords: defense, attack, robust, transformer
• Abstract: Recent studies have revealed the vulnerability of Deep Neural Network (DNN) models to backdoor attacks. However, existing backdoor attacks arbitrarily set the trigger mask or use a randomly selected trigger, which restricts the effectiveness and robustness of the generated backdoor triggers. In this paper, we propose a novel attention-based mask generation methodology that searches for the optimal trigger shape and location. We also introduce a Quality-of-Experience (QoE) term into the loss function and carefully adjust the transparency value of the trigger in order to make the backdoored samples to be more natural. To further improve the prediction accuracy of the victim model, we propose an alternating retraining algorithm in the backdoor injection process. The victim model is retrained with mixed poisoned datasets in even iterations and with only benign samples in odd iterations. Besides, we launch the backdoor attack under a co-optimized attack framework that alternately optimizes the backdoor trigger and backdoored model to further improve the attack performance. Apart from DNN models, we also extend our proposed attack method against vision transformers. We evaluate our proposed method with extensive experiments on VGG-Flower, CIFAR-10, GTSRB, CIFAR-100, and ImageNette datasets. It is shown that we can increase the attack success rate by as much as 82\% over baselines when the poison ratio is low and achieve a high QoE of the backdoored samples. Our proposed backdoor attack framework also showcases robustness against state-of-the-art backdoor defenses.

Title: A Hyperdimensional One Place Signature to Represent Them All: Stackable Descriptors For Visual Place Recognition

• Authors: Connor Malone, Somayeh Hussaini, Tobias Fischer, Michael Milford
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06153
• Pdf URL: https://arxiv.org/pdf/2412.06153
• Copy Paste: [[2412.06153]] A Hyperdimensional One Place Signature to Represent Them All: Stackable Descriptors For Visual Place Recognition(https://arxiv.org/abs/2412.06153)
• Keywords: robust
• Abstract: Visual Place Recognition (VPR) enables coarse localization by comparing query images to a reference database of geo-tagged images. Recent breakthroughs in deep learning architectures and training regimes have led to methods with improved robustness to factors like environment appearance change, but with the downside that the required training and/or matching compute scales with the number of distinct environmental conditions encountered. Here, we propose Hyperdimensional One Place Signatures (HOPS) to simultaneously improve the performance, compute and scalability of these state-of-the-art approaches by fusing the descriptors from multiple reference sets captured under different conditions. HOPS scales to any number of environmental conditions by leveraging the Hyperdimensional Computing framework. Extensive evaluations demonstrate that our approach is highly generalizable and consistently improves recall performance across all evaluated VPR methods and datasets by large margins. Arbitrarily fusing reference images without compute penalty enables numerous other useful possibilities, three of which we demonstrate here: descriptor dimensionality reduction with no performance penalty, stacking synthetic images, and coarse localization to an entire traverse or environmental section.

Title: MoSH: Modeling Multi-Objective Tradeoffs with Soft and Hard Bounds

• Authors: Edward Chen, Natalie Dullerud, Thomas Niedermayr, Elizabeth Kidd, Ransalu Senanayake, Pang Wei Koh, Sanmi Koyejo, Carlos Guestrin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06154
• Pdf URL: https://arxiv.org/pdf/2412.06154
• Copy Paste: [[2412.06154]] MoSH: Modeling Multi-Objective Tradeoffs with Soft and Hard Bounds(https://arxiv.org/abs/2412.06154)
• Keywords: robust, large language model
• Abstract: Countless science and engineering applications in multi-objective optimization (MOO) necessitate that decision-makers (DMs) select a Pareto-optimal solution which aligns with their preferences. Evaluating individual solutions is often expensive, necessitating cost-sensitive optimization techniques. Due to competing objectives, the space of trade-offs is also expansive -- thus, examining the full Pareto frontier may prove overwhelming to a DM. Such real-world settings generally have loosely-defined and context-specific desirable regions for each objective function that can aid in constraining the search over the Pareto frontier. We introduce a novel conceptual framework that operationalizes these priors using soft-hard functions, SHFs, which allow for the DM to intuitively impose soft and hard bounds on each objective -- which has been lacking in previous MOO frameworks. Leveraging a novel minimax formulation for Pareto frontier sampling, we propose a two-step process for obtaining a compact set of Pareto-optimal points which respect the user-defined soft and hard bounds: (1) densely sample the Pareto frontier using Bayesian optimization, and (2) sparsify the selected set to surface to the user, using robust submodular function optimization. We prove that (2) obtains the optimal compact Pareto-optimal set of points from (1). We further show that many practical problems fit within the SHF framework and provide extensive empirical validation on diverse domains, including brachytherapy, engineering design, and large language model personalization. Specifically, for brachytherapy, our approach returns a compact set of points with over 3% greater SHF-defined utility than the next best approach. Among the other diverse experiments, our approach consistently leads in utility, allowing the DM to reach >99% of their maximum possible desired utility within validation of 5 points.

Title: Membership Inference Attacks and Defenses in Federated Learning: A Survey

• Authors: Li Bai, Haibo Hu, Qingqing Ye, Haoyang Li, Leixia Wang, Jianliang Xu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06157
• Pdf URL: https://arxiv.org/pdf/2412.06157
• Copy Paste: [[2412.06157]] Membership Inference Attacks and Defenses in Federated Learning: A Survey(https://arxiv.org/abs/2412.06157)
• Keywords: privacy, defense, attack, membership infer, federate
• Abstract: Federated learning is a decentralized machine learning approach where clients train models locally and share model updates to develop a global model. This enables low-resource devices to collaboratively build a high-quality model without requiring direct access to the raw training data. However, despite only sharing model updates, federated learning still faces several privacy vulnerabilities. One of the key threats is membership inference attacks, which target clients' privacy by determining whether a specific example is part of the training set. These attacks can compromise sensitive information in real-world applications, such as medical diagnoses within a healthcare system. Although there has been extensive research on membership inference attacks, a comprehensive and up-to-date survey specifically focused on it within federated learning is still absent. To fill this gap, we categorize and summarize membership inference attacks and their corresponding defense strategies based on their characteristics in this setting. We introduce a unique taxonomy of existing attack research and provide a systematic overview of various countermeasures. For these studies, we thoroughly analyze the strengths and weaknesses of different approaches. Finally, we identify and discuss key future research directions for readers interested in advancing the field.

Title: ASGDiffusion: Parallel High-Resolution Generation with Asynchronous Structure Guidance

• Authors: Yuming Li, Peidong Jia, Daiwei Hong, Yueru Jia, Qi She, Rui Zhao, Ming Lu, Shanghang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06163
• Pdf URL: https://arxiv.org/pdf/2412.06163
• Copy Paste: [[2412.06163]] ASGDiffusion: Parallel High-Resolution Generation with Asynchronous Structure Guidance(https://arxiv.org/abs/2412.06163)
• Keywords: diffusion
• Abstract: Training-free high-resolution (HR) image generation has garnered significant attention due to the high costs of training large diffusion models. Most existing methods begin by reconstructing the overall structure and then proceed to refine the local details. Despite their advancements, they still face issues with repetitive patterns in HR image generation. Besides, HR generation with diffusion models incurs significant computational costs. Thus, parallel generation is essential for interactive applications. To solve the above limitations, we introduce a novel method named ASGDiffusion for parallel HR generation with Asynchronous Structure Guidance (ASG) using pre-trained diffusion models. To solve the pattern repetition problem of HR image generation, ASGDiffusion leverages the low-resolution (LR) noise weighted by the attention mask as the structure guidance for the denoising step to ensure semantic consistency. The proposed structure guidance can significantly alleviate the pattern repetition problem. To enable parallel generation, we further propose a parallelism strategy, which calculates the patch noises and structure guidance asynchronously. By leveraging multi-GPU parallel acceleration, we significantly accelerate generation speed and reduce memory usage per GPU. Extensive experiments demonstrate that our method effectively and efficiently addresses common issues like pattern repetition and achieves state-of-the-art HR generation.

Title: Out-of-Distribution Detection with Overlap Index

• Authors: Hao Fu, Prashanth Krishnamurthy, Siddharth Garg, Farshad Khorrami
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.06168
• Pdf URL: https://arxiv.org/pdf/2412.06168
• Copy Paste: [[2412.06168]] Out-of-Distribution Detection with Overlap Index(https://arxiv.org/abs/2412.06168)
• Keywords: robust, interpretability
• Abstract: Out-of-distribution (OOD) detection is crucial for the deployment of machine learning models in the open world. While existing OOD detectors are effective in identifying OOD samples that deviate significantly from in-distribution (ID) data, they often come with trade-offs. For instance, deep OOD detectors usually suffer from high computational costs, require tuning hyperparameters, and have limited interpretability, whereas traditional OOD detectors may have a low accuracy on large high-dimensional datasets. To address these limitations, we propose a novel effective OOD detection approach that employs an overlap index (OI)-based confidence score function to evaluate the likelihood of a given input belonging to the same distribution as the available ID samples. The proposed OI-based confidence score function is non-parametric, lightweight, and easy to interpret, hence providing strong flexibility and generality. Extensive empirical evaluations indicate that our OI-based OOD detector is competitive with state-of-the-art OOD detectors in terms of detection accuracy on a wide range of datasets while requiring less computation and memory costs. Lastly, we show that the proposed OI-based confidence score function inherits nice properties from OI (e.g., insensitivity to small distributional variations and robustness against Huber $\epsilon$-contamination) and is a versatile tool for estimating OI and model accuracy in specific contexts.

Title: Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity

• Authors: Huaxin Zhang, Xiaohao Xu, Xiang Wang, Jialong Zuo, Xiaonan Huang, Changxin Gao, Shanjun Zhang, Li Yu, Nong Sang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06171
• Pdf URL: https://arxiv.org/pdf/2412.06171
• Copy Paste: [[2412.06171]] Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity(https://arxiv.org/abs/2412.06171)
• Keywords: interpretability, large language model, segmentation
• Abstract: How can we enable models to comprehend video anomalies occurring over varying temporal scales and contexts? Traditional Video Anomaly Understanding (VAU) methods focus on frame-level anomaly prediction, often missing the interpretability of complex and diverse real-world anomalies. Recent multimodal approaches leverage visual and textual data but lack hierarchical annotations that capture both short-term and long-term anomalies. To address this challenge, we introduce HIVAU-70k, a large-scale benchmark for hierarchical video anomaly understanding across any granularity. We develop a semi-automated annotation engine that efficiently scales high-quality annotations by combining manual video segmentation with recursive free-text annotation using large language models (LLMs). This results in over 70,000 multi-granular annotations organized at clip-level, event-level, and video-level segments. For efficient anomaly detection in long videos, we propose the Anomaly-focused Temporal Sampler (ATS). ATS integrates an anomaly scorer with a density-aware sampler to adaptively select frames based on anomaly scores, ensuring that the multimodal LLM concentrates on anomaly-rich regions, which significantly enhances both efficiency and accuracy. Extensive experiments demonstrate that our hierarchical instruction data markedly improves anomaly comprehension. The integrated ATS and visual-language model outperform traditional methods in processing long videos. Our benchmark and model are publicly available at this https URL.

Title: Robust Noisy Correspondence Learning via Self-Drop and Dual-Weight

• Authors: Fan Liu, Chenwei Dong, Chuanyi Zhang, Hualiang Zhou, Jun Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06172
• Pdf URL: https://arxiv.org/pdf/2412.06172
• Copy Paste: [[2412.06172]] Robust Noisy Correspondence Learning via Self-Drop and Dual-Weight(https://arxiv.org/abs/2412.06172)
• Keywords: robust
• Abstract: Many researchers collect data from the internet through crowd-sourcing or web crawling to alleviate the data-hungry challenge associated with cross-modal matching. Although such practice does not require expensive annotations, it inevitably introduces mismatched pairs and results in a noisy correspondence problem. Current approaches leverage the memorization effect of deep neural networks to distinguish noise and perform re-weighting. However, briefly lowering the weight of noisy pairs cannot eliminate the negative impact of noisy correspondence in the training process. In this paper, we propose a novel self-drop and dual-weight approach, which achieves elaborate data processing by qua-partitioning the data. Specifically, our approach partitions all data into four types: clean and significant, clean yet insignificant, vague, and noisy. We analyze the effect of noisy and clean data pairs and find that for vision-language pre-training models, a small number of clean samples is more valuable than a majority of noisy ones. Based on this observation, we employ self-drop to discard noisy samples to effectively mitigate the impact of noise. In addition, we adopt a dual-weight strategy to ensure that the model focuses more on significant samples while appropriately leveraging vague samples. Compared to the prior works, our approach is more robust and demonstrates relatively more stable performance on noisy datasets, especially under a high noise ratio. Extensive experiments on three widely used datasets, including Flickr30K, MS-COCO, and Conceptual Captions, validate the effectiveness of our approach. The source code is available at this https URL.

Title: One-shot Human Motion Transfer via Occlusion-Robust Flow Prediction and Neural Texturing

• Authors: Yuzhu Ji, Chuanxia Zheng, Tat-Jen Cham
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06174
• Pdf URL: https://arxiv.org/pdf/2412.06174
• Copy Paste: [[2412.06174]] One-shot Human Motion Transfer via Occlusion-Robust Flow Prediction and Neural Texturing(https://arxiv.org/abs/2412.06174)
• Keywords: robust
• Abstract: Human motion transfer aims at animating a static source image with a driving video. While recent advances in one-shot human motion transfer have led to significant improvement in results, it remains challenging for methods with 2D body landmarks, skeleton and semantic mask to accurately capture correspondences between source and driving poses due to the large variation in motion and articulation complexity. In addition, the accuracy and precision of DensePose degrade the image quality for neural-rendering-based methods. To address the limitations and by both considering the importance of appearance and geometry for motion transfer, in this work, we proposed a unified framework that combines multi-scale feature warping and neural texture mapping to recover better 2D appearance and 2.5D geometry, partly by exploiting the information from DensePose, yet adapting to its inherent limited accuracy. Our model takes advantage of multiple modalities by jointly training and fusing them, which allows it to robust neural texture features that cope with geometric errors as well as multi-scale dense motion flow that better preserves appearance. Experimental results with full and half-view body video datasets demonstrate that our model can generalize well and achieve competitive results, and that it is particularly effective in handling challenging cases such as those with substantial self-occlusions.

Title: AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement

• Authors: Pranjal Aggarwal, Bryan Parno, Sean Welleck
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06176
• Pdf URL: https://arxiv.org/pdf/2412.06176
• Copy Paste: [[2412.06176]] AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement(https://arxiv.org/abs/2412.06176)
• Keywords: large language model
• Abstract: Automated code generation with large language models has gained significant traction, but there remains no guarantee on the correctness of generated code. We aim to use formal verification to provide mathematical guarantees that the generated code is correct. However, generating formally verified code with LLMs is hindered by the scarcity of training data and the complexity of formal proofs. To tackle this challenge, we introduce AlphaVerus, a self-improving framework that bootstraps formally verified code generation by iteratively translating programs from a higher-resource language and leveraging feedback from a verifier. AlphaVerus operates in three phases: exploration of candidate translations, Treefinement -- a novel tree search algorithm for program refinement using verifier feedback, and filtering misaligned specifications and programs to prevent reward hacking. Through this iterative process, AlphaVerus enables a LLaMA-3.1-70B model to generate verified code without human intervention or model finetuning. AlphaVerus shows an ability to generate formally verified solutions for HumanEval and MBPP, laying the groundwork for truly trustworthy code-generation agents.

Title: Enhancing Adversarial Resistance in LLMs with Recursion

• Authors: Bryan Li, Sounak Bagchi, Zizhan Wang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06181
• Pdf URL: https://arxiv.org/pdf/2412.06181
• Copy Paste: [[2412.06181]] Enhancing Adversarial Resistance in LLMs with Recursion(https://arxiv.org/abs/2412.06181)
• Keywords: security, defense, robust, large language model
• Abstract: The increasing integration of Large Language Models (LLMs) into society necessitates robust defenses against vulnerabilities from jailbreaking and adversarial prompts. This project proposes a recursive framework for enhancing the resistance of LLMs to manipulation through the use of prompt simplification techniques. By increasing the transparency of complex and confusing adversarial prompts, the proposed method enables more reliable detection and prevention of malicious inputs. Our findings attempt to address a critical problem in AI safety and security, providing a foundation for the development of systems able to distinguish harmless inputs from prompts containing malicious intent. As LLMs continue to be used in diverse applications, the importance of such safeguards will only grow.

Title: Adaptive Resolution Residual Networks -- Generalizing Across Resolutions Easily and Efficiently

• Authors: Léa Demeule, Mahtab Sandhu, Glen Berseth
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06195
• Pdf URL: https://arxiv.org/pdf/2412.06195
• Copy Paste: [[2412.06195]] Adaptive Resolution Residual Networks -- Generalizing Across Resolutions Easily and Efficiently(https://arxiv.org/abs/2412.06195)
• Keywords: robust
• Abstract: The majority of signal data captured in the real world uses numerous sensors with different resolutions. In practice, however, most deep learning architectures are fixed-resolution; they consider a single resolution at training time and inference time. This is convenient to implement but fails to fully take advantage of the diverse signal data that exists. In contrast, other deep learning architectures are adaptive-resolution; they directly allow various resolutions to be processed at training time and inference time. This benefits robustness and computational efficiency but introduces difficult design constraints that hinder mainstream use. In this work, we address the shortcomings of both fixed-resolution and adaptive-resolution methods by introducing Adaptive Resolution Residual Networks (ARRNs), which inherit the advantages of adaptive-resolution methods and the ease of use of fixed-resolution methods. We construct ARRNs from Laplacian residuals, which serve as generic adaptive-resolution adapters for fixed-resolution layers, and which allow casting high-resolution ARRNs into low-resolution ARRNs at inference time by simply omitting high-resolution Laplacian residuals, thus reducing computational cost on low-resolution signals without compromising performance. We complement this novel component with Laplacian dropout, which regularizes for robustness to a distribution of lower resolutions, and which also regularizes for errors that may be induced by approximate smoothing kernels in Laplacian residuals. We provide a solid grounding for the advantageous properties of ARRNs through a theoretical analysis based on neural operators, and empirically show that ARRNs embrace the challenge posed by diverse resolutions with greater flexibility, robustness, and computational efficiency.

Title: SparseAccelerate: Efficient Long-Context Inference for Mid-Range GPUs

• Authors: James Vo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06198
• Pdf URL: https://arxiv.org/pdf/2412.06198
• Copy Paste: [[2412.06198]] SparseAccelerate: Efficient Long-Context Inference for Mid-Range GPUs(https://arxiv.org/abs/2412.06198)
• Keywords: large language model
• Abstract: As Large Language Models (LLMs) scale to longer context windows, the computational cost of attention mechanisms, which traditionally grows quadratically with input length, presents a critical challenge for real-time and memory-constrained deployments. Existing sparse attention techniques have sought to reduce this complexity, but they often incur significant overhead or compromise accuracy, making them less practical for large contexts on mid-range hardware. In this paper, we introduce SparseAccelerate, a dynamic sparse attention method that adapts its sparsity patterns based on input characteristics, effectively flattening the attention complexity curve. Our approach is effective for input lengths starting at 16K tokens and scales efficiently up to 128K tokens on dual NVIDIA A5000 GPUs (24GB each). Experimental results show that SparseAccelerate achieves up to a 1.04x reduction in Time-To-First-Token (TTFT) latency at 32K tokens, while also providing substantial memory savings. These improvements yield practical gains for memory-intensive applications and long-context tasks that were previously infeasible with standard attention. Beyond latency reductions, SparseAccelerate fundamentally shifts the scaling trend, demonstrating the smallest TTFT growth gradient relative to context length among competing methods. Ongoing evaluations on diverse benchmarks confirm its scalability, positioning SparseAccelerate as a critical advancement toward efficient, real-time, and large-context LLM inference on accessible hardware.

Title: Applications of Positive Unlabeled (PU) and Negative Unlabeled (NU) Learning in Cybersecurity

• Authors: Robert Dilworth, Charan Gudla
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06203
• Pdf URL: https://arxiv.org/pdf/2412.06203
• Copy Paste: [[2412.06203]] Applications of Positive Unlabeled (PU) and Negative Unlabeled (NU) Learning in Cybersecurity(https://arxiv.org/abs/2412.06203)
• Keywords: security
• Abstract: This paper explores the relatively underexplored application of Positive Unlabeled (PU) Learning and Negative Unlabeled (NU) Learning in the cybersecurity domain. While these semi-supervised learning methods have been applied successfully in fields like medicine and marketing, their potential in cybersecurity remains largely untapped. The paper identifies key areas of cybersecurity--such as intrusion detection, vulnerability management, malware detection, and threat intelligence--where PU/NU learning can offer significant improvements, particularly in scenarios with imbalanced or limited labeled data. We provide a detailed problem formulation for each subfield, supported by mathematical reasoning, and highlight the specific challenges and research gaps in scaling these methods to real-time systems, addressing class imbalance, and adapting to evolving threats. Finally, we propose future directions to advance the integration of PU/NU learning in cybersecurity, offering solutions that can better detect, manage, and mitigate emerging cyber threats.

Title: You KAN Do It in a Single Shot: Plug-and-Play Methods with Single-Instance Priors

• Authors: Yanqi Cheng, Carola-Bibiane Schönlieb, Angelica I Aviles-Rivero
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06204
• Pdf URL: https://arxiv.org/pdf/2412.06204
• Copy Paste: [[2412.06204]] You KAN Do It in a Single Shot: Plug-and-Play Methods with Single-Instance Priors(https://arxiv.org/abs/2412.06204)
• Keywords: robust
• Abstract: The use of Plug-and-Play (PnP) methods has become a central approach for solving inverse problems, with denoisers serving as regularising priors that guide optimisation towards a clean solution. In this work, we introduce KAN-PnP, an optimisation framework that incorporates Kolmogorov-Arnold Networks (KANs) as denoisers within the Plug-and-Play (PnP) paradigm. KAN-PnP is specifically designed to solve inverse problems with single-instance priors, where only a single noisy observation is available, eliminating the need for large datasets typically required by traditional denoising methods. We show that KANs, based on the Kolmogorov-Arnold representation theorem, serve effectively as priors in such settings, providing a robust approach to denoising. We prove that the KAN denoiser is Lipschitz continuous, ensuring stability and convergence in optimisation algorithms like PnP-ADMM, even in the context of single-shot learning. Additionally, we provide theoretical guarantees for KAN-PnP, demonstrating its convergence under key conditions: the convexity of the data fidelity term, Lipschitz continuity of the denoiser, and boundedness of the regularisation functional. These conditions are crucial for stable and reliable optimisation. Our experimental results show, on super-resolution and joint optimisation, that KAN-PnP outperforms exiting methods, delivering superior performance in single-shot learning with minimal data. The method exhibits strong convergence properties, achieving high accuracy with fewer iterations.

Title: Applying Machine Learning Tools for Urban Resilience Against Floods

• Authors: Mahla Ardebili Pour, Mohammad B. Ghiasi, Ali Karkehabadi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06205
• Pdf URL: https://arxiv.org/pdf/2412.06205
• Copy Paste: [[2412.06205]] Applying Machine Learning Tools for Urban Resilience Against Floods(https://arxiv.org/abs/2412.06205)
• Keywords: robust
• Abstract: Floods are among the most prevalent and destructive natural disasters, often leading to severe social and economic impacts in urban areas due to the high concentration of assets and population density. In Iran, particularly in Tehran, recurring flood events underscore the urgent need for robust urban resilience strategies. This paper explores flood resilience models to identify the most effective approach for District 6 in Tehran. Through an extensive literature review, various resilience models were analyzed, with the Climate Disaster Resilience Index (CDRI) emerging as the most suitable model for this district due to its comprehensive resilience dimensions: Physical, Social, Economic, Organizational, and Natural Health resilience. Although the CDRI model provides a structured approach to resilience measurement, it remains a static model focused on spatial characteristics and lacks temporal adaptability. An extensive literature review enhances the CDRI model by integrating data from 2013 to 2022 in three-year intervals and applying machine learning techniques to predict resilience dimensions for 2025. This integration enables a dynamic resilience model that can accommodate temporal changes, providing a more adaptable and data driven foundation for urban flood resilience planning. By employing artificial intelligence to reflect evolving urban conditions, this model offers valuable insights for policymakers and urban planners to enhance flood resilience in Tehrans critical District 6.

Title: H-FedSN: Personalized Sparse Networks for Efficient and Accurate Hierarchical Federated Learning for IoT Applications

• Authors: Jiechao Gao, Yuangang Li, Yue Zhao, Brad Campbell
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06210
• Pdf URL: https://arxiv.org/pdf/2412.06210
• Copy Paste: [[2412.06210]] H-FedSN: Personalized Sparse Networks for Efficient and Accurate Hierarchical Federated Learning for IoT Applications(https://arxiv.org/abs/2412.06210)
• Keywords: privacy, federate
• Abstract: The proliferation of Internet of Things (IoT) has increased interest in federated learning (FL) for privacy-preserving distributed data utilization. However, traditional two-tier FL architectures inadequately adapt to multi-tier IoT environments. While Hierarchical Federated Learning (HFL) improves practicality in multi-tier IoT environments by multi-layer aggregation, it still faces challenges in communication efficiency and accuracy due to high data transfer volumes, data heterogeneity, and imbalanced device distribution, struggling to meet the low-latency and high-accuracy model training requirements of practical IoT scenarios. To overcome these limitations, we propose H-FedSN, an innovative approach for practical IoT environments. H-FedSN introduces a binary mask mechanism with shared and personalized layers to reduce communication overhead by creating a sparse network while keeping original weights frozen. To address data heterogeneity and imbalanced device distribution, we integrate personalized layers for local data adaptation and apply Bayesian aggregation with cumulative Beta distribution updates at edge and cloud levels, effectively balancing contributions from diverse client groups. Evaluations on three real-world IoT datasets and MNIST under non-IID settings demonstrate that H-FedSN significantly reduces communication costs by 58 to 238 times compared to HierFAVG while achieving high accuracy, making it highly effective for practical IoT applications in hierarchical federated learning scenarios.

Title: MSCrackMamba: Leveraging Vision Mamba for Crack Detection in Fused Multispectral Imagery

• Authors: Qinfeng Zhu, Yuan Fang, Lei Fan
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.06211
• Pdf URL: https://arxiv.org/pdf/2412.06211
• Copy Paste: [[2412.06211]] MSCrackMamba: Leveraging Vision Mamba for Crack Detection in Fused Multispectral Imagery(https://arxiv.org/abs/2412.06211)
• Keywords: segmentation
• Abstract: Crack detection is a critical task in structural health monitoring, aimed at assessing the structural integrity of bridges, buildings, and roads to prevent potential failures. Vision-based crack detection has become the mainstream approach due to its ease of implementation and effectiveness. Fusing infrared (IR) channels with red, green and blue (RGB) channels can enhance feature representation and thus improve crack detection. However, IR and RGB channels often differ in resolution. To align them, higher-resolution RGB images typically need to be downsampled to match the IR image resolution, which leads to the loss of fine details. Moreover, crack detection performance is restricted by the limited receptive fields and high computational complexity of traditional image segmentation networks. Inspired by the recently proposed Mamba neural architecture, this study introduces a two-stage paradigm called MSCrackMamba, which leverages Vision Mamba along with a super-resolution network to address these challenges. Specifically, to align IR and RGB channels, we first apply super-resolution to IR channels to match the resolution of RGB channels for data fusion. Vision Mamba is then adopted as the backbone network, while UperNet is employed as the decoder for crack detection. Our approach is validated on the large-scale Crack Detection dataset Crack900, demonstrating an improvement of 3.55% in mIoU compared to the best-performing baseline methods.

Title: A Self-guided Multimodal Approach to Enhancing Graph Representation Learning for Alzheimer's Diseases

• Authors: Zhepeng Wang, Runxue Bao, Yawen Wu, Guodong Liu, Lei Yang, Liang Zhan, Feng Zheng, Weiwen Jiang, Yanfu Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06212
• Pdf URL: https://arxiv.org/pdf/2412.06212
• Copy Paste: [[2412.06212]] A Self-guided Multimodal Approach to Enhancing Graph Representation Learning for Alzheimer's Diseases(https://arxiv.org/abs/2412.06212)
• Keywords: interpretability
• Abstract: Graph neural networks (GNNs) are powerful machine learning models designed to handle irregularly structured data. However, their generic design often proves inadequate for analyzing brain connectomes in Alzheimer's Disease (AD), highlighting the need to incorporate domain knowledge for optimal performance. Infusing AD-related knowledge into GNNs is a complicated task. Existing methods typically rely on collaboration between computer scientists and domain experts, which can be both time-intensive and resource-demanding. To address these limitations, this paper presents a novel self-guided, knowledge-infused multimodal GNN that autonomously incorporates domain knowledge into the model development process. Our approach conceptualizes domain knowledge as natural language and introduces a specialized multimodal GNN capable of leveraging this uncurated knowledge to guide the learning process of the GNN, such that it can improve the model performance and strengthen the interpretability of the predictions. To evaluate our framework, we curated a comprehensive dataset of recent peer-reviewed papers on AD and integrated it with multiple real-world AD datasets. Experimental results demonstrate the ability of our method to extract relevant domain knowledge, provide graph-based explanations for AD diagnosis, and improve the overall performance of the GNN. This approach provides a more scalable and efficient alternative to inject domain knowledge for AD compared with the manual design from the domain expert, advancing both prediction accuracy and interpretability in AD diagnosis.

Title: A Real-Time Defense Against Object Vanishing Adversarial Patch Attacks for Object Detection in Autonomous Vehicles

• Authors: Jaden Mu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06215
• Pdf URL: https://arxiv.org/pdf/2412.06215
• Copy Paste: [[2412.06215]] A Real-Time Defense Against Object Vanishing Adversarial Patch Attacks for Object Detection in Autonomous Vehicles(https://arxiv.org/abs/2412.06215)
• Keywords: defense, attack
• Abstract: Autonomous vehicles (AVs) increasingly use DNN-based object detection models in vision-based perception. Correct detection and classification of obstacles is critical to ensure safe, trustworthy driving decisions. Adversarial patches aim to fool a DNN with intentionally generated patterns concentrated in a localized region of an image. In particular, object vanishing patch attacks can cause object detection models to fail to detect most or all objects in a scene, posing a significant practical threat to AVs. This work proposes ADAV (Adversarial Defense for Autonomous Vehicles), a novel defense methodology against object vanishing patch attacks specifically designed for autonomous vehicles. Unlike existing defense methods which have high latency or are designed for static images, ADAV runs in real-time and leverages contextual information from prior frames in an AV's video feed. ADAV checks if the object detector's output for the target frame is temporally consistent with the output from a previous reference frame to detect the presence of a patch. If the presence of a patch is detected, ADAV uses gradient-based attribution to localize adversarial pixels that break temporal consistency. This two stage procedure allows ADAV to efficiently process clean inputs, and both stages are optimized to be low latency. ADAV is evaluated using real-world driving data from the Berkeley Deep Drive BDD100K dataset, and demonstrates high adversarial and clean performance.

Title: Data Free Backdoor Attacks

• Authors: Bochuan Cao, Jinyuan Jia, Chuxuan Hu, Wenbo Guo, Zhen Xiang, Jinghui Chen, Bo Li, Dawn Song
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06219
• Pdf URL: https://arxiv.org/pdf/2412.06219
• Copy Paste: [[2412.06219]] Data Free Backdoor Attacks(https://arxiv.org/abs/2412.06219)
• Keywords: defense, attack, steal, data-free
• Abstract: Backdoor attacks aim to inject a backdoor into a classifier such that it predicts any input with an attacker-chosen backdoor trigger as an attacker-chosen target class. Existing backdoor attacks require either retraining the classifier with some clean data or modifying the model's architecture. As a result, they are 1) not applicable when clean data is unavailable, 2) less efficient when the model is large, and 3) less stealthy due to architecture changes. In this work, we propose DFBA, a novel retraining-free and data-free backdoor attack without changing the model architecture. Technically, our proposed method modifies a few parameters of a classifier to inject a backdoor. Through theoretical analysis, we verify that our injected backdoor is provably undetectable and unremovable by various state-of-the-art defenses under mild assumptions. Our evaluation on multiple datasets further demonstrates that our injected backdoor: 1) incurs negligible classification loss, 2) achieves 100% attack success rates, and 3) bypasses six existing state-of-the-art defenses. Moreover, our comparison with a state-of-the-art non-data-free backdoor attack shows our attack is more stealthy and effective against various defenses while achieving less classification accuracy loss.

Title: Generative Densification: Learning to Densify Gaussians for High-Fidelity Generalizable 3D Reconstruction

• Authors: Seungtae Nam, Xiangyu Sun, Gyeongjin Kang, Younggeun Lee, Seungjun Oh, Eunbyung Park
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.06234
• Pdf URL: https://arxiv.org/pdf/2412.06234
• Copy Paste: [[2412.06234]] Generative Densification: Learning to Densify Gaussians for High-Fidelity Generalizable 3D Reconstruction(https://arxiv.org/abs/2412.06234)
• Keywords: generative
• Abstract: Generalized feed-forward Gaussian models have achieved significant progress in sparse-view 3D reconstruction by leveraging prior knowledge from large multi-view datasets. However, these models often struggle to represent high-frequency details due to the limited number of Gaussians. While the densification strategy used in per-scene 3D Gaussian splatting (3D-GS) optimization can be adapted to the feed-forward models, it may not be ideally suited for generalized scenarios. In this paper, we propose Generative Densification, an efficient and generalizable method to densify Gaussians generated by feed-forward models. Unlike the 3D-GS densification strategy, which iteratively splits and clones raw Gaussian parameters, our method up-samples feature representations from the feed-forward models and generates their corresponding fine Gaussians in a single forward pass, leveraging the embedded prior knowledge for enhanced generalization. Experimental results on both object-level and scene-level reconstruction tasks demonstrate that our method outperforms state-of-the-art approaches with comparable or smaller model sizes, achieving notable improvements in representing fine details.

Title: VariFace: Fair and Diverse Synthetic Dataset Generation for Face Recognition

• Authors: Michael Yeung, Toya Teramoto, Songtao Wu, Tatsuo Fujiwara, Kenji Suzuki, Tamaki Kojima
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06235
• Pdf URL: https://arxiv.org/pdf/2412.06235
• Copy Paste: [[2412.06235]] VariFace: Fair and Diverse Synthetic Dataset Generation for Face Recognition(https://arxiv.org/abs/2412.06235)
• Keywords: privacy, fair, diffusion
• Abstract: The use of large-scale, web-scraped datasets to train face recognition models has raised significant privacy and bias concerns. Synthetic methods mitigate these concerns and provide scalable and controllable face generation to enable fair and accurate face recognition. However, existing synthetic datasets display limited intraclass and interclass diversity and do not match the face recognition performance obtained using real datasets. Here, we propose VariFace, a two-stage diffusion-based pipeline to create fair and diverse synthetic face datasets to train face recognition models. Specifically, we introduce three methods: Face Recognition Consistency to refine demographic labels, Face Vendi Score Guidance to improve interclass diversity, and Divergence Score Conditioning to balance the identity preservation-intraclass diversity trade-off. When constrained to the same dataset size, VariFace considerably outperforms previous synthetic datasets (0.9200 $\rightarrow$ 0.9405) and achieves comparable performance to face recognition models trained with real data (Real Gap = -0.0065). In an unconstrained setting, VariFace not only consistently achieves better performance compared to previous synthetic methods across dataset sizes but also, for the first time, outperforms the real dataset (CASIA-WebFace) across six evaluation datasets. This sets a new state-of-the-art performance with an average face verification accuracy of 0.9567 (Real Gap = +0.0097) across LFW, CFP-FP, CPLFW, AgeDB, and CALFW datasets and 0.9366 (Real Gap = +0.0380) on the RFW dataset.

Title: Unseen Attack Detection in Software-Defined Networking Using a BERT-Based Large Language Model

• Authors: Mohammed N. Swileh (1), Shengli Zhang (1) ((1) College of Electronics and Information Engineering, Shenzhen University, Shenzhen, China)
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06239
• Pdf URL: https://arxiv.org/pdf/2412.06239
• Copy Paste: [[2412.06239]] Unseen Attack Detection in Software-Defined Networking Using a BERT-Based Large Language Model(https://arxiv.org/abs/2412.06239)
• Keywords: security, attack, robust, large language model
• Abstract: Software defined networking (SDN) represents a transformative shift in network architecture by decoupling the control plane from the data plane, enabling centralized and flexible management of network resources. However, this architectural shift introduces significant security challenges, as SDN's centralized control becomes an attractive target for various types of attacks. While current research has yielded valuable insights into attack detection in SDN, critical gaps remain. Addressing challenges in feature selection, broadening the scope beyond DDoS attacks, strengthening attack decisions based on multi flow analysis, and building models capable of detecting unseen attacks that they have not been explicitly trained on are essential steps toward advancing security in SDN. In this paper, we introduce a novel approach that leverages Natural Language Processing (NLP) and the pre trained BERT base model to enhance attack detection in SDN. Our approach transforms network flow data into a format interpretable by language models, allowing BERT to capture intricate patterns and relationships within network traffic. By using Random Forest for feature selection, we optimize model performance and reduce computational overhead, ensuring accurate detection. Attack decisions are made based on several flows, providing stronger and more reliable detection of malicious traffic. Furthermore, our approach is specifically designed to detect previously unseen attacks, offering a solution for identifying threats that the model was not explicitly trained on. To rigorously evaluate our approach, we conducted experiments in two scenarios: one focused on detecting known attacks, achieving 99.96% accuracy, and another on detecting unseen attacks, where our model achieved 99.96% accuracy, demonstrating the robustness of our approach in detecting evolving threats to improve the security of SDN networks.

Title: U-Know-DiffPAN: An Uncertainty-aware Knowledge Distillation Diffusion Framework with Details Enhancement for PAN-Sharpening

• Authors: Sungpyo Kim, Jeonghyeok Do, Jaehyup Lee, Munchurl Kim
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.06243
• Pdf URL: https://arxiv.org/pdf/2412.06243
• Copy Paste: [[2412.06243]] U-Know-DiffPAN: An Uncertainty-aware Knowledge Distillation Diffusion Framework with Details Enhancement for PAN-Sharpening(https://arxiv.org/abs/2412.06243)
• Keywords: robust, diffusion
• Abstract: Conventional methods for PAN-sharpening often struggle to restore fine details due to limitations in leveraging high-frequency information. Moreover, diffusion-based approaches lack sufficient conditioning to fully utilize Panchromatic (PAN) images and low-resolution multispectral (LRMS) inputs effectively. To address these challenges, we propose an uncertainty-aware knowledge distillation diffusion framework with details enhancement for PAN-sharpening, called U-Know-DiffPAN. The U-Know-DiffPAN incorporates uncertainty-aware knowledge distillation for effective transfer of feature details from our teacher model to a student one. The teacher model in our U-Know-DiffPAN captures frequency details through freqeuncy selective attention, facilitating accurate reverse process learning. By conditioning the encoder on compact vector representations of PAN and LRMS and the decoder on Wavelet transforms, we enable rich frequency utilization. So, the high-capacity teacher model distills frequency-rich features into a lightweight student model aided by an uncertainty map. From this, the teacher model can guide the student model to focus on difficult image regions for PAN-sharpening via the usage of the uncertainty map. Extensive experiments on diverse datasets demonstrate the robustness and superior performance of our U-Know-DiffPAN over very recent state-of-the-art PAN-sharpening methods.

Title: DenseVLM: A Retrieval and Decoupled Alignment Framework for Open-Vocabulary Dense Prediction

• Authors: Yunheng Li, Yuxuan Li, Quansheng Zeng, Wenhai Wang, Qibin Hou, Ming-Ming Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06244
• Pdf URL: https://arxiv.org/pdf/2412.06244
• Copy Paste: [[2412.06244]] DenseVLM: A Retrieval and Decoupled Alignment Framework for Open-Vocabulary Dense Prediction(https://arxiv.org/abs/2412.06244)
• Keywords: segmentation
• Abstract: Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated impressive zero-shot recognition capability, but still underperform in dense prediction tasks. Self-distillation recently is emerging as a promising approach for fine-tuning VLMs to better adapt to local regions without requiring extensive annotations. However, previous state-of-the-art approaches often suffer from significant `foreground bias', where models tend to wrongly identify background regions as foreground objects. To alleviate this issue, we propose DenseVLM, a framework designed to learn unbiased region-language alignment from powerful pre-trained VLM representations. By leveraging the pre-trained VLM to retrieve categories for unlabeled regions, DenseVLM effectively decouples the interference between foreground and background region features, ensuring that each region is accurately aligned with its corresponding category. We show that DenseVLM can be seamlessly integrated into open-vocabulary object detection and image segmentation tasks, leading to notable performance improvements. Furthermore, it exhibits promising zero-shot scalability when training on more extensive and diverse datasets.

Title: A Comparative Study of Learning Paradigms in Large Language Models via Intrinsic Dimension

• Authors: Saahith Janapati, Yangfeng Ji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06245
• Pdf URL: https://arxiv.org/pdf/2412.06245
• Copy Paste: [[2412.06245]] A Comparative Study of Learning Paradigms in Large Language Models via Intrinsic Dimension(https://arxiv.org/abs/2412.06245)
• Keywords: large language model
• Abstract: The performance of Large Language Models (LLMs) on natural language tasks can be improved through both supervised fine-tuning (SFT) and in-context learning (ICL), which operate via distinct mechanisms. Supervised fine-tuning updates the model's weights by minimizing loss on training data, whereas in-context learning leverages task demonstrations embedded in the prompt, without changing the model's parameters. This study investigates the effects of these learning paradigms on the hidden representations of LLMs using Intrinsic Dimension (ID). We use ID to estimate the number of degrees of freedom between representations extracted from LLMs as they perform specific natural language tasks. We first explore how the ID of LLM representations evolves during SFT and how it varies due to the number of demonstrations in ICL. We then compare the IDs induced by SFT and ICL and find that ICL consistently induces a higher ID compared to SFT, suggesting that representations generated during ICL reside in higher dimensional manifolds in the embedding space.

Title: Rendering-Refined Stable Diffusion for Privacy Compliant Synthetic Data

• Authors: Kartik Patwari, David Schneider, Xiaoxiao Sun, Chen-Nee Chuah, Lingjuan Lyu, Vivek Sharma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06248
• Pdf URL: https://arxiv.org/pdf/2412.06248
• Copy Paste: [[2412.06248]] Rendering-Refined Stable Diffusion for Privacy Compliant Synthetic Data(https://arxiv.org/abs/2412.06248)
• Keywords: privacy, protect, diffusion
• Abstract: Growing privacy concerns and regulations like GDPR and CCPA necessitate pseudonymization techniques that protect identity in image datasets. However, retaining utility is also essential. Traditional methods like masking and blurring degrade quality and obscure critical context, especially in human-centric images. We introduce Rendering-Refined Stable Diffusion (RefSD), a pipeline that combines 3D-rendering with Stable Diffusion, enabling prompt-based control over human attributes while preserving posture. Unlike standard diffusion models that fail to retain posture or GANs that lack realism and flexible attribute control, RefSD balances posture preservation, realism, and customization. We also propose HumanGenAI, a framework for human perception and utility evaluation. Human perception assessments reveal attribute-specific strengths and weaknesses of RefSD. Our utility experiments show that models trained on RefSD pseudonymized data outperform those trained on real data in detection tasks, with further performance gains when combining RefSD with real data. For classification tasks, we consistently observe performance improvements when using RefSD data with real data, confirming the utility of our pseudonymized data.

Title: Optimizing Multi-Task Learning for Enhanced Performance in Large Language Models

• Authors: Zhen Qi, Jiajing Chen, Shuo Wang, Bingying Liu, Hongye Zheng, Chihang Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06249
• Pdf URL: https://arxiv.org/pdf/2412.06249
• Copy Paste: [[2412.06249]] Optimizing Multi-Task Learning for Enhanced Performance in Large Language Models(https://arxiv.org/abs/2412.06249)
• Keywords: large language model
• Abstract: This study aims to explore the performance improvement method of large language models based on GPT-4 under the multi-task learning framework and conducts experiments on two tasks: text classification and automatic summary generation. Through the combined design of shared feature extractors and task-specific modules, we achieve knowledge-sharing and optimization of multiple tasks in the same model. The experiment uses multiple subtasks of the GLUE dataset to compare the performance of the multi-task model with the single-task GPT-4, the multi-task version of GPT-3, the BERT basic model, and the classic Bi-LSTM with Attention model. The results show that the proposed multi-task learning model outperforms other comparison models in terms of text classification accuracy and ROUGE value of summary generation, demonstrating the advantages of multi-task learning in improving model generalization ability and collaborative learning between tasks. The model maintains a stable loss convergence rate during training, showing good learning efficiency and adaptability to the test set. This study verifies the applicability of the multi-task learning framework in large language models, especially in improving the model's ability to balance different tasks. In the future, with the combination of large language models and multimodal data and the application of dynamic task adjustment technology, the framework based on multi-task learning is expected to play a greater role in practical applications across fields and provide new ideas for the development of general artificial intelligence.

Title: Splatter-360: Generalizable 360$^{\circ}$ Gaussian Splatting for Wide-baseline Panoramic Images

• Authors: Zheng Chen, Chenming Wu, Zhelun Shen, Chen Zhao, Weicai Ye, Haocheng Feng, Errui Ding, Song-Hai Zhang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.06250
• Pdf URL: https://arxiv.org/pdf/2412.06250
• Copy Paste: [[2412.06250]] Splatter-360: Generalizable 360$^{\circ}$ Gaussian Splatting for Wide-baseline Panoramic Images(https://arxiv.org/abs/2412.06250)
• Keywords: robust
• Abstract: Wide-baseline panoramic images are frequently used in applications like VR and simulations to minimize capturing labor costs and storage needs. However, synthesizing novel views from these panoramic images in real time remains a significant challenge, especially due to panoramic imagery's high resolution and inherent distortions. Although existing 3D Gaussian splatting (3DGS) methods can produce photo-realistic views under narrow baselines, they often overfit the training views when dealing with wide-baseline panoramic images due to the difficulty in learning precise geometry from sparse 360$^{\circ}$ views. This paper presents \textit{Splatter-360}, a novel end-to-end generalizable 3DGS framework designed to handle wide-baseline panoramic images. Unlike previous approaches, \textit{Splatter-360} performs multi-view matching directly in the spherical domain by constructing a spherical cost volume through a spherical sweep algorithm, enhancing the network's depth perception and geometry estimation. Additionally, we introduce a 3D-aware bi-projection encoder to mitigate the distortions inherent in panoramic images and integrate cross-view attention to improve feature interactions across multiple viewpoints. This enables robust 3D-aware feature representations and real-time rendering capabilities. Experimental results on the HM3D~\cite{hm3d} and Replica~\cite{replica} demonstrate that \textit{Splatter-360} significantly outperforms state-of-the-art NeRF and 3DGS methods (e.g., PanoGRF, MVSplat, DepthSplat, and HiSplat) in both synthesis quality and generalization performance for wide-baseline panoramic images. Code and trained models are available at \url{this https URL}.

Title: Towards a Comprehensive Framework for Cyber-Incident Response Decision Support in Smart Grids

• Authors: Omer Sen, Yanico Aust, Martin Neumuller, Immanuel Hacker, Andreas Ulbig
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06254
• Pdf URL: https://arxiv.org/pdf/2412.06254
• Copy Paste: [[2412.06254]] Towards a Comprehensive Framework for Cyber-Incident Response Decision Support in Smart Grids(https://arxiv.org/abs/2412.06254)
• Keywords: security, defense, attack
• Abstract: The modernization of power grid infrastructures necessitates the incorporation of decision support systems to effectively mitigate cybersecurity threats. This paper presents a comprehensive framework based on integrating Attack-Defense Trees and the Multi-Criteria Decision Making method to enhance smart grid cybersecurity. By analyzing risk attributes and optimizing defense strategies, this framework enables grid operators to prioritize critical security measures. Additionally, this paper incorporates findings on decision-making processes in intelligent power systems to present a comprehensive approach to grid cybersecurity. The proposed model aims to optimize the effectiveness and efficiency of grid cybersecurity efforts while offering insights into future grid management challenges.

Title: Simulation of Multi-Stage Attack and Defense Mechanisms in Smart Grids

• Authors: Omer Sen, Bozhidar Ivanov, Christian Kloos, Christoph Zol_, Philipp Lutat, Martin Henze, Andreas Ulbig
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06255
• Pdf URL: https://arxiv.org/pdf/2412.06255
• Copy Paste: [[2412.06255]] Simulation of Multi-Stage Attack and Defense Mechanisms in Smart Grids(https://arxiv.org/abs/2412.06255)
• Keywords: security, protect, defense, attack
• Abstract: The power grid is a critical infrastructure essential for public safety and welfare. As its reliance on digital technologies grows, so do its vulnerabilities to sophisticated cyber threats, which could severely disrupt operations. Effective protective measures, such as intrusion detection and decision support systems, are essential to mitigate these risks. Machine learning offers significant potential in this field, yet its effectiveness is constrained by the limited availability of high-quality data due to confidentiality and access restrictions. To address this, we introduce a simulation environment that replicates the power grid's infrastructure and communication dynamics. This environment enables the modeling of complex, multi-stage cyber attacks and defensive responses, using attack trees to outline attacker strategies and game-theoretic approaches to model defender actions. The framework generates diverse, realistic attack data to train machine learning algorithms for detecting and mitigating cyber threats. It also provides a controlled, flexible platform to evaluate emerging security technologies, including advanced decision support systems. The environment is modular and scalable, facilitating the integration of new scenarios without dependence on external components. It supports scenario generation, data modeling, mapping, power flow simulation, and communication traffic analysis in a cohesive chain, capturing all relevant data for cyber security investigations under consistent conditions. Detailed modeling of communication protocols and grid operations offers insights into attack propagation, while datasets undergo validation in laboratory settings to ensure real-world applicability. These datasets are leveraged to train machine learning models for intrusion detection, focusing on their ability to identify complex attack patterns within power grid operations.

Title: Vulnerability Coordination Under the Cyber Resilience Act

• Authors: Jukka Ruohonen, Paul Timmers
• Subjects: cs.CR, cs.CY, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.06261
• Pdf URL: https://arxiv.org/pdf/2412.06261
• Copy Paste: [[2412.06261]] Vulnerability Coordination Under the Cyber Resilience Act(https://arxiv.org/abs/2412.06261)
• Keywords: security
• Abstract: A new Cyber Resilience Act (CRA) was recently agreed upon in the European Union (EU). It imposes many new cyber security requirements practically to all information technology products, whether hardware or software. The paper examines and elaborates the CRA's new requirements for vulnerability coordination, including vulnerability disclosure. Although these requirements are only a part of the CRA's obligations for vendors, also some new vulnerability coordination mandates are present, including particularly with respect to so-called actively exploited vulnerabilities. The CRA further alters the coordination practices on the side of public administrations. With the examination, elaboration, and associated discussion, the paper contributes to the study of cyber security regulations, providing also a few practical takeaways.

Title: A Lightweight U-like Network Utilizing Neural Memory Ordinary Differential Equations for Slimming the Decoder

• Authors: Quansong He, Xiaojun Yao, Jun Wu, Zhang Yi, Tao He
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.06262
• Pdf URL: https://arxiv.org/pdf/2412.06262
• Copy Paste: [[2412.06262]] A Lightweight U-like Network Utilizing Neural Memory Ordinary Differential Equations for Slimming the Decoder(https://arxiv.org/abs/2412.06262)
• Keywords: segmentation
• Abstract: In recent years, advanced U-like networks have demonstrated remarkable performance in medical image segmentation tasks. However, their drawbacks, including excessive parameters, high computational complexity, and slow inference speed, pose challenges for practical implementation in scenarios with limited computational resources. Existing lightweight U-like networks have alleviated some of these problems, but they often have pre-designed structures and consist of inseparable modules, limiting their application scenarios. In this paper, we propose three plug-and-play decoders by employing different discretization methods of the neural memory Ordinary Differential Equations (nmODEs). These decoders integrate features at various levels of abstraction by processing information from skip connections and performing numerical operations on upward path. Through experiments on the PH2, ISIC2017, and ISIC2018 datasets, we embed these decoders into different U-like networks, demonstrating their effectiveness in significantly reducing the number of parameters and FLOPs while maintaining performance. In summary, the proposed discretized nmODEs decoders are capable of reducing the number of parameters by about 20% ~ 50% and FLOPs by up to 74%, while possessing the potential to adapt to all U-like networks. Our code is available at this https URL.

Title: iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models

• Authors: Lianyu Hu, Fanhua Shang, Liang Wan, Wei Feng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06263
• Pdf URL: https://arxiv.org/pdf/2412.06263
• Copy Paste: [[2412.06263]] iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models(https://arxiv.org/abs/2412.06263)
• Keywords: large language model
• Abstract: In this paper, we introduce iLLaVA, a simple method that can be seamlessly deployed upon current Large Vision-Language Models (LVLMs) to greatly increase the throughput with nearly lossless model performance, without a further requirement to train. iLLaVA achieves this by finding and gradually merging the redundant tokens with an accurate and fast algorithm, which can merge hundreds of tokens within only one step. While some previous methods have explored directly pruning or merging tokens in the inference stage to accelerate models, our method excels in both performance and throughput by two key designs. First, while most previous methods only try to save the computations of Large Language Models (LLMs), our method accelerates the forward pass of both image encoders and LLMs in LVLMs, which both occupy a significant part of time during inference. Second, our method recycles the beneficial information from the pruned tokens into existing tokens, which avoids directly dropping context tokens like previous methods to cause performance loss. iLLaVA can nearly 2$\times$ the throughput, and reduce the memory costs by half with only a 0.2\% - 0.5\% performance drop across models of different scales including 7B, 13B and 34B. On tasks across different domains including single-image, multi-images and videos, iLLaVA demonstrates strong generalizability with consistently promising efficiency. We finally offer abundant visualizations to show the merging processes of iLLaVA in each step, which show insights into the distribution of computing resources in LVLMs. Code is available at this https URL.

Title: Flow Matching Guide and Code

• Authors: Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q. Chen, David Lopez-Paz, Heli Ben-Hamu, Itai Gat
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06264
• Pdf URL: https://arxiv.org/pdf/2412.06264
• Copy Paste: [[2412.06264]] Flow Matching Guide and Code(https://arxiv.org/abs/2412.06264)
• Keywords: generative
• Abstract: Flow Matching (FM) is a recent framework for generative modeling that has achieved state-of-the-art performance across various domains, including image, video, audio, speech, and biological structures. This guide offers a comprehensive and self-contained review of FM, covering its mathematical foundations, design choices, and extensions. By also providing a PyTorch package featuring relevant examples (e.g., image and text generation), this work aims to serve as a resource for both novice and experienced researchers interested in understanding, applying and further developing FM.

Title: Table2Image: Interpretable Tabular data Classification with Realistic Image Transformations

• Authors: Seungeun Lee, Seungsang Oh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06265
• Pdf URL: https://arxiv.org/pdf/2412.06265
• Copy Paste: [[2412.06265]] Table2Image: Interpretable Tabular data Classification with Realistic Image Transformations(https://arxiv.org/abs/2412.06265)
• Keywords: interpretability
• Abstract: Recent advancements in deep learning for tabular data have demonstrated promising performance, yet interpretable models remain limited, with many relying on complex and large-scale architectures. This paper introduces Table2Image, an interpretable framework that transforms tabular data into realistic image representations for classification, achieving competitive performance with relatively lightweight models. Additionally, we propose variance inflation factor (VIF) initialization, which reflects the statistical properties of the data, and a novel interpretability framework that integrates insights from both the original tabular data and its image transformations. By leveraging Shapley additive explanations (SHAP) with methods to minimize distributional discrepancies, our approach combines tabular and image-based representations. Experiments on benchmark datasets showcase competitive classification accuracy, area under the curve (AUC), and improved interpretability, offering a scalable and reliable solution. Our code is available at this https URL.

Title: Open-Vocabulary High-Resolution 3D (OVHR3D) Data Segmentation and Annotation Framework

• Authors: Jiuyi Xu, Meida Chen, Andrew Feng, Yangming Shi, Zifan Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06268
• Pdf URL: https://arxiv.org/pdf/2412.06268
• Copy Paste: [[2412.06268]] Open-Vocabulary High-Resolution 3D (OVHR3D) Data Segmentation and Annotation Framework(https://arxiv.org/abs/2412.06268)
• Keywords: segmentation
• Abstract: In the domain of the U.S. Army modeling and simulation, the availability of high quality annotated 3D data is pivotal to creating virtual environments for training and simulations. Traditional methodologies for 3D semantic and instance segmentation, such as KpConv, RandLA, Mask3D, etc., are designed to train on extensive labeled datasets to obtain satisfactory performance in practical tasks. This requirement presents a significant challenge, given the inherent scarcity of manually annotated 3D datasets, particularly for the military use cases. Recognizing this gap, our previous research leverages the One World Terrain data repository manually annotated databases, as showcased at IITSEC 2019 and 2021, to enrich the training dataset for deep learning models. However, collecting and annotating large scale 3D data for specific tasks remains costly and inefficient. To this end, the objective of this research is to design and develop a comprehensive and efficient framework for 3D segmentation tasks to assist in 3D data annotation. This framework integrates Grounding DINO and Segment anything Model, augmented by an enhancement in 2D image rendering via 3D mesh. Furthermore, the authors have also developed a user friendly interface that facilitates the 3D annotation process, offering intuitive visualization of rendered images and the 3D point cloud.

Title: Methods for Legal Citation Prediction in the Age of LLMs: An Australian Law Case Study

• Authors: Ehsan Shareghi, Jiuzhou Han, Paul Burgess
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.06272
• Pdf URL: https://arxiv.org/pdf/2412.06272
• Copy Paste: [[2412.06272]] Methods for Legal Citation Prediction in the Age of LLMs: An Australian Law Case Study(https://arxiv.org/abs/2412.06272)
• Keywords: large language model
• Abstract: In recent years, Large Language Models (LLMs) have shown great potential across a wide range of legal tasks. Despite these advances, mitigating hallucination remains a significant challenge, with state-of-the-art LLMs still frequently generating incorrect legal references. In this paper, we focus on the problem of legal citation prediction within the Australian law context, where correctly identifying and citing relevant legislations or precedents is critical. We compare several approaches: prompting general purpose and law-specialised LLMs, retrieval-only pipelines with both generic and domain-specific embeddings, task-specific instruction-tuning of LLMs, and hybrid strategies that combine LLMs with retrieval augmentation, query expansion, or voting ensembles. Our findings indicate that domain-specific pre-training alone is insufficient for achieving satisfactory citation accuracy even after law-specialised pre-training. In contrast, instruction tuning on our task-specific dataset dramatically boosts performance reaching the best results across all settings. We also highlight that database granularity along with the type of embeddings play a critical role in the performance of retrieval systems. Among retrieval-based approaches, hybrid methods consistently outperform retrieval-only setups, and among these, ensemble voting delivers the best result by combining the predictive quality of instruction-tuned LLMs with the retrieval system.

Title: Omni-Scene: Omni-Gaussian Representation for Ego-Centric Sparse-View Scene Reconstruction

• Authors: Dongxu Wei, Zhiqi Li, Peidong Liu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.06273
• Pdf URL: https://arxiv.org/pdf/2412.06273
• Copy Paste: [[2412.06273]] Omni-Scene: Omni-Gaussian Representation for Ego-Centric Sparse-View Scene Reconstruction(https://arxiv.org/abs/2412.06273)
• Keywords: diffusion
• Abstract: Prior works employing pixel-based Gaussian representation have demonstrated efficacy in feed-forward sparse-view reconstruction. However, such representation necessitates cross-view overlap for accurate depth estimation, and is challenged by object occlusions and frustum truncations. As a result, these methods require scene-centric data acquisition to maintain cross-view overlap and complete scene visibility to circumvent occlusions and truncations, which limits their applicability to scene-centric reconstruction. In contrast, in autonomous driving scenarios, a more practical paradigm is ego-centric reconstruction, which is characterized by minimal cross-view overlap and frequent occlusions and truncations. The limitations of pixel-based representation thus hinder the utility of prior works in this task. In light of this, this paper conducts an in-depth analysis of different representations, and introduces Omni-Gaussian representation with tailored network design to complement their strengths and mitigate their drawbacks. Experiments show that our method significantly surpasses state-of-the-art methods, pixelSplat and MVSplat, in ego-centric reconstruction, and achieves comparable performance to prior works in scene-centric reconstruction. Furthermore, we extend our method with diffusion models, pioneering feed-forward multi-modal generation of 3D driving scenes.

Title: Neural Garment Dynamic Super-Resolution

• Authors: Meng Zhang, Jun Li
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.06285
• Pdf URL: https://arxiv.org/pdf/2412.06285
• Copy Paste: [[2412.06285]] Neural Garment Dynamic Super-Resolution(https://arxiv.org/abs/2412.06285)
• Keywords: robust
• Abstract: Achieving efficient, high-fidelity, high-resolution garment simulation is challenging due to its computational demands. Conversely, low-resolution garment simulation is more accessible and ideal for low-budget devices like smartphones. In this paper, we introduce a lightweight, learning-based method for garment dynamic super-resolution, designed to efficiently enhance high-resolution, high-frequency details in low-resolution garment simulations. Starting with low-resolution garment simulation and underlying body motion, we utilize a mesh-graph-net to compute super-resolution features based on coarse garment dynamics and garment-body interactions. These features are then used by a hyper-net to construct an implicit function of detailed wrinkle residuals for each coarse mesh triangle. Considering the influence of coarse garment shapes on detailed wrinkle performance, we correct the coarse garment shape and predict detailed wrinkle residuals using these implicit functions. Finally, we generate detailed high-resolution garment geometry by applying the detailed wrinkle residuals to the corrected coarse garment. Our method enables roll-out prediction by iteratively using its predictions as input for subsequent frames, producing fine-grained wrinkle details to enhance the low-resolution simulation. Despite training on a small dataset, our network robustly generalizes to different body shapes, motions, and garment types not present in the training data. We demonstrate significant improvements over state-of-the-art alternatives, particularly in enhancing the quality of high-frequency, fine-grained wrinkle details.

Title: No Annotations for Object Detection in Art through Stable Diffusion

• Authors: Patrick Ramos, Nicolas Gonthier, Selina Khan, Yuta Nakashima, Noa Garcia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06286
• Pdf URL: https://arxiv.org/pdf/2412.06286
• Copy Paste: [[2412.06286]] No Annotations for Object Detection in Art through Stable Diffusion(https://arxiv.org/abs/2412.06286)
• Keywords: diffusion
• Abstract: Object detection in art is a valuable tool for the digital humanities, as it allows for faster identification of objects in artistic and historical images compared to humans. However, annotating such images poses significant challenges due to the need for specialized domain expertise. We present NADA (no annotations for detection in art), a pipeline that leverages diffusion models' art-related knowledge for object detection in paintings without the need for full bounding box supervision. Our method, which supports both weakly-supervised and zero-shot scenarios and does not require any fine-tuning of its pretrained components, consists of a class proposer based on large vision-language models and a class-conditioned detector based on Stable Diffusion. NADA is evaluated on two artwork datasets, ArtDL 2.0 and IconArt, outperforming prior work in weakly-supervised detection, while being the first work for zero-shot object detection in art. Code is available at this https URL

Title: PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Models

• Authors: Qian Zhang, Panfeng Chen, Jiali Li, Linkun Feng, Shuyu Liu, Mei Chen, Hui Li, Yanhao Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06287
• Pdf URL: https://arxiv.org/pdf/2412.06287
• Copy Paste: [[2412.06287]] PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Models(https://arxiv.org/abs/2412.06287)
• Keywords: large language model
• Abstract: The emergence of Large Language Models (LLMs) in the medical domain has stressed a compelling need for standard datasets to evaluate their question-answering (QA) performance. Although there have been several benchmark datasets for medical QA, they either cover common knowledge across different departments or are specific to another department rather than pediatrics. Moreover, some of them are limited to objective questions and do not measure the generation capacity of LLMs. Therefore, they cannot comprehensively assess the QA ability of LLMs in pediatrics. To fill this gap, we construct PediaBench, the first Chinese pediatric dataset for LLM evaluation. Specifically, it contains 4,565 objective questions and 1,632 subjective questions spanning 12 pediatric disease groups. It adopts an integrated scoring criterion based on different difficulty levels to thoroughly assess the proficiency of an LLM in instruction following, knowledge understanding, clinical case analysis, etc. Finally, we validate the effectiveness of PediaBench with extensive experiments on 20 open-source and commercial LLMs. Through an in-depth analysis of experimental results, we offer insights into the ability of LLMs to answer pediatric questions in the Chinese context, highlighting their limitations for further improvements. Our code and data are published at this https URL.

Title: S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity

• Authors: Xinyu Yang, Jixuan Leng, Geyang Guo, Jiawei Zhao, Ryumei Nakada, Linjun Zhang, Huaxiu Yao, Beidi Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06289
• Pdf URL: https://arxiv.org/pdf/2412.06289
• Copy Paste: [[2412.06289]] S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity(https://arxiv.org/abs/2412.06289)
• Keywords: transformer
• Abstract: Current PEFT methods for LLMs can achieve either high quality, efficient training, or scalable serving, but not all three simultaneously. To address this limitation, we investigate sparse fine-tuning and observe a remarkable improvement in generalization ability. Utilizing this key insight, we propose a family of Structured Sparse Fine-Tuning (S$^{2}$FT) methods for LLMs, which concurrently achieve state-of-the-art fine-tuning performance, training efficiency, and inference scalability. S$^{2}$FT accomplishes this by "selecting sparsely and computing densely". It selects a few heads and channels in the MHA and FFN modules for each Transformer block, respectively. Next, it co-permutes weight matrices on both sides of the coupled structures in LLMs to connect the selected components in each layer into a dense submatrix. Finally, S$^{2}$FT performs in-place gradient updates on all submatrices. Through theoretical analysis and empirical results, our method prevents overfitting and forgetting, delivers SOTA performance on both commonsense and arithmetic reasoning with 4.6% and 1.3% average improvements compared to LoRA, and surpasses full FT by 11.5% when generalizing to various domains after instruction tuning. Using our partial backpropagation algorithm, S$^{2}$FT saves training memory up to 3$\times$ and improves latency by 1.5-2.7$\times$ compared to full FT, while delivering an average 10% improvement over LoRA on both metrics. We further demonstrate that the weight updates in S$^{2}$FT can be decoupled into adapters, enabling effective fusion, fast switch, and efficient parallelism for serving multiple fine-tuned models.

Title: The Hybrid ROA: A Flexible and Scalable Encoding Scheme for Route Origin Authorization

• Authors: Yanbiao Li, Hui Zou, Yuxuan Chen, Yinbo Xu, Zhuoran Ma, Di Ma, Ying Hu, Gaogang Xie
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06290
• Pdf URL: https://arxiv.org/pdf/2412.06290
• Copy Paste: [[2412.06290]] The Hybrid ROA: A Flexible and Scalable Encoding Scheme for Route Origin Authorization(https://arxiv.org/abs/2412.06290)
• Keywords: security, protect
• Abstract: On top of the Resource Public Key Infrastructure (RPKI), the Route Origin Authorization (ROA) creates a cryptographically verifiable binding of an autonomous system to a set of IP prefixes it is authorized to originate. By their design, ROAs can protect the inter-domain routing system against prefix and sub-prefix hijacks. However, it is hard for the state-of-the-art approach, the maxLength-based ROA encoding scheme, to guarantee security and scalability at the same time when facing various authorization scenarios. To this end, we propose a novel bitmap-based encoding scheme for ROAs to provide flexible and controllable compression. Furthermore, the hybrid ROA encoding scheme (h-ROA) is proposed, which encodes ROAs based on maxLength and bitmap jointly. This approach ensures strong security, provides flexibility and significantly improves system scalability, enabling it to effectively handle various authorization patterns. According to the performance evaluation with real-world data sets, h-ROA outperforms the state-of-the-art approach $1.99 \sim 3.28$ times in terms of the encoding speed, and it can reduce the cost of a router to synchronize all validated ROA payloads by $43.9\% \sim 56.6\%$.

Title: ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models

• Authors: Bingchen Gong, Diego Gomez, Abdullah Hamdi, Abdelrahman Eldesokey, Ahmed Abdelreheem, Peter Wonka, Maks Ovsjanikov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06292
• Pdf URL: https://arxiv.org/pdf/2412.06292
• Copy Paste: [[2412.06292]] ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models(https://arxiv.org/abs/2412.06292)
• Keywords: large language model
• Abstract: We propose a novel zero-shot approach for keypoint detection on 3D shapes. Point-level reasoning on visual data is challenging as it requires precise localization capability, posing problems even for powerful models like DINO or CLIP. Traditional methods for 3D keypoint detection rely heavily on annotated 3D datasets and extensive supervised training, limiting their scalability and applicability to new categories or domains. In contrast, our method utilizes the rich knowledge embedded within Multi-Modal Large Language Models (MLLMs). Specifically, we demonstrate, for the first time, that pixel-level annotations used to train recent MLLMs can be exploited for both extracting and naming salient keypoints on 3D models without any ground truth labels or supervision. Experimental evaluations demonstrate that our approach achieves competitive performance on standard benchmarks compared to supervised methods, despite not requiring any 3D keypoint annotations during training. Our results highlight the potential of integrating language models for localized 3D shape understanding. This work opens new avenues for cross-modal learning and underscores the effectiveness of MLLMs in contributing to 3D computer vision challenges.

Title: Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness

• Authors: Qifan Yu, Zhebei Shen, Zhongqi Yue, Yang Wu, Wenqiao Zhang, Yunfei Li, Juncheng Li, Siliang Tang, Yueting Zhuang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06293
• Pdf URL: https://arxiv.org/pdf/2412.06293
• Copy Paste: [[2412.06293]] Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness(https://arxiv.org/abs/2412.06293)
• Keywords: large language model
• Abstract: Instruction tuning fine-tunes pre-trained Multi-modal Large Language Models (MLLMs) to handle real-world tasks. However, the rapid expansion of visual instruction datasets introduces data redundancy, leading to excessive computational costs. We propose a collaborative framework, DataTailor, which leverages three key principles--informativeness, uniqueness, and representativeness--for effective data selection. We argue that a valuable sample should be informative of the task, non-redundant, and represent the sample distribution (i.e., not an outlier). We further propose practical ways to score against each principle, which automatically adapts to a given dataset without tedious hyperparameter tuning. Comprehensive experiments on various benchmarks demonstrate that DataTailor achieves 100.8% of the performance of full-data fine-tuning with only 15% of the data, significantly reducing computational costs while maintaining superior results. This exemplifies the "Less is More" philosophy in MLLM development.

Title: See Further When Clear: Curriculum Consistency Model

• Authors: Yunpeng Liu, Boxiao Liu, Yi Zhang, Xingzhong Hou, Guanglu Song, Yu Liu, Haihang You
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06295
• Pdf URL: https://arxiv.org/pdf/2412.06295
• Copy Paste: [[2412.06295]] See Further When Clear: Curriculum Consistency Model(https://arxiv.org/abs/2412.06295)
• Keywords: diffusion
• Abstract: Significant advances have been made in the sampling efficiency of diffusion models and flow matching models, driven by Consistency Distillation (CD), which trains a student model to mimic the output of a teacher model at a later timestep. However, we found that the learning complexity of the student model varies significantly across different timesteps, leading to suboptimal performance in this http URL address this issue, we propose the Curriculum Consistency Model (CCM), which stabilizes and balances the learning complexity across timesteps. Specifically, we regard the distillation process at each timestep as a curriculum and introduce a metric based on Peak Signal-to-Noise Ratio (PSNR) to quantify the learning complexity of this curriculum, then ensure that the curriculum maintains consistent learning complexity across different timesteps by having the teacher model iterate more steps when the noise intensity is low. Our method achieves competitive single-step sampling Fréchet Inception Distance (FID) scores of 1.64 on CIFAR-10 and 2.18 on ImageNet this http URL, we have extended our method to large-scale text-to-image models and confirmed that it generalizes well to both diffusion models (Stable Diffusion XL) and flow matching models (Stable Diffusion 3). The generated samples demonstrate improved image-text alignment and semantic structure, since CCM enlarges the distillation step at large timesteps and reduces the accumulated error.

Title: DSAI: Unbiased and Interpretable Latent Feature Extraction for Data-Centric AI

• Authors: Hyowon Cho, Soonwon Ka, Daechul Park, Jaewook Kang, Minjoon Seo, Bokyung Son
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06303
• Pdf URL: https://arxiv.org/pdf/2412.06303
• Copy Paste: [[2412.06303]] DSAI: Unbiased and Interpretable Latent Feature Extraction for Data-Centric AI(https://arxiv.org/abs/2412.06303)
• Keywords: extraction, large language model
• Abstract: Large language models (LLMs) often struggle to objectively identify latent characteristics in large datasets due to their reliance on pre-trained knowledge rather than actual data patterns. To address this data grounding issue, we propose Data Scientist AI (DSAI), a framework that enables unbiased and interpretable feature extraction through a multi-stage pipeline with quantifiable prominence metrics for evaluating extracted features. On synthetic datasets with known ground-truth features, DSAI demonstrates high recall in identifying expert-defined features while faithfully reflecting the underlying data. Applications on real-world datasets illustrate the framework's practical utility in uncovering meaningful patterns with minimal expert oversight, supporting use cases such as interpretable classification. The title of our paper is chosen from multiple candidates based on DSAI-generated criteria.

Title: LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations

• Authors: Mingjie Xu, Mengyang Wu, Yuzhi Zhao, Jason Chun Lok Li, Weifeng Ou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06322
• Pdf URL: https://arxiv.org/pdf/2412.06322
• Copy Paste: [[2412.06322]] LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations(https://arxiv.org/abs/2412.06322)
• Keywords: large language model
• Abstract: Scene Graph Generation (SGG) converts visual scenes into structured graph representations, providing deeper scene understanding for complex vision tasks. However, existing SGG models often overlook essential spatial relationships and struggle with generalization in open-vocabulary contexts. To address these limitations, we propose LLaVA-SpaceSGG, a multimodal large language model (MLLM) designed for open-vocabulary SGG with enhanced spatial relation modeling. To train it, we collect the SGG instruction-tuning dataset, named SpaceSGG. This dataset is constructed by combining publicly available datasets and synthesizing data using open-source models within our data construction pipeline. It combines object locations, object relations, and depth information, resulting in three data formats: spatial SGG description, question-answering, and conversation. To enhance the transfer of MLLMs' inherent capabilities to the SGG task, we introduce a two-stage training paradigm. Experiments show that LLaVA-SpaceSGG outperforms other open-vocabulary SGG methods, boosting recall by 8.6% and mean recall by 28.4% compared to the baseline. Our codebase, dataset, and trained models are publicly accessible on GitHub at the following URL: this https URL.

Title: HAIFAI: Human-AI Collaboration for Mental Face Reconstruction

• Authors: Florian Strohm, Mihai Bâce, Andreas Bulling
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06323
• Pdf URL: https://arxiv.org/pdf/2412.06323
• Copy Paste: [[2412.06323]] HAIFAI: Human-AI Collaboration for Mental Face Reconstruction(https://arxiv.org/abs/2412.06323)
• Keywords: generative
• Abstract: We present HAIFAI - a novel collaborative human-AI system to tackle the challenging task of reconstructing a visual representation of a face that exists only in a person's mind. Users iteratively rank images presented by the AI system based on their resemblance to a mental image. These rankings, in turn, allow the system to extract relevant image features, fuse them into a unified feature vector, and use a generative model to reconstruct the mental image. We also propose an extension called HAIFAI-X that allows users to manually refine and further improve the reconstruction using an easy-to-use slider interface. To avoid the need for tedious human data collection for model training, we introduce a computational user model of human ranking behaviour. For this, we collected a small face ranking dataset through an online crowd-sourcing study containing data from 275 participants. We evaluate HAIFAI and HAIFAI-X in a 12-participant user study and show that HAIFAI outperforms the previous state of the art regarding reconstruction quality, usability, perceived workload, and reconstruction speed. HAIFAI-X achieves even better reconstruction quality at the cost of reduced usability, perceived workload, and increased reconstruction time. We further validate the reconstructions in a subsequent face ranking study with 18 participants and show that HAIFAI-X achieves a new state-of-the-art identification rate of 60.6%. These findings represent a significant advancement towards developing new collaborative intelligent systems capable of reliably and effortlessly reconstructing a user's mental image.

Title: World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving

• Authors: Mingliang Zhai, Cheng Li, Zengyuan Guo, Ningrui Yang, Xiameng Qin, Yuwei Wu, Sanyuan Zhao, Junyu Han, Ji Tao, Yunde Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06324
• Pdf URL: https://arxiv.org/pdf/2412.06324
• Copy Paste: [[2412.06324]] World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving(https://arxiv.org/abs/2412.06324)
• Keywords: large language model
• Abstract: The Multi-modal Large Language Models (MLLMs) with extensive world knowledge have revitalized autonomous driving, particularly in reasoning tasks within perceivable regions. However, when faced with perception-limited areas (dynamic or static occlusion regions), MLLMs struggle to effectively integrate perception ability with world knowledge for reasoning. These perception-limited regions can conceal crucial safety information, especially for vulnerable road users. In this paper, we propose a framework, which aims to improve autonomous driving performance under perceptionlimited conditions by enhancing the integration of perception capabilities and world knowledge. Specifically, we propose a plug-and-play instruction-guided interaction module that bridges modality gaps and significantly reduces the input sequence length, allowing it to adapt effectively to multi-view video inputs. Furthermore, to better integrate world knowledge with driving-related tasks, we have collected and refined a large-scale multi-modal dataset that includes 2 million natural language QA pairs, 1.7 million grounding task data. To evaluate the model's utilization of world knowledge, we introduce an object-level risk assessment dataset comprising 200K QA pairs, where the questions necessitate multi-step reasoning leveraging world knowledge for resolution. Extensive experiments validate the effectiveness of our proposed method.

Title: Q-PnV: A Quantum Consensus Mechanism for Security Consortium Blockchains

• Authors: Jianming Lin, Hui Li, Hongjian Xing, Runhuai Huang, Weixiang Huang, Shaowen Deng, Yanping Zhang, Weimin Zeng, Ping Lu, Xiyu Wang, Tao Sun, Xiongyan Tang
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/2412.06325
• Pdf URL: https://arxiv.org/pdf/2412.06325
• Copy Paste: [[2412.06325]] Q-PnV: A Quantum Consensus Mechanism for Security Consortium Blockchains(https://arxiv.org/abs/2412.06325)
• Keywords: secure, security, attack, fair
• Abstract: Due to the rapid development of quantum computing, many classical blockchain technologies are now considered insecure. The emergence of quantum blockchain holds promise for addressing this issue. Various quantum consensus algorithms have been proposed so far, but there has not yet been a quantum consensus algorithm tailored specifically for consortium blockchain scenarios. In this paper, we propose a novel quantum consensus mechanism, named Q-PnV. This consensus mechanism is based on the classical Proof of Vote (PoV), integrating quantum voting, quantum digital signature and quantum random number generators (QRNGs). By combining Q-PnV with a quantum blockchain using weighted hypergraph states, we propose a comprehensive quantum blockchain solution for consortium blockchain scenarios. Compared to the classical method, the quantum blockchain based on Q-PnV can resist quantum attacks and shows significant improvements in security and fairness, making it better suit-ed for the future quantum era.

Title: Normalizing Flows are Capable Generative Models

• Authors: Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Angel Bautista, Navdeep Jaitly, Josh Susskind
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06329
• Pdf URL: https://arxiv.org/pdf/2412.06329
• Copy Paste: [[2412.06329]] Normalizing Flows are Capable Generative Models(https://arxiv.org/abs/2412.06329)
• Keywords: diffusion, transformer, generative
• Abstract: Normalizing Flows (NFs) are likelihood-based models for continuous inputs. They have demonstrated promising results on both density estimation and generative modeling tasks, but have received relatively little attention in recent years. In this work, we demonstrate that NFs are more powerful than previously believed. We present \textit{TarFlow}: a simple and scalable architecture that enables highly performant NF models. TarFlow can be thought of as a Transformer-based variant of Masked Autoregressive Flows (MAFs): it consists of a stack of autoregressive Transformer blocks on image patches, alternating the autoregression direction between layers. TarFlow is straightforward to train end-to-end, and capable of directly modeling and generating pixels. We also propose three key techniques to improve sample quality: Gaussian noise augmentation during training, a post training denoising procedure, and an effective guidance method for both class-conditional and unconditional settings. Putting these together, TarFlow sets new state-of-the-art results on likelihood estimation for images, beating the previous best methods by a large margin, and generates samples with quality and diversity comparable to diffusion models, for the first time with a stand-alone NF model. We make our code available at \href{this https URL}{this https URL}.

Title: TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions

• Authors: Ilya A. Petrov, Riccardo Marin, Julian Chibane, Gerard Pons-Moll
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06334
• Pdf URL: https://arxiv.org/pdf/2412.06334
• Copy Paste: [[2412.06334]] TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions(https://arxiv.org/abs/2412.06334)
• Keywords: diffusion, transformer
• Abstract: Modeling 3D human-object interaction (HOI) is a problem of great interest for computer vision and a key enabler for virtual and mixed-reality applications. Existing methods work in a one-way direction: some recover plausible human interactions conditioned on a 3D object; others recover the object pose conditioned on a human pose. Instead, we provide the first unified model - TriDi which works in any direction. Concretely, we generate Human, Object, and Interaction modalities simultaneously with a new three-way diffusion process, allowing to model seven distributions with one network. We implement TriDi as a transformer attending to the various modalities' tokens, thereby discovering conditional relations between them. The user can control the interaction either as a text description of HOI or a contact map. We embed these two representations into a shared latent space, combining the practicality of text descriptions with the expressiveness of contact maps. Using a single network, TriDi unifies all the special cases of prior work and extends to new ones, modeling a family of seven distributions. Remarkably, despite using a single model, TriDi generated samples surpass one-way specialized baselines on GRAB and BEHAVE in terms of both qualitative and quantitative metrics, and demonstrating better diversity. We show the applicability of TriDi to scene population, generating objects for human-contact datasets, and generalization to unseen object geometry. The project page is available at: this https URL.

Title: UniPaint: Unified Space-time Video Inpainting via Mixture-of-Experts

• Authors: Zhen Wan, Yue Ma, Chenyang Qi, Zhiheng Liu, Tao Gui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06340
• Pdf URL: https://arxiv.org/pdf/2412.06340
• Copy Paste: [[2412.06340]] UniPaint: Unified Space-time Video Inpainting via Mixture-of-Experts(https://arxiv.org/abs/2412.06340)
• Keywords: generative
• Abstract: In this paper, we present UniPaint, a unified generative space-time video inpainting framework that enables spatial-temporal inpainting and interpolation. Different from existing methods that treat video inpainting and video interpolation as two distinct tasks, we leverage a unified inpainting framework to tackle them and observe that these two tasks can mutually enhance synthesis performance. Specifically, we first introduce a plug-and-play space-time video inpainting adapter, which can be employed in various personalized models. The key insight is to propose a Mixture of Experts (MoE) attention to cover various tasks. Then, we design a spatial-temporal masking strategy during the training stage to mutually enhance each other and improve performance. UniPaint produces high-quality and aesthetically pleasing results, achieving the best quantitative results across various tasks and scale setups. The code and checkpoints will be available soon.

Title: SeFENet: Robust Deep Homography Estimation via Semantic-Driven Feature Enhancement

• Authors: Zeru Shi, Zengxi Zhang, Zhiying Jiang, Ruizhe An, Jinyuan Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06352
• Pdf URL: https://arxiv.org/pdf/2412.06352
• Copy Paste: [[2412.06352]] SeFENet: Robust Deep Homography Estimation via Semantic-Driven Feature Enhancement(https://arxiv.org/abs/2412.06352)
• Keywords: robust, extraction
• Abstract: Images captured in harsh environments often exhibit blurred details, reduced contrast, and color distortion, which hinder feature detection and matching, thereby affecting the accuracy and robustness of homography estimation. While visual enhancement can improve contrast and clarity, it may introduce visual-tolerant artifacts that obscure the structural integrity of images. Considering the resilience of semantic information against environmental interference, we propose a semantic-driven feature enhancement network for robust homography estimation, dubbed SeFENet. Concretely, we first introduce an innovative hierarchical scale-aware module to expand the receptive field by aggregating multi-scale information, thereby effectively extracting image features under diverse harsh conditions. Subsequently, we propose a semantic-guided constraint module combined with a high-level perceptual framework to achieve degradation-tolerant with semantic feature. A meta-learning-based training strategy is introduced to mitigate the disparity between semantic and structural features. By internal-external alternating optimization, the proposed network achieves implicit semantic-wise feature enhancement, thereby improving the robustness of homography estimation in adverse environments by strengthening the local feature comprehension and context information extraction. Experimental results under both normal and harsh conditions demonstrate that SeFENet significantly outperforms SOTA methods, reducing point match error by at least 41\% on the large-scale datasets.

Title: Is Self-Supervision Enough? Benchmarking Foundation Models Against End-to-End Training for Mitotic Figure Classification

• Authors: Jonathan Ganz, Jonas Ammeling, Emely Rosbach, Ludwig Lausser, Christof A. Bertram, Katharina Breininger, Marc Aubreville
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06365
• Pdf URL: https://arxiv.org/pdf/2412.06365
• Copy Paste: [[2412.06365]] Is Self-Supervision Enough? Benchmarking Foundation Models Against End-to-End Training for Mitotic Figure Classification(https://arxiv.org/abs/2412.06365)
• Keywords: robust
• Abstract: Foundation models (FMs), i.e., models trained on a vast amount of typically unlabeled data, have become popular and available recently for the domain of histopathology. The key idea is to extract semantically rich vectors from any input patch, allowing for the use of simple subsequent classification networks potentially reducing the required amounts of labeled data, and increasing domain robustness. In this work, we investigate to which degree this also holds for mitotic figure classification. Utilizing two popular public mitotic figure datasets, we compared linear probing of five publicly available FMs against models trained on ImageNet and a simple ResNet50 end-to-end-trained baseline. We found that the end-to-end-trained baseline outperformed all FM-based classifiers, regardless of the amount of data provided. Additionally, we did not observe the FM-based classifiers to be more robust against domain shifts, rendering both of the above assumptions incorrect.

Title: Exploring Memorization and Copyright Violation in Frontier LLMs: A Study of the New York Times v. OpenAI 2023 Lawsuit

• Authors: Joshua Freeman, Chloe Rippe, Edoardo Debenedetti, Maksym Andriushchenko
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06370
• Pdf URL: https://arxiv.org/pdf/2412.06370
• Copy Paste: [[2412.06370]] Exploring Memorization and Copyright Violation in Frontier LLMs: A Study of the New York Times v. OpenAI 2023 Lawsuit(https://arxiv.org/abs/2412.06370)
• Keywords: defense, generative
• Abstract: Copyright infringement in frontier LLMs has received much attention recently due to the New York Times v. OpenAI lawsuit, filed in December 2023. The New York Times claims that GPT-4 has infringed its copyrights by reproducing articles for use in LLM training and by memorizing the inputs, thereby publicly displaying them in LLM outputs. Our work aims to measure the propensity of OpenAI's LLMs to exhibit verbatim memorization in its outputs relative to other LLMs, specifically focusing on news articles. We discover that both GPT and Claude models use refusal training and output filters to prevent verbatim output of the memorized articles. We apply a basic prompt template to bypass the refusal training and show that OpenAI models are currently less prone to memorization elicitation than models from Meta, Mistral, and Anthropic. We find that as models increase in size, especially beyond 100 billion parameters, they demonstrate significantly greater capacity for memorization. Our findings have practical implications for training: more attention must be placed on preventing verbatim memorization in very large models. Our findings also have legal significance: in assessing the relative memorization capacity of OpenAI's LLMs, we probe the strength of The New York Times's copyright infringement claims and OpenAI's legal defenses, while underscoring issues at the intersection of generative AI, law, and policy.

Title: Low-Rank Matrix Factorizations with Volume-based Constraints and Regularizations

• Authors: Olivier Vu Thanh
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.06380
• Pdf URL: https://arxiv.org/pdf/2412.06380
• Copy Paste: [[2412.06380]] Low-Rank Matrix Factorizations with Volume-based Constraints and Regularizations(https://arxiv.org/abs/2412.06380)
• Keywords: interpretability
• Abstract: Low-rank matrix factorizations are a class of linear models widely used in various fields such as machine learning, signal processing, and data analysis. These models approximate a matrix as the product of two smaller matrices, where the left matrix captures latent features while the right matrix linearly decomposes the data based on these features. There are many ways to define what makes a component "important." Standard LRMFs, such as the truncated singular value decomposition, focus on minimizing the distance between the original matrix and its low-rank approximation. In this thesis, the notion of "importance" is closely linked to interpretability and uniqueness, which are key to obtaining reliable and meaningful results. This thesis thus focuses on volume-based constraints and regularizations designed to enhance interpretability and uniqueness. We first introduce two new volume-constrained LRMFs designed to enhance these properties. The first assumes that data points are naturally bounded (e.g., movie ratings between 1 and 5 stars) and can be explained by convex combinations of features within the same bounds, allowing them to be interpreted in the same way as the data. The second model is more general, constraining the factors to belong to convex polytopes. Then, two variants of volume-regularized LRMFs are proposed. The first minimizes the volume of the latent features, encouraging them to cluster closely together, while the second maximizes the volume of the decompositions, promoting sparse representations. Across all these models, uniqueness is achieved under the core principle that the factors must be "sufficiently scattered" within their respective feasible sets. Motivated by applications such as blind source separation and missing data imputation, this thesis also proposes efficient algorithms that make these models practical for real-world applications.

Title: Gentle robustness implies Generalization

• Authors: Khoat Than, Dat Phan, Giang Vu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.06381
• Pdf URL: https://arxiv.org/pdf/2412.06381
• Copy Paste: [[2412.06381]] Gentle robustness implies Generalization(https://arxiv.org/abs/2412.06381)
• Keywords: robust
• Abstract: Robustness and generalization ability of machine learning models are of utmost importance in various application domains. There is a wide interest in efficient ways to analyze those properties. One important direction is to analyze connection between those two properties. Prior theories suggest that a robust learning algorithm can produce trained models with a high generalization ability. However, we show in this work that the existing error bounds are vacuous for the Bayes optimal classifier which is the best among all measurable classifiers for a classification problem with overlapping classes. Those bounds cannot converge to the true error of this ideal classifier. This is undesirable, surprizing, and never known before. We then present a class of novel bounds, which are model-dependent and provably tighter than the existing robustness-based ones. Unlike prior ones, our bounds are guaranteed to converge to the true error of the best classifier, as the number of samples increases. We further provide an extensive experiment and find that two of our bounds are often non-vacuous for a large class of deep neural networks, pretrained from ImageNet.

Title: PyPulse: A Python Library for Biosignal Imputation

• Authors: Kevin Gao, Maxwell A. Xu, James M. Rehg, Alexander Moreno
• Subjects: cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.06382
• Pdf URL: https://arxiv.org/pdf/2412.06382
• Copy Paste: [[2412.06382]] PyPulse: A Python Library for Biosignal Imputation(https://arxiv.org/abs/2412.06382)
• Keywords: secure
• Abstract: We introduce PyPulse, a Python package for imputation of biosignals in both clinical and wearable sensor settings. Missingness is commonplace in these settings and can arise from multiple causes, such as insecure sensor attachment or data transmission loss. PyPulse's framework provides a modular and extendable framework with high ease-of-use for a broad userbase, including non-machine-learning bioresearchers. Specifically, its new capabilities include using pre-trained imputation methods out-of-the-box on custom datasets, running the full workflow of training or testing a baseline method with a single line of code, and comparing baseline methods in an interactive visualization tool. We released PyPulse under the MIT License on Github and PyPI. The source code can be found at: this https URL.

Title: Exploring the Impact of Synthetic Data on Human Gesture Recognition Tasks Using GANs

• Authors: George Kontogiannis, Pantelis Tzamalis, Sotiris Nikoletseas
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06389
• Pdf URL: https://arxiv.org/pdf/2412.06389
• Copy Paste: [[2412.06389]] Exploring the Impact of Synthetic Data on Human Gesture Recognition Tasks Using GANs(https://arxiv.org/abs/2412.06389)
• Keywords: privacy, generative
• Abstract: In the evolving domain of Human Activity Recognition (HAR) using Internet of Things (IoT) devices, there is an emerging interest in employing Deep Generative Models (DGMs) to address data scarcity, enhance data quality, and improve classification metrics scores. Among these types of models, Generative Adversarial Networks (GANs) have arisen as a powerful tool for generating synthetic data that mimic real-world scenarios with high fidelity. However, Human Gesture Recognition (HGR), a subset of HAR, particularly in healthcare applications, using time series data such as allergic gestures, remains highly unexplored. In this paper, we examine and evaluate the performance of two GANs in the generation of synthetic gesture motion data that compose a part of an open-source benchmark dataset. The data is related to the disease identification domain and healthcare, specifically to allergic rhinitis. We also focus on these AI models' performance in terms of fidelity, diversity, and privacy. Furthermore, we examine the scenario if the synthetic data can substitute real data, in training scenarios and how well models trained on synthetic data can be generalized for the allergic rhinitis gestures. In our work, these gestures are related to 6-axes accelerometer and gyroscope data, serving as multi-variate time series instances, and retrieved from smart wearable devices. To the best of our knowledge, this study is the first to explore the feasibility of synthesizing motion gestures for allergic rhinitis from wearable IoT device data using Generative Adversarial Networks (GANs) and testing their impact on the generalization of gesture recognition systems. It is worth noting that, even if our method has been applied to a specific category of gestures, it is designed to be generalized and can be deployed also to other motion data in the HGR domain.

Title: Edge Delayed Deep Deterministic Policy Gradient: efficient continuous control for edge scenarios

• Authors: Alberto Sinigaglia, Niccolò Turcato, Ruggero Carli, Gian Antonio Susto
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06390
• Pdf URL: https://arxiv.org/pdf/2412.06390
• Copy Paste: [[2412.06390]] Edge Delayed Deep Deterministic Policy Gradient: efficient continuous control for edge scenarios(https://arxiv.org/abs/2412.06390)
• Keywords: privacy
• Abstract: Deep Reinforcement Learning is gaining increasing attention thanks to its capability to learn complex policies in high-dimensional settings. Recent advancements utilize a dual-network architecture to learn optimal policies through the Q-learning algorithm. However, this approach has notable drawbacks, such as an overestimation bias that can disrupt the learning process and degrade the performance of the resulting policy. To address this, novel algorithms have been developed that mitigate overestimation bias by employing multiple Q-functions. Edge scenarios, which prioritize privacy, have recently gained prominence. In these settings, limited computational resources pose a significant challenge for complex Machine Learning approaches, making the efficiency of algorithms crucial for their performance. In this work, we introduce a novel Reinforcement Learning algorithm tailored for edge scenarios, called Edge Delayed Deep Deterministic Policy Gradient (EdgeD3). EdgeD3 enhances the Deep Deterministic Policy Gradient (DDPG) algorithm, achieving significantly improved performance with $25\%$ less Graphics Process Unit (GPU) time while maintaining the same memory usage. Additionally, EdgeD3 consistently matches or surpasses the performance of state-of-the-art methods across various benchmarks, all while using $30\%$ fewer computational resources and requiring $30\%$ less memory.

Title: Generative Lines Matching Models

• Authors: Ori Matityahu, Raanan Fattal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06403
• Pdf URL: https://arxiv.org/pdf/2412.06403
• Copy Paste: [[2412.06403]] Generative Lines Matching Models(https://arxiv.org/abs/2412.06403)
• Keywords: diffusion, generative
• Abstract: In this paper we identify the source of a singularity in the training loss of key denoising models, that causes the denoiser's predictions to collapse towards the mean of the source or target distributions. This degeneracy creates false basins of attraction, distorting the denoising trajectories and ultimately increasing the number of steps required to sample these models. We circumvent this artifact by leveraging the deterministic ODE-based samplers, offered by certain denoising diffusion and score-matching models, which establish a well-defined change-of-variables between the source and target distributions. Given this correspondence, we propose a new probability flow model, the Lines Matching Model (LMM), which matches globally straight lines interpolating the two distributions. We demonstrate that the flow fields produced by the LMM exhibit notable temporal consistency, resulting in trajectories with excellent straightness scores. Beyond its sampling efficiency, the LMM formulation allows us to enhance the fidelity of the generated samples by integrating domain-specific reconstruction and adversarial losses, and by optimizing its training for the sampling procedure used. Overall, the LMM achieves state-of-the-art FID scores with minimal NFEs on established benchmark datasets: 1.57/1.39 (NFE=1/2) on CIFAR-10, 1.47/1.17 on ImageNet 64x64, and 2.68/1.54 on AFHQ 64x64. Finally, we provide a theoretical analysis showing that the use of optimal transport to relate the two distributions suffers from a curse of dimensionality, where the pairing set size (mini-batch) must scale exponentially with the signal dimension.

Title: Federated Split Learning with Model Pruning and Gradient Quantization in Wireless Networks

• Authors: Junhe Zhang, Wanli Ni, Dongyu Wang
• Subjects: cs.LG, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2412.06414
• Pdf URL: https://arxiv.org/pdf/2412.06414
• Copy Paste: [[2412.06414]] Federated Split Learning with Model Pruning and Gradient Quantization in Wireless Networks(https://arxiv.org/abs/2412.06414)
• Keywords: federate
• Abstract: As a paradigm of distributed machine learning, federated learning typically requires all edge devices to train a complete model locally. However, with the increasing scale of artificial intelligence models, the limited resources on edge devices often become a bottleneck for efficient fine-tuning. To address this challenge, federated split learning (FedSL) implements collaborative training across the edge devices and the server through model splitting. In this paper, we propose a lightweight FedSL scheme, that further alleviates the training burden on resource-constrained edge devices by pruning the client-side model dynamicly and using quantized gradient updates to reduce computation overhead. Additionally, we apply random dropout to the activation values at the split layer to reduce communication overhead. We conduct theoretical analysis to quantify the convergence performance of the proposed scheme. Finally, simulation results verify the effectiveness and advantages of the proposed lightweight FedSL in wireless network environments.

Title: Continual Learning for Segment Anything Model Adaptation

• Authors: Jinglong Yang, Yichen Wu, Jun Cen, Wenjian Huang, Hong Wang, Jianguo Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06418
• Pdf URL: https://arxiv.org/pdf/2412.06418
• Copy Paste: [[2412.06418]] Continual Learning for Segment Anything Model Adaptation(https://arxiv.org/abs/2412.06418)
• Keywords: segmentation
• Abstract: Although the current different types of SAM adaptation methods have achieved promising performance for various downstream tasks, such as prompt-based ones and adapter-based ones, most of them belong to the one-step adaptation paradigm. In real-world scenarios, we are generally confronted with the dynamic scenario where the data comes in a streaming manner. Driven by the practical need, in this paper, we first propose a novel Continual SAM adaptation (CoSAM) benchmark with 8 different task domains and carefully analyze the limitations of the existing SAM one-step adaptation methods in the continual segmentation scenario. Then we propose a novel simple-yet-effective Mixture of Domain Adapters (MoDA) algorithm which utilizes the Global Feature Tokens (GFT) and Global Assistant Tokens (GAT) modules to help the SAM encoder extract well-separated features for different task domains, and then provide the accurate task-specific information for continual learning. Extensive experiments demonstrate that our proposed MoDA obviously surpasses the existing classic continual learning methods, as well as prompt-based and adapter-based approaches for continual segmentation. Moreover, after sequential learning on the CoSAM benchmark with diverse data distributions, our MoDA maintains highly competitive results in the natural image domain, approaching the zero-shot performance of the original SAM, demonstrating its superior capability in knowledge preservation. Notably, the proposed MoDA can be seamlessly integrated into various one-step adaptation methods of SAM, which can consistently bring obvious performance gains. Code is available at \url{this https URL}

Title: LLM-BIP: Structured Pruning for Large Language Models with Block-Wise Forward Importance Propagation

• Authors: Haihang Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06419
• Pdf URL: https://arxiv.org/pdf/2412.06419
• Copy Paste: [[2412.06419]] LLM-BIP: Structured Pruning for Large Language Models with Block-Wise Forward Importance Propagation(https://arxiv.org/abs/2412.06419)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable performance across various language tasks, but their widespread deployment is impeded by their large size and high computational costs. Structural pruning is a prevailing technique used to introduce sparsity into pre-trained models and facilitate direct hardware acceleration during inference by removing redundant connections (structurally-grouped parameters), such as channels and attention heads. Existing structural pruning approaches often employ either global or layer-wise pruning criteria; however, they are hindered by ineffectiveness stemming from inaccurate evaluation of connection importance. Global pruning methods typically assess component importance using near-zero and unreliable gradients, while layer-wise pruning approaches encounter significant pruning error accumulation issues. To this end, we propose a more accurate pruning metric based on the block-wise importance score propagation, termed LLM-BIP. Specifically, LLM-BIP precisely evaluates connection importance by gauging its influence on the respective transformer block output, which can be efficiently approximated in a single forward pass through an upper bound derived from the assumption of Lipschitz continuity. We evaluate the proposed method using LLaMA-7B, Vicuna-7B, and LLaMA-13B across common zero-shot tasks. The results demonstrate that our approach achieves an average of 3.26% increase in accuracy for common reasoning tasks compared to previous best baselines. It also reduces perplexity by 14.09 and 68.76 on average for the WikiText2 dataset and PTB dataset, respectively.

Title: Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design

• Authors: Marco Wrzalik, Adrian Ulges, Anne Uersfeld, Florian Faust
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06432
• Pdf URL: https://arxiv.org/pdf/2412.06432
• Copy Paste: [[2412.06432]] Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design(https://arxiv.org/abs/2412.06432)
• Keywords: extraction
• Abstract: We address the detection of emission reduction goals in corporate reports, an important task for monitoring companies' progress in addressing climate change. Specifically, we focus on the issue of integrating expert feedback in the form of labeled example passages into LLM-based pipelines, and compare the two strategies of (1) a dynamic selection of few-shot examples and (2) the automatic optimization of the prompt by the LLM itself. Our findings on a public dataset of 769 climate-related passages from real-world business reports indicate that automatic prompt optimization is the superior approach, while combining both methods provides only limited benefit. Qualitative results indicate that optimized prompts do indeed capture many intricacies of the targeted emission goal extraction task.

Title: Local Attention Transformers for High-Detail Optical Flow Upsampling

• Authors: Alexander Gielisse, Nergis Tömen, Jan van Gemert
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06439
• Pdf URL: https://arxiv.org/pdf/2412.06439
• Copy Paste: [[2412.06439]] Local Attention Transformers for High-Detail Optical Flow Upsampling(https://arxiv.org/abs/2412.06439)
• Keywords: transformer
• Abstract: Most recent works on optical flow use convex upsampling as the last step to obtain high-resolution flow. In this work, we show and discuss several issues and limitations of this currently widely adopted convex upsampling approach. We propose a series of changes, in an attempt to resolve current issues. First, we propose to decouple the weights for the final convex upsampler, making it easier to find the correct convex combination. For the same reason, we also provide extra contextual features to the convex upsampler. Then, we increase the convex mask size by using an attention-based alternative convex upsampler; Transformers for Convex Upsampling. This upsampler is based on the observation that convex upsampling can be reformulated as attention, and we propose to use local attention masks as a drop-in replacement for convex masks to increase the mask size. We provide empirical evidence that a larger mask size increases the likelihood of the existence of the convex combination. Lastly, we propose an alternative training scheme to remove bilinear interpolation artifacts from the model output. Our proposed ideas could theoretically be applied to almost every current state-of-the-art optical flow architecture. On the FlyingChairs + FlyingThings3D training setting we reduce the Sintel Clean training end-point-error of RAFT from 1.42 to 1.26, GMA from 1.31 to 1.18, and that of FlowFormer from 0.94 to 0.90, by solely adapting the convex upsampler.

Title: How Certain are Uncertainty Estimates? Three Novel Earth Observation Datasets for Benchmarking Uncertainty Quantification in Machine Learning

• Authors: Yuanyuan Wang, Qian Song, Dawood Wasif, Muhammad Shahzad, Christoph Koller, Jonathan Bamber, Xiao Xiang Zhu
• Subjects: cs.LG, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.06451
• Pdf URL: https://arxiv.org/pdf/2412.06451
• Copy Paste: [[2412.06451]] How Certain are Uncertainty Estimates? Three Novel Earth Observation Datasets for Benchmarking Uncertainty Quantification in Machine Learning(https://arxiv.org/abs/2412.06451)
• Keywords: segmentation
• Abstract: Uncertainty quantification (UQ) is essential for assessing the reliability of Earth observation (EO) products. However, the extensive use of machine learning models in EO introduces an additional layer of complexity, as those models themselves are inherently uncertain. While various UQ methods do exist for machine learning models, their performance on EO datasets remains largely unevaluated. A key challenge in the community is the absence of the ground truth for uncertainty, i.e. how certain the uncertainty estimates are, apart from the labels for the image/signal. This article fills this gap by introducing three benchmark datasets specifically designed for UQ in EO machine learning models. These datasets address three common problem types in EO: regression, image segmentation, and scene classification. They enable a transparent comparison of different UQ methods for EO machine learning models. We describe the creation and characteristics of each dataset, including data sources, preprocessing steps, and label generation, with a particular focus on calculating the reference uncertainty. We also showcase baseline performance of several machine learning models on each dataset, highlighting the utility of these benchmarks for model development and comparison. Overall, this article offers a valuable resource for researchers and practitioners working in artificial intelligence for EO, promoting a more accurate and reliable quality measure of the outputs of machine learning models. The dataset and code are accessible via this https URL.

Title: Ranked from Within: Ranking Large Multimodal Models for Visual Question Answering Without Labels

• Authors: Weijie Tu, Weijian Deng, Dylan Campbell, Yu Yao, Jiyang Zheng, Tom Gedeon, Tongliang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06461
• Pdf URL: https://arxiv.org/pdf/2412.06461
• Copy Paste: [[2412.06461]] Ranked from Within: Ranking Large Multimodal Models for Visual Question Answering Without Labels(https://arxiv.org/abs/2412.06461)
• Keywords: robust
• Abstract: As large multimodal models (LMMs) are increasingly deployed across diverse applications, the need for adaptable, real-world model ranking has become paramount. Traditional evaluation methods are largely dataset-centric, relying on fixed, labeled datasets and supervised metrics, which are resource-intensive and may lack generalizability to novel scenarios, highlighting the importance of unsupervised ranking. In this work, we explore unsupervised model ranking for LMMs by leveraging their uncertainty signals, such as softmax probabilities. We evaluate state-of-the-art LMMs (e.g., LLaVA) across visual question answering benchmarks, analyzing how uncertainty-based metrics can reflect model performance. Our findings show that uncertainty scores derived from softmax distributions provide a robust, consistent basis for ranking models across varied tasks. This finding enables the ranking of LMMs on real-world, unlabeled data for visual question answering, providing a practical approach for selecting models across diverse domains without requiring manual annotation.

Title: Gated Delta Networks: Improving Mamba2 with Delta Rule

• Authors: Songlin Yang, Jan Kautz, Ali Hatamizadeh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06464
• Pdf URL: https://arxiv.org/pdf/2412.06464
• Copy Paste: [[2412.06464]] Gated Delta Networks: Improving Mamba2 with Delta Rule(https://arxiv.org/abs/2412.06464)
• Keywords: transformer
• Abstract: Linear Transformers have gained attention as efficient alternatives to standard Transformers, but their performance in retrieval and long-context tasks has been limited. To address these limitations, recent work has explored two distinct mechanisms: gating for adaptive memory control and the delta update rule for precise memory modifications. We observe that these mechanisms are complementary: gating enables rapid memory erasure while the delta rule facilitates targeted updates. Building on this insight, we introduce the gated delta rule and develop a parallel training algorithm optimized for modern hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses existing models like Mamba2 and DeltaNet across multiple benchmarks, including language modeling, common-sense reasoning, in-context retrieval, length extrapolation, and long-context understanding. We further enhance performance by developing hybrid architectures that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, achieving both improved training efficiency and superior task performance.

Title: Active Learning with Context Sampling and One-vs-Rest Entropy for Semantic Segmentation

• Authors: Fei Wu, Pablo Marquez-Neila, Hedyeh Rafi-Tarii, Raphael Sznitman
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06470
• Pdf URL: https://arxiv.org/pdf/2412.06470
• Copy Paste: [[2412.06470]] Active Learning with Context Sampling and One-vs-Rest Entropy for Semantic Segmentation(https://arxiv.org/abs/2412.06470)
• Keywords: segmentation
• Abstract: Multi-class semantic segmentation remains a cornerstone challenge in computer vision. Yet, dataset creation remains excessively demanding in time and effort, especially for specialized domains. Active Learning (AL) mitigates this challenge by selecting data points for annotation strategically. However, existing patch-based AL methods often overlook boundary pixels critical information, essential for accurate segmentation. We present OREAL, a novel patch-based AL method designed for multi-class semantic segmentation. OREAL enhances boundary detection by employing maximum aggregation of pixel-wise uncertainty scores. Additionally, we introduce one-vs-rest entropy, a novel uncertainty score function that computes class-wise uncertainties while achieving implicit class balancing during dataset creation. Comprehensive experiments across diverse datasets and model architectures validate our hypothesis.

Title: From Uncertainty to Trust: Enhancing Reliability in Vision-Language Models with Uncertainty-Guided Dropout Decoding

• Authors: Yixiong Fang, Ziran Yang, Zhaorun Chen, Zhuokai Zhao, Jiawei Zhou
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06474
• Pdf URL: https://arxiv.org/pdf/2412.06474
• Copy Paste: [[2412.06474]] From Uncertainty to Trust: Enhancing Reliability in Vision-Language Models with Uncertainty-Guided Dropout Decoding(https://arxiv.org/abs/2412.06474)
• Keywords: robust
• Abstract: Large vision-language models (LVLMs) demonstrate remarkable capabilities in multimodal tasks but are prone to misinterpreting visual inputs, often resulting in hallucinations and unreliable outputs. To address these challenges, we propose Dropout Decoding, a novel inference-time approach that quantifies the uncertainty of visual tokens and selectively masks uncertain tokens to improve decoding. Our method measures the uncertainty of each visual token by projecting it onto the text space and decomposing it into aleatoric and epistemic components. Specifically, we focus on epistemic uncertainty, which captures perception-related errors more effectively. Inspired by dropout regularization, we introduce uncertainty-guided token dropout, which applies the dropout principle to input visual tokens instead of model parameters, and during inference rather than training. By aggregating predictions from an ensemble of masked decoding contexts, Dropout Decoding robustly mitigates errors arising from visual token misinterpretations. Evaluations on benchmarks including CHAIR, THRONE, and MMBench demonstrate that Dropout Decoding significantly reduces object hallucinations (OH) and enhances both reliability and quality of LVLM outputs across diverse visual contexts.

Title: SafeWorld: Geo-Diverse Safety Alignment

• Authors: Da Yin, Haoyi Qiu, Kung-Hsiang Huang, Kai-Wei Chang, Nanyun Peng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06483
• Pdf URL: https://arxiv.org/pdf/2412.06483
• Copy Paste: [[2412.06483]] SafeWorld: Geo-Diverse Safety Alignment(https://arxiv.org/abs/2412.06483)
• Keywords: large language model
• Abstract: In the rapidly evolving field of Large Language Models (LLMs), ensuring safety is a crucial and widely discussed topic. However, existing works often overlook the geo-diversity of cultural and legal standards across the world. To demonstrate the challenges posed by geo-diverse safety standards, we introduce SafeWorld, a novel benchmark specifically designed to evaluate LLMs' ability to generate responses that are not only helpful but also culturally sensitive and legally compliant across diverse global contexts. SafeWorld encompasses 2,342 test user queries, each grounded in high-quality, human-verified cultural norms and legal policies from 50 countries and 493 regions/races. On top of it, we propose a multi-dimensional automatic safety evaluation framework that assesses the contextual appropriateness, accuracy, and comprehensiveness of responses. Our evaluations reveal that current LLMs struggle to meet these criteria. To enhance LLMs' alignment with geo-diverse safety standards, we synthesize helpful preference pairs for Direct Preference Optimization (DPO) alignment training. The preference pair construction aims to encourage LLMs to behave appropriately and provide precise references to relevant cultural norms and policies when necessary. Our trained SafeWorldLM outperforms all competing models, including GPT-4o on all three evaluation dimensions by a large margin. Global human evaluators also note a nearly 20% higher winning rate in helpfulness and harmfulness evaluation. Our code and data can be found here: this https URL.

Title: Small Languages, Big Models: A Study of Continual Training on Languages of Norway

• Authors: David Samuel, Vladislav Mikhailov, Erik Velldal, Lilja Øvrelid, Lucas Georges Gabriel Charpentier, Andrey Kutuzov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06484
• Pdf URL: https://arxiv.org/pdf/2412.06484
• Copy Paste: [[2412.06484]] Small Languages, Big Models: A Study of Continual Training on Languages of Norway(https://arxiv.org/abs/2412.06484)
• Keywords: generative, large language model
• Abstract: Training large language models requires vast amounts of data, posing a challenge for less widely spoken languages like Norwegian and even more so for truly low-resource languages like Sámi. To address this issue, we present a novel three-stage continual training approach. We also experiment with combining causal and masked language modeling to get more flexible models. Based on our findings, we train, evaluate, and openly release a new large generative language model for Norwegian Bokmål, Nynorsk, and Northern Sámi with 11.4 billion parameters: NorMistral-11B.

Title: SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimation

• Authors: Catalin E. Brita, Stephan Bongers, Frans A. Oliehoek
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06486
• Pdf URL: https://arxiv.org/pdf/2412.06486
• Copy Paste: [[2412.06486]] SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimation(https://arxiv.org/abs/2412.06486)
• Keywords: robust
• Abstract: In offline reinforcement learning, deriving an effective policy from a pre-collected set of experiences is challenging due to the distribution mismatch between the target policy and the behavioral policy used to collect the data, as well as the limited sample size. Model-based reinforcement learning improves sample efficiency by generating simulated experiences using a learned dynamic model of the environment. However, these synthetic experiences often suffer from the same distribution mismatch. To address these challenges, we introduce SimuDICE, a framework that iteratively refines the initial policy derived from offline data using synthetically generated experiences from the world model. SimuDICE enhances the quality of these simulated experiences by adjusting the sampling probabilities of state-action pairs based on stationary DIstribution Correction Estimation (DICE) and the estimated confidence in the model's predictions. This approach guides policy improvement by balancing experiences similar to those frequently encountered with ones that have a distribution mismatch. Our experiments show that SimuDICE achieves performance comparable to existing algorithms while requiring fewer pre-collected experiences and planning steps, and it remains robust across varying data collection policies.

Title: PPT: Pre-Training with Pseudo-Labeled Trajectories for Motion Forecasting

• Authors: Yihong Xu, Yuan Yin, Tuan-Hung Vu, Alexandre Boulch, Éloi Zablocki, Matthieu Cord
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.06491
• Pdf URL: https://arxiv.org/pdf/2412.06491
• Copy Paste: [[2412.06491]] PPT: Pre-Training with Pseudo-Labeled Trajectories for Motion Forecasting(https://arxiv.org/abs/2412.06491)
• Keywords: robust
• Abstract: Motion forecasting (MF) for autonomous driving aims at anticipating trajectories of surrounding agents in complex urban scenarios. In this work, we investigate a mixed strategy in MF training that first pre-train motion forecasters on pseudo-labeled data, then fine-tune them on annotated data. To obtain pseudo-labeled trajectories, we propose a simple pipeline that leverages off-the-shelf single-frame 3D object detectors and non-learning trackers. The whole pre-training strategy including pseudo-labeling is coined as PPT. Our extensive experiments demonstrate that: (1) combining PPT with supervised fine-tuning on annotated data achieves superior performance on diverse testbeds, especially under annotation-efficient regimes, (2) scaling up to multiple datasets improves the previous state-of-the-art and (3) PPT helps enhance cross-dataset generalization. Our findings showcase PPT as a promising pre-training solution for robust motion forecasting in diverse autonomous driving contexts.

Title: A cautionary tale on the cost-effectiveness of collaborative AI in real-world medical applications

• Authors: Francesco Cremonesi, Lucia Innocenti, Sebastien Ourselin, Vicky Goh, Michela Antonelli, Marco Lorenzi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06494
• Pdf URL: https://arxiv.org/pdf/2412.06494
• Copy Paste: [[2412.06494]] A cautionary tale on the cost-effectiveness of collaborative AI in real-world medical applications(https://arxiv.org/abs/2412.06494)
• Keywords: federate
• Abstract: Background. Federated learning (FL) has gained wide popularity as a collaborative learning paradigm enabling collaborative AI in sensitive healthcare applications. Nevertheless, the practical implementation of FL presents technical and organizational challenges, as it generally requires complex communication infrastructures. In this context, consensus-based learning (CBL) may represent a promising collaborative learning alternative, thanks to the ability of combining local knowledge into a federated decision system, while potentially reducing deployment overhead. Methods. In this work we propose an extensive benchmark of the accuracy and cost-effectiveness of a panel of FL and CBL methods in a wide range of collaborative medical data analysis scenarios. The benchmark includes 7 different medical datasets, encompassing 3 machine learning tasks, 8 different data modalities, and multi-centric settings involving 3 to 23 clients. Findings. Our results reveal that CBL is a cost-effective alternative to FL. When compared across the panel of medical dataset in the considered benchmark, CBL methods provide equivalent accuracy to the one achieved by this http URL, CBL significantly reduces training time and communication cost (resp. 15 fold and 60 fold decrease) (p < 0.05). Interpretation. This study opens a novel perspective on the deployment of collaborative AI in real-world applications, whereas the adoption of cost-effective methods is instrumental to achieve sustainability and democratisation of AI by alleviating the need for extensive computational resources.

Title: Hybrid Attention Network: An efficient approach for anatomy-free landmark detection

• Authors: Xiaoqian Zhou, Zhen Huang, Heqin Zhu, Qingsong Yao, S.Kevin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06499
• Pdf URL: https://arxiv.org/pdf/2412.06499
• Copy Paste: [[2412.06499]] Hybrid Attention Network: An efficient approach for anatomy-free landmark detection(https://arxiv.org/abs/2412.06499)
• Keywords: robust, transformer
• Abstract: Accurate anatomical landmark detection in medical images is crucial for clinical applications. Existing methods often struggle to balance global context with computational efficiency, particularly with high-resolution images. This paper introduces the Hybrid Attention Network(HAN), a novel hybrid architecture integrating CNNs and Transformers. Its core is the BiFormer module, utilizing Bi-Level Routing Attention (BRA) for efficient attention to relevant image regions. This, combined with Convolutional Attention Blocks (CAB) enhanced by CBAM, enables precise local feature refinement guided by the global context. A Feature Fusion Correction Module (FFCM) integrates multi-scale features, mitigating resolution loss. Deep supervision with MSE loss on multi-resolution heatmaps optimizes the model. Experiments on five diverse datasets demonstrate state-of-the-art performance, surpassing existing methods in accuracy, robustness, and efficiency. The HAN provides a promising solution for accurate and efficient anatomical landmark detection in complex medical images. Our codes and data will be released soon at: \url{this https URL}.

Title: Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation

• Authors: Egor Cherepanov, Nikita Kachaev, Artem Zholus, Alexey K. Kovalev, Aleksandr I. Panov
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06531
• Pdf URL: https://arxiv.org/pdf/2412.06531
• Copy Paste: [[2412.06531]] Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation(https://arxiv.org/abs/2412.06531)
• Keywords: robust
• Abstract: The incorporation of memory into agents is essential for numerous tasks within the domain of Reinforcement Learning (RL). In particular, memory is paramount for tasks that require the utilization of past information, adaptation to novel environments, and improved sample efficiency. However, the term ``memory'' encompasses a wide range of concepts, which, coupled with the lack of a unified methodology for validating an agent's memory, leads to erroneous judgments about agents' memory capabilities and prevents objective comparison with other memory-enhanced agents. This paper aims to streamline the concept of memory in RL by providing practical precise definitions of agent memory types, such as long-term versus short-term memory and declarative versus procedural memory, inspired by cognitive science. Using these definitions, we categorize different classes of agent memory, propose a robust experimental methodology for evaluating the memory capabilities of RL agents, and standardize evaluations. Furthermore, we empirically demonstrate the importance of adhering to the proposed methodology when evaluating different types of agent memory by conducting experiments with different RL agents and what its violation leads to.

Title: Inverting Visual Representations with Detection Transformers

• Authors: Jan Rathjens, Shirin Reyhanian, David Kappel, Laurenz Wiskott
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06534
• Pdf URL: https://arxiv.org/pdf/2412.06534
• Copy Paste: [[2412.06534]] Inverting Visual Representations with Detection Transformers(https://arxiv.org/abs/2412.06534)
• Keywords: robust, transformer
• Abstract: Understanding the mechanisms underlying deep neural networks in computer vision remains a fundamental challenge. While many prior approaches have focused on visualizing intermediate representations within deep neural networks, particularly convolutional neural networks, these techniques have yet to be thoroughly explored in transformer-based vision models. In this study, we apply the approach of training inverse models to reconstruct input images from intermediate layers within a Detection Transformer, showing that this approach is efficient and feasible for transformer-based vision models. Through qualitative and quantitative evaluations of reconstructed images across model stages, we demonstrate critical properties of Detection Transformers, including contextual shape preservation, inter-layer correlation, and robustness to color perturbations, illustrating how these characteristics emerge within the model's architecture. Our findings contribute to a deeper understanding of transformer-based vision models. The code for reproducing our experiments will be made available at this http URL.

Title: Understanding Factual Recall in Transformers via Associative Memories

• Authors: Eshaan Nichani, Jason D. Lee, Alberto Bietti
• Subjects: cs.LG, cs.CL, cs.IT, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.06538
• Pdf URL: https://arxiv.org/pdf/2412.06538
• Copy Paste: [[2412.06538]] Understanding Factual Recall in Transformers via Associative Memories(https://arxiv.org/abs/2412.06538)
• Keywords: transformer, large language model
• Abstract: Large language models have demonstrated an impressive ability to perform factual recall. Prior work has found that transformers trained on factual recall tasks can store information at a rate proportional to their parameter count. In our work, we show that shallow transformers can use a combination of associative memories to obtain such near optimal storage capacity. We begin by proving that the storage capacities of both linear and MLP associative memories scale linearly with parameter count. We next introduce a synthetic factual recall task, and prove that a transformer with a single layer of self-attention followed by an MLP can obtain 100% accuracy on the task whenever either the total number of self-attention parameters or MLP parameters scales (up to log factors) linearly with the number of facts. In particular, the transformer can trade off between using the value matrices or the MLP as an associative memory to store the dataset of facts. We complement these expressivity results with an analysis of the gradient flow trajectory of a simplified linear attention model trained on our factual recall task, where we show that the model exhibits sequential learning behavior.

Title: Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families

• Authors: Felipe Maia Polo, Seamus Somerstep, Leshem Choshen, Yuekai Sun, Mikhail Yurochkin
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.06540
• Pdf URL: https://arxiv.org/pdf/2412.06540
• Copy Paste: [[2412.06540]] Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families(https://arxiv.org/abs/2412.06540)
• Keywords: large language model
• Abstract: Scaling laws for large language models (LLMs) predict model performance based on parameters like size and training data. However, differences in training configurations and data processing across model families lead to significant variations in benchmark performance, making it difficult for a single scaling law to generalize across all LLMs. On the other hand, training family-specific scaling laws requires training models of varying sizes for every family. In this work, we propose Skills Scaling Laws (SSLaws, pronounced as Sloth), a novel scaling law that leverages publicly available benchmark data and assumes LLM performance is driven by low-dimensional latent skills, such as reasoning and instruction following. These latent skills are influenced by computational resources like model size and training tokens but with varying efficiencies across model families. Sloth exploits correlations across benchmarks to provide more accurate and interpretable predictions while alleviating the need to train multiple LLMs per family. We present both theoretical results on parameter identification and empirical evaluations on 12 prominent benchmarks, from Open LLM Leaderboard v1/v2, demonstrating that Sloth predicts LLM performance efficiently and offers insights into scaling behaviors for downstream tasks such as coding and emotional intelligence applications.

Title: When Dimensionality Reduction Meets Graph (Drawing) Theory: Introducing a Common Framework, Challenges and Opportunities

• Authors: Fernando Paulovich, Alessio Arleo, Stef van den Elzen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06555
• Pdf URL: https://arxiv.org/pdf/2412.06555
• Copy Paste: [[2412.06555]] When Dimensionality Reduction Meets Graph (Drawing) Theory: Introducing a Common Framework, Challenges and Opportunities(https://arxiv.org/abs/2412.06555)
• Keywords: extraction
• Abstract: In the vast landscape of visualization research, Dimensionality Reduction (DR) and graph analysis are two popular subfields, often essential to most visual data analytics setups. DR aims to create representations to support neighborhood and similarity analysis on complex, large datasets. Graph analysis focuses on identifying the salient topological properties and key actors within networked data, with specialized research on investigating how such features could be presented to the user to ease the comprehension of the underlying structure. Although these two disciplines are typically regarded as disjoint subfields, we argue that both fields share strong similarities and synergies that can potentially benefit both. Therefore, this paper discusses and introduces a unifying framework to help bridge the gap between DR and graph (drawing) theory. Our goal is to use the strongly math-grounded graph theory to improve the overall process of creating DR visual representations. We propose how to break the DR process into well-defined stages, discussing how to match some of the DR state-of-the-art techniques to this framework and presenting ideas on how graph drawing, topology features, and some popular algorithms and strategies used in graph analysis can be employed to improve DR topology extraction, embedding generation, and result validation. We also discuss the challenges and identify opportunities for implementing and using our framework, opening directions for future visualization research.

Title: Vulnerability, Where Art Thou? An Investigation of Vulnerability Management in Android Smartphone Chipsets

• Authors: Daniel Klischies, Philipp Mackensen, Veelasha Moonsamy
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06556
• Pdf URL: https://arxiv.org/pdf/2412.06556
• Copy Paste: [[2412.06556]] Vulnerability, Where Art Thou? An Investigation of Vulnerability Management in Android Smartphone Chipsets(https://arxiv.org/abs/2412.06556)
• Keywords: security, attack
• Abstract: Vulnerabilities in Android smartphone chipsets have severe consequences, as recent real-world attacks have demonstrated that adversaries can leverage vulnerabilities to execute arbitrary code or exfiltrate confidential information. Despite the far-reaching impact of such attacks, the lifecycle of chipset vulnerabilities has yet to be investigated, with existing papers primarily investigating vulnerabilities in the Android operating system. This paper provides a comprehensive and empirical study of the current state of smartphone chipset vulnerability management within the Android ecosystem. For the first time, we create a unified knowledge base of 3,676 chipset vulnerabilities affecting 437 chipset models from all four major chipset manufacturers, combined with 6,866 smartphone models. Our analysis revealed that the same vulnerabilities are often included in multiple generations of chipsets, providing novel empirical evidence that vulnerabilities are inherited through multiple chipset generations. Furthermore, we demonstrate that the commonly accepted 90-day responsible vulnerability disclosure period is seldom adhered to. We find that a single vulnerability often affects hundreds to thousands of different smartphone models, for which update availability is, as we show, often unclear or heavily delayed. Leveraging the new insights gained from our empirical analysis, we recommend several changes that chipset manufacturers can implement to improve the security posture of their products. At the same time, our knowledge base enables academic researchers to conduct more representative evaluations of smartphone chipsets, accurately assess the impact of vulnerabilities they discover, and identify avenues for future research.

Title: DEX: Data Channel Extension for Efficient CNN Inference on Tiny AI Accelerators

• Authors: Taesik Gong, Fahim Kawsar, Chulhong Min
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06566
• Pdf URL: https://arxiv.org/pdf/2412.06566
• Copy Paste: [[2412.06566]] DEX: Data Channel Extension for Efficient CNN Inference on Tiny AI Accelerators(https://arxiv.org/abs/2412.06566)
• Keywords: privacy
• Abstract: Tiny machine learning (TinyML) aims to run ML models on small devices and is increasingly favored for its enhanced privacy, reduced latency, and low cost. Recently, the advent of tiny AI accelerators has revolutionized the TinyML field by significantly enhancing hardware processing power. These accelerators, equipped with multiple parallel processors and dedicated per-processor memory instances, offer substantial performance improvements over traditional microcontroller units (MCUs). However, their limited data memory often necessitates downsampling input images, resulting in accuracy degradation. To address this challenge, we propose Data channel EXtension (DEX), a novel approach for efficient CNN execution on tiny AI accelerators. DEX incorporates additional spatial information from original images into input images through patch-wise even sampling and channel-wise stacking, effectively extending data across input channels. By leveraging underutilized processors and data memory for channel extension, DEX facilitates parallel execution without increasing inference latency. Our evaluation with four models and four datasets on tiny AI accelerators demonstrates that this simple idea improves accuracy on average by 3.5%p while keeping the inference latency the same on the AI accelerator. The source code is available at this https URL.

Title: Data Quality Enhancement on the Basis of Diversity with Large Language Models for Text Classification: Uncovered, Difficult, and Noisy

• Authors: Min Zeng, Caiquan Liu, Shiqi Zhang, Li Xie, Chen Sang, Xiaoxin Chen, Xiaoxin Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06575
• Pdf URL: https://arxiv.org/pdf/2412.06575
• Copy Paste: [[2412.06575]] Data Quality Enhancement on the Basis of Diversity with Large Language Models for Text Classification: Uncovered, Difficult, and Noisy(https://arxiv.org/abs/2412.06575)
• Keywords: large language model
• Abstract: In recent years, the use of large language models (LLMs) for text classification has attracted widespread attention. Despite this, the classification accuracy of LLMs has not yet universally surpassed that of smaller models. LLMs can enhance their performance in text classification through fine-tuning. However, existing data quality research based on LLMs is challenging to apply directly to solve text classification problems. To further improve the performance of LLMs in classification tasks, this paper proposes a data quality enhancement (DQE) method for text classification based on LLMs. This method starts by using a greedy algorithm to select data, dividing the dataset into sampled and unsampled subsets, and then performing fine-tuning of the LLMs using the sampled data. Subsequently, this model is used to predict the outcomes for the unsampled data, categorizing incorrectly predicted data into uncovered, difficult, and noisy data. Experimental results demonstrate that our method effectively enhances the performance of LLMs in text classification tasks and significantly improves training efficiency, saving nearly half of the training time. Our method has achieved state-of-the-art performance in several open-source classification tasks.

Title: MoViE: Mobile Diffusion for Video Editing

• Authors: Adil Karjauv, Noor Fathima, Ioannis Lelekas, Fatih Porikli, Amir Ghodrati, Amirhossein Habibian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06578
• Pdf URL: https://arxiv.org/pdf/2412.06578
• Copy Paste: [[2412.06578]] MoViE: Mobile Diffusion for Video Editing(https://arxiv.org/abs/2412.06578)
• Keywords: diffusion
• Abstract: Recent progress in diffusion-based video editing has shown remarkable potential for practical applications. However, these methods remain prohibitively expensive and challenging to deploy on mobile devices. In this study, we introduce a series of optimizations that render mobile video editing feasible. Building upon the existing image editing model, we first optimize its architecture and incorporate a lightweight autoencoder. Subsequently, we extend classifier-free guidance distillation to multiple modalities, resulting in a threefold on-device speedup. Finally, we reduce the number of sampling steps to one by introducing a novel adversarial distillation scheme which preserves the controllability of the editing process. Collectively, these optimizations enable video editing at 12 frames per second on mobile devices, while maintaining high quality. Our results are available at this https URL

Title: Bridging the Divide: Reconsidering Softmax and Linear Attention

• Authors: Dongchen Han, Yifan Pu, Zhuofan Xia, Yizeng Han, Xuran Pan, Xiu Li, Jiwen Lu, Shiji Song, Gao Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06590
• Pdf URL: https://arxiv.org/pdf/2412.06590
• Copy Paste: [[2412.06590]] Bridging the Divide: Reconsidering Softmax and Linear Attention(https://arxiv.org/abs/2412.06590)
• Keywords: transformer
• Abstract: Widely adopted in modern Vision Transformer designs, Softmax attention can effectively capture long-range visual information; however, it incurs excessive computational cost when dealing with high-resolution inputs. In contrast, linear attention naturally enjoys linear complexity and has great potential to scale up to higher-resolution images. Nonetheless, the unsatisfactory performance of linear attention greatly limits its practical application in various scenarios. In this paper, we take a step forward to close the gap between the linear and Softmax attention with novel theoretical analyses, which demystify the core factors behind the performance deviations. Specifically, we present two key perspectives to understand and alleviate the limitations of linear attention: the injective property and the local modeling ability. Firstly, we prove that linear attention is not injective, which is prone to assign identical attention weights to different query vectors, thus adding to severe semantic confusion since different queries correspond to the same outputs. Secondly, we confirm that effective local modeling is essential for the success of Softmax attention, in which linear attention falls short. The aforementioned two fundamental differences significantly contribute to the disparities between these two attention paradigms, which is demonstrated by our substantial empirical validation in the paper. In addition, more experiment results indicate that linear attention, as long as endowed with these two properties, can outperform Softmax attention across various tasks while maintaining lower computation complexity. Code is available at this https URL.

Title: PrEditor3D: Fast and Precise 3D Shape Editing

• Authors: Ziya Erkoç, Can Gümeli, Chaoyang Wang, Matthias Nießner, Angela Dai, Peter Wonka, Hsin-Ying Lee, Peiye Zhuang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.06592
• Pdf URL: https://arxiv.org/pdf/2412.06592
• Copy Paste: [[2412.06592]] PrEditor3D: Fast and Precise 3D Shape Editing(https://arxiv.org/abs/2412.06592)
• Keywords: segmentation
• Abstract: We propose a training-free approach to 3D editing that enables the editing of a single shape within a few minutes. The edited 3D mesh aligns well with the prompts, and remains identical for regions that are not intended to be altered. To this end, we first project the 3D object onto 4-view images and perform synchronized multi-view image editing along with user-guided text prompts and user-provided rough masks. However, the targeted regions to be edited are ambiguous due to projection from 3D to 2D. To ensure precise editing only in intended regions, we develop a 3D segmentation pipeline that detects edited areas in 3D space, followed by a merging algorithm to seamlessly integrate edited 3D regions with the original input. Extensive experiments demonstrate the superiority of our method over previous approaches, enabling fast, high-quality editing while preserving unintended regions.

Title: Anchoring Bias in Large Language Models: An Experimental Study

• Authors: Jiaxu Lou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06593
• Pdf URL: https://arxiv.org/pdf/2412.06593
• Copy Paste: [[2412.06593]] Anchoring Bias in Large Language Models: An Experimental Study(https://arxiv.org/abs/2412.06593)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) like GPT-4 and Gemini have significantly advanced artificial intelligence by enabling machines to generate and comprehend human-like text. Despite their impressive capabilities, LLMs are not immune to limitations, including various biases. While much research has explored demographic biases, the cognitive biases in LLMs have not been equally scrutinized. This study delves into anchoring bias, a cognitive bias where initial information disproportionately influences judgment. Utilizing an experimental dataset, we examine how anchoring bias manifests in LLMs and verify the effectiveness of various mitigation strategies. Our findings highlight the sensitivity of LLM responses to biased hints. At the same time, our experiments show that, to mitigate anchoring bias, one needs to collect hints from comprehensive angles to prevent the LLMs from being anchored to individual pieces of information, while simple algorithms such as Chain-of-Thought, Thoughts of Principles, Ignoring Anchor Hints, and Reflection are not sufficient.

Title: Towards Controllable Speech Synthesis in the Era of Large Language Models: A Survey

• Authors: Tianxin Xie, Yan Rong, Pengfei Zhang, Li Liu
• Subjects: cs.CL, cs.AI, cs.LG, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.06602
• Pdf URL: https://arxiv.org/pdf/2412.06602
• Copy Paste: [[2412.06602]] Towards Controllable Speech Synthesis in the Era of Large Language Models: A Survey(https://arxiv.org/abs/2412.06602)
• Keywords: diffusion, large language model
• Abstract: Text-to-speech (TTS), also known as speech synthesis, is a prominent research area that aims to generate natural-sounding human speech from text. Recently, with the increasing industrial demand, TTS technologies have evolved beyond synthesizing human-like speech to enabling controllable speech generation. This includes fine-grained control over various attributes of synthesized speech such as emotion, prosody, timbre, and duration. Besides, advancements in deep learning, such as diffusion and large language models, have significantly enhanced controllable TTS over the past several years. In this paper, we conduct a comprehensive survey of controllable TTS, covering approaches ranging from basic control techniques to methods utilizing natural language prompts, aiming to provide a clear understanding of the current state of research. We examine the general controllable TTS pipeline, challenges, model architectures, and control strategies, offering a comprehensive and clear taxonomy of existing methods. Additionally, we provide a detailed summary of datasets and evaluation metrics and shed some light on the applications and future directions of controllable TTS. To the best of our knowledge, this survey paper provides the first comprehensive review of emerging controllable TTS methods, which can serve as a beneficial resource for both academic researchers and industry practitioners.

Title: Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions

• Authors: Jhih-Yi (Janet)Hsieh, Aditi Raghunathan, Nihar B. Shah
• Subjects: cs.LG, cs.CR, cs.DL
• Abstract URL: https://arxiv.org/abs/2412.06606
• Pdf URL: https://arxiv.org/pdf/2412.06606
• Copy Paste: [[2412.06606]] Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions(https://arxiv.org/abs/2412.06606)
• Keywords: secure, robust
• Abstract: In the peer review process of top-tier machine learning (ML) and artificial intelligence (AI) conferences, reviewers are assigned to papers through automated methods. These assignment algorithms consider two main factors: (1) reviewers' expressed interests indicated by their bids for papers, and (2) reviewers' domain expertise inferred from the similarity between the text of their previously published papers and the submitted manuscripts. A significant challenge these conferences face is the existence of collusion rings, where groups of researchers manipulate the assignment process to review each other's papers, providing positive evaluations regardless of their actual quality. Most efforts to combat collusion rings have focused on preventing bid manipulation, under the assumption that the text similarity component is secure. In this paper, we demonstrate that even in the absence of bidding, colluding reviewers and authors can exploit the machine learning based text-matching component of reviewer assignment used at top ML/AI venues to get assigned their target paper. We also highlight specific vulnerabilities within this system and offer suggestions to enhance its robustness.

Title: 3D Spatial Understanding in MLLMs: Disambiguation and Evaluation

• Authors: Chun-Peng Chang, Alain Pagani, Didier Stricker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06613
• Pdf URL: https://arxiv.org/pdf/2412.06613
• Copy Paste: [[2412.06613]] 3D Spatial Understanding in MLLMs: Disambiguation and Evaluation(https://arxiv.org/abs/2412.06613)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have made significant progress in tasks such as image captioning and question answering. However, while these models can generate realistic captions, they often struggle with providing precise instructions, particularly when it comes to localizing and disambiguating objects in complex 3D environments. This capability is critical as MLLMs become more integrated with collaborative robotic systems. In scenarios where a target object is surrounded by similar objects (distractors), robots must deliver clear, spatially-aware instructions to guide humans effectively. We refer to this challenge as contextual object localization and disambiguation, which imposes stricter constraints than conventional 3D dense captioning, especially regarding ensuring target exclusivity. In response, we propose simple yet effective techniques to enhance the model's ability to localize and disambiguate target objects. Our approach not only achieves state-of-the-art performance on conventional metrics that evaluate sentence similarity, but also demonstrates improved 3D spatial understanding through 3D visual grounding model.

Title: MVReward: Better Aligning and Evaluating Multi-View Diffusion Models with Human Preferences

• Authors: Weitao Wang, Haoran Xu, Yuxiao Yang, Zhifang Liu, Jun Meng, Haoqian Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06614
• Pdf URL: https://arxiv.org/pdf/2412.06614
• Copy Paste: [[2412.06614]] MVReward: Better Aligning and Evaluating Multi-View Diffusion Models with Human Preferences(https://arxiv.org/abs/2412.06614)
• Keywords: fair, diffusion
• Abstract: Recent years have witnessed remarkable progress in 3D content generation. However, corresponding evaluation methods struggle to keep pace. Automatic approaches have proven challenging to align with human preferences, and the mixed comparison of text- and image-driven methods often leads to unfair evaluations. In this paper, we present a comprehensive framework to better align and evaluate multi-view diffusion models with human preferences. To begin with, we first collect and filter a standardized image prompt set from DALL$\cdot$E and Objaverse, which we then use to generate multi-view assets with several multi-view diffusion models. Through a systematic ranking pipeline on these assets, we obtain a human annotation dataset with 16k expert pairwise comparisons and train a reward model, coined MVReward, to effectively encode human preferences. With MVReward, image-driven 3D methods can be evaluated against each other in a more fair and transparent manner. Building on this, we further propose Multi-View Preference Learning (MVP), a plug-and-play multi-view diffusion tuning strategy. Extensive experiments demonstrate that MVReward can serve as a reliable metric and MVP consistently enhances the alignment of multi-view diffusion models with human preferences.

Title: Copyright-Protected Language Generation via Adaptive Model Fusion

• Authors: Javier Abad, Konstantin Donhauser, Francesco Pinto, Fanny Yang
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06619
• Pdf URL: https://arxiv.org/pdf/2412.06619
• Copy Paste: [[2412.06619]] Copyright-Protected Language Generation via Adaptive Model Fusion(https://arxiv.org/abs/2412.06619)
• Keywords: protect, robust
• Abstract: The risk of language models reproducing copyrighted material from their training data has led to the development of various protective measures. Among these, inference-time strategies that impose constraints via post-processing have shown promise in addressing the complexities of copyright regulation. However, they often incur prohibitive computational costs or suffer from performance trade-offs. To overcome these limitations, we introduce Copyright-Protecting Model Fusion (CP-Fuse), a novel approach that combines models trained on disjoint sets of copyrighted material during inference. In particular, CP-Fuse adaptively aggregates the model outputs to minimize the reproduction of copyrighted content, adhering to a crucial balancing property that prevents the regurgitation of memorized data. Through extensive experiments, we show that CP-Fuse significantly reduces the reproduction of protected material without compromising the quality of text and code generation. Moreover, its post-hoc nature allows seamless integration with other protective measures, further enhancing copyright safeguards. Lastly, we show that CP-Fuse is robust against common techniques for extracting training data.

Title: MAVias: Mitigate any Visual Bias

• Authors: Ioannis Sarridis, Christos Koutlis, Symeon Papadopoulos, Christos Diou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06632
• Pdf URL: https://arxiv.org/pdf/2412.06632
• Copy Paste: [[2412.06632]] MAVias: Mitigate any Visual Bias(https://arxiv.org/abs/2412.06632)
• Keywords: large language model
• Abstract: Mitigating biases in computer vision models is an essential step towards the trustworthiness of artificial intelligence models. Existing bias mitigation methods focus on a small set of predefined biases, limiting their applicability in visual datasets where multiple, possibly unknown biases exist. To address this limitation, we introduce MAVias, an open-set bias mitigation approach leveraging foundation models to discover spurious associations between visual attributes and target classes. MAVias first captures a wide variety of visual features in natural language via a foundation image tagging model, and then leverages a large language model to select those visual features defining the target class, resulting in a set of language-coded potential visual biases. We then translate this set of potential biases into vision-language embeddings and introduce an in-processing bias mitigation approach to prevent the model from encoding information related to them. Our experiments on diverse datasets, including CelebA, Waterbirds, ImageNet, and UrbanCars, show that MAVias effectively detects and mitigates a wide range of biases in visual recognition tasks outperforming current state-of-the-art.

Title: Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers

• Authors: Johanna Vielhaben, Dilyara Bareeva, Jim Berend, Wojciech Samek, Nils Strodthoff
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06639
• Pdf URL: https://arxiv.org/pdf/2412.06639
• Copy Paste: [[2412.06639]] Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers(https://arxiv.org/abs/2412.06639)
• Keywords: transformer
• Abstract: Vision transformers (ViTs) can be trained using various learning paradigms, from fully supervised to self-supervised. Diverse training protocols often result in significantly different feature spaces, which are usually compared through alignment analysis. However, current alignment measures quantify this relationship in terms of a single scalar value, obscuring the distinctions between common and unique features in pairs of representations that share the same scalar alignment. We address this limitation by combining alignment analysis with concept discovery, which enables a breakdown of alignment into single concepts encoded in feature space. This fine-grained comparison reveals both universal and unique concepts across different representations, as well as the internal structure of concepts within each of them. Our methodological contributions address two key prerequisites for concept-based alignment: 1) For a description of the representation in terms of concepts that faithfully capture the geometry of the feature space, we define concepts as the most general structure they can possibly form - arbitrary manifolds, allowing hidden features to be described by their proximity to these manifolds. 2) To measure distances between concept proximity scores of two representations, we use a generalized Rand index and partition it for alignment between pairs of concepts. We confirm the superiority of our novel concept definition for alignment analysis over existing linear baselines in a sanity check. The concept-based alignment analysis of representations from four different ViTs reveals that increased supervision correlates with a reduction in the semantic structure of learned representations.

Title: Detecting Facial Image Manipulations with Multi-Layer CNN Models

• Authors: Alejandro Marco Montejano, Angela Sanchez Perez, Javier Barrachina, David Ortiz-Perez, Manuel Benavent-Lledo, Jose Garcia-Rodriguez
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06643
• Pdf URL: https://arxiv.org/pdf/2412.06643
• Copy Paste: [[2412.06643]] Detecting Facial Image Manipulations with Multi-Layer CNN Models(https://arxiv.org/abs/2412.06643)
• Keywords: robust, extraction, diffusion
• Abstract: The rapid evolution of digital image manipulation techniques poses significant challenges for content verification, with models such as stable diffusion and mid-journey producing highly realistic, yet synthetic, images that can deceive human perception. This research develops and evaluates convolutional neural networks (CNNs) specifically tailored for the detection of these manipulated images. The study implements a comparative analysis of three progressively complex CNN architectures, assessing their ability to classify and localize manipulations across various facial image modifications. Regularization and optimization techniques were systematically incorporated to improve feature extraction and performance. The results indicate that the proposed models achieve an accuracy of up to 76\% in distinguishing manipulated images from genuine ones, surpassing traditional approaches. This research not only highlights the potential of CNNs in enhancing the robustness of digital media verification tools, but also provides insights into effective architectural adaptations and training strategies for low-computation environments. Future work will build on these findings by extending the architectures to handle more diverse manipulation techniques and integrating multi-modal data for improved detection capabilities.

Title: Object Detection using Event Camera: A MoE Heat Conduction based Detector and A New Benchmark Dataset

• Authors: Xiao Wang, Yu Jin, Wentao Wu, Wei Zhang, Lin Zhu, Bo Jiang, Yonghong Tian
• Subjects: cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2412.06647
• Pdf URL: https://arxiv.org/pdf/2412.06647
• Copy Paste: [[2412.06647]] Object Detection using Event Camera: A MoE Heat Conduction based Detector and A New Benchmark Dataset(https://arxiv.org/abs/2412.06647)
• Keywords: extraction, transformer
• Abstract: Object detection in event streams has emerged as a cutting-edge research area, demonstrating superior performance in low-light conditions, scenarios with motion blur, and rapid movements. Current detectors leverage spiking neural networks, Transformers, or convolutional neural networks as their core architectures, each with its own set of limitations including restricted performance, high computational overhead, or limited local receptive fields. This paper introduces a novel MoE (Mixture of Experts) heat conduction-based object detection algorithm that strikingly balances accuracy and computational efficiency. Initially, we employ a stem network for event data embedding, followed by processing through our innovative MoE-HCO blocks. Each block integrates various expert modules to mimic heat conduction within event streams. Subsequently, an IoU-based query selection module is utilized for efficient token extraction, which is then channeled into a detection head for the final object detection process. Furthermore, we are pleased to introduce EvDET200K, a novel benchmark dataset for event-based object detection. Captured with a high-definition Prophesee EVK4-HD event camera, this dataset encompasses 10 distinct categories, 200,000 bounding boxes, and 10,054 samples, each spanning 2 to 5 seconds. We also provide comprehensive results from over 15 state-of-the-art detectors, offering a solid foundation for future research and comparison. The source code of this paper will be released on: this https URL

Title: Efficiency Meets Fidelity: A Novel Quantization Framework for Stable Diffusion

• Authors: Shuaiting Li, Juncan Deng, Zeyu Wang, Hong Gu, Kedong Xu, Haibin Shen, Kejie Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06661
• Pdf URL: https://arxiv.org/pdf/2412.06661
• Copy Paste: [[2412.06661]] Efficiency Meets Fidelity: A Novel Quantization Framework for Stable Diffusion(https://arxiv.org/abs/2412.06661)
• Keywords: diffusion
• Abstract: Text-to-image generation of Stable Diffusion models has achieved notable success due to its remarkable generation ability. However, the repetitive denoising process is computationally intensive during inference, which renders Diffusion models less suitable for real-world applications that require low latency and scalability. Recent studies have employed post-training quantization (PTQ) and quantization-aware training (QAT) methods to compress Diffusion models. Nevertheless, prior research has often neglected to examine the consistency between results generated by quantized models and those from floating-point models. This consistency is crucial in fields such as content creation, design, and edge deployment, as it can significantly enhance both efficiency and system stability for practitioners. To ensure that quantized models generate high-quality and consistent images, we propose an efficient quantization framework for Stable Diffusion models. Our approach features a Serial-to-Parallel calibration pipeline that addresses the consistency of both the calibration and inference processes, as well as ensuring training stability. Based on this pipeline, we further introduce a mix-precision quantization strategy, multi-timestep activation quantization, and time information precalculation techniques to ensure high-fidelity generation in comparison to floating-point models. Through extensive experiments with Stable Diffusion v1-4, v2-1, and XL 1.0, we have demonstrated that our method outperforms the current state-of-the-art techniques when tested on prompts from the COCO validation dataset and the Stable-Diffusion-Prompts dataset. Under W4A8 quantization settings, our approach enhances both distribution similarity and visual similarity by 45%-60%.

Title: Knowledge Transfer and Domain Adaptation for Fine-Grained Remote Sensing Image Segmentation

• Authors: Shun Zhang, Xuechao Zou, Kai Li, Congyan Lang, Shiying Wang, Pin Tao, Tengfei Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06664
• Pdf URL: https://arxiv.org/pdf/2412.06664
• Copy Paste: [[2412.06664]] Knowledge Transfer and Domain Adaptation for Fine-Grained Remote Sensing Image Segmentation(https://arxiv.org/abs/2412.06664)
• Keywords: transformer, segmentation
• Abstract: Fine-grained remote sensing image segmentation is essential for accurately identifying detailed objects in remote sensing images. Recently, vision transformer models (VTM) pretrained on large-scale datasets have shown strong zero-shot generalization, indicating that they have learned the general knowledge of object understanding. We introduce a novel end-to-end learning paradigm combining knowledge guidance with domain refinement to enhance performance. We present two key components: the Feature Alignment Module (FAM) and the Feature Modulation Module (FMM). FAM aligns features from a CNN-based backbone with those from the pretrained VTM's encoder using channel transformation and spatial interpolation, and transfers knowledge via KL divergence and L2 normalization constraint. FMM further adapts the knowledge to the specific domain to address domain shift. We also introduce a fine-grained grass segmentation dataset and demonstrate, through experiments on two datasets, that our method achieves a significant improvement of 2.57 mIoU on the grass dataset and 3.73 mIoU on the cloud dataset. The results highlight the potential of combining knowledge transfer and domain adaptation to overcome domain-related challenges and data limitations. The project page is available at this https URL.

Title: ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance

• Authors: Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, Hang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06673
• Pdf URL: https://arxiv.org/pdf/2412.06673
• Copy Paste: [[2412.06673]] ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance(https://arxiv.org/abs/2412.06673)
• Keywords: large language model
• Abstract: In this paper, we introduce ILLUME, a unified multimodal large language model (MLLM) that seamlessly integrates multimodal understanding and generation capabilities within a single large language model through a unified next-token prediction formulation. To address the large dataset size typically required for image-text alignment, we propose to enhance data efficiency through the design of a vision tokenizer that incorporates semantic information and a progressive multi-stage training procedure. This approach reduces the dataset size to just 15M for pretraining -- over four times fewer than what is typically needed -- while achieving competitive or even superior performance with existing unified MLLMs, such as Janus. Additionally, to promote synergistic enhancement between understanding and generation capabilities, which is under-explored in previous works, we introduce a novel self-enhancing multimodal alignment scheme. This scheme supervises the MLLM to self-assess the consistency between text descriptions and self-generated images, facilitating the model to interpret images more accurately and avoid unrealistic and incorrect predictions caused by misalignment in image generation. Based on extensive experiments, our proposed ILLUME stands out and competes with state-of-the-art unified MLLMs and specialized models across various benchmarks for multimodal understanding, generation, and editing.

Title: EMOv2: Pushing 5M Vision Model Frontier

• Authors: Jiangning Zhang, Teng Hu, Haoyang He, Zhucun Xue, Yabiao Wang, Chengjie Wang, Yong Liu, Xiangtai Li, Dacheng Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06674
• Pdf URL: https://arxiv.org/pdf/2412.06674
• Copy Paste: [[2412.06674]] EMOv2: Pushing 5M Vision Model Frontier(https://arxiv.org/abs/2412.06674)
• Keywords: robust, transformer
• Abstract: This work focuses on developing parameter-efficient and lightweight models for dense predictions while trading off parameters, FLOPs, and performance. Our goal is to set up the new frontier of the 5M magnitude lightweight model on various downstream tasks. Inverted Residual Block (IRB) serves as the infrastructure for lightweight CNNs, but no counterparts have been recognized by attention-based design. Our work rethinks the lightweight infrastructure of efficient IRB and practical components in Transformer from a unified perspective, extending CNN-based IRB to attention-based models and abstracting a one-residual Meta Mobile Block (MMBlock) for lightweight model design. Following neat but effective design criterion, we deduce a modern Improved Inverted Residual Mobile Block (i2RMB) and improve a hierarchical Efficient MOdel (EMOv2) with no elaborate complex structures. Considering the imperceptible latency for mobile users when downloading models under 4G/5G bandwidth and ensuring model performance, we investigate the performance upper limit of lightweight models with a magnitude of 5M. Extensive experiments on various vision recognition, dense prediction, and image generation tasks demonstrate the superiority of our EMOv2 over state-of-the-art methods, e.g., EMOv2-1M/2M/5M achieve 72.3, 75.8, and 79.4 Top-1 that surpass equal-order CNN-/Attention-based models significantly. At the same time, EMOv2-5M equipped RetinaNet achieves 41.5 mAP for object detection tasks that surpasses the previous EMO-5M by +2.6. When employing the more robust training recipe, our EMOv2-5M eventually achieves 82.9 Top-1 accuracy, which elevates the performance of 5M magnitude models to a new level. Code is available at this https URL.

Title: I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token

• Authors: Roi Cohen, Konstantin Dobler, Eden Biran, Gerard de Melo
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06676
• Pdf URL: https://arxiv.org/pdf/2412.06676
• Copy Paste: [[2412.06676]] I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token(https://arxiv.org/abs/2412.06676)
• Keywords: large language model
• Abstract: Large Language Models are known to capture real-world knowledge, allowing them to excel in many downstream tasks. Despite recent advances, these models are still prone to what are commonly known as hallucinations, causing them to emit unwanted and factually incorrect text. In this work, we propose a novel calibration method that can be used to combat hallucinations. We add a special [IDK] ("I don't know") token to the model's vocabulary and introduce an objective function that shifts probability mass to the [IDK] token for incorrect predictions. This approach allows the model to express uncertainty in its output explicitly. We evaluate our proposed method across multiple model architectures and factual downstream tasks. We find that models trained with our method are able to express uncertainty in places where they would previously make mistakes while suffering only a small loss of encoded knowledge. We further perform extensive ablation studies of multiple variations of our approach and provide a detailed analysis of the precision-recall tradeoff of our method.

Title: Exploring Critical Testing Scenarios for Decision-Making Policies: An LLM Approach

• Authors: Weichao Xu, Huaxin Pei, Jingxuan Yang, Yuchen Shi, Yi Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06684
• Pdf URL: https://arxiv.org/pdf/2412.06684
• Copy Paste: [[2412.06684]] Exploring Critical Testing Scenarios for Decision-Making Policies: An LLM Approach(https://arxiv.org/abs/2412.06684)
• Keywords: large language model
• Abstract: Recent years have witnessed surprising achievements of decision-making policies across various fields, such as autonomous driving and robotics. Testing for decision-making policies is crucial with the existence of critical scenarios that may threaten their reliability. Numerous research efforts have been dedicated to testing these policies. However, there are still significant challenges, such as low testing efficiency and diversity due to the complexity of the policies and environments under test. Inspired by the remarkable capabilities of large language models (LLMs), in this paper, we propose an LLM-driven online testing framework for efficiently testing decision-making policies. The main idea is to employ an LLM-based test scenario generator to intelligently generate challenging test cases through contemplation and reasoning. Specifically, we first design a "generate-test-feedback" pipeline and apply templated prompt engineering to fully leverage the knowledge and reasoning abilities of LLMs. Then, we introduce a multi-scale scenario generation strategy to address the inherent challenges LLMs face in making fine adjustments, further enhancing testing efficiency. Finally, we evaluate the LLM-driven approach on five widely used benchmarks. The experimental results demonstrate that our method significantly outperforms baseline approaches in uncovering both critical and diverse scenarios.

Title: Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone

• Authors: Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, Aviral Kumar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06685
• Pdf URL: https://arxiv.org/pdf/2412.06685
• Copy Paste: [[2412.06685]] Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone(https://arxiv.org/abs/2412.06685)
• Keywords: diffusion, transformer
• Abstract: Recent advances in learning decision-making policies can largely be attributed to training expressive policy models, largely via imitation learning. While imitation learning discards non-expert data, reinforcement learning (RL) can still learn from suboptimal data. However, instantiating RL training of a new policy class often presents a different challenge: most deep RL machinery is co-developed with assumptions on the policy class and backbone, resulting in poor performance when the policy class changes. For instance, SAC utilizes a low-variance reparameterization policy gradient for Gaussian policies, but this is unstable for diffusion policies and intractable for autoregressive categorical policies. To address this issue, we develop an offline RL and online fine-tuning approach called policy-agnostic RL (PA-RL) that can effectively train multiple policy classes, with varying architectures and sizes. We build off the basic idea that a universal supervised learning loss can replace the policy improvement step in RL, as long as it is applied on "optimized" actions. To obtain these optimized actions, we first sample multiple actions from a base policy, and run global optimization (i.e., re-ranking multiple action samples using the Q-function) and local optimization (i.e., running gradient steps on an action sample) to maximize the critic on these candidates. PA-RL enables fine-tuning diffusion and transformer policies with either autoregressive tokens or continuous action outputs, at different sizes, entirely via actor-critic RL. Moreover, PA-RL improves the performance and sample-efficiency by up to 2 times compared to existing offline RL and online fine-tuning methods. We show the first result that successfully fine-tunes OpenVLA, a 7B generalist robot policy, autonomously with Cal-QL, an online RL fine-tuning algorithm, improving from 40% to 70% in the real world in 40 minutes.

Title: Some Best Practices in Operator Learning

• Authors: Dustin Enyeart, Guang Lin
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2412.06686
• Pdf URL: https://arxiv.org/pdf/2412.06686
• Copy Paste: [[2412.06686]] Some Best Practices in Operator Learning(https://arxiv.org/abs/2412.06686)
• Keywords: robust
• Abstract: Hyperparameters searches are computationally expensive. This paper studies some general choices of hyperparameters and training methods specifically for operator learning. It considers the architectures DeepONets, Fourier neural operators and Koopman autoencoders for several differential equations to find robust trends. Some options considered are activation functions, dropout and stochastic weight averaging.

Title: Impact of Privacy Parameters on Deep Learning Models for Image Classification

• Authors: Basanta Chaulagain
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06689
• Pdf URL: https://arxiv.org/pdf/2412.06689
• Copy Paste: [[2412.06689]] Impact of Privacy Parameters on Deep Learning Models for Image Classification(https://arxiv.org/abs/2412.06689)
• Keywords: privacy
• Abstract: The project aims to develop differentially private deep learning models for image classification on CIFAR-10 datasets \cite{cifar10} and analyze the impact of various privacy parameters on model accuracy. We have implemented five different deep learning models, namely ConvNet, ResNet18, EfficientNet, ViT, and DenseNet121 and three supervised classifiers namely K-Nearest Neighbors, Naive Bayes Classifier and Support Vector Machine. We evaluated the performance of these models under varying settings. Our best performing model to date is EfficientNet with test accuracy of $59.63\%$ with the following parameters (Adam optimizer, batch size 256, epoch size 100, epsilon value 5.0, learning rate $1e-3$, clipping threshold 1.0, and noise multiplier 0.912).

Title: OmniEvalKit: A Modular, Lightweight Toolbox for Evaluating Large Language Model and its Omni-Extensions

• Authors: Yi-Kai Zhang, Xu-Xiang Zhong, Shiyin Lu, Qing-Guo Chen, De-Chuan Zhan, Han-Jia Ye
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.06693
• Pdf URL: https://arxiv.org/pdf/2412.06693
• Copy Paste: [[2412.06693]] OmniEvalKit: A Modular, Lightweight Toolbox for Evaluating Large Language Model and its Omni-Extensions(https://arxiv.org/abs/2412.06693)
• Keywords: large language model
• Abstract: The rapid advancements in Large Language Models (LLMs) have significantly expanded their applications, ranging from multilingual support to domain-specific tasks and multimodal integration. In this paper, we present OmniEvalKit, a novel benchmarking toolbox designed to evaluate LLMs and their omni-extensions across multilingual, multidomain, and multimodal capabilities. Unlike existing benchmarks that often focus on a single aspect, OmniEvalKit provides a modular, lightweight, and automated evaluation system. It is structured with a modular architecture comprising a Static Builder and Dynamic Data Flow, promoting the seamless integration of new models and datasets. OmniEvalKit supports over 100 LLMs and 50 evaluation datasets, covering comprehensive evaluations across thousands of model-dataset combinations. OmniEvalKit is dedicated to creating an ultra-lightweight and fast-deployable evaluation framework, making downstream applications more convenient and versatile for the AI community.

Title: Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy

• Authors: Yuxuan Xue, Xianghui Xie, Riccardo Marin, Gerard Pons-Moll
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06698
• Pdf URL: https://arxiv.org/pdf/2412.06698
• Copy Paste: [[2412.06698]] Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy(https://arxiv.org/abs/2412.06698)
• Keywords: diffusion
• Abstract: Creating realistic 3D objects and clothed avatars from a single RGB image is an attractive yet challenging problem. Due to its ill-posed nature, recent works leverage powerful prior from 2D diffusion models pretrained on large datasets. Although 2D diffusion models demonstrate strong generalization capability, they cannot guarantee the generated multi-view images are 3D consistent. In this paper, we propose Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy. We leverage a pre-trained 2D diffusion model and a 3D diffusion model via our elegantly designed process that synchronizes two diffusion models at both training and sampling time. The synergy between the 2D and 3D diffusion models brings two major advantages: 1) 2D helps 3D in generalization: the pretrained 2D model has strong generalization ability to unseen images, providing strong shape priors for the 3D diffusion model; 2) 3D helps 2D in multi-view consistency: the 3D diffusion model enhances the 3D consistency of 2D multi-view sampling process, resulting in more accurate multi-view generation. We validate our idea through extensive experiments in image-based objects and clothed avatar generation tasks. Results show that our method generates realistic 3D objects and avatars with high-fidelity geometry and texture. Extensive ablations also validate our design choices and demonstrate the strong generalization ability to diverse clothing and compositional shapes. Our code and pretrained models will be publicly released on this https URL.

Title: You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale

• Authors: Baorui Ma, Huachen Gao, Haoge Deng, Zhengxiong Luo, Tiejun Huang, Lulu Tang, Xinlong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06699
• Pdf URL: https://arxiv.org/pdf/2412.06699
• Copy Paste: [[2412.06699]] You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale(https://arxiv.org/abs/2412.06699)
• Keywords: diffusion
• Abstract: Recent 3D generation models typically rely on limited-scale 3D `gold-labels' or 2D diffusion priors for 3D content creation. However, their performance is upper-bounded by constrained 3D priors due to the lack of scalable learning paradigms. In this work, we present See3D, a visual-conditional multi-view diffusion model trained on large-scale Internet videos for open-world 3D creation. The model aims to Get 3D knowledge by solely Seeing the visual contents from the vast and rapidly growing video data -- You See it, You Got it. To achieve this, we first scale up the training data using a proposed data curation pipeline that automatically filters out multi-view inconsistencies and insufficient observations from source videos. This results in a high-quality, richly diverse, large-scale dataset of multi-view images, termed WebVi3D, containing 320M frames from 16M video clips. Nevertheless, learning generic 3D priors from videos without explicit 3D geometry or camera pose annotations is nontrivial, and annotating poses for web-scale videos is prohibitively expensive. To eliminate the need for pose conditions, we introduce an innovative visual-condition - a purely 2D-inductive visual signal generated by adding time-dependent noise to the masked video data. Finally, we introduce a novel visual-conditional 3D generation framework by integrating See3D into a warping-based pipeline for high-fidelity 3D generation. Our numerical and visual comparisons on single and sparse reconstruction benchmarks show that See3D, trained on cost-effective and scalable video data, achieves notable zero-shot and open-world generation capabilities, markedly outperforming models trained on costly and constrained 3D datasets. Please refer to our project page at: this https URL

Title: Facade: High-Precision Insider Threat Detection Using Deep Contextual Anomaly Detection

• Authors: Alex Kantchelian, Casper Neo, Ryan Stevens, Hyungwon Kim, Zhaohao Fu, Sadegh Momeni, Birkett Huber, Elie Bursztein, Yanis Pavlidis, Senaka Buthpitiya, Martin Cochran, Massimiliano Poletto
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06700
• Pdf URL: https://arxiv.org/pdf/2412.06700
• Copy Paste: [[2412.06700]] Facade: High-Precision Insider Threat Detection Using Deep Contextual Anomaly Detection(https://arxiv.org/abs/2412.06700)
• Keywords: secure, defense, attack, robust
• Abstract: We present Facade (Fast and Accurate Contextual Anomaly DEtection): a high-precision deep-learning-based anomaly detection system deployed at Google (a large technology company) as the last line of defense against insider threats since 2018. Facade is an innovative unsupervised action-context system that detects suspicious actions by considering the context surrounding each action, including relevant facts about the user and other entities involved. It is built around a new multi-modal model that is trained on corporate document access, SQL query, and HTTP/RPC request logs. To overcome the scarcity of incident data, Facade harnesses a novel contrastive learning strategy that relies solely on benign data. Its use of history and implicit social network featurization efficiently handles the frequent out-of-distribution events that occur in a rapidly changing corporate environment, and sustains Facade's high precision performance for a full year after training. Beyond the core model, Facade contributes an innovative clustering approach based on user and action embeddings to improve detection robustness and achieve high precision, multi-scale detection. Functionally what sets Facade apart from existing anomaly detection systems is its high precision. It detects insider attackers with an extremely low false positive rate, lower than 0.01%. For single rogue actions, such as the illegitimate access to a sensitive document, the false positive rate is as low as 0.0003%. To the best of our knowledge, Facade is the only published insider risk anomaly detection system that helps secure such a large corporate environment.

Title: FlexEvent: Event Camera Object Detection at Arbitrary Frequencies

• Authors: Dongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, Wei Tsang Ooi
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.06708
• Pdf URL: https://arxiv.org/pdf/2412.06708
• Copy Paste: [[2412.06708]] FlexEvent: Event Camera Object Detection at Arbitrary Frequencies(https://arxiv.org/abs/2412.06708)
• Keywords: robust
• Abstract: Event cameras offer unparalleled advantages for real-time perception in dynamic environments, thanks to their microsecond-level temporal resolution and asynchronous operation. Existing event-based object detection methods, however, are limited by fixed-frequency paradigms and fail to fully exploit the high-temporal resolution and adaptability of event cameras. To address these limitations, we propose FlexEvent, a novel event camera object detection framework that enables detection at arbitrary frequencies. Our approach consists of two key components: FlexFuser, an adaptive event-frame fusion module that integrates high-frequency event data with rich semantic information from RGB frames, and FAL, a frequency-adaptive learning mechanism that generates frequency-adjusted labels to enhance model generalization across varying operational frequencies. This combination allows our method to detect objects with high accuracy in both fast-moving and static scenarios, while adapting to dynamic environments. Extensive experiments on large-scale event camera datasets demonstrate that our approach surpasses state-of-the-art methods, achieving significant improvements in both standard and high-frequency settings. Notably, our method maintains robust performance when scaling from 20 Hz to 90 Hz and delivers accurate detection up to 180 Hz, proving its effectiveness in extreme conditions. Our framework sets a new benchmark for event-based object detection and paves the way for more adaptable, real-time vision systems.

Title: Parkinson's Disease Diagnosis Through Deep Learning: A Novel LSTM-Based Approach for Freezing of Gait Detection

• Authors: Aqib Nazir Mir, Iqra Nissar, Mumtaz Ahmed, Sarfaraz Masood, Danish Raza Rizvi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06709
• Pdf URL: https://arxiv.org/pdf/2412.06709
• Copy Paste: [[2412.06709]] Parkinson's Disease Diagnosis Through Deep Learning: A Novel LSTM-Based Approach for Freezing of Gait Detection(https://arxiv.org/abs/2412.06709)
• Keywords: generative
• Abstract: Deep learning holds tremendous potential in healthcare for uncovering hidden patterns within extensive clinical datasets, aiding in the diagnosis of various diseases. Parkinson's disease (PD) is a neurodegenerative condition characterized by the deterioration of brain function. In the initial stages of PD, automatic diagnosis poses a challenge due to the similarity in behavior between individuals with PD and those who are healthy. Our objective is to propose an effective model that can aid in the early detection of Parkinson's disease. We employed the VGRF gait signal dataset sourced from Physionet for distinguishing between healthy individuals and those diagnosed with Parkinson's disease. This paper introduces a novel deep learning architecture based on the LSTM network for automatically detecting freezing of gait episodes in Parkinson's disease patients. In contrast to conventional machine learning algorithms, this method eliminates manual feature engineering and proficiently captures prolonged temporal dependencies in gait patterns, thereby improving the diagnosis of Parkinson's disease. The LSTM network resolves the issue of vanishing gradients by employing memory blocks in place of self-connected hidden units, allowing for optimal information assimilation. To prevent overfitting, dropout and L2 regularization techniques have been employed. Additionally, the stochastic gradient-based optimizer Adam is used for the optimization process. The results indicate that our proposed approach surpasses current state-of-the-art models in FOG episode detection, achieving an accuracy of 97.71%, sensitivity of 99%, precision of 98%, and specificity of 96%. This demonstrates its potential as a superior classification method for Parkinson's disease detection.

Title: MISFEAT: Feature Selection for Subgroups with Systematic Missing Data

• Authors: Bar Genossar, Thinh On, Md. Mouinul Islam, Ben Eliav, Senjuti Basu Roy, Avigdor Gal
• Subjects: cs.LG, cs.DB, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.06711
• Pdf URL: https://arxiv.org/pdf/2412.06711
• Copy Paste: [[2412.06711]] MISFEAT: Feature Selection for Subgroups with Systematic Missing Data(https://arxiv.org/abs/2412.06711)
• Keywords: privacy
• Abstract: We investigate the problem of selecting features for datasets that can be naturally partitioned into subgroups (e.g., according to socio-demographic groups and age), each with its own dominant set of features. Within this subgroup-oriented framework, we address the challenge of systematic missing data, a scenario in which some feature values are missing for all tuples of a subgroup, due to flawed data integration, regulatory constraints, or privacy concerns. Feature selection is governed by finding mutual Information, a popular quantification of correlation, between features and a target variable. Our goal is to identify top-K feature subsets of some fixed size with the highest joint mutual information with a target variable. In the presence of systematic missing data, the closed form of mutual information could not simply be applied. We argue that in such a setting, leveraging relationships between available feature mutual information within a subgroup or across subgroups can assist inferring missing mutual information values. We propose a generalizable model based on heterogeneous graph neural network to identify interdependencies between feature-subgroup-target variable connections by modeling it as a multiplex graph, and employing information propagation between its nodes. We address two distinct scalability challenges related to training and propose principled solutions to tackle them. Through an extensive empirical evaluation, we demonstrate the efficacy of the proposed solutions both qualitatively and running time wise.

Title: Toward Non-Invasive Diagnosis of Bankart Lesions with Deep Learning

• Authors: Sahil Sethi, Sai Reddy, Mansi Sakarvadia, Jordan Serotte, Darlington Nwaudo, Nicholas Maassen, Lewis Shi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06717
• Pdf URL: https://arxiv.org/pdf/2412.06717
• Copy Paste: [[2412.06717]] Toward Non-Invasive Diagnosis of Bankart Lesions with Deep Learning(https://arxiv.org/abs/2412.06717)
• Keywords: transformer
• Abstract: Bankart lesions, or anterior-inferior glenoid labral tears, are diagnostically challenging on standard MRIs due to their subtle imaging features-often necessitating invasive MRI arthrograms (MRAs). This study develops deep learning (DL) models to detect Bankart lesions on both standard MRIs and MRAs, aiming to improve diagnostic accuracy and reduce reliance on MRAs. We curated a dataset of 586 shoulder MRIs (335 standard, 251 MRAs) from 558 patients who underwent arthroscopy. Ground truth labels were derived from intraoperative findings, the gold standard for Bankart lesion diagnosis. Separate DL models for MRAs and standard MRIs were trained using the Swin Transformer architecture, pre-trained on a public knee MRI dataset. Predictions from sagittal, axial, and coronal views were ensembled to optimize performance. The models were evaluated on a 20% hold-out test set (117 MRIs: 46 MRAs, 71 standard MRIs). Bankart lesions were identified in 31.9% of MRAs and 8.6% of standard MRIs. The models achieved AUCs of 0.87 (86% accuracy, 83% sensitivity, 86% specificity) and 0.90 (85% accuracy, 82% sensitivity, 86% specificity) on standard MRIs and MRAs, respectively. These results match or surpass radiologist performance on our dataset and reported literature metrics. Notably, our model's performance on non-invasive standard MRIs matched or surpassed the radiologists interpreting MRAs. This study demonstrates the feasibility of using DL to address the diagnostic challenges posed by subtle pathologies like Bankart lesions. Our models demonstrate potential to improve diagnostic confidence, reduce reliance on invasive imaging, and enhance accessibility to care.

Title: ICtoken: An NFT for Hardware IP Protection

• Authors: Shashank Balla, Yiming Zhao, Farinaz Koushanfar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06726
• Pdf URL: https://arxiv.org/pdf/2412.06726
• Copy Paste: [[2412.06726]] ICtoken: An NFT for Hardware IP Protection(https://arxiv.org/abs/2412.06726)
• Keywords: secure, protect, attack, robust
• Abstract: Protecting integrated circuits (ICs) from piracy and theft throughout their lifecycle is a persistent and complex challenge. In order to safeguard against illicit piracy attacks, this work proposes a novel framework utilizing Non-Fungible Tokens (NFTs) called ICtokens, uniquely linked to their corresponding physical ICs. Each ICtoken contains comprehensive information, including authentication data, supply chain stage and status, ownership details, and other IC metadata, while also making provision for the secure integration of a logic-locking key. Designed to be publicly logged, ICtokens securely obscure metering information without compromising functionality. In addition, the ICtracker, a distributed ledger technology powered by a swift and energy-efficient consortium blockchain, is used to register and manage ICtokens and their respective owners, tracking all associated interactions. This robust ledger guarantees the traceability and auditing of ICtokens while simultaneously developing a product-level NFT at every transaction point within the supply chain. Consequently, a scalable framework is established, creating unique, immutable digital twins for ICs and IC-embedded products in the form of ICtokens and their transactions. This provides a robust and reliable supply chain trail back to the original IP owner, while also offering unprecedented assurance to consumers of IC-embedded products. The rich information contained within ICtokens facilitates more detailed audits than previous proposals for IC supply chain monitoring. A proof-of-concept, implemented as an open-source solution, ensures the ease of adoption of the proposed framework.

Title: Take Fake as Real: Realistic-like Robust Black-box Adversarial Attack to Evade AIGC Detection

• Authors: Caiyun Xie, Dengpan Ye, Yunming Zhang, Long Tang, Yunna Lv, Jiacheng Deng, Jiawei Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06727
• Pdf URL: https://arxiv.org/pdf/2412.06727
• Copy Paste: [[2412.06727]] Take Fake as Real: Realistic-like Robust Black-box Adversarial Attack to Evade AIGC Detection(https://arxiv.org/abs/2412.06727)
• Keywords: security, attack, robust, diffusion
• Abstract: The security of AI-generated content (AIGC) detection based on GANs and diffusion models is closely related to the credibility of multimedia content. Malicious adversarial attacks can evade these developing AIGC detection. However, most existing adversarial attacks focus only on GAN-generated facial images detection, struggle to be effective on multi-class natural images and diffusion-based detectors, and exhibit poor invisibility. To fill this gap, we first conduct an in-depth analysis of the vulnerability of AIGC detectors and discover the feature that detectors vary in vulnerability to different post-processing. Then, considering the uncertainty of detectors in real-world scenarios, and based on the discovery, we propose a Realistic-like Robust Black-box Adversarial attack (R$^2$BA) with post-processing fusion optimization. Unlike typical perturbations, R$^2$BA uses real-world post-processing, i.e., Gaussian blur, JPEG compression, Gaussian noise and light spot to generate adversarial examples. Specifically, we use a stochastic particle swarm algorithm with inertia decay to optimize post-processing fusion intensity and explore the detector's decision boundary. Guided by the detector's fake probability, R$^2$BA enhances/weakens the detector-vulnerable/detector-robust post-processing intensity to strike a balance between adversariality and invisibility. Extensive experiments on popular/commercial AIGC detectors and datasets demonstrate that R$^2$BA exhibits impressive anti-detection performance, excellent invisibility, and strong robustness in GAN-based and diffusion-based cases. Compared to state-of-the-art white-box and black-box attacks, R$^2$BA shows significant improvements of 15% and 21% in anti-detection performance under the original and robust scenario respectively, offering valuable insights for the security of AIGC detection in real-world applications.

Title: JAPAGEN: Efficient Few/Zero-shot Learning via Japanese Training Dataset Generation with LLM

• Authors: Takuro Fujii, Satoru Katsumata
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06738
• Pdf URL: https://arxiv.org/pdf/2412.06738
• Copy Paste: [[2412.06738]] JAPAGEN: Efficient Few/Zero-shot Learning via Japanese Training Dataset Generation with LLM(https://arxiv.org/abs/2412.06738)
• Keywords: robust, large language model
• Abstract: Recently some studies have highlighted the potential of Large Language Models (LLMs) as effective generators of supervised training data, offering advantages such as enhanced inference efficiency and reduced costs associated with data collection. However, these studies have predominantly focused on English language tasks. In this paper, we address the fundamental research question: Can LLMs serve as proficient training data generators for other language tasks? Specifically, we leverage LLMs to synthesize supervised training data under few-shot and zero-shot learning scenarios across six diverse Japanese downstream tasks. Subsequently, we utilize this synthesized data to train compact models (e.g., BERT). This novel methodology is termed JAPAGEN. Our experimental findings underscore that JAPAGEN achieves robust performance in classification tasks that necessitate formal text inputs, demonstrating competitive results compared to conventional LLM prompting strategies.

Title: ContRail: A Framework for Realistic Railway Image Synthesis using ControlNet

• Authors: Andrei-Robert Alexandrescu, Razvan-Gabriel Petec, Alexandru Manole, Laura-Silvia Diosan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06742
• Pdf URL: https://arxiv.org/pdf/2412.06742
• Copy Paste: [[2412.06742]] ContRail: A Framework for Realistic Railway Image Synthesis using ControlNet(https://arxiv.org/abs/2412.06742)
• Keywords: diffusion, segmentation
• Abstract: Deep Learning became an ubiquitous paradigm due to its extraordinary effectiveness and applicability in numerous domains. However, the approach suffers from the high demand of data required to achieve the potential of this type of model. An ever-increasing sub-field of Artificial Intelligence, Image Synthesis, aims to address this limitation through the design of intelligent models capable of creating original and realistic images, endeavour which could drastically reduce the need for real data. The Stable Diffusion generation paradigm recently propelled state-of-the-art approaches to exceed all previous benchmarks. In this work, we propose the ContRail framework based on the novel Stable Diffusion model ControlNet, which we empower through a multi-modal conditioning method. We experiment with the task of synthetic railway image generation, where we improve the performance in rail-specific tasks, such as rail semantic segmentation by enriching the dataset with realistic synthetic images.

Title: 3D Graph Attention Networks for High Fidelity Pediatric Glioma Segmentation

• Authors: Harish Thangaraj, Diya Katariya, Eshaan Joshi, Sangeetha N
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06743
• Pdf URL: https://arxiv.org/pdf/2412.06743
• Copy Paste: [[2412.06743]] 3D Graph Attention Networks for High Fidelity Pediatric Glioma Segmentation(https://arxiv.org/abs/2412.06743)
• Keywords: segmentation
• Abstract: Pediatric brain tumors, particularly gliomas, represent a significant cause of cancer related mortality in children with complex infiltrative growth patterns that complicate treatment. Early, accurate segmentation of these tumors in neuroimaging data is crucial for effective diagnosis and intervention planning. This study presents a novel 3D UNet architecture with a spatial attention mechanism tailored for automated segmentation of pediatric gliomas. Using the BraTS pediatric glioma dataset with multiparametric MRI data, the proposed model captures multi-scale features and selectively attends to tumor relevant regions, enhancing segmentation precision and reducing interference from surrounding tissue. The model's performance is quantitatively evaluated using the Dice similarity coefficient and HD95, demonstrating improved delineation of complex glioma structured. This approach offers a promising advancement in automating pediatric glioma segmentation, with the potential to improve clinical decision making and outcomes.

Title: ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities

• Authors: Adhiraj Ghosh, Sebastian Dziadzio, Ameya Prabhu, Vishaal Udandarao, Samuel Albanie, Matthias Bethge
• Subjects: cs.LG, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06745
• Pdf URL: https://arxiv.org/pdf/2412.06745
• Copy Paste: [[2412.06745]] ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities(https://arxiv.org/abs/2412.06745)
• Keywords: robust
• Abstract: Traditional fixed test sets fall short in evaluating open-ended capabilities of foundation models. To address this, we propose ONEBench(OpeN-Ended Benchmarking), a new testing paradigm that consolidates individual evaluation datasets into a unified, ever-expanding sample pool. ONEBench allows users to generate custom, open-ended evaluation benchmarks from this pool, corresponding to specific capabilities of interest. By aggregating samples across test sets, ONEBench enables the assessment of diverse capabilities beyond those covered by the original test sets, while mitigating overfitting and dataset bias. Most importantly, it frames model evaluation as a collective process of selecting and aggregating sample-level tests. The shift from task-specific benchmarks to ONEBench introduces two challenges: (1)heterogeneity and (2)incompleteness. Heterogeneity refers to the aggregation over diverse metrics, while incompleteness describes comparing models evaluated on different data subsets. To address these challenges, we explore algorithms to aggregate sparse measurements into reliable model scores. Our aggregation algorithm ensures identifiability(asymptotically recovering ground-truth scores) and rapid convergence, enabling accurate model ranking with less data. On homogenous datasets, we show our aggregation algorithm provides rankings that highly correlate with those produced by average scores. We also demonstrate robustness to ~95% of measurements missing, reducing evaluation cost by up to 20x with little-to-no change in model rankings. We introduce ONEBench-LLM for language models and ONEBench-LMM for vision-language models, unifying evaluations across these domains. Overall, we present a technique for open-ended evaluation, which can aggregate over incomplete, heterogeneous sample-level measurements to continually grow a benchmark alongside the rapidly developing foundation models.

Title: Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models

• Authors: Neel Jain, Aditya Shrivastava, Chenyang Zhu, Daben Liu, Alfy Samuel, Ashwinee Panda, Anoop Kumar, Micah Goldblum, Tom Goldstein
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06748
• Pdf URL: https://arxiv.org/pdf/2412.06748
• Copy Paste: [[2412.06748]] Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models(https://arxiv.org/abs/2412.06748)
• Keywords: large language model
• Abstract: A key component of building safe and reliable language models is enabling the models to appropriately refuse to follow certain instructions or answer certain questions. We may want models to output refusal messages for various categories of user queries, for example, ill-posed questions, instructions for committing illegal acts, or queries which require information past the model's knowledge horizon. Engineering models that refuse to answer such questions is complicated by the fact that an individual may want their model to exhibit varying levels of sensitivity for refusing queries of various categories, and different users may want different refusal rates. The current default approach involves training multiple models with varying proportions of refusal messages from each category to achieve the desired refusal rates, which is computationally expensive and may require training a new model to accommodate each user's desired preference over refusal rates. To address these challenges, we propose refusal tokens, one such token for each refusal category or a single refusal token, which are prepended to the model's responses during training. We then show how to increase or decrease the probability of generating the refusal token for each category during inference to steer the model's refusal behavior. Refusal tokens enable controlling a single model's refusal rates without the need of any further fine-tuning, but only by selectively intervening during generation.

Title: InstantRestore: Single-Step Personalized Face Restoration with Shared-Image Attention

• Authors: Howard Zhang, Yuval Alaluf, Sizhuo Ma, Achuta Kadambi, Jian Wang, Kfir Aberman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06753
• Pdf URL: https://arxiv.org/pdf/2412.06753
• Copy Paste: [[2412.06753]] InstantRestore: Single-Step Personalized Face Restoration with Shared-Image Attention(https://arxiv.org/abs/2412.06753)
• Keywords: diffusion
• Abstract: Face image restoration aims to enhance degraded facial images while addressing challenges such as diverse degradation types, real-time processing demands, and, most crucially, the preservation of identity-specific features. Existing methods often struggle with slow processing times and suboptimal restoration, especially under severe degradation, failing to accurately reconstruct finer-level identity details. To address these issues, we introduce InstantRestore, a novel framework that leverages a single-step image diffusion model and an attention-sharing mechanism for fast and personalized face restoration. Additionally, InstantRestore incorporates a novel landmark attention loss, aligning key facial landmarks to refine the attention maps, enhancing identity preservation. At inference time, given a degraded input and a small (~4) set of reference images, InstantRestore performs a single forward pass through the network to achieve near real-time performance. Unlike prior approaches that rely on full diffusion processes or per-identity model tuning, InstantRestore offers a scalable solution suitable for large-scale applications. Extensive experiments demonstrate that InstantRestore outperforms existing methods in quality and speed, making it an appealing choice for identity-preserving face restoration.

Title: Training Large Language Models to Reason in a Continuous Latent Space

• Authors: Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, Yuandong Tian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06769
• Pdf URL: https://arxiv.org/pdf/2412.06769
• Copy Paste: [[2412.06769]] Training Large Language Models to Reason in a Continuous Latent Space(https://arxiv.org/abs/2412.06769)
• Keywords: large language model
• Abstract: Large language models (LLMs) are restricted to reason in the "language space", where they typically express the reasoning process with a chain-of-thought (CoT) to solve a complex reasoning problem. However, we argue that language space may not always be optimal for reasoning. For example, most word tokens are primarily for textual coherence and not essential for reasoning, while some critical tokens require complex planning and pose huge challenges to LLMs. To explore the potential of LLM reasoning in an unrestricted latent space instead of using natural language, we introduce a new paradigm Coconut (Chain of Continuous Thought). We utilize the last hidden state of the LLM as a representation of the reasoning state (termed "continuous thought"). Rather than decoding this into a word token, we feed it back to the LLM as the subsequent input embedding directly in the continuous space. Experiments show that Coconut can effectively augment the LLM on several reasoning tasks. This novel latent reasoning paradigm leads to emergent advanced reasoning patterns: the continuous thought can encode multiple alternative next reasoning steps, allowing the model to perform a breadth-first search (BFS) to solve the problem, rather than prematurely committing to a single deterministic path like CoT. Coconut outperforms CoT in certain logical reasoning tasks that require substantial backtracking during planning, with fewer thinking tokens during inference. These findings demonstrate the promise of latent reasoning and offer valuable insights for future research.

Title: Visual Lexicon: Rich Image Features in Language Space

• Authors: XuDong Wang, Xingyi Zhou, Alireza Fathi, Trevor Darrell, Cordelia Schmid
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06774
• Pdf URL: https://arxiv.org/pdf/2412.06774
• Copy Paste: [[2412.06774]] Visual Lexicon: Rich Image Features in Language Space(https://arxiv.org/abs/2412.06774)
• Keywords: diffusion
• Abstract: We present Visual Lexicon, a novel visual language that encodes rich image information into the text space of vocabulary tokens while retaining intricate visual details that are often challenging to convey in natural language. Unlike traditional methods that prioritize either high-level semantics (e.g., CLIP) or pixel-level reconstruction (e.g., VAE), ViLex simultaneously captures rich semantic content and fine visual details, enabling high-quality image generation and comprehensive visual scene understanding. Through a self-supervised learning pipeline, ViLex generates tokens optimized for reconstructing input images using a frozen text-to-image (T2I) diffusion model, preserving the detailed information necessary for high-fidelity semantic-level reconstruction. As an image embedding in the language space, ViLex tokens leverage the compositionality of natural languages, allowing them to be used independently as "text tokens" or combined with natural language tokens to prompt pretrained T2I models with both visual and textual inputs, mirroring how we interact with vision-language models (VLMs). Experiments demonstrate that ViLex achieves higher fidelity in image reconstruction compared to text embeddings--even with a single ViLex token. Moreover, ViLex successfully performs various DreamBooth tasks in a zero-shot, unsupervised manner without fine-tuning T2I models. Additionally, ViLex serves as a powerful vision encoder, consistently improving vision-language model performance across 15 benchmarks relative to a strong SigLIP baseline.

Title: Diverse Score Distillation

• Authors: Yanbo Xu, Jayanth Srinivasa, Gaowen Liu, Shubham Tulsiani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06780
• Pdf URL: https://arxiv.org/pdf/2412.06780
• Copy Paste: [[2412.06780]] Diverse Score Distillation(https://arxiv.org/abs/2412.06780)
• Keywords: diffusion
• Abstract: Score distillation of 2D diffusion models has proven to be a powerful mechanism to guide 3D optimization, for example enabling text-based 3D generation or single-view reconstruction. A common limitation of existing score distillation formulations, however, is that the outputs of the (mode-seeking) optimization are limited in diversity despite the underlying diffusion model being capable of generating diverse samples. In this work, inspired by the sampling process in denoising diffusion, we propose a score formulation that guides the optimization to follow generation paths defined by random initial seeds, thus ensuring diversity. We then present an approximation to adopt this formulation for scenarios where the optimization may not precisely follow the generation paths (e.g. a 3D representation whose renderings evolve in a co-dependent manner). We showcase the applications of our `Diverse Score Distillation' (DSD) formulation across tasks such as 2D optimization, text-based 3D inference, and single-view reconstruction. We also empirically validate DSD against prior score distillation formulations and show that it significantly improves sample diversity while preserving fidelity.

Title: Around the World in 80 Timesteps: A Generative Approach to Global Visual Geolocation

• Authors: Nicolas Dufour, David Picard, Vicky Kalogeiton, Loic Landrieu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06781
• Pdf URL: https://arxiv.org/pdf/2412.06781
• Copy Paste: [[2412.06781]] Around the World in 80 Timesteps: A Generative Approach to Global Visual Geolocation(https://arxiv.org/abs/2412.06781)
• Keywords: diffusion, generative
• Abstract: Global visual geolocation predicts where an image was captured on Earth. Since images vary in how precisely they can be localized, this task inherently involves a significant degree of ambiguity. However, existing approaches are deterministic and overlook this aspect. In this paper, we aim to close the gap between traditional geolocalization and modern generative methods. We propose the first generative geolocation approach based on diffusion and Riemannian flow matching, where the denoising process operates directly on the Earth's surface. Our model achieves state-of-the-art performance on three visual geolocation benchmarks: OpenStreetView-5M, YFCC-100M, and iNat21. In addition, we introduce the task of probabilistic visual geolocation, where the model predicts a probability distribution over all possible locations instead of a single point. We introduce new metrics and baselines for this task, demonstrating the advantages of our diffusion-based approach. Codes and models will be made available.

Title: Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation

• Authors: Ruihan Gao, Kangle Deng, Gengshan Yang, Wenzhen Yuan, Jun-Yan Zhu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.06785
• Pdf URL: https://arxiv.org/pdf/2412.06785
• Copy Paste: [[2412.06785]] Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation(https://arxiv.org/abs/2412.06785)
• Keywords: diffusion
• Abstract: 3D generation methods have shown visually compelling results powered by diffusion image priors. However, they often fail to produce realistic geometric details, resulting in overly smooth surfaces or geometric details inaccurately baked in albedo maps. To address this, we introduce a new method that incorporates touch as an additional modality to improve the geometric details of generated 3D assets. We design a lightweight 3D texture field to synthesize visual and tactile textures, guided by 2D diffusion model priors on both visual and tactile domains. We condition the visual texture generation on high-resolution tactile normals and guide the patch-based tactile texture refinement with a customized TextureDreambooth. We further present a multi-part generation pipeline that enables us to synthesize different textures across various regions. To our knowledge, we are the first to leverage high-resolution tactile sensing to enhance geometric details for 3D generation tasks. We evaluate our method in both text-to-3D and image-to-3D settings. Our experiments demonstrate that our method provides customized and realistic fine geometric textures while maintaining accurate alignment between two modalities of vision and touch.

Title: Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis

• Authors: M. Hamza Mughal, Rishabh Dabral, Merel C.J. Scholman, Vera Demberg, Christian Theobalt
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06786
• Pdf URL: https://arxiv.org/pdf/2412.06786
• Copy Paste: [[2412.06786]] Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis(https://arxiv.org/abs/2412.06786)
• Keywords: diffusion
• Abstract: Non-verbal communication often comprises of semantically rich gestures that help convey the meaning of an utterance. Producing such semantic co-speech gestures has been a major challenge for the existing neural systems that can generate rhythmic beat gestures, but struggle to produce semantically meaningful gestures. Therefore, we present RAG-Gesture, a diffusion-based gesture generation approach that leverages Retrieval Augmented Generation (RAG) to produce natural-looking and semantically rich gestures. Our neuro-explicit gesture generation approach is designed to produce semantic gestures grounded in interpretable linguistic knowledge. We achieve this by using explicit domain knowledge to retrieve exemplar motions from a database of co-speech gestures. Once retrieved, we then inject these semantic exemplar gestures into our diffusion-based gesture generation pipeline using DDIM inversion and retrieval guidance at the inference time without any need of training. Further, we propose a control paradigm for guidance, that allows the users to modulate the amount of influence each retrieval insertion has over the generated sequence. Our comparative evaluations demonstrate the validity of our approach against recent gesture generation approaches. The reader is urged to explore the results on our project page.

Title: [MASK] is All You Need

• Authors: Vincent Tao Hu, Björn Ommer
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06787
• Pdf URL: https://arxiv.org/pdf/2412.06787
• Copy Paste: [[2412.06787]] [MASK] is All You Need(https://arxiv.org/abs/2412.06787)
• Keywords: diffusion, generative, segmentation
• Abstract: In generative models, two paradigms have gained attraction in various applications: next-set prediction-based Masked Generative Models and next-noise prediction-based Non-Autoregressive Models, e.g., Diffusion Models. In this work, we propose using discrete-state models to connect them and explore their scalability in the vision domain. First, we conduct a step-by-step analysis in a unified design space across two types of models including timestep-independence, noise schedule, temperature, guidance strength, etc in a scalable manner. Second, we re-cast typical discriminative tasks, e.g., image segmentation, as an unmasking process from [MASK]tokens on a discrete-state model. This enables us to perform various sampling processes, including flexible conditional sampling by only training once to model the joint distribution. All aforementioned explorations lead to our framework named Discrete Interpolants, which enables us to achieve state-of-the-art or competitive performance compared to previous discrete-state based methods in various benchmarks, like ImageNet256, MS COCO, and video dataset FaceForensics. In summary, by leveraging [MASK] in discrete-state models, we can bridge Masked Generative and Non-autoregressive Diffusion models, as well as generative and discriminative tasks.