2025-01-10

Title: Video Summarisation with Incident and Context Information using Generative AI

• Authors: Ulindu De Silva, Leon Fernando, Kalinga Bandara, Rashmika Nawaratne
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04764
• Pdf URL: https://arxiv.org/pdf/2501.04764
• Copy Paste: [[2501.04764]] Video Summarisation with Incident and Context Information using Generative AI(https://arxiv.org/abs/2501.04764)
• Keywords: robust, generative
• Abstract: The proliferation of video content production has led to vast amounts of data, posing substantial challenges in terms of analysis efficiency and resource utilization. Addressing this issue calls for the development of robust video analysis tools. This paper proposes a novel approach leveraging Generative Artificial Intelligence (GenAI) to facilitate streamlined video analysis. Our tool aims to deliver tailored textual summaries of user-defined queries, offering a focused insight amidst extensive video datasets. Unlike conventional frameworks that offer generic summaries or limited action recognition, our method harnesses the power of GenAI to distil relevant information, enhancing analysis precision and efficiency. Employing YOLO-V8 for object detection and Gemini for comprehensive video and text analysis, our solution achieves heightened contextual accuracy. By combining YOLO with Gemini, our approach furnishes textual summaries extracted from extensive CCTV footage, enabling users to swiftly navigate and verify pertinent events without the need for exhaustive manual review. The quantitative evaluation revealed a similarity of 72.8%, while the qualitative assessment rated an accuracy of 85%, demonstrating the capability of the proposed method.

Title: TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training

• Authors: Felix Krause, Timy Phan, Vincent Tao Hu, Björn Ommer
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04765
• Pdf URL: https://arxiv.org/pdf/2501.04765
• Copy Paste: [[2501.04765]] TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training(https://arxiv.org/abs/2501.04765)
• Keywords: diffusion, transformer
• Abstract: Diffusion models have emerged as the mainstream approach for visual generation. However, these models usually suffer from sample inefficiency and high training costs. This issue is particularly pronounced in the standard diffusion transformer architecture due to its quadratic complexity relative to input length. Recent works have addressed this by reducing the number of tokens processed in the model, often through masking. In contrast, this work aims to improve the training efficiency of the diffusion backbone by using predefined routes that store this information until it is reintroduced to deeper layers of the model, rather than discarding these tokens entirely. Further, we combine multiple routes and introduce an adapted auxiliary loss that accounts for all applied routes. Our method is not limited to the common transformer-based model - it can also be applied to state-space models. Unlike most current approaches, TREAD achieves this without architectural modifications. Finally, we show that our method reduces the computational cost and simultaneously boosts model performance on the standard benchmark ImageNet-1K 256 x 256 in class-conditional synthesis. Both of these benefits multiply to a convergence speedup of 9.55x at 400K training iterations compared to DiT and 25.39x compared to the best benchmark performance of DiT at 7M training iterations.

Title: GaussianVideo: Efficient Video Representation via Hierarchical Gaussian Splatting

• Authors: Andrew Bond, Jui-Hsien Wang, Long Mai, Erkut Erdem, Aykut Erdem
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04782
• Pdf URL: https://arxiv.org/pdf/2501.04782
• Copy Paste: [[2501.04782]] GaussianVideo: Efficient Video Representation via Hierarchical Gaussian Splatting(https://arxiv.org/abs/2501.04782)
• Keywords: robust
• Abstract: Efficient neural representations for dynamic video scenes are critical for applications ranging from video compression to interactive simulations. Yet, existing methods often face challenges related to high memory usage, lengthy training times, and temporal consistency. To address these issues, we introduce a novel neural video representation that combines 3D Gaussian splatting with continuous camera motion modeling. By leveraging Neural ODEs, our approach learns smooth camera trajectories while maintaining an explicit 3D scene representation through Gaussians. Additionally, we introduce a spatiotemporal hierarchical learning strategy, progressively refining spatial and temporal features to enhance reconstruction quality and accelerate convergence. This memory-efficient approach achieves high-quality rendering at impressive speeds. Experimental results show that our hierarchical learning, combined with robust camera motion modeling, captures complex dynamic scenes with strong temporal consistency, achieving state-of-the-art performance across diverse video datasets in both high- and low-motion scenarios.

Title: Leveraging Registers in Vision Transformers for Robust Adaptation

• Authors: Srikar Yellapragada, Kowshik Thopalli, Vivek Narayanaswamy, Wesam Sakla, Yang Liu, Yamen Mubarka, Dimitris Samaras, Jayaraman J. Thiagarajan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04784
• Pdf URL: https://arxiv.org/pdf/2501.04784
• Copy Paste: [[2501.04784]] Leveraging Registers in Vision Transformers for Robust Adaptation(https://arxiv.org/abs/2501.04784)
• Keywords: robust, transformer
• Abstract: Vision Transformers (ViTs) have shown success across a variety of tasks due to their ability to capture global image representations. Recent studies have identified the existence of high-norm tokens in ViTs, which can interfere with unsupervised object discovery. To address this, the use of "registers" which are additional tokens that isolate high norm patch tokens while capturing global image-level information has been proposed. While registers have been studied extensively for object discovery, their generalization properties particularly in out-of-distribution (OOD) scenarios, remains underexplored. In this paper, we examine the utility of register token embeddings in providing additional features for improving generalization and anomaly rejection. To that end, we propose a simple method that combines the special CLS token embedding commonly employed in ViTs with the average-pooled register embeddings to create feature representations which are subsequently used for training a downstream classifier. We find that this enhances OOD generalization and anomaly rejection, while maintaining in-distribution (ID) performance. Extensive experiments across multiple ViT backbones trained with and without registers reveal consistent improvements of 2-4\% in top-1 OOD accuracy and a 2-3\% reduction in false positive rates for anomaly detection. Importantly, these gains are achieved without additional computational overhead.

Title: Decentralised Resource Sharing in TinyML: Wireless Bilayer Gossip Parallel SGD for Collaborative Learning

• Authors: Ziyuan Bao, Eiman Kanjo, Soumya Banerjee, Hasib-Al Rashid, Tinoosh Mohsenin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04817
• Pdf URL: https://arxiv.org/pdf/2501.04817
• Copy Paste: [[2501.04817]] Decentralised Resource Sharing in TinyML: Wireless Bilayer Gossip Parallel SGD for Collaborative Learning(https://arxiv.org/abs/2501.04817)
• Keywords: privacy, federate
• Abstract: With the growing computational capabilities of microcontroller units (MCUs), edge devices can now support machine learning models. However, deploying decentralised federated learning (DFL) on such devices presents key challenges, including intermittent connectivity, limited communication range, and dynamic network topologies. This paper proposes a novel framework, bilayer Gossip Decentralised Parallel Stochastic Gradient Descent (GD PSGD), designed to address these issues in resource-constrained environments. The framework incorporates a hierarchical communication structure using Distributed Kmeans (DKmeans) clustering for geographic grouping and a gossip protocol for efficient model aggregation across two layers: intra-cluster and inter-cluster. We evaluate the framework's performance against the Centralised Federated Learning (CFL) baseline using the MCUNet model on the CIFAR-10 dataset under IID and Non-IID conditions. Results demonstrate that the proposed method achieves comparable accuracy to CFL on IID datasets, requiring only 1.8 additional rounds for convergence. On Non-IID datasets, the accuracy loss remains under 8\% for moderate data imbalance. These findings highlight the framework's potential to support scalable and privacy-preserving learning on edge devices with minimal performance trade-offs.

Title: Building Foundations for Natural Language Processing of Historical Turkish: Resources and Models

• Authors: Şaziye Betül Özateş, Tarık Emre Tıraş, Ece Elif Adak, Berat Doğan, Fatih Burak Karagöz, Efe Eren Genç, Esma F. Bilgin Taşdemir
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04828
• Pdf URL: https://arxiv.org/pdf/2501.04828
• Copy Paste: [[2501.04828]] Building Foundations for Natural Language Processing of Historical Turkish: Resources and Models(https://arxiv.org/abs/2501.04828)
• Keywords: transformer
• Abstract: This paper introduces foundational resources and models for natural language processing (NLP) of historical Turkish, a domain that has remained underexplored in computational linguistics. We present the first named entity recognition (NER) dataset, HisTR and the first Universal Dependencies treebank, OTA-BOUN for a historical form of the Turkish language along with transformer-based models trained using these datasets for named entity recognition, dependency parsing, and part-of-speech tagging tasks. Additionally, we introduce Ottoman Text Corpus (OTC), a clean corpus of transliterated historical Turkish texts that spans a wide range of historical periods. Our experimental results show significant improvements in the computational analysis of historical Turkish, achieving promising results in tasks that require understanding of historical linguistic structures. They also highlight existing challenges, such as domain adaptation and language variations across time periods. All of the presented resources and models are made available at this https URL to serve as a benchmark for future progress in historical Turkish NLP.

Title: Blockchain-Based Secure Vehicle Auction System with Smart Contracts

• Authors: Ka Wai Wu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.04841
• Pdf URL: https://arxiv.org/pdf/2501.04841
• Copy Paste: [[2501.04841]] Blockchain-Based Secure Vehicle Auction System with Smart Contracts(https://arxiv.org/abs/2501.04841)
• Keywords: secure, security, privacy
• Abstract: The problem of a single point of failure in centralized systems poses a great challenge to the stability of such systems. Meanwhile, the tamperability of data within centralized systems makes users reluctant to trust and use centralized applications in many scenarios, including the financial and business sectors. Blockchain, as a new decentralized technology, addresses these issues effectively. As a typical decentralized system, blockchain can be utilized to build a data-sharing model. Users in a blockchain do not need to trust other users; instead, they trust that the majority of miner nodes are honest. Smart contracts enable developers to write distributed programs based on blockchain systems, ensuring that all code is immutable and secure. In this paper, we analyze the security of blockchain technology to illustrate its advantages and justify its use. Furthermore, we design a new system for storing and trading vehicle information based on the Ethereum blockchain and smart contract technology. Specifically, our system allows users to upload vehicle information and auction vehicles to transfer ownership. Our application provides great convenience to buyers and owners, while the use of smart contracts enhances the security and privacy of the system.

Title: EDMB: Edge Detector with Mamba

• Authors: Yachuan Li, Xavier Soria Poma, Yun Bai, Qian Xiao, Chaozhi Yang, Guanlin Li, Zongmin Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04846
• Pdf URL: https://arxiv.org/pdf/2501.04846
• Copy Paste: [[2501.04846]] EDMB: Edge Detector with Mamba(https://arxiv.org/abs/2501.04846)
• Keywords: transformer
• Abstract: Transformer-based models have made significant progress in edge detection, but their high computational cost is prohibitive. Recently, vision Mamba have shown excellent ability in efficiently capturing long-range dependencies. Drawing inspiration from this, we propose a novel edge detector with Mamba, termed EDMB, to efficiently generate high-quality multi-granularity edges. In EDMB, Mamba is combined with a global-local architecture, therefore it can focus on both global information and fine-grained cues. The fine-grained cues play a crucial role in edge detection, but are usually ignored by ordinary Mamba. We design a novel decoder to construct learnable Gaussian distributions by fusing global features and fine-grained features. And the multi-grained edges are generated by sampling from the distributions. In order to make multi-granularity edges applicable to single-label data, we introduce Evidence Lower Bound loss to supervise the learning of the distributions. On the multi-label dataset BSDS500, our proposed EDMB achieves competitive single-granularity ODS 0.837 and multi-granularity ODS 0.851 without multi-scale test or extra PASCAL-VOC data. Remarkably, EDMB can be extended to single-label datasets such as NYUDv2 and BIPED. The source code is available at this https URL.

Title: Exploring Large Language Models for Semantic Analysis and Categorization of Android Malware

• Authors: Brandon J Walton, Mst Eshita Khatun, James M Ghawaly, Aisha Ali-Gombe
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04848
• Pdf URL: https://arxiv.org/pdf/2501.04848
• Copy Paste: [[2501.04848]] Exploring Large Language Models for Semantic Analysis and Categorization of Android Malware(https://arxiv.org/abs/2501.04848)
• Keywords: robust, large language model
• Abstract: Malware analysis is a complex process of examining and evaluating malicious software's functionality, origin, and potential impact. This arduous process typically involves dissecting the software to understand its components, infection vector, propagation mechanism, and payload. Over the years, deep reverse engineering of malware has become increasingly tedious, mainly due to modern malicious codebases' fast evolution and sophistication. Essentially, analysts are tasked with identifying the elusive needle in the haystack within the complexities of zero-day malware, all while under tight time constraints. Thus, in this paper, we explore leveraging Large Language Models (LLMs) for semantic malware analysis to expedite the analysis of known and novel samples. Built on GPT-4o-mini model, \msp is designed to augment malware analysis for Android through a hierarchical-tiered summarization chain and strategic prompt engineering. Additionally, \msp performs malware categorization, distinguishing potential malware from benign applications, thereby saving time during the malware reverse engineering process. Despite not being fine-tuned for Android malware analysis, we demonstrate that through optimized and advanced prompt engineering \msp can achieve up to 77% classification accuracy while providing highly robust summaries at functional, class, and package levels. In addition, leveraging the backward tracing of the summaries from package to function levels allowed us to pinpoint the precise code snippets responsible for malicious behavior.

Title: LayerMix: Enhanced Data Augmentation through Fractal Integration for Robust Deep Learning

• Authors: Hafiz Mughees Ahmad, Dario Morle, Afshin Rahimi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04861
• Pdf URL: https://arxiv.org/pdf/2501.04861
• Copy Paste: [[2501.04861]] LayerMix: Enhanced Data Augmentation through Fractal Integration for Robust Deep Learning(https://arxiv.org/abs/2501.04861)
• Keywords: attack, robust
• Abstract: Deep learning models have demonstrated remarkable performance across various computer vision tasks, yet their vulnerability to distribution shifts remains a critical challenge. Despite sophisticated neural network architectures, existing models often struggle to maintain consistent performance when confronted with Out-of-Distribution (OOD) samples, including natural corruptions, adversarial perturbations, and anomalous patterns. We introduce LayerMix, an innovative data augmentation approach that systematically enhances model robustness through structured fractal-based image synthesis. By meticulously integrating structural complexity into training datasets, our method generates semantically consistent synthetic samples that significantly improve neural network generalization capabilities. Unlike traditional augmentation techniques that rely on random transformations, LayerMix employs a structured mixing pipeline that preserves original image semantics while introducing controlled variability. Extensive experiments across multiple benchmark datasets, including CIFAR-10, CIFAR-100, ImageNet-200, and ImageNet-1K demonstrate LayerMixs superior performance in classification accuracy and substantially enhances critical Machine Learning (ML) safety metrics, including resilience to natural image corruptions, robustness against adversarial attacks, improved model calibration and enhanced prediction consistency. LayerMix represents a significant advancement toward developing more reliable and adaptable artificial intelligence systems by addressing the fundamental challenges of deep learning generalization. The code is available at this https URL.

Title: Real-Time Textless Dialogue Generation

• Authors: Long Mai, Julie Carson-Berndsen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04877
• Pdf URL: https://arxiv.org/pdf/2501.04877
• Copy Paste: [[2501.04877]] Real-Time Textless Dialogue Generation(https://arxiv.org/abs/2501.04877)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have led to significant progress in text-based dialogue systems. These systems can now generate high-quality responses that are accurate and coherent across a wide range of topics and tasks. However, spoken dialogue systems still lag behind in terms of naturalness. They tend to produce robotic interactions, with issues such as slow response times, overly generic or cautious replies, and a lack of natural rhythm and fluid turn-taking. This shortcoming is largely due to the over-reliance on the traditional cascaded design, which involve separate, sequential components, as well as the use of text as an intermediate representation. This paper propose a real-time, textless spoken dialogue generation model (RTTL-DG) that aims to overcome these challenges. Our system enables fluid turn-taking and generates responses with minimal delay by processing streaming spoken conversation directly. Additionally, our model incorporates backchannels, filters, laughter, and other paralinguistic signals, which are often absent in cascaded dialogue systems, to create more natural and human-like interactions. The implementations and generated samples are available in our repository: this https URL

Title: Leveraging Log Probabilities in Language Models to Forecast Future Events

• Authors: Tommaso Soru, Jim Marshall
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04880
• Pdf URL: https://arxiv.org/pdf/2501.04880
• Copy Paste: [[2501.04880]] Leveraging Log Probabilities in Language Models to Forecast Future Events(https://arxiv.org/abs/2501.04880)
• Keywords: large language model
• Abstract: In the constantly changing field of data-driven decision making, accurately predicting future events is crucial for strategic planning in various sectors. The emergence of Large Language Models (LLMs) marks a significant advancement in this area, offering advanced tools that utilise extensive text data for prediction. In this industry paper, we introduce a novel method for AI-driven foresight using LLMs. Building on top of previous research, we employ data on current trends and their trajectories for generating forecasts on 15 different topics. Subsequently, we estimate their probabilities via a multi-step approach based on log probabilities. We show we achieve a Brier score of 0.186, meaning a +26% improvement over random chance and a +19% improvement over widely-available AI systems.

Title: A Look into How Machine Learning is Reshaping Engineering Models: the Rise of Analysis Paralysis, Optimal yet Infeasible Solutions, and the Inevitable Rashomon Paradox

• Authors: MZ Naser
• Subjects: cs.LG, stat.ME
• Abstract URL: https://arxiv.org/abs/2501.04894
• Pdf URL: https://arxiv.org/pdf/2501.04894
• Copy Paste: [[2501.04894]] A Look into How Machine Learning is Reshaping Engineering Models: the Rise of Analysis Paralysis, Optimal yet Infeasible Solutions, and the Inevitable Rashomon Paradox(https://arxiv.org/abs/2501.04894)
• Keywords: interpretability, explainability
• Abstract: The widespread acceptance of empirically derived codal provisions and equations in civil engineering stands in stark contrast to the skepticism facing machine learning (ML) models, despite their shared statistical foundations. This paper examines this philosophical tension through the lens of structural engineering and explores how integrating ML challenges traditional engineering philosophies and professional identities. Recent efforts have documented how ML enhances predictive accuracy, optimizes designs, and analyzes complex behaviors. However, one might also raise concerns about the diminishing role of human intuition and the interpretability of algorithms. To showcase this rarely explored front, this paper presents how ML can be successfully integrated into various engineering problems by means of formulation via deduction, induction, and abduction. Then, this paper identifies three principal paradoxes that could arise when adopting ML: analysis paralysis (increased prediction accuracy leading to a reduced understanding of physical mechanisms), infeasible solutions (optimization resulting in unconventional designs that challenge engineering intuition), and the Rashomon effect (where contradictions in explainability methods and physics arise). This paper concludes by addressing these paradoxes and arguing the need to rethink epistemological shifts in engineering and engineering education and methodologies to harmonize traditional principles with ML.

Title: Online Continual Learning: A Systematic Literature Review of Approaches, Challenges, and Benchmarks

• Authors: Seyed Amir Bidaki, Amir Mohammadkhah, Kiyan Rezaee, Faeze Hassani, Sadegh Eskandari, Maziar Salahi, Mohammad M. Ghassemi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04897
• Pdf URL: https://arxiv.org/pdf/2501.04897
• Copy Paste: [[2501.04897]] Online Continual Learning: A Systematic Literature Review of Approaches, Challenges, and Benchmarks(https://arxiv.org/abs/2501.04897)
• Keywords: generative
• Abstract: Online Continual Learning (OCL) is a critical area in machine learning, focusing on enabling models to adapt to evolving data streams in real-time while addressing challenges such as catastrophic forgetting and the stability-plasticity trade-off. This study conducts the first comprehensive Systematic Literature Review (SLR) on OCL, analyzing 81 approaches, extracting over 1,000 features (specific tasks addressed by these approaches), and identifying more than 500 components (sub-models within approaches, including algorithms and tools). We also review 83 datasets spanning applications like image classification, object detection, and multimodal vision-language tasks. Our findings highlight key challenges, including reducing computational overhead, developing domain-agnostic solutions, and improving scalability in resource-constrained environments. Furthermore, we identify promising directions for future research, such as leveraging self-supervised learning for multimodal and sequential data, designing adaptive memory mechanisms that integrate sparse retrieval and generative replay, and creating efficient frameworks for real-world applications with noisy or evolving task boundaries. By providing a rigorous and structured synthesis of the current state of OCL, this review offers a valuable resource for advancing this field and addressing its critical challenges and opportunities. The complete SLR methodology steps and extracted data are publicly available through the provided link: this https URL Systematic-Literature-Review-on-Online-Continual-Learning

Title: SUGAR: Leveraging Contextual Confidence for Smarter Retrieval

• Authors: Hanna Zubkova, Ji-Hoon Park, Seong-Whan Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04899
• Pdf URL: https://arxiv.org/pdf/2501.04899
• Copy Paste: [[2501.04899]] SUGAR: Leveraging Contextual Confidence for Smarter Retrieval(https://arxiv.org/abs/2501.04899)
• Keywords: large language model
• Abstract: Bearing in mind the limited parametric knowledge of Large Language Models (LLMs), retrieval-augmented generation (RAG) which supplies them with the relevant external knowledge has served as an approach to mitigate the issue of hallucinations to a certain extent. However, uniformly retrieving supporting context makes response generation source-inefficient, as triggering the retriever is not always necessary, or even inaccurate, when a model gets distracted by noisy retrieved content and produces an unhelpful answer. Motivated by these issues, we introduce Semantic Uncertainty Guided Adaptive Retrieval (SUGAR), where we leverage context-based entropy to actively decide whether to retrieve and to further determine between single-step and multi-step retrieval. Our empirical results show that selective retrieval guided by semantic uncertainty estimation improves the performance across diverse question answering tasks, as well as achieves a more efficient inference.

Title: Beyond Life: A Digital Will Solution for Posthumous Data Management

• Authors: Xinzhang Chen, Arash Shaghaghi, Jesse Laeuchli, Salil Kanhere
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.04900
• Pdf URL: https://arxiv.org/pdf/2501.04900
• Copy Paste: [[2501.04900]] Beyond Life: A Digital Will Solution for Posthumous Data Management(https://arxiv.org/abs/2501.04900)
• Keywords: secure
• Abstract: In the digital era, managing posthumous data presents a growing challenge, with current technical solutions often falling short in practicality. Existing tools are typically closed-source, lack transparency, fail to offer cross-platform support, and provide limited access control. This paper introduces `Beyond Life', a cross-platform digital will management solution designed to securely handle and distribute digital assets after death. At the core of this solution is a customized Ciphertext-Policy Attribute-Based Encryption (CP-ABE) scheme, referred to as PD-CP-ABE, which enables efficient, fine-grained control over access to will content at scale. Unlike existing systems, Beyond Life operates independently of service providers, offering users greater transparency and control over how their will is generated, stored, and executed. The system is also designed to be portable, allowing users to change their will service provider. The proposed system has been fully developed and rigorously evaluated to ensure performance and real-world feasibility. The system implementation is made publicly available.

Title: JELLY: Joint Emotion Recognition and Context Reasoning with LLMs for Conversational Speech Synthesis

• Authors: Jun-Hyeok Cha, Seung-Bin Kim, Hyung-Seok Oh, Seong-Whan Lee
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.04904
• Pdf URL: https://arxiv.org/pdf/2501.04904
• Copy Paste: [[2501.04904]] JELLY: Joint Emotion Recognition and Context Reasoning with LLMs for Conversational Speech Synthesis(https://arxiv.org/abs/2501.04904)
• Keywords: large language model
• Abstract: Recently, there has been a growing demand for conversational speech synthesis (CSS) that generates more natural speech by considering the conversational context. To address this, we introduce JELLY, a novel CSS framework that integrates emotion recognition and context reasoning for generating appropriate speech in conversation by fine-tuning a large language model (LLM) with multiple partial LoRA modules. We propose an Emotion-aware Q-former encoder, which enables the LLM to perceive emotions in speech. The encoder is trained to align speech emotions with text, utilizing datasets of emotional speech. The entire model is then fine-tuned with conversational speech data to infer emotional context for generating emotionally appropriate speech in conversation. Our experimental results demonstrate that JELLY excels in emotional context modeling, synthesizing speech that naturally aligns with conversation, while mitigating the scarcity of emotional conversational speech datasets.

Title: A Machine Learning Model for Crowd Density Classification in Hajj Video Frames

• Authors: Afnan A.Shah
• Subjects: cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.04911
• Pdf URL: https://arxiv.org/pdf/2501.04911
• Copy Paste: [[2501.04911]] A Machine Learning Model for Crowd Density Classification in Hajj Video Frames(https://arxiv.org/abs/2501.04911)
• Keywords: robust, extraction
• Abstract: Managing the massive annual gatherings of Hajj and Umrah presents significant challenges, particularly as the Saudi government aims to increase the number of pilgrims. Currently, around two million pilgrims attend Hajj and 26 million attend Umrah making crowd control especially in critical areas like the Grand Mosque during Tawaf, a major concern. Additional risks arise in managing dense crowds at key sites such as Arafat where the potential for stampedes, fires and pandemics poses serious threats to public safety. This research proposes a machine learning model to classify crowd density into three levels: moderate crowd, overcrowded and very dense crowd in video frames recorded during Hajj, with a flashing red light to alert organizers in real-time when a very dense crowd is detected. While current research efforts in processing Hajj surveillance videos focus solely on using CNN to detect abnormal behaviors, this research focuses more on high-risk crowds that can lead to disasters. Hazardous crowd conditions require a robust method, as incorrect classification could trigger unnecessary alerts and government intervention, while failure to classify could result in disaster. The proposed model integrates Local Binary Pattern (LBP) texture analysis, which enhances feature extraction for differentiating crowd density levels, along with edge density and area-based features. The model was tested on the KAU-Smart Crowd 'HAJJv2' dataset which contains 18 videos from various key locations during Hajj including 'Massaa', 'Jamarat', 'Arafat' and 'Tawaf'. The model achieved an accuracy rate of 87% with a 2.14% error percentage (misclassification rate), demonstrating its ability to detect and classify various crowd conditions effectively. That contributes to enhanced crowd management and safety during large-scale events like Hajj.

Title: From Mesh Completion to AI Designed Crown

• Authors: Golriz Hosseinimanesh, Farnoosh Ghadiri, Francois Guibault, Farida Cheriet, Julia Keren
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04914
• Pdf URL: https://arxiv.org/pdf/2501.04914
• Copy Paste: [[2501.04914]] From Mesh Completion to AI Designed Crown(https://arxiv.org/abs/2501.04914)
• Keywords: transformer
• Abstract: Designing a dental crown is a time-consuming and labor intensive process. Our goal is to simplify crown design and minimize the tediousness of making manual adjustments while still ensuring the highest level of accuracy and consistency. To this end, we present a new end- to-end deep learning approach, coined Dental Mesh Completion (DMC), to generate a crown mesh conditioned on a point cloud context. The dental context includes the tooth prepared to receive a crown and its surroundings, namely the two adjacent teeth and the three closest teeth in the opposing jaw. We formulate crown generation in terms of completing this point cloud context. A feature extractor first converts the input point cloud into a set of feature vectors that represent local regions in the point cloud. The set of feature vectors is then fed into a transformer to predict a new set of feature vectors for the missing region (crown). Subsequently, a point reconstruction head, followed by a multi-layer perceptron, is used to predict a dense set of points with normals. Finally, a differentiable point-to-mesh layer serves to reconstruct the crown surface mesh. We compare our DMC method to a graph-based convolutional neural network which learns to deform a crown mesh from a generic crown shape to the target geometry. Extensive experiments on our dataset demonstrate the effectiveness of our method, which attains an average of 0.062 Chamfer this http URL code is available at:this https URL

Title: SpecTf: Transformers Enable Data-Driven Imaging Spectroscopy Cloud Detection

• Authors: Jake H. Lee, Michael Kiper, David R. Thompson, Philip G. Brodrick
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04916
• Pdf URL: https://arxiv.org/pdf/2501.04916
• Copy Paste: [[2501.04916]] SpecTf: Transformers Enable Data-Driven Imaging Spectroscopy Cloud Detection(https://arxiv.org/abs/2501.04916)
• Keywords: interpretability, transformer
• Abstract: Current and upcoming generations of visible-shortwave infrared (VSWIR) imaging spectrometers promise unprecedented capacity to quantify Earth System processes across the globe. However, reliable cloud screening remains a fundamental challenge for these instruments, where traditional spatial and temporal approaches are limited by cloud variability and limited temporal coverage. The Spectroscopic Transformer (SpecTf) addresses these challenges with a spectroscopy-specific deep learning architecture that performs cloud detection using only spectral information (no spatial or temporal data are required). By treating spectral measurements as sequences rather than image channels, SpecTf learns fundamental physical relationships without relying on spatial context. Our experiments demonstrate that SpecTf significantly outperforms the current baseline approach implemented for the EMIT instrument, and performs comparably with other machine learning methods with orders of magnitude fewer learned parameters. Critically, we demonstrate SpecTf's inherent interpretability through its attention mechanism, revealing physically meaningful spectral features the model has learned. Finally, we present SpecTf's potential for cross-instrument generalization by applying it to a different instrument on a different platform without modifications, opening the door to instrument agnostic data driven algorithms for future imaging spectroscopy tasks.

Title: Investigating Numerical Translation with Large Language Models

• Authors: Wei Tang, Jiawei Yu, Yuang Li, Yanqing Zhao, Weidong Zhang, Wei Feng, Min Zhang, Hao Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04927
• Pdf URL: https://arxiv.org/pdf/2501.04927
• Copy Paste: [[2501.04927]] Investigating Numerical Translation with Large Language Models(https://arxiv.org/abs/2501.04927)
• Keywords: security, large language model
• Abstract: The inaccurate translation of numbers can lead to significant security issues, ranging from financial setbacks to medical inaccuracies. While large language models (LLMs) have made significant advancements in machine translation, their capacity for translating numbers has not been thoroughly explored. This study focuses on evaluating the reliability of LLM-based machine translation systems when handling numerical data. In order to systematically test the numerical translation capabilities of currently open source LLMs, we have constructed a numerical translation dataset between Chinese and English based on real business data, encompassing ten types of numerical translation. Experiments on the dataset indicate that errors in numerical translation are a common issue, with most open-source LLMs faltering when faced with our test scenarios. Especially when it comes to numerical types involving large units like ``million", ``billion", and "yi", even the latest llama3.1 8b model can have error rates as high as 20%. Finally, we introduce three potential strategies to mitigate the numerical mistranslations for large units.

Title: Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency

• Authors: Shiji Zhao, Ranjie Duan, Fengxiang Wang, Chi Chen, Caixin Kang, Jialing Tao, YueFeng Chen, Hui Xue, Xingxing Wei
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04931
• Pdf URL: https://arxiv.org/pdf/2501.04931
• Copy Paste: [[2501.04931]] Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency(https://arxiv.org/abs/2501.04931)
• Keywords: attack, large language model
• Abstract: Multimodal Large Language Models (MLLMs) have achieved impressive performance and have been put into practical use in commercial applications, but they still have potential safety mechanism vulnerabilities. Jailbreak attacks are red teaming methods that aim to bypass safety mechanisms and discover MLLMs' potential risks. Existing MLLMs' jailbreak methods often bypass the model's safety mechanism through complex optimization methods or carefully designed image and text prompts. Despite achieving some progress, they have a low attack success rate on commercial closed-source MLLMs. Unlike previous research, we empirically find that there exists a Shuffle Inconsistency between MLLMs' comprehension ability and safety ability for the shuffled harmful instruction. That is, from the perspective of comprehension ability, MLLMs can understand the shuffled harmful text-image instructions well. However, they can be easily bypassed by the shuffled harmful instructions from the perspective of safety ability, leading to harmful responses. Then we innovatively propose a text-image jailbreak attack named SI-Attack. Specifically, to fully utilize the Shuffle Inconsistency and overcome the shuffle randomness, we apply a query-based black-box optimization method to select the most harmful shuffled inputs based on the feedback of the toxic judge model. A series of experiments show that SI-Attack can improve the attack's performance on three benchmarks. In particular, SI-Attack can obviously improve the attack success rate for commercial MLLMs such as GPT-4o or Claude-3.5-Sonnet.

Title: Plug-and-Play DISep: Separating Dense Instances for Scene-to-Pixel Weakly-Supervised Change Detection in High-Resolution Remote Sensing Images

• Authors: Zhenghui Zhao, Chen Wu, Lixiang Ru, Di Wang, Hongruixuan Chen, Cuiqun Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04934
• Pdf URL: https://arxiv.org/pdf/2501.04934
• Copy Paste: [[2501.04934]] Plug-and-Play DISep: Separating Dense Instances for Scene-to-Pixel Weakly-Supervised Change Detection in High-Resolution Remote Sensing Images(https://arxiv.org/abs/2501.04934)
• Keywords: transformer
• Abstract: Existing Weakly-Supervised Change Detection (WSCD) methods often encounter the problem of "instance lumping" under scene-level supervision, particularly in scenarios with a dense distribution of changed instances (i.e., changed objects). In these scenarios, unchanged pixels between changed instances are also mistakenly identified as changed, causing multiple changes to be mistakenly viewed as one. In practical applications, this issue prevents the accurate quantification of the number of changes. To address this issue, we propose a Dense Instance Separation (DISep) method as a plug-and-play solution, refining pixel features from a unified instance perspective under scene-level supervision. Specifically, our DISep comprises a three-step iterative training process: 1) Instance Localization: We locate instance candidate regions for changed pixels using high-pass class activation maps. 2) Instance Retrieval: We identify and group these changed pixels into different instance IDs through connectivity searching. Then, based on the assigned instance IDs, we extract corresponding pixel-level features on a per-instance basis. 3) Instance Separation: We introduce a separation loss to enforce intra-instance pixel consistency in the embedding space, thereby ensuring separable instance feature representations. The proposed DISep adds only minimal training cost and no inference cost. It can be seamlessly integrated to enhance existing WSCD methods. We achieve state-of-the-art performance by enhancing {three Transformer-based and four ConvNet-based methods} on the LEVIR-CD, WHU-CD, DSIFN-CD, SYSU-CD, and CDD datasets. Additionally, our DISep can be used to improve fully-supervised change detection methods. Code is available at this https URL.

Title: Multi-Context Temporal Consistent Modeling for Referring Video Object Segmentation

• Authors: Sun-Hyuk Choi, Hayoung Jo, Seong-Whan Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04939
• Pdf URL: https://arxiv.org/pdf/2501.04939
• Copy Paste: [[2501.04939]] Multi-Context Temporal Consistent Modeling for Referring Video Object Segmentation(https://arxiv.org/abs/2501.04939)
• Keywords: transformer, segmentation
• Abstract: Referring video object segmentation aims to segment objects within a video corresponding to a given text description. Existing transformer-based temporal modeling approaches face challenges related to query inconsistency and the limited consideration of context. Query inconsistency produces unstable masks of different objects in the middle of the video. The limited consideration of context leads to the segmentation of incorrect objects by failing to adequately account for the relationship between the given text and instances. To address these issues, we propose the Multi-context Temporal Consistency Module (MTCM), which consists of an Aligner and a Multi-Context Enhancer (MCE). The Aligner removes noise from queries and aligns them to achieve query consistency. The MCE predicts text-relevant queries by considering multi-context. We applied MTCM to four different models, increasing performance across all of them, particularly achieving 47.6 J&F on the MeViS. Code is available at this https URL.

Title: A New Perspective on Privacy Protection in Federated Learning with Granular-Ball Computing

• Authors: Guannan Lai, Yihui Feng, Xin Yang, Xiaoyu Deng, Hao Yu, Shuyin Xia, Guoyin Wang, Tianrui Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04940
• Pdf URL: https://arxiv.org/pdf/2501.04940
• Copy Paste: [[2501.04940]] A New Perspective on Privacy Protection in Federated Learning with Granular-Ball Computing(https://arxiv.org/abs/2501.04940)
• Keywords: privacy, protect, robust, federate, segmentation
• Abstract: Federated Learning (FL) facilitates collaborative model training while prioritizing privacy by avoiding direct data sharing. However, most existing articles attempt to address challenges within the model's internal parameters and corresponding outputs, while neglecting to solve them at the input level. To address this gap, we propose a novel framework called Granular-Ball Federated Learning (GrBFL) for image classification. GrBFL diverges from traditional methods that rely on the finest-grained input data. Instead, it segments images into multiple regions with optimal coarse granularity, which are then reconstructed into a graph structure. We designed a two-dimensional binary search segmentation algorithm based on variance constraints for GrBFL, which effectively removes redundant information while preserving key representative features. Extensive theoretical analysis and experiments demonstrate that GrBFL not only safeguards privacy and enhances efficiency but also maintains robust utility, consistently outperforming other state-of-the-art FL methods. The code is available at this https URL.

Title: MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classification

• Authors: Yapeng Li, Yong Luo, Lefei Zhang, Zengmao Wang, Bo Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04944
• Pdf URL: https://arxiv.org/pdf/2501.04944
• Copy Paste: [[2501.04944]] MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classification(https://arxiv.org/abs/2501.04944)
• Keywords: transformer
• Abstract: Transformer has been extensively explored for hyperspectral image (HSI) classification. However, transformer poses challenges in terms of speed and memory usage because of its quadratic computational complexity. Recently, the Mamba model has emerged as a promising approach, which has strong long-distance modeling capabilities while maintaining a linear computational complexity. However, representing the HSI is challenging for the Mamba due to the requirement for an integrated spatial and spectral understanding. To remedy these drawbacks, we propose a novel HSI classification model based on a Mamba model, named MambaHSI, which can simultaneously model long-range interaction of the whole image and integrate spatial and spectral information in an adaptive manner. Specifically, we design a spatial Mamba block (SpaMB) to model the long-range interaction of the whole image at the pixel-level. Then, we propose a spectral Mamba block (SpeMB) to split the spectral vector into multiple groups, mine the relations across different spectral groups, and extract spectral features. Finally, we propose a spatial-spectral fusion module (SSFM) to adaptively integrate spatial and spectral features of a HSI. To our best knowledge, this is the first image-level HSI classification model based on the Mamba. We conduct extensive experiments on four diverse HSI datasets. The results demonstrate the effectiveness and superiority of the proposed model for HSI classification. This reveals the great potential of Mamba to be the next-generation backbone for HSI models. Codes are available at this https URL .

Title: Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models

• Authors: Qingyu Ren, Jie Zeng, Qianyu He, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, Fei Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04945
• Pdf URL: https://arxiv.org/pdf/2501.04945
• Copy Paste: [[2501.04945]] Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models(https://arxiv.org/abs/2501.04945)
• Keywords: large language model
• Abstract: It is crucial for large language models (LLMs) to follow instructions that involve multiple constraints. However, soft constraints are semantically related and difficult to verify through automated methods. These constraints remain a significant challenge for LLMs. To enhance the ability of LLMs to follow soft constraints, we initially design a pipeline to obtain high-quality outputs automatically. Additionally, to fully utilize the acquired data, we introduce a training paradigm based on curriculum learning. We experimentally evaluate the effectiveness of our methods in improving LLMs' soft constraint following ability and analyze the factors driving the improvements. The datasets and code are publicly available at this https URL.

Title: Seeing with Partial Certainty: Conformal Prediction for Robotic Scene Recognition in Built Environments

• Authors: Yifan Xu, Vineet Kamat, Carol Menassa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04947
• Pdf URL: https://arxiv.org/pdf/2501.04947
• Copy Paste: [[2501.04947]] Seeing with Partial Certainty: Conformal Prediction for Robotic Scene Recognition in Built Environments(https://arxiv.org/abs/2501.04947)
• Keywords: large language model
• Abstract: In assistive robotics serving people with disabilities (PWD), accurate place recognition in built environments is crucial to ensure that robots navigate and interact safely within diverse indoor spaces. Language interfaces, particularly those powered by Large Language Models (LLM) and Vision Language Models (VLM), hold significant promise in this context, as they can interpret visual scenes and correlate them with semantic information. However, such interfaces are also known for their hallucinated predictions. In addition, language instructions provided by humans can also be ambiguous and lack precise details about specific locations, objects, or actions, exacerbating the hallucination issue. In this work, we introduce Seeing with Partial Certainty (SwPC) - a framework designed to measure and align uncertainty in VLM-based place recognition, enabling the model to recognize when it lacks confidence and seek assistance when necessary. This framework is built on the theory of conformal prediction to provide statistical guarantees on place recognition while minimizing requests for human help in complex indoor environment settings. Through experiments on the widely used richly-annotated scene dataset Matterport3D, we show that SwPC significantly increases the success rate and decreases the amount of human intervention required relative to the prior art. SwPC can be utilized with any VLMs directly without requiring model fine-tuning, offering a promising, lightweight approach to uncertainty modeling that complements and scales alongside the expanding capabilities of foundational models.

Title: Open Problems in Machine Unlearning for AI Safety

• Authors: Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O'Gara, Robert Kirk, Ben Bucknall, Tim Fist, Luke Ong, Philip Torr, Kwok-Yan Lam, Robert Trager, David Krueger, Sören Mindermann, José Hernandez-Orallo, Mor Geva, Yarin Gal
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.04952
• Pdf URL: https://arxiv.org/pdf/2501.04952
• Copy Paste: [[2501.04952]] Open Problems in Machine Unlearning for AI Safety(https://arxiv.org/abs/2501.04952)
• Keywords: security, privacy, robust
• Abstract: As AI systems become more capable, widely deployed, and increasingly autonomous in critical areas such as cybersecurity, biological research, and healthcare, ensuring their safety and alignment with human values is paramount. Machine unlearning -- the ability to selectively forget or suppress specific types of knowledge -- has shown promise for privacy and data removal tasks, which has been the primary focus of existing research. More recently, its potential application to AI safety has gained attention. In this paper, we identify key limitations that prevent unlearning from serving as a comprehensive solution for AI safety, particularly in managing dual-use knowledge in sensitive domains like cybersecurity and chemical, biological, radiological, and nuclear (CBRN) safety. In these contexts, information can be both beneficial and harmful, and models may combine seemingly harmless information for harmful purposes -- unlearning this information could strongly affect beneficial uses. We provide an overview of inherent constraints and open problems, including the broader side effects of unlearning dangerous knowledge, as well as previously unexplored tensions between unlearning and existing safety mechanisms. Finally, we investigate challenges related to evaluation, robustness, and the preservation of safety features during unlearning. By mapping these limitations and open challenges, we aim to guide future research toward realistic applications of unlearning within a broader AI safety framework, acknowledging its limitations and highlighting areas where alternative approaches may be required.

Title: Addressing Domain Shift via Imbalance-Aware Domain Adaptation in Embryo Development Assessment

• Authors: Lei Li, Xinglin Zhang, Jun Liang, Tao Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04958
• Pdf URL: https://arxiv.org/pdf/2501.04958
• Copy Paste: [[2501.04958]] Addressing Domain Shift via Imbalance-Aware Domain Adaptation in Embryo Development Assessment(https://arxiv.org/abs/2501.04958)
• Keywords: robust
• Abstract: Deep learning models in medical imaging face dual challenges: domain shift, where models perform poorly when deployed in settings different from their training environment, and class imbalance, where certain disease conditions are naturally underrepresented. We present Imbalance-Aware Domain Adaptation (IADA), a novel framework that simultaneously tackles both challenges through three key components: (1) adaptive feature learning with class-specific attention mechanisms, (2) balanced domain alignment with dynamic weighting, and (3) adaptive threshold optimization. Our theoretical analysis establishes convergence guarantees and complexity bounds. Through extensive experiments on embryo development assessment across four imaging modalities, IADA demonstrates significant improvements over existing methods, achieving up to 25.19\% higher accuracy while maintaining balanced performance across classes. In challenging scenarios with low-quality imaging systems, IADA shows robust generalization with AUC improvements of up to 12.56\%. These results demonstrate IADA's potential for developing reliable and equitable medical imaging systems for diverse clinical settings. The code is made public available at \url{this https URL}

Title: Demystifying Domain-adaptive Post-training for Financial LLMs

• Authors: Zixuan Ke, Yifei Ming, Xuan-Phi Nguyen, Caiming Xiong, Shafiq Joty
• Subjects: cs.CL, cs.AI, cs.CE, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04961
• Pdf URL: https://arxiv.org/pdf/2501.04961
• Copy Paste: [[2501.04961]] Demystifying Domain-adaptive Post-training for Financial LLMs(https://arxiv.org/abs/2501.04961)
• Keywords: generative, large language model
• Abstract: Domain-adaptive post-training of large language models (LLMs) has emerged as a promising approach for specialized domains such as medicine and finance. However, significant challenges remain in identifying optimal adaptation criteria and training strategies across varying data and model configurations. To address these challenges, we introduce FINDAP, a systematic and fine-grained investigation into domain-adaptive post-training of LLMs for the finance domain. Our approach begins by identifying the core capabilities required for the target domain and designing a comprehensive evaluation suite aligned with these needs. We then analyze the effectiveness of key post-training stages, including continual pretraining, instruction tuning, and preference alignment. Building on these insights, we propose an effective training recipe centered on a novel preference data distillation method, which leverages process signals from a generative reward model. The resulting model, Llama-Fin, achieves state-of-the-art performance across a wide range of financial tasks. Our analysis also highlights how each post-training stage contributes to distinct capabilities, uncovering specific challenges and effective solutions, providing valuable insights for domain adaptation of LLMs. Project page: this https URL

Title: VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models

• Authors: Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, Irwin King
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.04962
• Pdf URL: https://arxiv.org/pdf/2501.04962
• Copy Paste: [[2501.04962]] VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models(https://arxiv.org/abs/2501.04962)
• Keywords: robust
• Abstract: With the growing demand for developing speech-based interaction models, end-to-end Spoken Language Models (SLMs) have emerged as a promising solution. When engaging in conversations with humans, it is essential for these models to comprehend a wide range of world knowledge. In this paper, we introduce VoxEval, a novel speech question-answering benchmark specifically designed to assess SLMs' knowledge understanding through purely speech-based interactions. Unlike existing AudioQA benchmarks, VoxEval maintains speech format for both questions and answers, evaluates model robustness across diverse audio conditions (varying timbres, audio qualities, and speaking styles), and pioneers the assessment of challenging domains like mathematical problem-solving in spoken format. Our comprehensive evaluation of recent SLMs using VoxEval reveals significant performance limitations in current models, highlighting crucial areas for future improvements.

Title: Shelving it rather than Ditching it: Dynamically Debloating DEX and Native Methods of Android Applications without APK Modification

• Authors: Zicheng Zhang, Jiakun Liu, Ferdian Thung, Haoyu Ma, Rui Li, Yan Naing Tun, Wei Minn, Lwin Khin Shar, Shahar Maoz, Eran Toch, David Lo, Joshua Wong, Debin Gao
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.04963
• Pdf URL: https://arxiv.org/pdf/2501.04963
• Copy Paste: [[2501.04963]] Shelving it rather than Ditching it: Dynamically Debloating DEX and Native Methods of Android Applications without APK Modification(https://arxiv.org/abs/2501.04963)
• Keywords: security, attack
• Abstract: Today's Android developers tend to include numerous features to accommodate diverse user requirements, which inevitably leads to bloated apps. Yet more often than not, only a fraction of these features are frequently utilized by users, thus a bloated app costs dearly in potential vulnerabilities, expanded attack surfaces, and additional resource consumption. Especially in the event of severe security incidents, users have the need to block vulnerable functionalities immediately. Existing works have proposed various code debloating approaches for identifying and removing features of executable components. However, they typically involve static modification of files (and, for Android apps, repackaging of APKs, too), which lacks user convenience let alone undermining the security model of Android due to the compromising of public key verification and code integrity checks. This paper introduces 3DNDroid, a Dynamic Debloating approach targeting both DEX and Native methods in AnDroid apps. Using an unprivileged management app in tandem with a customized Android OS, 3DNDroid dynamically reduces unnecessary code loading during app execution based on a pre-generated debloating schema from static or dynamic analyses. It intercepts invocations of debloated bytecode methods to prevent their interpretation, compilation, and execution, while zero-filling memory spaces of debloated native methods during code loading. Evaluation demonstrates 3DNDroid's ability to debloat 187 DEX methods and 30 native methods across 55 real-world apps, removing over 10K Return-Oriented Programming (ROP) gadgets. Case studies confirm its effectiveness in mitigating vulnerabilities, and performance assessments highlight its resource-saving advantages over non-debloated apps.

Title: Battling the Non-stationarity in Time Series Forecasting via Test-time Adaptation

• Authors: HyunGi Kim, Siwon Kim, Jisoo Mok, Sungroh Yoon
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04970
• Pdf URL: https://arxiv.org/pdf/2501.04970
• Copy Paste: [[2501.04970]] Battling the Non-stationarity in Time Series Forecasting via Test-time Adaptation(https://arxiv.org/abs/2501.04970)
• Keywords: robust
• Abstract: Deep Neural Networks have spearheaded remarkable advancements in time series forecasting (TSF), one of the major tasks in time series modeling. Nonetheless, the non-stationarity of time series undermines the reliability of pre-trained source time series forecasters in mission-critical deployment settings. In this study, we introduce a pioneering test-time adaptation framework tailored for TSF (TSF-TTA). TAFAS, the proposed approach to TSF-TTA, flexibly adapts source forecasters to continuously shifting test distributions while preserving the core semantic information learned during pre-training. The novel utilization of partially-observed ground truth and gated calibration module enables proactive, robust, and model-agnostic adaptation of source forecasters. Experiments on diverse benchmark datasets and cutting-edge architectures demonstrate the efficacy and generality of TAFAS, especially in long-term forecasting scenarios that suffer from significant distribution shifts. The code is available at this https URL.

Title: V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer

• Authors: Hangzhou He, Lei Zhu, Xinliang Zhang, Shuang Zeng, Qian Chen, Yanye Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04975
• Pdf URL: https://arxiv.org/pdf/2501.04975
• Copy Paste: [[2501.04975]] V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer(https://arxiv.org/abs/2501.04975)
• Keywords: interpretability, large language model
• Abstract: Concept Bottleneck Models (CBMs) offer inherent interpretability by initially translating images into human-comprehensible concepts, followed by a linear combination of these concepts for classification. However, the annotation of concepts for visual recognition tasks requires extensive expert knowledge and labor, constraining the broad adoption of CBMs. Recent approaches have leveraged the knowledge of large language models to construct concept bottlenecks, with multimodal models like CLIP subsequently mapping image features into the concept feature space for classification. Despite this, the concepts produced by language models can be verbose and may introduce non-visual attributes, which hurts accuracy and interpretability. In this study, we investigate to avoid these issues by constructing CBMs directly from multimodal models. To this end, we adopt common words as base concept vocabulary and leverage auxiliary unlabeled images to construct a Vision-to-Concept (V2C) tokenizer that can explicitly quantize images into their most relevant visual concepts, thus creating a vision-oriented concept bottleneck tightly coupled with the multimodal model. This leads to our V2C-CBM which is training efficient and interpretable with high accuracy. Our V2C-CBM has matched or outperformed LLM-supervised CBMs on various visual classification benchmarks, validating the efficacy of our approach.

Title: SpaLLM-Guard: Pairing SMS Spam Detection Using Open-source and Commercial LLMs

• Authors: Muhammad Salman, Muhammad Ikram, Nardine Basta, Mohamed Ali Kaafar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.04985
• Pdf URL: https://arxiv.org/pdf/2501.04985
• Copy Paste: [[2501.04985]] SpaLLM-Guard: Pairing SMS Spam Detection Using Open-source and Commercial LLMs(https://arxiv.org/abs/2501.04985)
• Keywords: attack, robust, large language model
• Abstract: The increasing threat of SMS spam, driven by evolving adversarial techniques and concept drift, calls for more robust and adaptive detection methods. In this paper, we evaluate the potential of large language models (LLMs), both open-source and commercial, for SMS spam detection, comparing their performance across zero-shot, few-shot, fine-tuning, and chain-of-thought prompting approaches. Using a comprehensive dataset of SMS messages, we assess the spam detection capabilities of prominent LLMs such as GPT-4, DeepSeek, LLAMA-2, and Mixtral. Our findings reveal that while zero-shot learning provides convenience, it is unreliable for effective spam detection. Few-shot learning, particularly with carefully selected examples, improves detection but exhibits variability across models. Fine-tuning emerges as the most effective strategy, with Mixtral achieving 98.6% accuracy and a balanced false positive and false negative rate below 2%, meeting the criteria for robust spam detection. Furthermore, we explore the resilience of these models to adversarial attacks, finding that fine-tuning significantly enhances robustness against both perceptible and imperceptible manipulations. Lastly, we investigate the impact of concept drift and demonstrate that fine-tuned LLMs, especially when combined with few-shot learning, can mitigate its effects, maintaining high performance even on evolving spam datasets. This study highlights the importance of fine-tuning and tailored learning strategies to deploy LLMs effectively for real-world SMS spam detection

Title: TreeKV: Smooth Key-Value Cache Compression with Tree Structures

• Authors: Ziwei He, Jian Yuan, Haoli Bai, Jingwen Leng, Bo Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04987
• Pdf URL: https://arxiv.org/pdf/2501.04987
• Copy Paste: [[2501.04987]] TreeKV: Smooth Key-Value Cache Compression with Tree Structures(https://arxiv.org/abs/2501.04987)
• Keywords: transformer, large language model
• Abstract: Efficient key-value (KV) cache compression is critical for scaling transformer-based Large Language Models (LLMs) in long sequences and resource-limited settings. Existing methods evict tokens based on their positions or importance scores, but position-based strategies can miss crucial information outside predefined regions, while those relying on global importance scores resulting in strong regional biases, limiting the KV cache's overall context retention and potentially impairing the performance of LLMs on complex tasks. Our wavelet analysis reveals that as tokens approach the end of sequence, their contributions to generation gradually increase and tends to diverge more from neighboring tokens, indicating a smooth transition with increasing complexity and variability from distant to nearby context. Motivated by this observation, we propose TreeKV, an intuitive, training-free method that employs a tree structure for smooth cache compression. TreeKV maintains a fixed cache size, allowing LLMs to deliver high-quality output even in long text scenarios. Unlike most compression methods, TreeKV is applicable to both the generation and prefilling stages. It consistently surpasses all baseline models in language modeling tasks on PG19 and OpenWebText2, allowing LLMs trained with short context window to generalize to longer window with a 16x cache reduction. On the Longbench benchmark, TreeKV achieves the best performance with only 6\% of the budget at optimal efficiency.

Title: IPDN: Image-enhanced Prompt Decoding Network for 3D Referring Expression Segmentation

• Authors: Qi Chen, Changli Wu, Jiayi Ji, Yiwei Ma, Danni Yang, Xiaoshuai Sun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04995
• Pdf URL: https://arxiv.org/pdf/2501.04995
• Copy Paste: [[2501.04995]] IPDN: Image-enhanced Prompt Decoding Network for 3D Referring Expression Segmentation(https://arxiv.org/abs/2501.04995)
• Keywords: segmentation
• Abstract: 3D Referring Expression Segmentation (3D-RES) aims to segment point cloud scenes based on a given expression. However, existing 3D-RES approaches face two major challenges: feature ambiguity and intent ambiguity. Feature ambiguity arises from information loss or distortion during point cloud acquisition due to limitations such as lighting and viewpoint. Intent ambiguity refers to the model's equal treatment of all queries during the decoding process, lacking top-down task-specific guidance. In this paper, we introduce an Image enhanced Prompt Decoding Network (IPDN), which leverages multi-view images and task-driven information to enhance the model's reasoning capabilities. To address feature ambiguity, we propose the Multi-view Semantic Embedding (MSE) module, which injects multi-view 2D image information into the 3D scene and compensates for potential spatial information loss. To tackle intent ambiguity, we designed a Prompt-Aware Decoder (PAD) that guides the decoding process by deriving task-driven signals from the interaction between the expression and visual features. Comprehensive experiments demonstrate that IPDN outperforms the state-ofthe-art by 1.9 and 4.2 points in mIoU metrics on the 3D-RES and 3D-GRES tasks, respectively.

Title: A CT Image Classification Network Framework for Lung Tumors Based on Pre-trained MobileNetV2 Model and Transfer learning, And Its Application and Market Analysis in the Medical field

• Authors: Ziyang Gao, Yong Tian, Shih-Chi Lin, Junghua Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04996
• Pdf URL: https://arxiv.org/pdf/2501.04996
• Copy Paste: [[2501.04996]] A CT Image Classification Network Framework for Lung Tumors Based on Pre-trained MobileNetV2 Model and Transfer learning, And Its Application and Market Analysis in the Medical field(https://arxiv.org/abs/2501.04996)
• Keywords: extraction
• Abstract: In the medical field, accurate diagnosis of lung cancer is crucial for treatment. Traditional manual analysis methods have significant limitations in terms of accuracy and efficiency. To address this issue, this paper proposes a deep learning network framework based on the pre-trained MobileNetV2 model, initialized with weights from the ImageNet-1K dataset (version 2). The last layer of the model (the fully connected layer) is replaced with a new fully connected layer, and a softmax activation function is added to efficiently classify three types of lung cancer CT scan images. Experimental results show that the model achieves an accuracy of 99.6% on the test set, with significant improvements in feature extraction compared to traditional this http URL the rapid development of artificial intelligence technologies, deep learning applications in medical image processing are bringing revolutionary changes to the healthcare industry. AI-based lung cancer detection systems can significantly improve diagnostic efficiency, reduce the workload of doctors, and occupy an important position in the global healthcare market. The potential of AI to improve diagnostic accuracy, reduce medical costs, and promote precision medicine will have a profound impact on the future development of the healthcare industry.

Title: Load Forecasting for Households and Energy Communities: Are Deep Learning Models Worth the Effort?

• Authors: Lukas Moosbrugger, Valentin Seiler, Philipp Wohlgenannt, Sebastian Hegenbart, Sashko Ristov, Peter Kepplinger
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05000
• Pdf URL: https://arxiv.org/pdf/2501.05000
• Copy Paste: [[2501.05000]] Load Forecasting for Households and Energy Communities: Are Deep Learning Models Worth the Effort?(https://arxiv.org/abs/2501.05000)
• Keywords: transformer
• Abstract: Accurate load forecasting is crucial for predictive control in many energy domain applications, with significant economic and ecological implications. To address these implications, this study provides an extensive benchmark of state-of-the-art deep learning models for short-term load forecasting in energy communities. Namely, LSTM, xLSTM, and Transformers are compared with benchmarks such as KNNs, synthetic load models, and persistence forecasting models. This comparison considers different scales of aggregation (e.g., number of household loads) and varying training data availability (e.g., training data time spans). Further, the impact of transfer learning from synthetic (standard) load profiles and the deep learning model size (i.e., parameter count) is investigated in terms of forecasting error. Implementations are publicly available and other researchers are encouraged to benchmark models using this framework. Additionally, a comprehensive case study, comprising an energy community of 50 households and a battery storage demonstrates the beneficial financial implications of accurate predictions. Key findings of this research include: (1) Simple persistence benchmarks outperform deep learning models for short-term load forecasting when the available training data is limited to six months or less; (2) Pretraining with publicly available synthetic load profiles improves the normalized Mean Absolute Error (nMAE) by an average of 1.28%pt during the first nine months of training data; (3) Increased aggregation significantly enhances the performance of deep learning models relative to persistence benchmarks; (4) Improved load forecasting, with an nMAE reduction of 1.1%pt, translates to an economic benefit of approximately 600EUR per year in an energy community comprising 50 households.

Title: On Measuring Unnoticeability of Graph Adversarial Attacks: Observations, New Measure, and Applications

• Authors: Hyeonsoo Jo, Hyunjin Hwang, Fanchen Bu, Soo Yong Lee, Chanyoung Park, Kijung Shin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05015
• Pdf URL: https://arxiv.org/pdf/2501.05015
• Copy Paste: [[2501.05015]] On Measuring Unnoticeability of Graph Adversarial Attacks: Observations, New Measure, and Applications(https://arxiv.org/abs/2501.05015)
• Keywords: attack, robust
• Abstract: Adversarial attacks are allegedly unnoticeable. Prior studies have designed attack noticeability measures on graphs, primarily using statistical tests to compare the topology of original and (possibly) attacked graphs. However, we observe two critical limitations in the existing measures. First, because the measures rely on simple rules, attackers can readily enhance their attacks to bypass them, reducing their attack "noticeability" and, yet, maintaining their attack performance. Second, because the measures naively leverage global statistics, such as degree distributions, they may entirely overlook attacks until severe perturbations occur, letting the attacks be almost "totally unnoticeable." To address the limitations, we introduce HideNSeek, a learnable measure for graph attack noticeability. First, to mitigate the bypass problem, HideNSeek learns to distinguish the original and (potential) attack edges using a learnable edge scorer (LEO), which scores each edge on its likelihood of being an attack. Second, to mitigate the overlooking problem, HideNSeek conducts imbalance-aware aggregation of all the edge scores to obtain the final noticeability score. Using six real-world graphs, we empirically demonstrate that HideNSeek effectively alleviates the observed limitations, and LEO (i.e., our learnable edge scorer) outperforms eleven competitors in distinguishing attack edges under five different attack methods. For an additional application, we show that LEO boost the performance of robust GNNs by removing attack-like edges.

Title: ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark

• Authors: Ronghao Dang, Yuqian Yuan, Wenqi Zhang, Yifei Xin, Boqiang Zhang, Long Li, Liuyi Wang, Qinyang Zeng, Xin Li, Lidong Bing
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.05031
• Pdf URL: https://arxiv.org/pdf/2501.05031
• Copy Paste: [[2501.05031]] ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark(https://arxiv.org/abs/2501.05031)
• Keywords: fair
• Abstract: The enhancement of generalization in robots by large vision-language models (LVLMs) is increasingly evident. Therefore, the embodied cognitive abilities of LVLMs based on egocentric videos are of great interest. However, current datasets for embodied video question answering lack comprehensive and systematic evaluation frameworks. Critical embodied cognitive issues, such as robotic self-cognition, dynamic scene perception, and hallucination, are rarely addressed. To tackle these challenges, we propose ECBench, a high-quality benchmark designed to systematically evaluate the embodied cognitive abilities of LVLMs. ECBench features a diverse range of scene video sources, open and varied question formats, and 30 dimensions of embodied cognition. To ensure quality, balance, and high visual dependence, ECBench uses class-independent meticulous human annotation and multi-round question screening strategies. Additionally, we introduce ECEval, a comprehensive evaluation system that ensures the fairness and rationality of the indicators. Utilizing ECBench, we conduct extensive evaluations of proprietary, open-source, and task-specific LVLMs. ECBench is pivotal in advancing the embodied cognitive capabilities of LVLMs, laying a solid foundation for developing reliable core models for embodied agents. All data and code are available at this https URL.

Title: Enhancing Human-Like Responses in Large Language Models

• Authors: Ethem Yağız Çalık, Talha Rüzgar Akkuş
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05032
• Pdf URL: https://arxiv.org/pdf/2501.05032
• Copy Paste: [[2501.05032]] Enhancing Human-Like Responses in Large Language Models(https://arxiv.org/abs/2501.05032)
• Keywords: large language model
• Abstract: This paper explores the advancements in making large language models (LLMs) more human-like. We focus on techniques that enhance natural language understanding, conversational coherence, and emotional intelligence in AI systems. The study evaluates various approaches, including fine-tuning with diverse datasets, incorporating psychological principles, and designing models that better mimic human reasoning patterns. Our findings demonstrate that these enhancements not only improve user interactions but also open new possibilities for AI applications across different domains. Future work will address the ethical implications and potential biases introduced by these human-like attributes.

Title: Towards Fingerprint Mosaicking Artifact Detection: A Self-Supervised Deep Learning Approach

• Authors: Laurenz Ruzicka, Alexander Spenke, Stephan Bergmann, Gerd Nolden, Bernhard Kohn, Clemens Heitzinger
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05034
• Pdf URL: https://arxiv.org/pdf/2501.05034
• Copy Paste: [[2501.05034]] Towards Fingerprint Mosaicking Artifact Detection: A Self-Supervised Deep Learning Approach(https://arxiv.org/abs/2501.05034)
• Keywords: robust, biometric
• Abstract: Fingerprint mosaicking, which is the process of combining multiple fingerprint images into a single master fingerprint, is an essential process in modern biometric systems. However, it is prone to errors that can significantly degrade fingerprint image quality. This paper proposes a novel deep learning-based approach to detect and score mosaicking artifacts in fingerprint images. Our method leverages a self-supervised learning framework to train a model on large-scale unlabeled fingerprint data, eliminating the need for manual artifact annotation. The proposed model effectively identifies mosaicking errors, achieving high accuracy on various fingerprint modalities, including contactless, rolled, and pressed fingerprints and furthermore proves to be robust to different data sources. Additionally, we introduce a novel mosaicking artifact score to quantify the severity of errors, enabling automated evaluation of fingerprint images. By addressing the challenges of mosaicking artifact detection, our work contributes to improving the accuracy and reliability of fingerprint-based biometric systems.

Title: LongViTU: Instruction Tuning for Long-Form Video Understanding

• Authors: Rujie Wu, Xiaojian Ma, Hai Ci, Yue Fan, Yuxuan Wang, Haozhe Zhao, Qing Li, Yizhou Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05037
• Pdf URL: https://arxiv.org/pdf/2501.05037
• Copy Paste: [[2501.05037]] LongViTU: Instruction Tuning for Long-Form Video Understanding(https://arxiv.org/abs/2501.05037)
• Keywords: robust
• Abstract: This paper introduce LongViTU, a large-scale (~121k QA pairs, ~900h videos), automatically generated dataset for long-form video understanding. We developed a systematic approach that organizes videos into a hierarchical tree structure and incorporates self-revision mechanisms to ensure high-quality QA pairs. Each QA pair in LongViTU features: 1) long-term context (average certificate length of 4.6 minutes); 2) rich knowledge and condensed reasoning (commonsense, causality, planning, etc.); and 3) explicit timestamp labels for relevant events. LongViTU also serves as a benchmark for instruction following in long-form and streaming video understanding. We evaluate the open-source state-of-the-art long video understanding model, LongVU, and the commercial model, Gemini-1.5-Pro, on our benchmark. They achieve GPT-4 scores of 49.9 and 52.3, respectively, underscoring the substantial challenge posed by our benchmark. Further supervised fine-tuning (SFT) on LongVU led to performance improvements of 12.0% on our benchmark, 2.2% on the in-distribution (ID) benchmark EgoSchema, 1.0%, 2.2% and 1.2% on the out-of-distribution (OOD) benchmarks VideoMME (Long), WorldQA and OpenEQA, respectively. These outcomes demonstrate LongViTU's high data quality and robust OOD generalizability.

Title: SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution

• Authors: Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, Kai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05040
• Pdf URL: https://arxiv.org/pdf/2501.05040
• Copy Paste: [[2501.05040]] SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution(https://arxiv.org/abs/2501.05040)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable proficiency across a variety of complex tasks. One significant application of LLMs is in tackling software engineering challenges, particularly in resolving real-world tasks on GitHub by fixing code based on the issues reported by the users. However, many current approaches rely on proprietary LLMs, which limits reproducibility, accessibility, and transparency. The critical components of LLMs for addressing software engineering issues and how their capabilities can be effectively enhanced remain unclear. To address these challenges, we introduce SWE-Fixer, a novel open-source LLM designed to effectively and efficiently resolve GitHub issues. SWE-Fixer comprises two essential modules: a code file retrieval module and a code editing module. The retrieval module employs BM25 along with a lightweight LLM model to achieve coarse-to-fine file retrieval. Subsequently, the code editing module utilizes the other LLM model to generate patches for the identified files. Then, to mitigate the lack of publicly available datasets, we compile an extensive dataset that includes 110K GitHub issues along with their corresponding patches, and train the two modules of SWE-Fixer separately. We assess our approach on the SWE-Bench Lite and Verified benchmarks, achieving state-of-the-art performance among open-source models with scores of 23.3% and 30.2%, respectively. These outcomes highlight the efficacy of our approach. We will make our model, dataset, and code publicly available at this https URL.

Title: TAPFed: Threshold Secure Aggregation for Privacy-Preserving Federated Learning

• Authors: Runhua Xu, Bo Li, Chao Li, James B.D. Joshi, Shuai Ma, Jianxin Li
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05053
• Pdf URL: https://arxiv.org/pdf/2501.05053
• Copy Paste: [[2501.05053]] TAPFed: Threshold Secure Aggregation for Privacy-Preserving Federated Learning(https://arxiv.org/abs/2501.05053)
• Keywords: secure, security, privacy, attack, federate
• Abstract: Federated learning is a computing paradigm that enhances privacy by enabling multiple parties to collaboratively train a machine learning model without revealing personal data. However, current research indicates that traditional federated learning platforms are unable to ensure privacy due to privacy leaks caused by the interchange of gradients. To achieve privacy-preserving federated learning, integrating secure aggregation mechanisms is essential. Unfortunately, existing solutions are vulnerable to recently demonstrated inference attacks such as the disaggregation attack. This paper proposes TAPFed, an approach for achieving privacy-preserving federated learning in the context of multiple decentralized aggregators with malicious actors. TAPFed uses a proposed threshold functional encryption scheme and allows for a certain number of malicious aggregators while maintaining security and privacy. We provide formal security and privacy analyses of TAPFed and compare it to various baselines through experimental evaluation. Our results show that TAPFed offers equivalent performance in terms of model quality compared to state-of-the-art approaches while reducing transmission overhead by 29%-45% across different model training scenarios. Most importantly, TAPFed can defend against recently demonstrated inference attacks caused by curious aggregators, which the majority of existing approaches are susceptible to.

Title: Improving Skeleton-based Action Recognition with Interactive Object Information

• Authors: Hao Wen, Ziqian Lu, Fengli Shen, Zhe-Ming Lu, Jialin Cui
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05066
• Pdf URL: https://arxiv.org/pdf/2501.05066
• Copy Paste: [[2501.05066]] Improving Skeleton-based Action Recognition with Interactive Object Information(https://arxiv.org/abs/2501.05066)
• Keywords: attack
• Abstract: Human skeleton information is important in skeleton-based action recognition, which provides a simple and efficient way to describe human pose. However, existing skeleton-based methods focus more on the skeleton, ignoring the objects interacting with humans, resulting in poor performance in recognizing actions that involve object interactions. We propose a new action recognition framework introducing object nodes to supplement absent interactive object information. We also propose Spatial Temporal Variable Graph Convolutional Networks (ST-VGCN) to effectively model the Variable Graph (VG) containing object nodes. Specifically, in order to validate the role of interactive object information, by leveraging a simple self-training approach, we establish a new dataset, JXGC 24, and an extended dataset, NTU RGB+D+Object 60, including more than 2 million additional object nodes. At the same time, we designe the Variable Graph construction method to accommodate a variable number of nodes for graph structure. Additionally, we are the first to explore the overfitting issue introduced by incorporating additional object information, and we propose a VG-based data augmentation method to address this issue, called Random Node Attack. Finally, regarding the network structure, we introduce two fusion modules, CAF and WNPool, along with a novel Node Balance Loss, to enhance the comprehensive performance by effectively fusing and balancing skeleton and object node information. Our method surpasses the previous state-of-the-art on multiple skeleton-based action recognition benchmarks. The accuracy of our method on NTU RGB+D 60 cross-subject split is 96.7\%, and on cross-view split, it is 99.2\%.

Title: LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understanding

• Authors: Jiaxing Zhao, Boyuan Sun, Xiang Chen, Xihan Wei, Qibin Hou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05067
• Pdf URL: https://arxiv.org/pdf/2501.05067
• Copy Paste: [[2501.05067]] LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understanding(https://arxiv.org/abs/2501.05067)
• Keywords: large language model
• Abstract: In this paper, we introduce LLaVA-Octopus, a novel video multimodal large language model. LLaVA-Octopus adaptively weights features from different visual projectors based on user instructions, enabling us to leverage the complementary strengths of each projector. We observe that different visual projectors exhibit distinct characteristics when handling specific tasks. For instance, some projectors excel at capturing static details, while others are more effective at processing temporal information, and some are better suited for tasks requiring temporal coherence. By dynamically adjusting feature weights according to user instructions, LLaVA-Octopus dynamically selects and combines the most suitable features, significantly enhancing the model's performance in multimodal tasks. Experimental results demonstrate that LLaVA-Octopus achieves excellent performance across multiple benchmarks, especially in tasks such as multimodal understanding, visual question answering, and video understanding, highlighting its broad application potential.

Title: Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning

• Authors: Huabin Liu, Filip Ilievski, Cees G. M. Snoek
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05069
• Pdf URL: https://arxiv.org/pdf/2501.05069
• Copy Paste: [[2501.05069]] Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning(https://arxiv.org/abs/2501.05069)
• Keywords: fair
• Abstract: This paper proposes the first video-grounded entailment tree reasoning method for commonsense video question answering (VQA). Despite the remarkable progress of large visual-language models (VLMs), there are growing concerns that they learn spurious correlations between videos and likely answers, reinforced by their black-box nature and remaining benchmarking biases. Our method explicitly grounds VQA tasks to video fragments in four steps: entailment tree construction, video-language entailment verification, tree reasoning, and dynamic tree expansion. A vital benefit of the method is its generalizability to current video and image-based VLMs across reasoning types. To support fair evaluation, we devise a de-biasing procedure based on large-language models that rewrites VQA benchmark answer sets to enforce model reasoning. Systematic experiments on existing and de-biased benchmarks highlight the impact of our method components across benchmarks, VLMs, and reasoning types.

Title: TipSegNet: Fingertip Segmentation in Contactless Fingerprint Imaging

• Authors: Laurenz Ruzicka, Bernhard Kohn, Clemens Heitzinger
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05076
• Pdf URL: https://arxiv.org/pdf/2501.05076
• Copy Paste: [[2501.05076]] TipSegNet: Fingertip Segmentation in Contactless Fingerprint Imaging(https://arxiv.org/abs/2501.05076)
• Keywords: robust, biometric, extraction, segmentation
• Abstract: Contactless fingerprint recognition systems offer a hygienic, user-friendly, and efficient alternative to traditional contact-based methods. However, their accuracy heavily relies on precise fingertip detection and segmentation, particularly under challenging background conditions. This paper introduces TipSegNet, a novel deep learning model that achieves state-of-the-art performance in segmenting fingertips directly from grayscale hand images. TipSegNet leverages a ResNeXt-101 backbone for robust feature extraction, combined with a Feature Pyramid Network (FPN) for multi-scale representation, enabling accurate segmentation across varying finger poses and image qualities. Furthermore, we employ an extensive data augmentation strategy to enhance the model's generalizability and robustness. TipSegNet outperforms existing methods, achieving a mean Intersection over Union (mIoU) of 0.987 and an accuracy of 0.999, representing a significant advancement in contactless fingerprint segmentation. This enhanced accuracy has the potential to substantially improve the reliability and effectiveness of contactless biometric systems in real-world applications.

Title: Analyzing Memorization in Large Language Models through the Lens of Model Attribution

• Authors: Tarun Ram Menta, Susmit Agrawal, Chirag Agarwal
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05078
• Pdf URL: https://arxiv.org/pdf/2501.05078
• Copy Paste: [[2501.05078]] Analyzing Memorization in Large Language Models through the Lens of Model Attribution(https://arxiv.org/abs/2501.05078)
• Keywords: privacy, transformer, large language model
• Abstract: Large Language Models (LLMs) are prevalent in modern applications but often memorize training data, leading to privacy breaches and copyright issues. Existing research has mainly focused on posthoc analyses, such as extracting memorized content or developing memorization metrics, without exploring the underlying architectural factors that contribute to memorization. In this work, we investigate memorization from an architectural lens by analyzing how attention modules at different layers impact its memorization and generalization performance. Using attribution techniques, we systematically intervene in the LLM architecture by bypassing attention modules at specific blocks while keeping other components like layer normalization and MLP transformations intact. We provide theorems analyzing our intervention mechanism from a mathematical view, bounding the difference in layer outputs with and without our attributions. Our theoretical and empirical analyses reveal that attention modules in deeper transformer blocks are primarily responsible for memorization, whereas earlier blocks are crucial for the models generalization and reasoning capabilities. We validate our findings through comprehensive experiments on different LLM families (Pythia and GPTNeo) and five benchmark datasets. Our insights offer a practical approach to mitigate memorization in LLMs while preserving their performance, contributing to safer and more ethical deployment in real world applications.

Title: DriVLM: Domain Adaptation of Vision-Language Models in Autonomous Driving

• Authors: Xuran Zheng, Chang D. Yoo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05081
• Pdf URL: https://arxiv.org/pdf/2501.05081
• Copy Paste: [[2501.05081]] DriVLM: Domain Adaptation of Vision-Language Models in Autonomous Driving(https://arxiv.org/abs/2501.05081)
• Keywords: large language model
• Abstract: In recent years, large language models have had a very impressive performance, which largely contributed to the development and application of artificial intelligence, and the parameters and performance of the models are still growing rapidly. In particular, multimodal large language models (MLLM) can combine multiple modalities such as pictures, videos, sounds, texts, etc., and have great potential in various tasks. However, most MLLMs require very high computational resources, which is a major challenge for most researchers and developers. In this paper, we explored the utility of small-scale MLLMs and applied small-scale MLLMs to the field of autonomous driving. We hope that this will advance the application of MLLMs in real-world scenarios.

Title: ResPanDiff: Diffusion Model with Disentangled Modulations for Image Fusion

• Authors: Shiqi Cao, Liangjian Deng, Shangqi Deng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05091
• Pdf URL: https://arxiv.org/pdf/2501.05091
• Copy Paste: [[2501.05091]] ResPanDiff: Diffusion Model with Disentangled Modulations for Image Fusion(https://arxiv.org/abs/2501.05091)
• Keywords: diffusion
• Abstract: The implementation of diffusion-based pansharpening task is predominantly constrained by its slow inference speed, which results from numerous sampling steps. Despite the existing techniques aiming to accelerate sampling, they often compromise performance when fusing multi-source images. To ease this limitation, we introduce a novel and efficient diffusion model named Diffusion Model for Pansharpening by Inferring Residual Inference (ResPanDiff), which significantly reduces the number of diffusion steps without sacrificing the performance to tackle pansharpening task. In ResPanDiff, we innovatively propose a Markov chain that transits from noisy residuals to the residuals between the LRMS and HRMS images, thereby reducing the number of sampling steps and enhancing performance. Additionally, we design the latent space to help model extract more features at the encoding stage, Shallow Cond-Injection~(SC-I) to help model fetch cond-injected hidden features with higher dimensions, and loss functions to give a better guidance for the residual generation task. enabling the model to achieve superior performance in residual generation. Furthermore, experimental evaluations on pansharpening datasets demonstrate that the proposed method achieves superior outcomes compared to recent state-of-the-art~(SOTA) techniques, requiring only 15 sampling steps, which reduces over $90\%$ step compared with the benchmark diffusion models. Our experiments also include thorough discussions and ablation studies to underscore the effectiveness of our approach.

Title: Advancing ALS Applications with Large-Scale Pre-training: Dataset Development and Downstream Assessment

• Authors: Haoyi Xiu, Xin Liu, Taehoon Kim, Kyoung-Sook Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05095
• Pdf URL: https://arxiv.org/pdf/2501.05095
• Copy Paste: [[2501.05095]] Advancing ALS Applications with Large-Scale Pre-training: Dataset Development and Downstream Assessment(https://arxiv.org/abs/2501.05095)
• Keywords: segmentation
• Abstract: The pre-training and fine-tuning paradigm has revolutionized satellite remote sensing applications. However, this approach remains largely underexplored for airborne laser scanning (ALS), an important technology for applications such as forest management and urban planning. In this study, we address this gap by constructing a large-scale ALS point cloud dataset and evaluating its impact on downstream applications. Our dataset comprises ALS point clouds collected across the contiguous United States, provided by the United States Geological Survey's 3D Elevation Program. To ensure efficient data collection while capturing diverse land cover and terrain types, we introduce a geospatial sampling method that selects point cloud tiles based on land cover maps and digital elevation models. As a baseline self-supervised learning model, we adopt BEV-MAE, a state-of-the-art masked autoencoder for 3D outdoor point clouds, and pre-train it on the constructed dataset. The pre-trained models are subsequently fine-tuned for downstream tasks, including tree species classification, terrain scene recognition, and point cloud semantic segmentation. Our results show that the pre-trained models significantly outperform their scratch counterparts across all downstream tasks, demonstrating the transferability of the representations learned from the proposed dataset. Furthermore, we observe that scaling the dataset using our geospatial sampling method consistently enhances performance, whereas pre-training on datasets constructed with random sampling fails to achieve similar improvements. These findings highlight the utility of the constructed dataset and the effectiveness of our sampling strategy in the pre-training and fine-tuning paradigm. The source code and pre-trained models will be made publicly available at \url{this https URL}.

Title: Optimizing Multitask Industrial Processes with Predictive Action Guidance

• Authors: Naval Kishore Mehta, Arvind, Shyam Sunder Prasad, Sumeet Saurav, Sanjay Singh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05108
• Pdf URL: https://arxiv.org/pdf/2501.05108
• Copy Paste: [[2501.05108]] Optimizing Multitask Industrial Processes with Predictive Action Guidance(https://arxiv.org/abs/2501.05108)
• Keywords: transformer
• Abstract: Monitoring complex assembly processes is critical for maintaining productivity and ensuring compliance with assembly standards. However, variability in human actions and subjective task preferences complicate accurate task anticipation and guidance. To address these challenges, we introduce the Multi-Modal Transformer Fusion and Recurrent Units (MMTFRU) Network for egocentric activity anticipation, utilizing multimodal fusion to improve prediction accuracy. Integrated with the Operator Action Monitoring Unit (OAMU), the system provides proactive operator guidance, preventing deviations in the assembly process. OAMU employs two strategies: (1) Top-5 MMTF-RU predictions, combined with a reference graph and an action dictionary, for next-step recommendations; and (2) Top-1 MMTF-RU predictions, integrated with a reference graph, for detecting sequence deviations and predicting anomaly scores via an entropy-informed confidence mechanism. We also introduce Time-Weighted Sequence Accuracy (TWSA) to evaluate operator efficiency and ensure timely task completion. Our approach is validated on the industrial Meccano dataset and the largescale EPIC-Kitchens-55 dataset, demonstrating its effectiveness in dynamic environments.

Title: EquiBoost: An Equivariant Boosting Approach to Molecular Conformation Generation

• Authors: Yixuan Yang, Xingyu Fang, Zhaowen Cheng, Pengju Yan, Xiaolin Li
• Subjects: cs.LG, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2501.05109
• Pdf URL: https://arxiv.org/pdf/2501.05109
• Copy Paste: [[2501.05109]] EquiBoost: An Equivariant Boosting Approach to Molecular Conformation Generation(https://arxiv.org/abs/2501.05109)
• Keywords: robust, diffusion, transformer
• Abstract: Molecular conformation generation plays key roles in computational drug design. Recently developed deep learning methods, particularly diffusion models have reached competitive performance over traditional cheminformatical approaches. However, these methods are often time-consuming or require extra support from traditional methods. We propose EquiBoost, a boosting model that stacks several equivariant graph transformers as weak learners, to iteratively refine 3D conformations of molecules. Without relying on diffusion techniques, EquiBoost balances accuracy and efficiency more effectively than diffusion-based methods. Notably, compared to the previous state-of-the-art diffusion method, EquiBoost improves generation quality and preserves diversity, achieving considerably better precision of Average Minimum RMSD (AMR) on the GEOM datasets. This work rejuvenates boosting and sheds light on its potential to be a robust alternative to diffusion models in certain scenarios.

Title: Learning In-Distribution Representations for Anomaly Detection

• Authors: William T. Lunardi, Abdulrahman Banabila, Dania Herzalla, Martin L. Andreoni
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05130
• Pdf URL: https://arxiv.org/pdf/2501.05130
• Copy Paste: [[2501.05130]] Learning In-Distribution Representations for Anomaly Detection(https://arxiv.org/abs/2501.05130)
• Keywords: robust
• Abstract: Anomaly detection involves identifying data patterns that deviate from the anticipated norm. Traditional methods struggle in high-dimensional spaces due to the curse of dimensionality. In recent years, self-supervised learning, particularly through contrastive objectives, has driven advances in anomaly detection. However, vanilla contrastive learning struggles to align with the unique demands of anomaly detection, as it lacks a pretext task tailored to the homogeneous nature of In-Distribution (ID) data and the diversity of Out-of-Distribution (OOD) anomalies. Methods that attempt to address these challenges, such as introducing hard negatives through synthetic outliers, Outlier Exposure (OE), and supervised objectives, often rely on pretext tasks that fail to balance compact clustering of ID samples with sufficient separation from OOD data. In this work, we propose Focused In-distribution Representation Modeling (FIRM), a contrastive learning objective specifically designed for anomaly detection. Unlike existing approaches, FIRM incorporates synthetic outliers into its pretext task in a way that actively shapes the representation space, promoting compact clustering of ID samples while enforcing strong separation from outliers. This formulation addresses the challenges of class collision, enhancing both the compactness of ID representations and the discriminative power of the learned feature space. We show that FIRM surpasses other contrastive methods in standard benchmarks, significantly enhancing anomaly detection compared to both traditional and supervised contrastive learning objectives. Our ablation studies confirm that FIRM consistently improves the quality of representations and shows robustness across a range of scoring methods. The code is available at: this https URL.

Title: CorrDiff: Adaptive Delay-aware Detector with Temporal Cue Inputs for Real-time Object Detection

• Authors: Xiang Zhang, Chenchen Fu, Yufei Cui, Lan Yi, Yuyang Sun, Weiwei Wu, Xue Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05132
• Pdf URL: https://arxiv.org/pdf/2501.05132
• Copy Paste: [[2501.05132]] CorrDiff: Adaptive Delay-aware Detector with Temporal Cue Inputs for Real-time Object Detection(https://arxiv.org/abs/2501.05132)
• Keywords: robust
• Abstract: Real-time object detection takes an essential part in the decision-making process of numerous real-world applications, including collision avoidance and path planning in autonomous driving systems. This paper presents a novel real-time streaming perception method named CorrDiff, designed to tackle the challenge of delays in real-time detection systems. The main contribution of CorrDiff lies in its adaptive delay-aware detector, which is able to utilize runtime-estimated temporal cues to predict objects' locations for multiple future frames, and selectively produce predictions that matches real-world time, effectively compensating for any communication and computational delays. The proposed model outperforms current state-of-the-art methods by leveraging motion estimation and feature enhancement, both for 1) single-frame detection for the current frame or the next frame, in terms of the metric mAP, and 2) the prediction for (multiple) future frame(s), in terms of the metric sAP (The sAP metric is to evaluate object detection algorithms in streaming scenarios, factoring in both latency and accuracy). It demonstrates robust performance across a range of devices, from powerful Tesla V100 to modest RTX 2080Ti, achieving the highest level of perceptual accuracy on all platforms. Unlike most state-of-the-art methods that struggle to complete computation within a single frame on less powerful devices, CorrDiff meets the stringent real-time processing requirements on all kinds of devices. The experimental results emphasize the system's adaptability and its potential to significantly improve the safety and reliability for many real-world systems, such as autonomous driving. Our code is completely open-sourced and is available at this https URL.

Title: Biomedical Relation Extraction via Adaptive Document-Relation Cross-Mapping and Concept Unique Identifier

• Authors: Yufei Shang, Yanrong Guo, Shijie Hao, Richang Hong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05155
• Pdf URL: https://arxiv.org/pdf/2501.05155
• Copy Paste: [[2501.05155]] Biomedical Relation Extraction via Adaptive Document-Relation Cross-Mapping and Concept Unique Identifier(https://arxiv.org/abs/2501.05155)
• Keywords: extraction, large language model
• Abstract: Document-Level Biomedical Relation Extraction (Bio-RE) aims to identify relations between biomedical entities within extensive texts, serving as a crucial subfield of biomedical text mining. Existing Bio-RE methods struggle with cross-sentence inference, which is essential for capturing relations spanning multiple sentences. Moreover, previous methods often overlook the incompleteness of documents and lack the integration of external knowledge, limiting contextual richness. Besides, the scarcity of annotated data further hampers model training. Recent advancements in large language models (LLMs) have inspired us to explore all the above issues for document-level Bio-RE. Specifically, we propose a document-level Bio-RE framework via LLM Adaptive Document-Relation Cross-Mapping (ADRCM) Fine-Tuning and Concept Unique Identifier (CUI) Retrieval-Augmented Generation (RAG). First, we introduce the Iteration-of-REsummary (IoRs) prompt for solving the data scarcity issue. In this way, Bio-RE task-specific synthetic data can be generated by guiding ChatGPT to focus on entity relations and iteratively refining synthetic data. Next, we propose ADRCM fine-tuning, a novel fine-tuning recipe that establishes mappings across different documents and relations, enhancing the model's contextual understanding and cross-sentence inference capabilities. Finally, during the inference, a biomedical-specific RAG approach, named CUI RAG, is designed to leverage CUIs as indexes for entities, narrowing the retrieval scope and enriching the relevant document contexts. Experiments conducted on three Bio-RE datasets (GDA, CDR, and BioRED) demonstrate the state-of-the-art performance of our proposed method by comparing it with other related works.

Title: FaceMe: Robust Blind Face Restoration with Personal Identification

• Authors: Siyu Liu, Zheng-Peng Duan, Jia OuYang, Jiayi Fu, Hyunhee Park, Zikun Liu, Chun-Le Guo, Chongyi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05177
• Pdf URL: https://arxiv.org/pdf/2501.05177
• Copy Paste: [[2501.05177]] FaceMe: Robust Blind Face Restoration with Personal Identification(https://arxiv.org/abs/2501.05177)
• Keywords: robust, diffusion
• Abstract: Blind face restoration is a highly ill-posed problem due to the lack of necessary context. Although existing methods produce high-quality outputs, they often fail to faithfully preserve the individual's identity. In this paper, we propose a personalized face restoration method, FaceMe, based on a diffusion model. Given a single or a few reference images, we use an identity encoder to extract identity-related features, which serve as prompts to guide the diffusion model in restoring high-quality and identity-consistent facial images. By simply combining identity-related features, we effectively minimize the impact of identity-irrelevant features during training and support any number of reference image inputs during inference. Additionally, thanks to the robustness of the identity encoder, synthesized images can be used as reference images during training, and identity changing during inference does not require fine-tuning the model. We also propose a pipeline for constructing a reference image training pool that simulates the poses and expressions that may appear in real-world scenarios. Experimental results demonstrate that our FaceMe can restore high-quality facial images while maintaining identity consistency, achieving excellent performance and robustness.

Title: Compression with Global Guidance: Towards Training-free High-Resolution MLLMs Acceleration

• Authors: Xuyang Liu, Ziming Wang, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Bo Zheng, Linfeng Zhang, Siteng Huang, Honggang Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05179
• Pdf URL: https://arxiv.org/pdf/2501.05179
• Copy Paste: [[2501.05179]] Compression with Global Guidance: Towards Training-free High-Resolution MLLMs Acceleration(https://arxiv.org/abs/2501.05179)
• Keywords: large language model
• Abstract: Multimodal large language models (MLLMs) have attracted considerable attention due to their exceptional performance in visual content understanding and reasoning. However, their inference efficiency has been a notable concern, as the increasing length of multimodal contexts leads to quadratic complexity. Token compression techniques, which reduce the number of visual tokens, have demonstrated their effectiveness in reducing computational costs. Yet, these approaches have struggled to keep pace with the rapid advancements in MLLMs, especially the AnyRes strategy in the context of high-resolution image understanding. In this paper, we propose a novel token compression method, GlobalCom$^2$, tailored for high-resolution MLLMs that receive both the thumbnail and multiple crops. GlobalCom$^2$ treats the tokens derived from the thumbnail as the ``commander'' of the entire token compression process, directing the allocation of retention ratios and the specific compression for each crop. In this way, redundant tokens are eliminated while important local details are adaptively preserved to the highest extent feasible. Empirical results across 10 benchmarks reveal that GlobalCom$^2$ achieves an optimal balance between performance and efficiency, and consistently outperforms state-of-the-art token compression methods with LLaVA-NeXT-7B/13B models. Our code is released at \url{this https URL}.

Title: An Algorithmic Approach for Causal Health Equity: A Look at Race Differentials in Intensive Care Unit (ICU) Outcomes

• Authors: Drago Plecko, Paul Secombe, Andrea Clarke, Amelia Fiske, Samarra Toby, Donisha Duff, David Pilcher, Leo Anthony Celi, Rinaldo Bellomo, Elias Bareinboim
• Subjects: cs.LG, cs.AI, stat.AP, stat.ME
• Abstract URL: https://arxiv.org/abs/2501.05197
• Pdf URL: https://arxiv.org/pdf/2501.05197
• Copy Paste: [[2501.05197]] An Algorithmic Approach for Causal Health Equity: A Look at Race Differentials in Intensive Care Unit (ICU) Outcomes(https://arxiv.org/abs/2501.05197)
• Keywords: protect
• Abstract: The new era of large-scale data collection and analysis presents an opportunity for diagnosing and understanding the causes of health inequities. In this study, we describe a framework for systematically analyzing health disparities using causal inference. The framework is illustrated by investigating racial and ethnic disparities in intensive care unit (ICU) outcome between majority and minority groups in Australia (Indigenous vs. Non-Indigenous) and the United States (African-American vs. White). We demonstrate that commonly used statistical measures for quantifying inequity are insufficient, and focus on attributing the observed disparity to the causal mechanisms that generate it. We find that minority patients are younger at admission, have worse chronic health, are more likely to be admitted for urgent and non-elective reasons, and have higher illness severity. At the same time, however, we find a protective direct effect of belonging to a minority group, with minority patients showing improved survival compared to their majority counterparts, with all other variables kept equal. We demonstrate that this protective effect is related to the increased probability of being admitted to ICU, with minority patients having an increased risk of ICU admission. We also find that minority patients, while showing improved survival, are more likely to be readmitted to ICU. Thus, due to worse access to primary health care, minority patients are more likely to end up in ICU for preventable conditions, causing a reduction in the mortality rates and creating an effect that appears to be protective. Since the baseline risk of ICU admission may serve as proxy for lack of access to primary care, we developed the Indigenous Intensive Care Equity (IICE) Radar, a monitoring system for tracking the over-utilization of ICU resources by the Indigenous population of Australia across geographical areas.

Title: MHAFF: Multi-Head Attention Feature Fusion of CNN and Transformer for Cattle Identification

• Authors: Rabin Dulal, Lihong Zheng, Muhammad Ashad Kabir
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05209
• Pdf URL: https://arxiv.org/pdf/2501.05209
• Copy Paste: [[2501.05209]] MHAFF: Multi-Head Attention Feature Fusion of CNN and Transformer for Cattle Identification(https://arxiv.org/abs/2501.05209)
• Keywords: transformer
• Abstract: Convolutional Neural Networks (CNNs) have drawn researchers' attention to identifying cattle using muzzle images. However, CNNs often fail to capture long-range dependencies within the complex patterns of the muzzle. The transformers handle these challenges. This inspired us to fuse the strengths of CNNs and transformers in muzzle-based cattle identification. Addition and concatenation have been the most commonly used techniques for feature fusion. However, addition fails to preserve discriminative information, while concatenation results in an increase in dimensionality. Both methods are simple operations and cannot discover the relationships or interactions between fusing features. This research aims to overcome the issues faced by addition and concatenation. This research introduces a novel approach called Multi-Head Attention Feature Fusion (MHAFF) for the first time in cattle identification. MHAFF captures relations between the different types of fusing features while preserving their originality. The experiments show that MHAFF outperformed addition and concatenation techniques and the existing cattle identification methods in accuracy on two publicly available cattle datasets. MHAFF demonstrates excellent performance and quickly converges to achieve optimum accuracy of 99.88% and 99.52% in two cattle datasets simultaneously.

Title: EVA-S2PLoR: A Secure Element-wise Multiplication Meets Logistic Regression on Heterogeneous Database

• Authors: Tianle Tao, Shizhao Peng, Tianyu Mei, Shoumo Li, Haogang Zhu
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05223
• Pdf URL: https://arxiv.org/pdf/2501.05223
• Copy Paste: [[2501.05223]] EVA-S2PLoR: A Secure Element-wise Multiplication Meets Logistic Regression on Heterogeneous Database(https://arxiv.org/abs/2501.05223)
• Keywords: secure, security, privacy, robust
• Abstract: Accurate nonlinear computation is a key challenge in privacy-preserving machine learning (PPML). Most existing frameworks approximate it through linear operations, resulting in significant precision loss. This paper proposes an efficient, verifiable and accurate security 2-party logistic regression framework (EVA-S2PLoR), which achieves accurate nonlinear function computation through a novel secure element-wise multiplication protocol and its derived protocols. Our framework primarily includes secure 2-party vector element-wise multiplication, addition to multiplication, reciprocal, and sigmoid function based on data disguising technology, where high efficiency and accuracy are guaranteed by the simple computation flow based on the real number domain and the few number of fixed communication rounds. We provide secure and robust anomaly detection through dimension transformation and Monte Carlo methods. EVA-S2PLoR outperforms many advanced frameworks in terms of precision (improving the performance of the sigmoid function by about 10 orders of magnitude compared to most frameworks) and delivers the best overall performance in secure logistic regression experiments.

Title: Leveraging Large Language Models for Zero-shot Lay Summarisation in Biomedicine and Beyond

• Authors: Tomas Goldsack, Carolina Scarton, Chenghua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05224
• Pdf URL: https://arxiv.org/pdf/2501.05224
• Copy Paste: [[2501.05224]] Leveraging Large Language Models for Zero-shot Lay Summarisation in Biomedicine and Beyond(https://arxiv.org/abs/2501.05224)
• Keywords: large language model
• Abstract: In this work, we explore the application of Large Language Models to zero-shot Lay Summarisation. We propose a novel two-stage framework for Lay Summarisation based on real-life processes, and find that summaries generated with this method are increasingly preferred by human judges for larger models. To help establish best practices for employing LLMs in zero-shot settings, we also assess the ability of LLMs as judges, finding that they are able to replicate the preferences of human judges. Finally, we take the initial steps towards Lay Summarisation for Natural Language Processing (NLP) articles, finding that LLMs are able to generalise to this new domain, and further highlighting the greater utility of summaries generated by our proposed approach via an in-depth human evaluation.

Title: Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes

• Authors: Ludwic Leonard, Nils Thuerey, Ruediger Westermann
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05226
• Pdf URL: https://arxiv.org/pdf/2501.05226
• Copy Paste: [[2501.05226]] Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes(https://arxiv.org/abs/2501.05226)
• Keywords: diffusion, generative
• Abstract: We introduce a single-view reconstruction technique of volumetric fields in which multiple light scattering effects are omnipresent, such as in clouds. We model the unknown distribution of volumetric fields using an unconditional diffusion model trained on a novel benchmark dataset comprising 1,000 synthetically simulated volumetric density fields. The neural diffusion model is trained on the latent codes of a novel, diffusion-friendly, monoplanar representation. The generative model is used to incorporate a tailored parametric diffusion posterior sampling technique into different reconstruction tasks. A physically-based differentiable volume renderer is employed to provide gradients with respect to light transport in the latent space. This stands in contrast to classic NeRF approaches and makes the reconstructions better aligned with observed data. Through various experiments, we demonstrate single-view reconstruction of volumetric clouds at a previously unattainable quality.

Title: Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection

• Authors: Pei-Kang Lee, Jun-Cheng Chen, Ja-Ling Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05228
• Pdf URL: https://arxiv.org/pdf/2501.05228
• Copy Paste: [[2501.05228]] Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection(https://arxiv.org/abs/2501.05228)
• Keywords: robust, extraction, large language model
• Abstract: Out-of-distribution (OOD) detection has seen significant advancements with zero-shot approaches by leveraging the powerful Vision-Language Models (VLMs) such as CLIP. However, prior research works have predominantly focused on enhancing Far-OOD performance, while potentially compromising Near-OOD efficacy, as observed from our pilot study. To address this issue, we propose a novel strategy to enhance zero-shot OOD detection performances for both Far-OOD and Near-OOD scenarios by innovatively harnessing Large Language Models (LLMs) and VLMs. Our approach first exploit an LLM to generate superclasses of the ID labels and their corresponding background descriptions followed by feature extraction using CLIP. We then isolate the core semantic features for ID data by subtracting background features from the superclass features. The refined representation facilitates the selection of more appropriate negative labels for OOD data from a comprehensive candidate label set of WordNet, thereby enhancing the performance of zero-shot OOD detection in both scenarios. Furthermore, we introduce novel few-shot prompt tuning and visual prompt tuning to adapt the proposed framework to better align with the target distribution. Experimental results demonstrate that the proposed approach consistently outperforms current state-of-the-art methods across multiple benchmarks, with an improvement of up to 2.9% in AUROC and a reduction of up to 12.6% in FPR95. Additionally, our method exhibits superior robustness against covariate shift across different domains, further highlighting its effectiveness in real-world scenarios.

Title: Optimizing Estonian TV Subtitles with Semi-supervised Learning and LLMs

• Authors: Artem Fedorchenko, Tanel Alumäe
• Subjects: cs.CL, cs.AI, cs.LG, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.05234
• Pdf URL: https://arxiv.org/pdf/2501.05234
• Copy Paste: [[2501.05234]] Optimizing Estonian TV Subtitles with Semi-supervised Learning and LLMs(https://arxiv.org/abs/2501.05234)
• Keywords: large language model
• Abstract: This paper presents an approach for generating high-quality, same-language subtitles for Estonian TV content. We fine-tune the Whisper model on human-generated Estonian subtitles and enhance it with iterative pseudo-labeling and large language model (LLM) based post-editing. Our experiments demonstrate notable subtitle quality improvement through pseudo-labeling with an unlabeled dataset. We find that applying LLM-based editing at test time enhances subtitle accuracy, while its use during training does not yield further gains. This approach holds promise for creating subtitle quality close to human standard and could be extended to real-time applications.

Title: Automated external cervical resorption segmentation in cone-beam CT using local texture features

• Authors: Sadhana Ravikumar, Asma A. Khan, Matthew C. Davis, Beatriz Paniagua
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05236
• Pdf URL: https://arxiv.org/pdf/2501.05236
• Copy Paste: [[2501.05236]] Automated external cervical resorption segmentation in cone-beam CT using local texture features(https://arxiv.org/abs/2501.05236)
• Keywords: segmentation
• Abstract: External cervical resorption (ECR) is a resorptive process affecting teeth. While in some patients, active resorption ceases and gets replaced by osseous tissue, in other cases, the resorption progresses and ultimately results in tooth loss. For proper ECR assessment, cone-beam computed tomography (CBCT) is the recommended imaging modality, enabling a 3-D characterization of these lesions. While it is possible to manually identify and measure ECR resorption in CBCT scans, this process can be time intensive and highly subject to human error. Therefore, there is an urgent need to develop an automated method to identify and quantify the severity of ECR resorption using CBCT. Here, we present a method for ECR lesion segmentation that is based on automatic, binary classification of locally extracted voxel-wise texture features. We evaluate our method on 6 longitudinal CBCT datasets and show that certain texture-features can be used to accurately detect subtle CBCT signal changes due to ECR. We also present preliminary analyses clustering texture features within a lesion to stratify the defects and identify patterns indicative of calcification. These methods are important steps in developing prognostic biomarkers to predict whether ECR will continue to progress or cease, ultimately informing treatment decisions.

Title: FOCUS: Towards Universal Foreground Segmentation

• Authors: Zuyao You, Lingyu Kong, Lingchen Meng, Zuxuan Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05238
• Pdf URL: https://arxiv.org/pdf/2501.05238
• Copy Paste: [[2501.05238]] FOCUS: Towards Universal Foreground Segmentation(https://arxiv.org/abs/2501.05238)
• Keywords: segmentation
• Abstract: Foreground segmentation is a fundamental task in computer vision, encompassing various subdivision tasks. Previous research has typically designed task-specific architectures for each task, leading to a lack of unification. Moreover, they primarily focus on recognizing foreground objects without effectively distinguishing them from the background. In this paper, we emphasize the importance of the background and its relationship with the foreground. We introduce FOCUS, the Foreground ObjeCts Universal Segmentation framework that can handle multiple foreground tasks. We develop a multi-scale semantic network using the edge information of objects to enhance image features. To achieve boundary-aware segmentation, we propose a novel distillation method, integrating the contrastive learning strategy to refine the prediction mask in multi-modal feature space. We conduct extensive experiments on a total of 13 datasets across 5 tasks, and the results demonstrate that FOCUS consistently outperforms the state-of-the-art task-specific models on most metrics.

Title: Is Your Autonomous Vehicle Safe? Understanding the Threat of Electromagnetic Signal Injection Attacks on Traffic Scene Perception

• Authors: Wenhao Liao, Sineng Yan, Youqian Zhang, Xinwei Zhai, Yuanyuan Wang, Eugene Yujun Fu
• Subjects: cs.CR, cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2501.05239
• Pdf URL: https://arxiv.org/pdf/2501.05239
• Copy Paste: [[2501.05239]] Is Your Autonomous Vehicle Safe? Understanding the Threat of Electromagnetic Signal Injection Attacks on Traffic Scene Perception(https://arxiv.org/abs/2501.05239)
• Keywords: secure, attack, robust
• Abstract: Autonomous vehicles rely on camera-based perception systems to comprehend their driving environment and make crucial decisions, thereby ensuring vehicles to steer safely. However, a significant threat known as Electromagnetic Signal Injection Attacks (ESIA) can distort the images captured by these cameras, leading to incorrect AI decisions and potentially compromising the safety of autonomous vehicles. Despite the serious implications of ESIA, there is limited understanding of its impacts on the robustness of AI models across various and complex driving scenarios. To address this gap, our research analyzes the performance of different models under ESIA, revealing their vulnerabilities to the attacks. Moreover, due to the challenges in obtaining real-world attack data, we develop a novel ESIA simulation method and generate a simulated attack dataset for different driving scenarios. Our research provides a comprehensive simulation and evaluation framework, aiming to enhance the development of more robust AI models and secure intelligent systems, ultimately contributing to the advancement of safer and more reliable technology across various fields.

Title: Contrast-Free Myocardial Scar Segmentation in Cine MRI using Motion and Texture Fusion

• Authors: Guang Yang, Jingkun Chen, Xicheng Sheng, Shan Yang, Xiahai Zhuang, Betty Raman, Lei Li, Vicente Grau
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05241
• Pdf URL: https://arxiv.org/pdf/2501.05241
• Copy Paste: [[2501.05241]] Contrast-Free Myocardial Scar Segmentation in Cine MRI using Motion and Texture Fusion(https://arxiv.org/abs/2501.05241)
• Keywords: segmentation
• Abstract: Late gadolinium enhancement MRI (LGE MRI) is the gold standard for the detection of myocardial scars for post myocardial infarction (MI). LGE MRI requires the injection of a contrast agent, which carries potential side effects and increases scanning time and patient discomfort. To address these issues, we propose a novel framework that combines cardiac motion observed in cine MRI with image texture information to segment the myocardium and scar tissue in the left ventricle. Cardiac motion tracking can be formulated as a full cardiac image cycle registration problem, which can be solved via deep neural networks. Experimental results prove that the proposed method can achieve scar segmentation based on non-contrasted cine images with comparable accuracy to LGE MRI. This demonstrates its potential as an alternative to contrast-enhanced techniques for scar detection.

Title: Domain-Incremental Semantic Segmentation for Autonomous Driving under Adverse Driving Conditions

• Authors: Shishir Muralidhara, René Schuster, Didier Stricker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05246
• Pdf URL: https://arxiv.org/pdf/2501.05246
• Copy Paste: [[2501.05246]] Domain-Incremental Semantic Segmentation for Autonomous Driving under Adverse Driving Conditions(https://arxiv.org/abs/2501.05246)
• Keywords: segmentation
• Abstract: Semantic segmentation for autonomous driving is an even more challenging task when faced with adverse driving conditions. Standard models trained on data recorded under ideal conditions show a deteriorated performance in unfavorable weather or illumination conditions. Fine-tuning on the new task or condition would lead to overwriting the previously learned information resulting in catastrophic forgetting. Adapting to the new conditions through traditional domain adaption methods improves the performance on the target domain at the expense of the source domain. Addressing these issues, we propose an architecture-based domain-incremental learning approach called Progressive Semantic Segmentation (PSS). PSS is a task-agnostic, dynamically growing collection of domain-specific segmentation models. The task of inferring the domain and subsequently selecting the appropriate module for segmentation is carried out using a collection of convolutional autoencoders. We extensively evaluate our proposed approach using several datasets at varying levels of granularity in the categorization of adverse driving conditions. Furthermore, we demonstrate the generalization of the proposed approach to similar and unseen domains.

Title: Deriving Coding-Specific Sub-Models from LLMs using Resource-Efficient Pruning

• Authors: Laura Puccioni, Alireza Farshin, Mariano Scazzariello, Changjie Wang, Marco Chiesa, Dejan Kostic
• Subjects: cs.LG, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.05248
• Pdf URL: https://arxiv.org/pdf/2501.05248
• Copy Paste: [[2501.05248]] Deriving Coding-Specific Sub-Models from LLMs using Resource-Efficient Pruning(https://arxiv.org/abs/2501.05248)
• Keywords: extraction, large language model
• Abstract: Large Language Models (LLMs) have demonstrated their exceptional performance in various complex code generation tasks. However, their broader adoption is limited by significant computational demands and high resource requirements, particularly memory and processing power. To mitigate such requirements, model pruning techniques are used to create more compact models with significantly fewer parameters. However, current approaches do not focus on the efficient extraction of programming-language-specific sub-models. In this work, we explore the idea of efficiently deriving coding-specific sub-models through unstructured pruning (i.e., Wanda). We investigate the impact of different domain-specific calibration datasets on pruning outcomes across three distinct domains and extend our analysis to extracting four language-specific sub-models: Python, Java, C++, and JavaScript. We are the first to efficiently extract programming-language-specific sub-models using appropriate calibration datasets while maintaining acceptable accuracy w.r.t. full models. We are also the first to provide analytical evidence that domain-specific tasks activate distinct regions within LLMs, supporting the creation of specialized sub-models through unstructured pruning. We believe that this work has significant potential to enhance LLM accessibility for coding by reducing computational requirements to enable local execution on consumer-grade hardware, and supporting faster inference times critical for real-time development feedback.

Title: RAG-WM: An Efficient Black-Box Watermarking Approach for Retrieval-Augmented Generation of Large Language Models

• Authors: Peizhuo Lv, Mengjie Sun, Hao Wang, Xiaofeng Wang, Shengzhi Zhang, Yuxuan Chen, Kai Chen, Limin Sun
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05249
• Pdf URL: https://arxiv.org/pdf/2501.05249
• Copy Paste: [[2501.05249]] RAG-WM: An Efficient Black-Box Watermarking Approach for Retrieval-Augmented Generation of Large Language Models(https://arxiv.org/abs/2501.05249)
• Keywords: privacy, protect, attack, robust, steal, watermark, large language model
• Abstract: In recent years, tremendous success has been witnessed in Retrieval-Augmented Generation (RAG), widely used to enhance Large Language Models (LLMs) in domain-specific, knowledge-intensive, and privacy-sensitive tasks. However, attackers may steal those valuable RAGs and deploy or commercialize them, making it essential to detect Intellectual Property (IP) infringement. Most existing ownership protection solutions, such as watermarks, are designed for relational databases and texts. They cannot be directly applied to RAGs because relational database watermarks require white-box access to detect IP infringement, which is unrealistic for the knowledge base in RAGs. Meanwhile, post-processing by the adversary's deployed LLMs typically destructs text watermark information. To address those problems, we propose a novel black-box "knowledge watermark" approach, named RAG-WM, to detect IP infringement of RAGs. RAG-WM uses a multi-LLM interaction framework, comprising a Watermark Generator, Shadow LLM & RAG, and Watermark Discriminator, to create watermark texts based on watermark entity-relationship tuples and inject them into the target RAG. We evaluate RAG-WM across three domain-specific and two privacy-sensitive tasks on four benchmark LLMs. Experimental results show that RAG-WM effectively detects the stolen RAGs in various deployed LLMs. Furthermore, RAG-WM is robust against paraphrasing, unrelated content removal, knowledge insertion, and knowledge expansion attacks. Lastly, RAG-WM can also evade watermark detection approaches, highlighting its promising application in detecting IP infringement of RAG systems.

Title: Enhancing Plagiarism Detection in Marathi with a Weighted Ensemble of TF-IDF and BERT Embeddings for Low-Resource Language Processing

• Authors: Atharva Mutsaddi, Aditya Choudhary
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05260
• Pdf URL: https://arxiv.org/pdf/2501.05260
• Copy Paste: [[2501.05260]] Enhancing Plagiarism Detection in Marathi with a Weighted Ensemble of TF-IDF and BERT Embeddings for Low-Resource Language Processing(https://arxiv.org/abs/2501.05260)
• Keywords: robust, extraction, transformer
• Abstract: Plagiarism involves using another person's work or concepts without proper attribution, presenting them as original creations. With the growing amount of data communicated in regional languages such as Marathi -- one of India's regional languages -- it is crucial to design robust plagiarism detection systems tailored for low-resource languages. Language models like Bidirectional Encoder Representations from Transformers (BERT) have demonstrated exceptional capability in text representation and feature extraction, making them essential tools for semantic analysis and plagiarism detection. However, the application of BERT for low-resource languages remains under-explored, particularly in the context of plagiarism detection. This paper presents a method to enhance the accuracy of plagiarism detection for Marathi texts using BERT sentence embeddings in conjunction with Term Frequency-Inverse Document Frequency (TF-IDF) feature representation. This approach effectively captures statistical, semantic, and syntactic aspects of text features through a weighted voting ensemble of machine learning models.

Title: Patch-GAN Transfer Learning with Reconstructive Models for Cloud Removal

• Authors: Wanli Ma, Oktay Karakus, Paul L. Rosin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05265
• Pdf URL: https://arxiv.org/pdf/2501.05265
• Copy Paste: [[2501.05265]] Patch-GAN Transfer Learning with Reconstructive Models for Cloud Removal(https://arxiv.org/abs/2501.05265)
• Keywords: generative
• Abstract: Cloud removal plays a crucial role in enhancing remote sensing image analysis, yet accurately reconstructing cloud-obscured regions remains a significant challenge. Recent advancements in generative models have made the generation of realistic images increasingly accessible, offering new opportunities for this task. Given the conceptual alignment between image generation and cloud removal tasks, generative models present a promising approach for addressing cloud removal in remote sensing. In this work, we propose a deep transfer learning approach built on a generative adversarial network (GAN) framework to explore the potential of the novel masked autoencoder (MAE) image reconstruction model in cloud removal. Due to the complexity of remote sensing imagery, we further propose using a patch-wise discriminator to determine whether each patch of the image is real or not. The proposed reconstructive transfer learning approach demonstrates significant improvements in cloud removal performance compared to other GAN-based methods. Additionally, whilst direct comparisons with some of the state-of-the-art cloud removal techniques are limited due to unclear details regarding their train/test data splits, the proposed model achieves competitive results based on available benchmarks.

Title: CellViT++: Energy-Efficient and Adaptive Cell Segmentation and Classification Using Foundation Models

• Authors: Fabian Hörst, Moritz Rempe, Helmut Becker, Lukas Heine, Julius Keyl, Jens Kleesiek
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05269
• Pdf URL: https://arxiv.org/pdf/2501.05269
• Copy Paste: [[2501.05269]] CellViT++: Energy-Efficient and Adaptive Cell Segmentation and Classification Using Foundation Models(https://arxiv.org/abs/2501.05269)
• Keywords: transformer, segmentation
• Abstract: Digital Pathology is a cornerstone in the diagnosis and treatment of diseases. A key task in this field is the identification and segmentation of cells in hematoxylin and eosin-stained images. Existing methods for cell segmentation often require extensive annotated datasets for training and are limited to a predefined cell classification scheme. To overcome these limitations, we propose $\text{CellViT}^{\scriptscriptstyle ++}$, a framework for generalized cell segmentation in digital pathology. $\text{CellViT}^{\scriptscriptstyle ++}$ utilizes Vision Transformers with foundation models as encoders to compute deep cell features and segmentation masks simultaneously. To adapt to unseen cell types, we rely on a computationally efficient approach. It requires minimal data for training and leads to a drastically reduced carbon footprint. We demonstrate excellent performance on seven different datasets, covering a broad spectrum of cell types, organs, and clinical settings. The framework achieves remarkable zero-shot segmentation and data-efficient cell-type classification. Furthermore, we show that $\text{CellViT}^{\scriptscriptstyle ++}$ can leverage immunofluorescence stainings to generate training datasets without the need for pathologist annotations. The automated dataset generation approach surpasses the performance of networks trained on manually labeled data, demonstrating its effectiveness in creating high-quality training datasets without expert annotations. To advance digital pathology, $\text{CellViT}^{\scriptscriptstyle ++}$ is available as an open-source framework featuring a user-friendly, web-based interface for visualization and annotation. The code is available under this https URL.

Title: Comparison Study: Glacier Calving Front Delineation in Synthetic Aperture Radar Images With Deep Learning

• Authors: Nora Gourmelon, Konrad Heidler, Erik Loebel, Daniel Cheng, Julian Klink, Anda Dong, Fei Wu, Noah Maul, Moritz Koch, Marcel Dreier, Dakota Pyles, Thorsten Seehaus, Matthias Braun, Andreas Maier, Vincent Christlein
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05281
• Pdf URL: https://arxiv.org/pdf/2501.05281
• Copy Paste: [[2501.05281]] Comparison Study: Glacier Calving Front Delineation in Synthetic Aperture Radar Images With Deep Learning(https://arxiv.org/abs/2501.05281)
• Keywords: transformer
• Abstract: Calving front position variation of marine-terminating glaciers is an indicator of ice mass loss and a crucial parameter in numerical glacier models. Deep Learning (DL) systems can automatically extract this position from Synthetic Aperture Radar (SAR) imagery, enabling continuous, weather- and illumination-independent, large-scale monitoring. This study presents the first comparison of DL systems on a common calving front benchmark dataset. A multi-annotator study with ten annotators is performed to contrast the best-performing DL system against human performance. The best DL model's outputs deviate 221 m on average, while the average deviation of the human annotators is 38 m. This significant difference shows that current DL systems do not yet match human performance and that further research is needed to enable fully automated monitoring of glacier calving fronts. The study of Vision Transformers, foundation models, and the inclusion and processing strategy of more information are identified as avenues for future research.

Title: Distributed Learning and Inference Systems: A Networking Perspective

• Authors: Hesham G. Moussa, Arashmid Akhavain, S. Maryam Hosseini, Bill McCormick
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.05323
• Pdf URL: https://arxiv.org/pdf/2501.05323
• Copy Paste: [[2501.05323]] Distributed Learning and Inference Systems: A Networking Perspective(https://arxiv.org/abs/2501.05323)
• Keywords: privacy
• Abstract: Machine learning models have achieved, and in some cases surpassed, human-level performance in various tasks, mainly through centralized training of static models and the use of large models stored in centralized clouds for inference. However, this centralized approach has several drawbacks, including privacy concerns, high storage demands, a single point of failure, and significant computing requirements. These challenges have driven interest in developing alternative decentralized and distributed methods for AI training and inference. Distribution introduces additional complexity, as it requires managing multiple moving parts. To address these complexities and fill a gap in the development of distributed AI systems, this work proposes a novel framework, Data and Dynamics-Aware Inference and Training Networks (DA-ITN). The different components of DA-ITN and their functions are explored, and the associated challenges and research areas are highlighted.

Title: Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction

• Authors: Hantao Lou, Jiaming Ji, Kaile Wang, Yaodong Yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05336
• Pdf URL: https://arxiv.org/pdf/2501.05336
• Copy Paste: [[2501.05336]] Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction(https://arxiv.org/abs/2501.05336)
• Keywords: large language model
• Abstract: The rapid advancement of large language models (LLMs) has led to significant improvements in their capabilities, but also to increased concerns about their alignment with human values and intentions. Current alignment strategies, including adaptive training and inference-time methods, have demonstrated potential in this area. However, these approaches still struggle to balance deployment complexity and capability across various tasks and difficulties. In this work, we introduce the Streaming Distribution Induce Aligner (Stream Aligner), a novel alignment paradigm that combines efficiency with enhanced performance in various tasks throughout the generation process. Stream Aligner achieves dynamic sentence-level correction by using a small model to learn the preferences of the suffix sentence, iteratively correcting the suffix sentence output by the upstream model, and then using the corrected sentence to replace the suffix sentence in subsequent generations. Compared to Aligner, our experiments demonstrate that Stream Aligner reduces reliance on the capabilities of additional models, enhances the reasoning abilities of LLMs, and decreases latency during user interaction. Specifically, Stream Aligner-2B model has achieved an improvement of 76.1% in helpfulness, 36.0% in harmlessness on the tested Llama2-70B-chat model, and Stream Aligner-8B has achieved an improvement of 3.5% on the math ability of the tested Llama3-70B-Instruct model.

Title: Cybersecurity in Transportation Systems: Policies and Technology Directions

• Authors: Ostonya Thomas, M Sabbir Salek, Jean-Michel Tine, Mizanur Rahman, Trayce Hockstad, Mashrur Chowdhury
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.05356
• Pdf URL: https://arxiv.org/pdf/2501.05356
• Copy Paste: [[2501.05356]] Cybersecurity in Transportation Systems: Policies and Technology Directions(https://arxiv.org/abs/2501.05356)
• Keywords: secure, security
• Abstract: The transportation industry is experiencing vast digitalization as a plethora of technologies are being implemented to improve efficiency, functionality, and safety. Although technological advancements bring many benefits to transportation, integrating cyberspace across transportation sectors has introduced new and deliberate cyber threats. In the past, public agencies assumed digital infrastructure was secured since its vulnerabilities were unknown to adversaries. However, with the expansion of cyberspace, this assumption has become invalid. With the rapid advancement of wireless technologies, transportation systems are increasingly interconnected with both transportation and non-transportation networks in an internet-of-things ecosystem, expanding cyberspace in transportation and increasing threats and vulnerabilities. This study investigates some prominent reasons for the increase in cyber vulnerabilities in transportation. In addition, this study presents various collaborative strategies among stakeholders that could help improve cybersecurity in the transportation industry. These strategies address programmatic and policy aspects and suggest avenues for technological research and development. The latter highlights opportunities for future research to enhance the cybersecurity of transportation systems and infrastructure by leveraging hybrid approaches and emerging technologies.

Title: CROPS: Model-Agnostic Training-Free Framework for Safe Image Synthesis with Latent Diffusion Models

• Authors: Junha Park, Ian Ryu, Jaehui Hwang, Hyungkeun Park, Jiyoon Kim, Jong-Seok Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05359
• Pdf URL: https://arxiv.org/pdf/2501.05359
• Copy Paste: [[2501.05359]] CROPS: Model-Agnostic Training-Free Framework for Safe Image Synthesis with Latent Diffusion Models(https://arxiv.org/abs/2501.05359)
• Keywords: attack, robust, diffusion
• Abstract: With advances in diffusion models, image generation has shown significant performance improvements. This raises concerns about the potential abuse of image generation, such as the creation of explicit or violent images, commonly referred to as Not Safe For Work (NSFW) content. To address this, the Stable Diffusion model includes several safety checkers to censor initial text prompts and final output images generated from the model. However, recent research has shown that these safety checkers have vulnerabilities against adversarial attacks, allowing them to generate NSFW images. In this paper, we find that these adversarial attacks are not robust to small changes in text prompts or input latents. Based on this, we propose CROPS (Circular or RandOm Prompts for Safety), a model-agnostic framework that easily defends against adversarial attacks generating NSFW images without requiring additional training. Moreover, we develop an approach that utilizes one-step diffusion models for efficient NSFW detection (CROPS-1), further reducing computational resources. We demonstrate the superiority of our method in terms of performance and applicability.

Title: No-Regret Linear Bandits under Gap-Adjusted Misspecification

• Authors: Chong Liu, Dan Qiao, Ming Yin, Ilija Bogunovic, Yu-Xiang Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05361
• Pdf URL: https://arxiv.org/pdf/2501.05361
• Copy Paste: [[2501.05361]] No-Regret Linear Bandits under Gap-Adjusted Misspecification(https://arxiv.org/abs/2501.05361)
• Keywords: robust
• Abstract: This work studies linear bandits under a new notion of gap-adjusted misspecification and is an extension of Liu et al. (2023). When the underlying reward function is not linear, existing linear bandits work usually relies on a uniform misspecification parameter $\epsilon$ that measures the sup-norm error of the best linear approximation. This results in an unavoidable linear regret whenever $\epsilon > 0$. We propose a more natural model of misspecification which only requires the approximation error at each input $x$ to be proportional to the suboptimality gap at $x$. It captures the intuition that, for optimization problems, near-optimal regions should matter more and we can tolerate larger approximation errors in suboptimal regions. Quite surprisingly, we show that the classical LinUCB algorithm -- designed for the realizable case -- is automatically robust against such $\rho$-gap-adjusted misspecification with parameter $\rho$ diminishing at $O(1/(d \sqrt{\log T}))$. It achieves a near-optimal $O(\sqrt{T})$ regret for problems that the best-known regret is almost linear in time horizon $T$. We further advance this frontier by presenting a novel phased elimination-based algorithm whose gap-adjusted misspecification parameter $\rho = O(1/\sqrt{d})$ does not scale with $T$. This algorithm attains optimal $O(\sqrt{T})$ regret and is deployment-efficient, requiring only $\log T$ batches of exploration. It also enjoys an adaptive $O(\log T)$ regret when a constant suboptimality gap exists. Technically, our proof relies on a novel self-bounding argument that bounds the part of the regret due to misspecification by the regret itself, and a new inductive lemma that limits the misspecification error within the suboptimality gap for all valid actions in each batch selected by G-optimal design.

Title: 1-2-1: Renaissance of Single-Network Paradigm for Virtual Try-On

• Authors: Shuliang Ning, Yipeng Qin, Xiaoguang Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05369
• Pdf URL: https://arxiv.org/pdf/2501.05369
• Copy Paste: [[2501.05369]] 1-2-1: Renaissance of Single-Network Paradigm for Virtual Try-On(https://arxiv.org/abs/2501.05369)
• Keywords: extraction, generative
• Abstract: Virtual Try-On (VTON) has become a crucial tool in ecommerce, enabling the realistic simulation of garments on individuals while preserving their original appearance and pose. Early VTON methods relied on single generative networks, but challenges remain in preserving fine-grained garment details due to limitations in feature extraction and fusion. To address these issues, recent approaches have adopted a dual-network paradigm, incorporating a complementary "ReferenceNet" to enhance garment feature extraction and fusion. While effective, this dual-network approach introduces significant computational overhead, limiting its scalability for high-resolution and long-duration image/video VTON applications. In this paper, we challenge the dual-network paradigm by proposing a novel single-network VTON method that overcomes the limitations of existing techniques. Our method, namely MNVTON, introduces a Modality-specific Normalization strategy that separately processes text, image and video inputs, enabling them to share the same attention layers in a VTON network. Extensive experimental results demonstrate the effectiveness of our approach, showing that it consistently achieves higher-quality, more detailed results for both image and video VTON tasks. Our results suggest that the single-network paradigm can rival the performance of dualnetwork approaches, offering a more efficient alternative for high-quality, scalable VTON applications.

Title: Accelerated Diffusion Models via Speculative Sampling

• Authors: Valentin De Bortoli, Alexandre Galashov, Arthur Gretton, Arnaud Doucet
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.05370
• Pdf URL: https://arxiv.org/pdf/2501.05370
• Copy Paste: [[2501.05370]] Accelerated Diffusion Models via Speculative Sampling(https://arxiv.org/abs/2501.05370)
• Keywords: diffusion, large language model
• Abstract: Speculative sampling is a popular technique for accelerating inference in Large Language Models by generating candidate tokens using a fast draft model and accepting or rejecting them based on the target model's distribution. While speculative sampling was previously limited to discrete sequences, we extend it to diffusion models, which generate samples via continuous, vector-valued Markov chains. In this context, the target model is a high-quality but computationally expensive diffusion model. We propose various drafting strategies, including a simple and effective approach that does not require training a draft model and is applicable out of the box to any diffusion model. Our experiments demonstrate significant generation speedup on various diffusion models, halving the number of function evaluations, while generating exact samples from the target model.

Title: Integrating Explainable AI for Effective Malware Detection in Encrypted Network Traffic

• Authors: Sileshi Nibret Zeleke, Amsalu Fentie Jember, Mario Bochicchio
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05387
• Pdf URL: https://arxiv.org/pdf/2501.05387
• Copy Paste: [[2501.05387]] Integrating Explainable AI for Effective Malware Detection in Encrypted Network Traffic(https://arxiv.org/abs/2501.05387)
• Keywords: security, privacy, attack, robust
• Abstract: Encrypted network communication ensures confidentiality, integrity, and privacy between endpoints. However, attackers are increasingly exploiting encryption to conceal malicious behavior. Detecting unknown encrypted malicious traffic without decrypting the payloads remains a significant challenge. In this study, we investigate the integration of explainable artificial intelligence (XAI) techniques to detect malicious network traffic. We employ ensemble learning models to identify malicious activity using multi-view features extracted from various aspects of encrypted communication. To effectively represent malicious communication, we compiled a robust dataset with 1,127 unique connections, more than any other available open-source dataset, and spanning 54 malware families. Our models were benchmarked against the CTU-13 dataset, achieving performance of over 99% accuracy, precision, and F1-score. Additionally, the eXtreme Gradient Boosting (XGB) model demonstrated 99.32% accuracy, 99.53% precision, and 99.43% F1-score on our custom dataset. By leveraging Shapley Additive Explanations (SHAP), we identified that the maximum packet size, mean inter-arrival time of packets, and transport layer security version used are the most critical features for the global model explanation. Furthermore, key features were identified as important for local explanations across both datasets for individual traffic samples. These insights provide a deeper understanding of the model decision-making process, enhancing the transparency and reliability of detecting malicious encrypted traffic.

Title: FairCode: Evaluating Social Bias of LLMs in Code Generation

• Authors: Yongkang Du, Jen-tse Huang, Jieyu Zhao, Lu Lin
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.05396
• Pdf URL: https://arxiv.org/pdf/2501.05396
• Copy Paste: [[2501.05396]] FairCode: Evaluating Social Bias of LLMs in Code Generation(https://arxiv.org/abs/2501.05396)
• Keywords: fair, large language model
• Abstract: Large language models (LLMs) have demonstrated significant capability in code generation, drawing increasing attention to the evaluation of the quality and safety of their outputs. However, research on bias in code generation remains limited. Existing studies typically assess bias by applying malicious prompts or reapply tasks and dataset for discriminative models. Given that LLMs are often aligned with human values and that prior datasets are not fully optimized for code-related tasks, there is a pressing need for benchmarks specifically designed for evaluating code models. In this study, we introduce FairCode, a novel benchmark for evaluating bias in code generation. FairCode comprises two tasks: function implementation and test case generation, each evaluating social bias through diverse scenarios. Additionally, we propose a new metric, FairScore, to assess model performance on this benchmark. We conduct experiments on widely used LLMs and provide a comprehensive analysis of the results. The findings reveal that all tested LLMs exhibit bias. The code is available at this https URL.

Title: BRATI: Bidirectional Recurrent Attention for Time-Series Imputation

• Authors: Armando Collado-Villaverde, Pablo Muñoz, Maria D. R-Moreno
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05401
• Pdf URL: https://arxiv.org/pdf/2501.05401
• Copy Paste: [[2501.05401]] BRATI: Bidirectional Recurrent Attention for Time-Series Imputation(https://arxiv.org/abs/2501.05401)
• Keywords: robust
• Abstract: Missing data in time-series analysis poses significant challenges, affecting the reliability of downstream applications. Imputation, the process of estimating missing values, has emerged as a key solution. This paper introduces BRATI, a novel deep-learning model designed to address multivariate time-series imputation by combining Bidirectional Recurrent Networks and Attention mechanisms. BRATI processes temporal dependencies and feature correlations across long and short time horizons, utilizing two imputation blocks that operate in opposite temporal directions. Each block integrates recurrent layers and attention mechanisms to effectively resolve long-term dependencies. We evaluate BRATI on three real-world datasets under diverse missing-data scenarios: randomly missing values, fixed-length missing sequences, and variable-length missing sequences. Our findings demonstrate that BRATI consistently outperforms state-of-the-art models, delivering superior accuracy and robustness in imputing multivariate time-series data.

Title: TimeDP: Learning to Generate Multi-Domain Time Series with Domain Prompts

• Authors: Yu-Hao Huang, Chang Xu, Yueying Wu, Wu-Jun Li, Jiang Bian
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05403
• Pdf URL: https://arxiv.org/pdf/2501.05403
• Copy Paste: [[2501.05403]] TimeDP: Learning to Generate Multi-Domain Time Series with Domain Prompts(https://arxiv.org/abs/2501.05403)
• Keywords: privacy, diffusion
• Abstract: Time series generation models are crucial for applications like data augmentation and privacy preservation. Most existing time series generation models are typically designed to generate data from one specified domain. While leveraging data from other domain for better generalization is proved to work in other application areas, this approach remains challenging for time series modeling due to the large divergence in patterns among different real world time series categories. In this paper, we propose a multi-domain time series diffusion model with domain prompts, named TimeDP. In TimeDP, we utilize a time series semantic prototype module which defines time series prototypes to represent time series basis, each prototype vector serving as "word" representing some elementary time series feature. A prototype assignment module is applied to extract the extract domain specific prototype weights, for learning domain prompts as generation condition. During sampling, we extract "domain prompt" with few-shot samples from the target domain and use the domain prompts as condition to generate time series samples. Experiments demonstrate that our method outperforms baselines to provide the state-of-the-art in-domain generation quality and strong unseen domain generation capability.

Title: Uncertainty-aware Knowledge Tracing

• Authors: Weihua Cheng, Hanwen Du, Chunxiao Li, Ersheng Ni, Liangdi Tan, Tianqi Xu, Yongxin Ni
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05415
• Pdf URL: https://arxiv.org/pdf/2501.05415
• Copy Paste: [[2501.05415]] Uncertainty-aware Knowledge Tracing(https://arxiv.org/abs/2501.05415)
• Keywords: robust
• Abstract: Knowledge Tracing (KT) is crucial in education assessment, which focuses on depicting students' learning states and assessing students' mastery of subjects. With the rise of modern online learning platforms, particularly massive open online courses (MOOCs), an abundance of interaction data has greatly advanced the development of the KT technology. Previous research commonly adopts deterministic representation to capture students' knowledge states, which neglects the uncertainty during student interactions and thus fails to model the true knowledge state in learning process. In light of this, we propose an Uncertainty-Aware Knowledge Tracing model (UKT) which employs stochastic distribution embeddings to represent the uncertainty in student interactions, with a Wasserstein self-attention mechanism designed to capture the transition of state distribution in student learning behaviors. Additionally, we introduce the aleatory uncertainty-aware contrastive learning loss, which strengthens the model's robustness towards different types of uncertainties. Extensive experiments on six real-world datasets demonstrate that UKT not only significantly surpasses existing deep learning-based models in KT prediction, but also shows unique advantages in handling the uncertainty of student interactions.

Title: Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation

• Authors: Xuyi Meng, Chen Wang, Jiahui Lei, Kostas Daniilidis, Jiatao Gu, Lingjie Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05427
• Pdf URL: https://arxiv.org/pdf/2501.05427
• Copy Paste: [[2501.05427]] Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation(https://arxiv.org/abs/2501.05427)
• Keywords: diffusion, generative
• Abstract: Recent advances in 2D image generation have achieved remarkable quality,largely driven by the capacity of diffusion models and the availability of large-scale datasets. However, direct 3D generation is still constrained by the scarcity and lower fidelity of 3D datasets. In this paper, we introduce Zero-1-to-G, a novel approach that addresses this problem by enabling direct single-view generation on Gaussian splats using pretrained 2D diffusion models. Our key insight is that Gaussian splats, a 3D representation, can be decomposed into multi-view images encoding different attributes. This reframes the challenging task of direct 3D generation within a 2D diffusion framework, allowing us to leverage the rich priors of pretrained 2D diffusion models. To incorporate 3D awareness, we introduce cross-view and cross-attribute attention layers, which capture complex correlations and enforce 3D consistency across generated splats. This makes Zero-1-to-G the first direct image-to-3D generative model to effectively utilize pretrained 2D diffusion priors, enabling efficient training and improved generalization to unseen objects. Extensive experiments on both synthetic and in-the-wild datasets demonstrate superior performance in 3D object generation, offering a new approach to high-quality 3D generation.

Title: The GAN is dead; long live the GAN! A Modern GAN Baseline

• Authors: Yiwen Huang, Aaron Gokaslan, Volodymyr Kuleshov, James Tompkin
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05441
• Pdf URL: https://arxiv.org/pdf/2501.05441
• Copy Paste: [[2501.05441]] The GAN is dead; long live the GAN! A Modern GAN Baseline(https://arxiv.org/abs/2501.05441)
• Keywords: diffusion
• Abstract: There is a widely-spread claim that GANs are difficult to train, and GAN architectures in the literature are littered with empirical tricks. We provide evidence against this claim and build a modern GAN baseline in a more principled manner. First, we derive a well-behaved regularized relativistic GAN loss that addresses issues of mode dropping and non-convergence that were previously tackled via a bag of ad-hoc tricks. We analyze our loss mathematically and prove that it admits local convergence guarantees, unlike most existing relativistic losses. Second, our new loss allows us to discard all ad-hoc tricks and replace outdated backbones used in common GANs with modern architectures. Using StyleGAN2 as an example, we present a roadmap of simplification and modernization that results in a new minimalist baseline -- R3GAN. Despite being simple, our approach surpasses StyleGAN2 on FFHQ, ImageNet, CIFAR, and Stacked MNIST datasets, and compares favorably against state-of-the-art GANs and diffusion models.

Title: Progressive Growing of Video Tokenizers for Highly Compressed Latent Spaces

• Authors: Aniruddha Mahapatra, Long Mai, Yitian Zhang, David Bourgin, Feng Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05442
• Pdf URL: https://arxiv.org/pdf/2501.05442
• Copy Paste: [[2501.05442]] Progressive Growing of Video Tokenizers for Highly Compressed Latent Spaces(https://arxiv.org/abs/2501.05442)
• Keywords: diffusion
• Abstract: Video tokenizers are essential for latent video diffusion models, converting raw video data into spatiotemporally compressed latent spaces for efficient training. However, extending state-of-the-art video tokenizers to achieve a temporal compression ratio beyond 4x without increasing channel capacity poses significant challenges. In this work, we propose an alternative approach to enhance temporal compression. We find that the reconstruction quality of temporally subsampled videos from a low-compression encoder surpasses that of high-compression encoders applied to original videos. This indicates that high-compression models can leverage representations from lower-compression models. Building on this insight, we develop a bootstrapped high-temporal-compression model that progressively trains high-compression blocks atop well-trained lower-compression models. Our method includes a cross-level feature-mixing module to retain information from the pretrained low-compression model and guide higher-compression blocks to capture the remaining details from the full video sequence. Evaluation of video benchmarks shows that our method significantly improves reconstruction quality while increasing temporal compression compared to direct extensions of existing video tokenizers. Furthermore, the resulting compact latent space effectively trains a video diffusion model for high-quality video generation with a reduced token budget.

Title: A survey of textual cyber abuse detection using cutting-edge language models and large language models

• Authors: Jose A. Diaz-Garcia, Joao Paulo Carvalho
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05443
• Pdf URL: https://arxiv.org/pdf/2501.05443
• Copy Paste: [[2501.05443]] A survey of textual cyber abuse detection using cutting-edge language models and large language models(https://arxiv.org/abs/2501.05443)
• Keywords: large language model
• Abstract: The success of social media platforms has facilitated the emergence of various forms of online abuse within digital communities. This abuse manifests in multiple ways, including hate speech, cyberbullying, emotional abuse, grooming, and sexting. In this paper, we present a comprehensive analysis of the different forms of abuse prevalent in social media, with a particular focus on how emerging technologies, such as Language Models (LMs) and Large Language Models (LLMs), are reshaping both the detection and generation of abusive content within these networks. We delve into the mechanisms through which social media abuse is perpetuated, exploring the psychological and social impact. Additionally, we examine the dual role of advanced language models-highlighting their potential to enhance automated detection systems for abusive behavior while also acknowledging their capacity to generate harmful content. This paper aims to contribute to the ongoing discourse on online safety and ethics, offering insights into the evolving landscape of cyberabuse and the technological innovations that both mitigate and exacerbate it.

Title: Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark

• Authors: Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, Yu Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05444
• Pdf URL: https://arxiv.org/pdf/2501.05444
• Copy Paste: [[2501.05444]] Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark(https://arxiv.org/abs/2501.05444)
• Keywords: large language model
• Abstract: The ability to organically reason over and with both text and images is a pillar of human intelligence, yet the ability of Multimodal Large Language Models (MLLMs) to perform such multimodal reasoning remains under-explored. Existing benchmarks often emphasize text-dominant reasoning or rely on shallow visual cues, failing to adequately assess integrated visual and textual reasoning. We introduce EMMA (Enhanced MultiModal reAsoning), a benchmark targeting organic multimodal reasoning across mathematics, physics, chemistry, and coding. EMMA tasks demand advanced cross-modal reasoning that cannot be addressed by reasoning independently in each modality, offering an enhanced test suite for MLLMs' reasoning capabilities. Our evaluation of state-of-the-art MLLMs on EMMA reveals significant limitations in handling complex multimodal and multi-step reasoning tasks, even with advanced techniques like Chain-of-Thought prompting and test-time compute scaling underperforming. These findings underscore the need for improved multimodal architectures and training paradigms to close the gap between human and model reasoning in multimodality.

Title: Consistent Flow Distillation for Text-to-3D Generation

• Authors: Runjie Yan, Yinbo Chen, Xiaolong Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05445
• Pdf URL: https://arxiv.org/pdf/2501.05445
• Copy Paste: [[2501.05445]] Consistent Flow Distillation for Text-to-3D Generation(https://arxiv.org/abs/2501.05445)
• Keywords: diffusion, generative
• Abstract: Score Distillation Sampling (SDS) has made significant strides in distilling image-generative models for 3D generation. However, its maximum-likelihood-seeking behavior often leads to degraded visual quality and diversity, limiting its effectiveness in 3D applications. In this work, we propose Consistent Flow Distillation (CFD), which addresses these limitations. We begin by leveraging the gradient of the diffusion ODE or SDE sampling process to guide the 3D generation. From the gradient-based sampling perspective, we find that the consistency of 2D image flows across different viewpoints is important for high-quality 3D generation. To achieve this, we introduce multi-view consistent Gaussian noise on the 3D object, which can be rendered from various viewpoints to compute the flow gradient. Our experiments demonstrate that CFD, through consistent flows, significantly outperforms previous methods in text-to-3D generation.

Title: Decentralized Diffusion Models

• Authors: David McAllister, Matthew Tancik, Jiaming Song, Angjoo Kanazawa
• Subjects: cs.CV, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05450
• Pdf URL: https://arxiv.org/pdf/2501.05450
• Copy Paste: [[2501.05450]] Decentralized Diffusion Models(https://arxiv.org/abs/2501.05450)
• Keywords: diffusion
• Abstract: Large-scale AI model training divides work across thousands of GPUs, then synchronizes gradients across them at each step. This incurs a significant network burden that only centralized, monolithic clusters can support, driving up infrastructure costs and straining power systems. We propose Decentralized Diffusion Models, a scalable framework for distributing diffusion model training across independent clusters or datacenters by eliminating the dependence on a centralized, high-bandwidth networking fabric. Our method trains a set of expert diffusion models over partitions of the dataset, each in full isolation from one another. At inference time, the experts ensemble through a lightweight router. We show that the ensemble collectively optimizes the same objective as a single model trained over the whole dataset. This means we can divide the training burden among a number of "compute islands," lowering infrastructure costs and improving resilience to localized GPU failures. Decentralized diffusion models empower researchers to take advantage of smaller, more cost-effective and more readily available compute like on-demand GPU nodes rather than central integrated systems. We conduct extensive experiments on ImageNet and LAION Aesthetics, showing that decentralized diffusion models FLOP-for-FLOP outperform standard diffusion models. We finally scale our approach to 24 billion parameters, demonstrating that high-quality diffusion models can now be trained with just eight individual GPU nodes in less than a week.

Title: ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding

• Authors: Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Corring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei Florencio, Cha Zhang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05452
• Pdf URL: https://arxiv.org/pdf/2501.05452
• Copy Paste: [[2501.05452]] ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding(https://arxiv.org/abs/2501.05452)
• Keywords: large language model
• Abstract: Structured image understanding, such as interpreting tables and charts, requires strategically refocusing across various structures and texts within an image, forming a reasoning sequence to arrive at the final answer. However, current multimodal large language models (LLMs) lack this multihop selective attention capability. In this work, we introduce ReFocus, a simple yet effective framework that equips multimodal LLMs with the ability to generate "visual thoughts" by performing visual editing on the input image through code, shifting and refining their visual focuses. Specifically, ReFocus enables multimodal LLMs to generate Python codes to call tools and modify the input image, sequentially drawing boxes, highlighting sections, and masking out areas, thereby enhancing the visual reasoning process. We experiment upon a wide range of structured image understanding tasks involving tables and charts. ReFocus largely improves performance on all tasks over GPT-4o without visual editing, yielding an average gain of 11.0% on table tasks and 6.8% on chart tasks. We present an in-depth analysis of the effects of different visual edits, and reasons why ReFocus can improve the performance without introducing additional information. Further, we collect a 14k training set using ReFocus, and prove that such visual chain-of-thought with intermediate information offers a better supervision than standard VQA data, reaching a 8.0% average gain over the same model trained with QA pairs and 2.6% over CoT.

Title: An Empirical Study of Autoregressive Pre-training from Videos

• Authors: Jathushan Rajasegaran, Ilija Radosavovic, Rahul Ravishankar, Yossi Gandelsman, Christoph Feichtenhofer, Jitendra Malik
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05453
• Pdf URL: https://arxiv.org/pdf/2501.05453
• Copy Paste: [[2501.05453]] An Empirical Study of Autoregressive Pre-training from Videos(https://arxiv.org/abs/2501.05453)
• Keywords: transformer
• Abstract: We empirically study autoregressive pre-training from videos. To perform our study, we construct a series of autoregressive video models, called Toto. We treat videos as sequences of visual tokens and train transformer models to autoregressively predict future tokens. Our models are pre-trained on a diverse dataset of videos and images comprising over 1 trillion visual tokens. We explore different architectural, training, and inference design choices. We evaluate the learned visual representations on a range of downstream tasks including image recognition, video classification, object tracking, and robotics. Our results demonstrate that, despite minimal inductive biases, autoregressive pre-training leads to competitive performance across all benchmarks. Finally, we find that scaling our video models results in similar scaling curves to those seen in language models, albeit with a different rate. More details at this https URL