2025-01-08

Title: Performance Comparison of Security Credential Management Systems for V2X: North American Standard IEEE 1609.2.1 and European Standard ETSI TS 102 941

• Authors: Abel C. H. Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.03237
• Pdf URL: https://arxiv.org/pdf/2501.03237
• Copy Paste: [[2501.03237]] Performance Comparison of Security Credential Management Systems for V2X: North American Standard IEEE 1609.2.1 and European Standard ETSI TS 102 941(https://arxiv.org/abs/2501.03237)
• Keywords: security
• Abstract: This study examines the performance and structural differences between the two primary standards for Security Credential Management Systems (SCMS) in Vehicular-to-Everything (V2X) communication: the North American IEEE standards and the European ETSI standards. It focuses on comparing their respective Public Key Infrastructure (PKI) architectures, certificate application workflows, and the formats of request/response messages used during certificate applications. The research includes a theoretical analysis of message length and security features inherent in each standard. Additionally, practical implementations of both systems are conducted to evaluate their efficiency in certificate management processes. Based on the findings, the study provides recommendations to guide future development of SCMS.

Title: gECC: A GPU-based high-throughput framework for Elliptic Curve Cryptography

• Authors: Qian Xiong, Weiliang Ma, Xuanhua Shi, Yongluan Zhou, Hai Jin, Kaiyi Huang, Haozhou Wang, Zhengru Wang
• Subjects: cs.CR, cs.AR, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.03245
• Pdf URL: https://arxiv.org/pdf/2501.03245
• Copy Paste: [[2501.03245]] gECC: A GPU-based high-throughput framework for Elliptic Curve Cryptography(https://arxiv.org/abs/2501.03245)
• Keywords: secure, security
• Abstract: Elliptic Curve Cryptography (ECC) is an encryption method that provides security comparable to traditional techniques like Rivest-Shamir-Adleman (RSA) but with lower computational complexity and smaller key sizes, making it a competitive option for applications such as blockchain, secure multi-party computation, and database security. However, the throughput of ECC is still hindered by the significant performance overhead associated with elliptic curve (EC) operations. This paper presents gECC, a versatile framework for ECC optimized for GPU architectures, specifically engineered to achieve high-throughput performance in EC operations. gECC incorporates batch-based execution of EC operations and microarchitecture-level optimization of modular arithmetic. It employs Montgomery's trick to enable batch EC computation and incorporates novel computation parallelization and memory management techniques to maximize the computation parallelism and minimize the access overhead of GPU global memory. Also, we analyze the primary bottleneck in modular multiplication by investigating how the user codes of modular multiplication are compiled into hardware instructions and what these instructions' issuance rates are. We identify that the efficiency of modular multiplication is highly dependent on the number of Integer Multiply-Add (IMAD) instructions. To eliminate this bottleneck, we propose techniques to minimize the number of IMAD instructions by leveraging predicate registers to pass the carry information and using addition and subtraction instructions (IADD3) to replace IMAD instructions. Our results show that, for ECDSA and ECDH, gECC can achieve performance improvements of 5.56x and 4.94x, respectively, compared to the state-of-the-art GPU-based system. In a real-world blockchain application, we can achieve performance improvements of 1.56x, compared to the state-of-the-art CPU-based system.

Title: Homomorphic Encryption Based on Lattice Post-Quantum Cryptography

• Authors: Abel C. H. Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.03249
• Pdf URL: https://arxiv.org/pdf/2501.03249
• Copy Paste: [[2501.03249]] Homomorphic Encryption Based on Lattice Post-Quantum Cryptography(https://arxiv.org/abs/2501.03249)
• Keywords: security, attack, federate
• Abstract: As quantum computing technology continues to advance, post-quantum cryptographic methods capable of resisting quantum attacks have emerged as a critical area of focus. Given the potential vulnerability of existing homomorphic encryption methods, such as RSA, ElGamal, and Paillier, to quantum computing attacks, this study proposes a lattice-based post-quantum homomorphic encryption scheme. The approach leverages lattice cryptography to build resilience against quantum threats while enabling practical homomorphic encryption applications. This research provides mathematical proofs and computational examples, alongside a security analysis of the lattice-based post-quantum homomorphic encryption scheme. The findings are intended to serve as a reference for developers of homomorphic encryption applications, such as federated learning systems.

Title: Machine Learning and Deep Learning Techniques used in Cybersecurity and Digital Forensics: a Review

• Authors: Jaouhar Fattahi
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03250
• Pdf URL: https://arxiv.org/pdf/2501.03250
• Copy Paste: [[2501.03250]] Machine Learning and Deep Learning Techniques used in Cybersecurity and Digital Forensics: a Review(https://arxiv.org/abs/2501.03250)
• Keywords: security, attack
• Abstract: In the paced realms of cybersecurity and digital forensics machine learning (ML) and deep learning (DL) have emerged as game changing technologies that introduce methods to identify stop and analyze cyber risks. This review presents an overview of the ML and DL approaches used in these fields showcasing their advantages drawbacks and possibilities. It covers a range of AI techniques used in spotting intrusions in systems and classifying malware to prevent cybersecurity attacks, detect anomalies and enhance resilience. This study concludes by highlighting areas where further research is needed and suggesting ways to create transparent and scalable ML and DL solutions that are suited to the evolving landscape of cybersecurity and digital forensics.

Title: AI-ANNE: (A) (N)eural (N)et for (E)xploration: Transferring Deep Learning Models onto Microcontrollers and Embedded Systems

• Authors: Dennis Klinkhammer
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03256
• Pdf URL: https://arxiv.org/pdf/2501.03256
• Copy Paste: [[2501.03256]] AI-ANNE: (A) (N)eural (N)et for (E)xploration: Transferring Deep Learning Models onto Microcontrollers and Embedded Systems(https://arxiv.org/abs/2501.03256)
• Keywords: privacy
• Abstract: This working paper explores the integration of neural networks onto resource-constrained embedded systems like a Raspberry Pi Pico / Raspberry Pi Pico 2. A TinyML aproach transfers neural networks directly on these microcontrollers, enabling real-time, low-latency, and energy-efficient inference while maintaining data privacy. Therefore, AI-ANNE: (A) (N)eural (N)et for (E)xploration will be presented, which facilitates the transfer of pre-trained models from high-performance platforms like TensorFlow and Keras onto microcontrollers, using a lightweight programming language like MicroPython. This approach demonstrates how neural network architectures, such as neurons, layers, density and activation functions can be implemented in MicroPython in order to deal with the computational limitations of embedded systems. Based on the Raspberry Pi Pico / Raspberry Pi Pico 2, two different neural networks on microcontrollers are presented for an example of data classification. As an further application example, such a microcontroller can be used for condition monitoring, where immediate corrective measures are triggered on the basis of sensor data. Overall, this working paper presents a very easy-to-implement way of using neural networks on energy-efficient devices such as microcontrollers. This makes AI-ANNE: (A) (N)eural (N)et for (E)xploration not only suited for practical use, but also as an educational tool with clear insights into how neural networks operate.

Title: Toward Inclusive Educational AI: Auditing Frontier LLMs through a Multiplexity Lens

• Authors: Abdullah Mushtaq, Muhammad Rafay Naeem, Muhammad Imran Taj, Ibrahim Ghaznavi, Junaid Qadir
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.03259
• Pdf URL: https://arxiv.org/pdf/2501.03259
• Copy Paste: [[2501.03259]] Toward Inclusive Educational AI: Auditing Frontier LLMs through a Multiplexity Lens(https://arxiv.org/abs/2501.03259)
• Keywords: generative, large language model
• Abstract: As large language models (LLMs) like GPT-4 and Llama 3 become integral to educational contexts, concerns are mounting over the cultural biases, power imbalances, and ethical limitations embedded within these technologies. Though generative AI tools aim to enhance learning experiences, they often reflect values rooted in Western, Educated, Industrialized, Rich, and Democratic (WEIRD) cultural paradigms, potentially sidelining diverse global perspectives. This paper proposes a framework to assess and mitigate cultural bias within LLMs through the lens of applied multiplexity. Multiplexity, inspired by Senturk et al. and rooted in Islamic and other wisdom traditions, emphasizes the coexistence of diverse cultural viewpoints, supporting a multi-layered epistemology that integrates both empirical sciences and normative values. Our analysis reveals that LLMs frequently exhibit cultural polarization, with biases appearing in both overt responses and subtle contextual cues. To address inherent biases and incorporate multiplexity in LLMs, we propose two strategies: \textit{Contextually-Implemented Multiplex LLMs}, which embed multiplex principles directly into the system prompt, influencing LLM outputs at a foundational level and independent of individual prompts, and \textit{Multi-Agent System (MAS)-Implemented Multiplex LLMs}, where multiple LLM agents, each representing distinct cultural viewpoints, collaboratively generate a balanced, synthesized response. Our findings demonstrate that as mitigation strategies evolve from contextual prompting to MAS-implementation, cultural inclusivity markedly improves, evidenced by a significant rise in the Perspectives Distribution Score (PDS) and a PDS Entropy increase from 3.25\% at baseline to 98\% with the MAS-Implemented Multiplex LLMs. Sentiment analysis further shows a shift towards positive sentiment across cultures,...

Title: REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models

• Authors: Jian Hu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03262
• Pdf URL: https://arxiv.org/pdf/2501.03262
• Copy Paste: [[2501.03262]] REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models(https://arxiv.org/abs/2501.03262)
• Keywords: large language model
• Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a critical approach for aligning large language models with human preferences, witnessing rapid algorithmic evolution through methods such as Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), REINFORCE Leave One-Out (RLOO), ReMax, and Group Relative Policy Optimization (GRPO). We present REINFORCE++, an enhanced variant of the classical REINFORCE algorithm that incorporates key optimization techniques from PPO while eliminating the need for a critic network. REINFORCE++ achieves three primary objectives: (1) simplicity (2) enhanced training stability, and (3) reduced computational overhead. Through extensive empirical evaluation, we demonstrate that REINFORCE++ exhibits superior stability compared to GRPO and achieves greater computational efficiency than PPO while maintaining comparable performance. The implementation is available at \url{this https URL}.

Title: Heterogeneous Graph Pre-training Based Model for Secure and Efficient Prediction of Default Risk Propagation among Bond Issuers

• Authors: Xurui Li, Xin Shan, Wenhao Yin, Haijiao Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03268
• Pdf URL: https://arxiv.org/pdf/2501.03268
• Copy Paste: [[2501.03268]] Heterogeneous Graph Pre-training Based Model for Secure and Efficient Prediction of Default Risk Propagation among Bond Issuers(https://arxiv.org/abs/2501.03268)
• Keywords: secure, security, privacy
• Abstract: Efficient prediction of default risk for bond-issuing enterprises is pivotal for maintaining stability and fostering growth in the bond market. Conventional methods usually rely solely on an enterprise's internal data for risk assessment. In contrast, graph-based techniques leverage interconnected corporate information to enhance default risk identification for targeted bond issuers. Traditional graph techniques such as label propagation algorithm or deepwalk fail to effectively integrate a enterprise's inherent attribute information with its topological network data. Additionally, due to data scarcity and security privacy concerns between enterprises, end-to-end graph neural network (GNN) algorithms may struggle in delivering satisfactory performance for target tasks. To address these challenges, we present a novel two-stage model. In the first stage, we employ an innovative Masked Autoencoders for Heterogeneous Graph (HGMAE) to pre-train on a vast enterprise knowledge graph. Subsequently, in the second stage, a specialized classifier model is trained to predict default risk propagation probabilities. The classifier leverages concatenated feature vectors derived from the pre-trained encoder with the enterprise's task-specific feature vectors. Through the two-stage training approach, our model not only boosts the importance of unique bond characteristics for specific default prediction tasks, but also securely and efficiently leverage the global information pre-trained from other enterprises. Experimental results demonstrate that our proposed model outperforms existing approaches in predicting default risk for bond issuers.

Title: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization

• Authors: Amitava Das, Suranjana Trivedy, Danush Khanna, Rajarshi Roy, Gurpreet Singh, Basab Ghosh, Yaswanth Narsupalli, Vinija Jain, Vasu Sharma, Aishwarya Naresh Reganti, Aman Chadha
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03271
• Pdf URL: https://arxiv.org/pdf/2501.03271
• Copy Paste: [[2501.03271]] A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization(https://arxiv.org/abs/2501.03271)
• Keywords: robust, large language model
• Abstract: The rapid rise of large language models (LLMs) has unlocked many applications but also underscores the challenge of aligning them with diverse values and preferences. Direct Preference Optimization (DPO) is central to alignment but constrained by fixed divergences and limited feature transformations. We propose DPO-Kernels, which integrates kernel methods to address these issues through four key contributions: (i) Kernelized Representations with polynomial, RBF, Mahalanobis, and spectral kernels for richer transformations, plus a hybrid loss combining embedding-based and probability-based objectives; (ii) Divergence Alternatives (Jensen-Shannon, Hellinger, Renyi, Bhattacharyya, Wasserstein, and f-divergences) for greater stability; (iii) Data-Driven Selection metrics that automatically choose the best kernel-divergence pair; and (iv) a Hierarchical Mixture of Kernels for both local precision and global modeling. Evaluations on 12 datasets demonstrate state-of-the-art performance in factuality, safety, reasoning, and instruction following. Grounded in Heavy-Tailed Self-Regularization, DPO-Kernels maintains robust generalization for LLMs, offering a comprehensive resource for further alignment research.

Title: Backdoor Token Unlearning: Exposing and Defending Backdoors in Pretrained Language Models

• Authors: Peihai Jiang, Xixiang Lyu, Yige Li, Jing Ma
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03272
• Pdf URL: https://arxiv.org/pdf/2501.03272
• Copy Paste: [[2501.03272]] Backdoor Token Unlearning: Exposing and Defending Backdoors in Pretrained Language Models(https://arxiv.org/abs/2501.03272)
• Keywords: defense, attack
• Abstract: Supervised fine-tuning has become the predominant method for adapting large pretrained models to downstream tasks. However, recent studies have revealed that these models are vulnerable to backdoor attacks, where even a small number of malicious samples can successfully embed backdoor triggers into the model. While most existing defense methods focus on post-training backdoor defense, efficiently defending against backdoor attacks during training phase remains largely unexplored. To address this gap, we propose a novel defense method called Backdoor Token Unlearning (BTU), which proactively detects and neutralizes trigger tokens during the training stage. Our work is based on two key findings: 1) backdoor learning causes distinctive differences between backdoor token parameters and clean token parameters in word embedding layers, and 2) the success of backdoor attacks heavily depends on backdoor token parameters. The BTU defense leverages these properties to identify aberrant embedding parameters and subsequently removes backdoor behaviors using a fine-grained unlearning technique. Extensive evaluations across three datasets and four types of backdoor attacks demonstrate that BTU effectively defends against these threats while preserving the model's performance on primary tasks. Our code is available at this https URL.

Title: Strategic Fusion Optimizes Transformer Compression

• Authors: Md Shoaibur Rahman
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03273
• Pdf URL: https://arxiv.org/pdf/2501.03273
• Copy Paste: [[2501.03273]] Strategic Fusion Optimizes Transformer Compression(https://arxiv.org/abs/2501.03273)
• Keywords: transformer
• Abstract: This study investigates transformer model compression by systematically pruning its layers. We evaluated 14 pruning strategies across nine diverse datasets, including 12 strategies based on different signals obtained from layer activations, mutual information, gradients, weights, and attention. To address the limitations of single-signal strategies, we introduced two fusion strategies, linear regression and random forest, which combine individual strategies (i.e., strategic fusion), for more informed pruning decisions. Additionally, we applied knowledge distillation to mitigate any accuracy loss during layer pruning. Our results reveal that random forest strategic fusion outperforms individual strategies in seven out of nine datasets and achieves near-optimal performance in the other two. The distilled random forest surpasses the original accuracy in six datasets and mitigates accuracy drops in the remaining three. Knowledge distillation also improves the accuracy-to-size ratio by an average factor of 18.84 across all datasets. Supported by mathematical foundations and biological analogies, our findings suggest that strategically combining multiple signals can lead to efficient, high-performing transformer models for resource-constrained applications.

Title: ComMer: a Framework for Compressing and Merging User Data for Personalization

• Authors: Yoel Zeldes, Amir Zait, Ilia Labzovsky, Danny Karmon, Efrat Farkash
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03276
• Pdf URL: https://arxiv.org/pdf/2501.03276
• Copy Paste: [[2501.03276]] ComMer: a Framework for Compressing and Merging User Data for Personalization(https://arxiv.org/abs/2501.03276)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) excel at a wide range of tasks, but adapting them to new data, particularly for personalized applications, poses significant challenges due to resource and computational constraints. Existing methods either rely on exposing fresh data to the model through the prompt, which is limited by context size and computationally expensive at inference time, or fine-tuning, which incurs substantial training and update costs. In this paper, we introduce ComMer - Compress and Merge - a novel framework that efficiently personalizes LLMs by compressing users' documents into compact representations, which are then merged and fed into a frozen LLM. We evaluate ComMer on two types of personalization tasks - personalized skill learning, using the tweet paraphrasing dataset and the personalized news headline generation dataset from the LaMP benchmark, and knowledge-intensive, using the PerLTQA dataset. Our experiments demonstrate that in constrained inference budget scenarios ComMer achieves superior quality in skill learning tasks, while highlighting limitations in knowledge-intensive settings due to the loss of detailed information. These results offer insights into trade-offs and potential optimizations in multi-document compression for personalization.

Title: Revolutionizing Encrypted Traffic Classification with MH-Net: A Multi-View Heterogeneous Graph Model

• Authors: Haozhen Zhang, Haodong Yue, Xi Xiao, Le Yu, Qing Li, Zhen Ling, Ye Zhang
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03279
• Pdf URL: https://arxiv.org/pdf/2501.03279
• Copy Paste: [[2501.03279]] Revolutionizing Encrypted Traffic Classification with MH-Net: A Multi-View Heterogeneous Graph Model(https://arxiv.org/abs/2501.03279)
• Keywords: security, robust
• Abstract: With the growing significance of network security, the classification of encrypted traffic has emerged as an urgent challenge. Traditional byte-based traffic analysis methods are constrained by the rigid granularity of information and fail to fully exploit the diverse correlations between bytes. To address these limitations, this paper introduces MH-Net, a novel approach for classifying network traffic that leverages multi-view heterogeneous traffic graphs to model the intricate relationships between traffic bytes. The essence of MH-Net lies in aggregating varying numbers of traffic bits into multiple types of traffic units, thereby constructing multi-view traffic graphs with diverse information granularities. By accounting for different types of byte correlations, such as header-payload relationships, MH-Net further endows the traffic graph with heterogeneity, significantly enhancing model performance. Notably, we employ contrastive learning in a multi-task manner to strengthen the robustness of the learned traffic unit representations. Experiments conducted on the ISCX and CIC-IoT datasets for both the packet-level and flow-level traffic classification tasks demonstrate that MH-Net achieves the best overall performance compared to dozens of SOTA methods.

Title: Sensorformer: Cross-patch attention with global-patch compression is effective for high-dimensional multivariate time series forecasting

• Authors: Liyang Qin, Xiaoli Wang, Chunhua Yang, Huaiwen Zou, Haochuan Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03284
• Pdf URL: https://arxiv.org/pdf/2501.03284
• Copy Paste: [[2501.03284]] Sensorformer: Cross-patch attention with global-patch compression is effective for high-dimensional multivariate time series forecasting(https://arxiv.org/abs/2501.03284)
• Keywords: transformer
• Abstract: Among the existing Transformer-based multivariate time series forecasting methods, iTransformer, which treats each variable sequence as a token and only explicitly extracts cross-variable dependencies, and PatchTST, which adopts a channel-independent strategy and only explicitly extracts cross-time dependencies, both significantly outperform most Channel-Dependent Transformer that simultaneously extract cross-time and cross-variable dependencies. This indicates that existing Transformer-based multivariate time series forecasting methods still struggle to effectively fuse these two types of information. We attribute this issue to the dynamic time lags in the causal relationships between different variables. Therefore, we propose a new multivariate time series forecasting Transformer, Sensorformer, which first compresses the global patch information and then simultaneously extracts cross-variable and cross-time dependencies from the compressed representations. Sensorformer can effectively capture the correct inter-variable correlations and causal relationships, even in the presence of dynamic causal lags between variables, while also reducing the computational complexity of pure cross-patch self-attention from $O(D^2 \cdot Patch\_num^2 \cdot d\_model)$ to $O(D^2 \cdot Patch\_num \cdot d\_model)$. Extensive comparative and ablation experiments on 9 mainstream real-world multivariate time series forecasting datasets demonstrate the superiority of Sensorformer. The implementation of Sensorformer, following the style of the Time-series-library and scripts for reproducing the main results, is publicly available at this https URL

Title: Adaptive Pruning of Pretrained Transformer via Differential Inclusions

• Authors: Yizhuo Ding, Ke Fan, Yikai Wang, Xinwei Sun, Yanwei Fu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03289
• Pdf URL: https://arxiv.org/pdf/2501.03289
• Copy Paste: [[2501.03289]] Adaptive Pruning of Pretrained Transformer via Differential Inclusions(https://arxiv.org/abs/2501.03289)
• Keywords: transformer
• Abstract: Large transformers have demonstrated remarkable success, making it necessary to compress these models to reduce inference costs while preserving their perfor-mance. Current compression algorithms prune transformers at fixed compression ratios, requiring a unique pruning process for each ratio, which results in high computational costs. In contrast, we propose pruning of pretrained transformers at any desired ratio within a single pruning stage, based on a differential inclusion for a mask parameter. This dynamic can generate the whole regularization solution path of the mask parameter, whose support set identifies the network structure. Therefore, the solution path identifies a Transformer weight family with various sparsity levels, offering greater flexibility and customization. In this paper, we introduce such an effective pruning method, termed SPP (Solution Path Pruning). To achieve effective pruning, we segment the transformers into paired modules, including query-key pairs, value-projection pairs, and sequential linear layers, and apply low-rank compression to these pairs, maintaining the output structure while enabling structural compression within the inner states. Extensive experiments conducted on various well-known transformer backbones have demonstrated the efficacy of SPP.

Title: A Decision-Based Heterogenous Graph Attention Network for Multi-Class Fake News Detection

• Authors: Batool Lakzaei, Mostafa Haghir Chehreghani, Alireza Bagheri
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.03290
• Pdf URL: https://arxiv.org/pdf/2501.03290
• Copy Paste: [[2501.03290]] A Decision-Based Heterogenous Graph Attention Network for Multi-Class Fake News Detection(https://arxiv.org/abs/2501.03290)
• Keywords: robust
• Abstract: A promising tool for addressing fake news detection is Graph Neural Networks (GNNs). However, most existing GNN-based methods rely on binary classification, categorizing news as either real or fake. Additionally, traditional GNN models use a static neighborhood for each node, making them susceptible to issues like over-squashing. In this paper, we introduce a novel model named Decision-based Heterogeneous Graph Attention Network (DHGAT) for fake news detection in a semi-supervised setting. DHGAT effectively addresses the limitations of traditional GNNs by dynamically optimizing and selecting the neighborhood type for each node in every layer. It represents news data as a heterogeneous graph where nodes (news items) are connected by various types of edges. The architecture of DHGAT consists of a decision network that determines the optimal neighborhood type and a representation network that updates node embeddings based on this selection. As a result, each node learns an optimal and task-specific computational graph, enhancing both the accuracy and efficiency of the fake news detection process. We evaluate DHGAT on the LIAR dataset, a large and challenging dataset for multi-class fake news detection, which includes news items categorized into six classes. Our results demonstrate that DHGAT outperforms existing methods, improving accuracy by approximately 4% and showing robustness with limited labeled data.

Title: ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning

• Authors: Pengwei Tang, Xiaolin Hu, Yong Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03291
• Pdf URL: https://arxiv.org/pdf/2501.03291
• Copy Paste: [[2501.03291]] ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning(https://arxiv.org/abs/2501.03291)
• Keywords: large language model
• Abstract: Prompt Tuning (PT) enables the adaptation of Pre-trained Large Language Models (PLMs) to downstream tasks by optimizing a small amount of soft virtual tokens, which are prepended to the input token embeddings. Recently, Decomposed Prompt Tuning (DePT) has demonstrated superior adaptation capabilities by decomposing the soft prompt into a shorter soft prompt and a pair of low-rank matrices. The product of the pair of low-rank matrices is added to the input token embeddings to offset them. Additionally, DePT achieves faster inference compared to PT due to the shorter soft prompt. However, in this paper, we find that the position-based token embedding offsets of DePT restricts its ability to generalize across diverse model inputs, and that the shared embedding offsets across many token embeddings result in sub-optimization. To tackle these issues, we introduce \textbf{A}daptive \textbf{De}composed \textbf{P}rompt \textbf{T}uning (ADePT), which is composed of a short soft prompt and a shallow token-shared feed-forward neural network. ADePT utilizes the token-shared feed-forward neural network to learn the embedding offsets for each token, enabling adaptive embedding offsets that vary according to the model input and better optimization of token embedding offsets. This enables ADePT to achieve superior adaptation performance without requiring more inference time or additional trainable parameters compared to vanilla PT and its variants. In comprehensive experiments across 23 natural language processing (NLP) tasks and 4 typical PLMs of different scales, we show that ADePT consistently surpasses the leading parameter-efficient fine-tuning (PEFT) methods, and even outperforms the full fine-tuning baseline in certain scenarios. Code is available at \url{this https URL}.

Title: Multi-Modal One-Shot Federated Ensemble Learning for Medical Data with Vision Large Language Model

• Authors: Naibo Wang, Yuchen Deng, Shichen Fan, Jianwei Yin, See-Kiong Ng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03292
• Pdf URL: https://arxiv.org/pdf/2501.03292
• Copy Paste: [[2501.03292]] Multi-Modal One-Shot Federated Ensemble Learning for Medical Data with Vision Large Language Model(https://arxiv.org/abs/2501.03292)
• Keywords: privacy, federate, large language model
• Abstract: Federated learning (FL) has attracted considerable interest in the medical domain due to its capacity to facilitate collaborative model training while maintaining data privacy. However, conventional FL methods typically necessitate multiple communication rounds, leading to significant communication overhead and delays, especially in environments with limited bandwidth. One-shot federated learning addresses these issues by conducting model training and aggregation in a single communication round, thereby reducing communication costs while preserving privacy. Among these, one-shot federated ensemble learning combines independently trained client models using ensemble techniques such as voting, further boosting performance in non-IID data scenarios. On the other hand, existing machine learning methods in healthcare predominantly use unimodal data (e.g., medical images or textual reports), which restricts their diagnostic accuracy and comprehensiveness. Therefore, the integration of multi-modal data is proposed to address these shortcomings. In this paper, we introduce FedMME, an innovative one-shot multi-modal federated ensemble learning framework that utilizes multi-modal data for medical image analysis. Specifically, FedMME capitalizes on vision large language models to produce textual reports from medical images, employs a BERT model to extract textual features from these reports, and amalgamates these features with visual features to improve diagnostic accuracy. Experimental results show that our method demonstrated superior performance compared to existing one-shot federated learning methods in healthcare scenarios across four datasets with various data distributions. For instance, it surpasses existing one-shot federated learning approaches by more than 17.5% in accuracy on the RSNA dataset when applying a Dirichlet distribution with ($\alpha$ = 0.3).

Title: A Soft Sensor Method with Uncertainty-Awareness and Self-Explanation Based on Large Language Models Enhanced by Domain Knowledge Retrieval

• Authors: Shuo Tong, Runyuan Guo, Wenqing Wang, Xueqiong Tian, Lingyun Wei, Lin Zhang, Huayong Wu, Ding Liu, Youmin Zhang
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2501.03295
• Pdf URL: https://arxiv.org/pdf/2501.03295
• Copy Paste: [[2501.03295]] A Soft Sensor Method with Uncertainty-Awareness and Self-Explanation Based on Large Language Models Enhanced by Domain Knowledge Retrieval(https://arxiv.org/abs/2501.03295)
• Keywords: robust, interpretability, large language model
• Abstract: Data-driven soft sensors are crucial in predicting key performance indicators in industrial systems. However, current methods predominantly rely on the supervised learning paradigms of parameter updating, which inherently faces challenges such as high development costs, poor robustness, training instability, and lack of interpretability. Recently, large language models (LLMs) have demonstrated significant potential across various domains, notably through In-Context Learning (ICL), which enables high-performance task execution with minimal input-label demonstrations and no prior training. This paper aims to replace supervised learning with the emerging ICL paradigm for soft sensor modeling to address existing challenges and explore new avenues for advancement. To achieve this, we propose a novel framework called the Few-shot Uncertainty-aware and self-Explaining Soft Sensor (LLM-FUESS), which includes the Zero-shot Auxiliary Variable Selector (LLM-ZAVS) and the Uncertainty-aware Few-shot Soft Sensor (LLM-UFSS). The LLM-ZAVS retrieves from the Industrial Knowledge Vector Storage to enhance LLMs' domain-specific knowledge, enabling zero-shot auxiliary variable selection. In the LLM-UFSS, we utilize text-based context demonstrations of structured data to prompt LLMs to execute ICL for predicting and propose a context sample retrieval augmentation strategy to improve performance. Additionally, we explored LLMs' AIGC and probabilistic characteristics to propose self-explanation and uncertainty quantification methods for constructing a trustworthy soft sensor. Extensive experiments demonstrate that our method achieved state-of-the-art predictive performance, strong robustness, and flexibility, effectively mitigates training instability found in traditional methods. To the best of our knowledge, this is the first work to establish soft sensor utilizing LLMs.

Title: Dynamic Data Defense: Unveiling the Database in motion Chaos Encryption (DaChE) Algorithm -- A Breakthrough in Chaos Theory for Enhanced Database Security

• Authors: Abraham Itzhak Weinberg
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.03296
• Pdf URL: https://arxiv.org/pdf/2501.03296
• Copy Paste: [[2501.03296]] Dynamic Data Defense: Unveiling the Database in motion Chaos Encryption (DaChE) Algorithm -- A Breakthrough in Chaos Theory for Enhanced Database Security(https://arxiv.org/abs/2501.03296)
• Keywords: security, protect, defense, attack
• Abstract: Amidst the burgeoning landscape of database architectures, the surge in NoSQL databases has heralded a transformative era, liberating data storage from traditional relational constraints and ushering in unprecedented scalability. As organizations grapple with the escalating security threats posed by database breaches, a novel theoretical framework emerges at the nexus of chaos theory and topology: the Database in motion Chaos Encryption (DaChE) Algorithm. This paradigm-shifting approach challenges the static nature of data storage, advocating for dynamic data motion to fortify database security. By incorporating chaos theory, this innovative strategy not only enhances database defenses against evolving attack vectors but also redefines the boundaries of data protection, offering a paradigmatic shift in safeguarding critical information assets. Additionally, it enables parallel processing, facilitating on-the-fly processing and optimizing the performance of the proposed framework.

Title: Rethinking Byzantine Robustness in Federated Recommendation from Sparse Aggregation Perspective

• Authors: Zhongjian Zhang, Mengmei Zhang, Xiao Wang, Lingjuan Lyu, Bo Yan, Junping Du, Chuan Shi
• Subjects: cs.CR, cs.AI, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03301
• Pdf URL: https://arxiv.org/pdf/2501.03301
• Copy Paste: [[2501.03301]] Rethinking Byzantine Robustness in Federated Recommendation from Sparse Aggregation Perspective(https://arxiv.org/abs/2501.03301)
• Keywords: security, privacy, defense, attack, robust, federate
• Abstract: To preserve user privacy in recommender systems, federated recommendation (FR) based on federated learning (FL) emerges, keeping the personal data on the local client and updating a model collaboratively. Unlike FL, FR has a unique sparse aggregation mechanism, where the embedding of each item is updated by only partial clients, instead of full clients in a dense aggregation of general FL. Recently, as an essential principle of FL, model security has received increasing attention, especially for Byzantine attacks, where malicious clients can send arbitrary updates. The problem of exploring the Byzantine robustness of FR is particularly critical since in the domains applying FR, e.g., e-commerce, malicious clients can be injected easily by registering new accounts. However, existing Byzantine works neglect the unique sparse aggregation of FR, making them unsuitable for our problem. Thus, we make the first effort to investigate Byzantine attacks on FR from the perspective of sparse aggregation, which is non-trivial: it is not clear how to define Byzantine robustness under sparse aggregations and design Byzantine attacks under limited knowledge/capability. In this paper, we reformulate the Byzantine robustness under sparse aggregation by defining the aggregation for a single item as the smallest execution unit. Then we propose a family of effective attack strategies, named Spattack, which exploit the vulnerability in sparse aggregation and are categorized along the adversary's knowledge and capability. Extensive experimental results demonstrate that Spattack can effectively prevent convergence and even break down defenses under a few malicious clients, raising alarms for securing FR systems.

Title: Plant Leaf Disease Detection and Classification Using Deep Learning: A Review and A Proposed System on Bangladesh's Perspective

• Authors: Md. Jalal Uddin Chowdhury, Zumana Islam Mou, Rezwana Afrin, Shafkat Kibria
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03305
• Pdf URL: https://arxiv.org/pdf/2501.03305
• Copy Paste: [[2501.03305]] Plant Leaf Disease Detection and Classification Using Deep Learning: A Review and A Proposed System on Bangladesh's Perspective(https://arxiv.org/abs/2501.03305)
• Keywords: security
• Abstract: A very crucial part of Bangladeshi people's employment, GDP contribution, and mainly livelihood is agriculture. It plays a vital role in decreasing poverty and ensuring food security. Plant diseases are a serious stumbling block in agricultural production in Bangladesh. At times, humans can't detect the disease from an infected leaf with the naked eye. Using inorganic chemicals or pesticides in plants when it's too late leads in vain most of the time, deposing all the previous labor. The deep-learning technique of leaf-based image classification, which has shown impressive results, can make the work of recognizing and classifying all diseases trouble-less and more precise. In this paper, we've mainly proposed a better model for the detection of leaf diseases. Our proposed paper includes the collection of data on three different kinds of crops: bell peppers, tomatoes, and potatoes. For training and testing the proposed CNN model, the plant leaf disease dataset collected from Kaggle is used, which has 17,430 images. The images are labeled with 14 separate classes of damage. The developed CNN model performs efficiently and could successfully detect and classify the tested diseases. The proposed CNN model may have great potency in crop disease management.

Title: The Robustness of Spiking Neural Networks in Federated Learning with Compression Against Non-omniscient Byzantine Attacks

• Authors: Manh V. Nguyen, Liang Zhao, Bobin Deng, Shaoen Wu
• Subjects: cs.CR, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03306
• Pdf URL: https://arxiv.org/pdf/2501.03306
• Copy Paste: [[2501.03306]] The Robustness of Spiking Neural Networks in Federated Learning with Compression Against Non-omniscient Byzantine Attacks(https://arxiv.org/abs/2501.03306)
• Keywords: privacy, attack, robust, federate
• Abstract: Spiking Neural Networks (SNNs), which offer exceptional energy efficiency for inference, and Federated Learning (FL), which offers privacy-preserving distributed training, is a rising area of interest that highly beneficial towards Internet of Things (IoT) devices. Despite this, research that tackles Byzantine attacks and bandwidth limitation in FL-SNNs, both poses significant threats on model convergence and training times, still remains largely unexplored. Going beyond proposing a solution for both of these problems, in this work we highlight the dual benefits of FL-SNNs, against non-omniscient Byzantine adversaries (ones that restrict attackers access to local clients datasets), and greater communication efficiency, over FL-ANNs. Specifically, we discovered that a simple integration of Top-\k{appa} sparsification into the FL apparatus can help leverage the advantages of the SNN models in both greatly reducing bandwidth usage and significantly boosting the robustness of FL training against non-omniscient Byzantine adversaries. Most notably, we saw a massive improvement of roughly 40% accuracy gain in FL-SNNs training under the lethal MinMax attack

Title: Analyzing Bias in Swiss Federal Supreme Court Judgments Using Facebook's Holistic Bias Dataset: Implications for Language Model Training

• Authors: Sabine Wehnert, Muhammet Ertas, Ernesto William De Luca
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03324
• Pdf URL: https://arxiv.org/pdf/2501.03324
• Copy Paste: [[2501.03324]] Analyzing Bias in Swiss Federal Supreme Court Judgments Using Facebook's Holistic Bias Dataset: Implications for Language Model Training(https://arxiv.org/abs/2501.03324)
• Keywords: fair
• Abstract: Natural Language Processing (NLP) is vital for computers to process and respond accurately to human language. However, biases in training data can introduce unfairness, especially in predicting legal judgment. This study focuses on analyzing biases within the Swiss Judgment Prediction Dataset (SJP-Dataset). Our aim is to ensure unbiased factual descriptions essential for fair decision making by NLP models in legal contexts. We analyze the dataset using social bias descriptors from the Holistic Bias dataset and employ advanced NLP techniques, including attention visualization, to explore the impact of dispreferred descriptors on model predictions. The study identifies biases and examines their influence on model behavior. Challenges include dataset imbalance and token limits affecting model performance.

Title: CM3T: Framework for Efficient Multimodal Learning for Inhomogeneous Interaction Datasets

• Authors: Tanay Agrawal, Mohammed Guermal, Michal Balazia, Francois Bremond
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03332
• Pdf URL: https://arxiv.org/pdf/2501.03332
• Copy Paste: [[2501.03332]] CM3T: Framework for Efficient Multimodal Learning for Inhomogeneous Interaction Datasets(https://arxiv.org/abs/2501.03332)
• Keywords: transformer
• Abstract: Challenges in cross-learning involve inhomogeneous or even inadequate amount of training data and lack of resources for retraining large pretrained models. Inspired by transfer learning techniques in NLP, adapters and prefix tuning, this paper presents a new model-agnostic plugin architecture for cross-learning, called CM3T, that adapts transformer-based models to new or missing information. We introduce two adapter blocks: multi-head vision adapters for transfer learning and cross-attention adapters for multimodal learning. Training becomes substantially efficient as the backbone and other plugins do not need to be finetuned along with these additions. Comparative and ablation studies on three datasets Epic-Kitchens-100, MPIIGroupInteraction and UDIVA v0.5 show efficacy of this framework on different recording settings and tasks. With only 12.8% trainable parameters compared to the backbone to process video input and only 22.3% trainable parameters for two additional modalities, we achieve comparable and even better results than the state-of-the-art. CM3T has no specific requirements for training or pretraining and is a step towards bridging the gap between a general model and specific practical applications of video classification.

Title: FTA-FTL: A Fine-Tuned Aggregation Federated Transfer Learning Scheme for Lithology Microscopic Image Classification

• Authors: Keyvan RahimiZadeh, Ahmad Taheri, Jan Baumbach, Esmael Makarian, Abbas Dehghani, Bahman Ravaei, Bahman Javadi, Amin Beheshti
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03349
• Pdf URL: https://arxiv.org/pdf/2501.03349
• Copy Paste: [[2501.03349]] FTA-FTL: A Fine-Tuned Aggregation Federated Transfer Learning Scheme for Lithology Microscopic Image Classification(https://arxiv.org/abs/2501.03349)
• Keywords: security, privacy, robust, federate
• Abstract: Lithology discrimination is a crucial activity in characterizing oil reservoirs, and processing lithology microscopic images is an essential technique for investigating fossils and minerals and geological assessment of shale oil exploration. In this way, Deep Learning (DL) technique is a powerful approach for building robust classifier models. However, there is still a considerable challenge to collect and produce a large dataset. Transfer-learning and data augmentation techniques have emerged as popular approaches to tackle this problem. Furthermore, due to different reasons, especially data privacy, individuals, organizations, and industry companies often are not willing to share their sensitive data and information. Federated Learning (FL) has emerged to train a highly accurate central model across multiple decentralized edge servers without transferring sensitive data, preserving sensitive data, and enhancing security. This study involves two phases; the first phase is to conduct Lithology microscopic image classification on a small dataset using transfer learning. In doing so, various pre-trained DL model architectures are comprehensively compared for the classification task. In the second phase, we formulated the classification task to a Federated Transfer Learning (FTL) scheme and proposed a Fine-Tuned Aggregation strategy for Federated Learning (FTA-FTL). In order to perform a comprehensive experimental study, several metrics such as accuracy, f1 score, precision, specificity, sensitivity (recall), and confusion matrix are taken into account. The results are in excellent agreement and confirm the efficiency of the proposed scheme, and show that the proposed FTA-FTL algorithm is capable enough to achieve approximately the same results obtained by the centralized implementation for Lithology microscopic images classification task.

Title: Data integrity vs. inference accuracy in large AIS datasets

• Authors: Adam Kiersztyn, Dariusz Czerwiński, Aneta Oniszczuk-Jastrzabek, Ernest Czermański, Agnieszka Rzepka
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03358
• Pdf URL: https://arxiv.org/pdf/2501.03358
• Copy Paste: [[2501.03358]] Data integrity vs. inference accuracy in large AIS datasets(https://arxiv.org/abs/2501.03358)
• Keywords: protect
• Abstract: Automatic Ship Identification Systems (AIS) play a key role in monitoring maritime traffic, providing the data necessary for analysis and decision-making. The integrity of this data is fundamental to the correctness of infer-ence and decision-making in the context of maritime safety, traffic manage-ment and environmental protection. This paper analyzes the impact of data integrity in large AIS datasets, on classification accuracy. It also presents er-ror detection and correction methods and data verification techniques that can improve the reliability of AIS systems. The results show that improving the integrity of AIS data significantly improves the quality of inference, which has a direct impact on operational efficiency and safety at sea.

Title: Advanced Machine Learning Techniques for Social Support Detection on Social Media

• Authors: Olga Kolesnikova, Moein Shahiki Tash, Zahra Ahani, Ameeta Agrawal, Raul Monroy, Grigori Sidorov
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03370
• Pdf URL: https://arxiv.org/pdf/2501.03370
• Copy Paste: [[2501.03370]] Advanced Machine Learning Techniques for Social Support Detection on Social Media(https://arxiv.org/abs/2501.03370)
• Keywords: transformer
• Abstract: The widespread use of social media highlights the need to understand its impact, particularly the role of online social support. This study uses a dataset focused on online social support, which includes binary and multiclass classifications of social support content on social media. The classification of social support is divided into three tasks. The first task focuses on distinguishing between supportive and non-supportive. The second task aims to identify whether the support is directed toward an individual or a group. The third task categorizes the specific type of social support, grouping it into categories such as Nation, LGBTQ, Black people, Women, Religion, and Other (if it does not fit into the previously mentioned categories). To address data imbalances in these tasks, we employed K-means clustering for balancing the dataset and compared the results with the original unbalanced data. Using advanced machine learning techniques, including transformers and zero-shot learning approaches with GPT3, GPT4, and GPT4-o, we predict social support levels in various contexts. The effectiveness of the dataset is evaluated using baseline models across different learning approaches, with transformer-based methods demonstrating superior performance. Additionally, we achieved a 0.4\% increase in the macro F1 score for the second task and a 0.7\% increase for the third task, compared to previous work utilizing traditional machine learning with psycholinguistic and unigram-based TF-IDF values.

Title: License Plate Images Generation with Diffusion Models

• Authors: Mariia Shpir, Nadiya Shvai, Amir Nakib
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03374
• Pdf URL: https://arxiv.org/pdf/2501.03374
• Copy Paste: [[2501.03374]] License Plate Images Generation with Diffusion Models(https://arxiv.org/abs/2501.03374)
• Keywords: privacy, protect, diffusion
• Abstract: Despite the evident practical importance of license plate recognition (LPR), corresponding research is limited by the volume of publicly available datasets due to privacy regulations such as the General Data Protection Regulation (GDPR). To address this challenge, synthetic data generation has emerged as a promising approach. In this paper, we propose to synthesize realistic license plates (LPs) using diffusion models, inspired by recent advances in image and video generation. In our experiments a diffusion model was successfully trained on a Ukrainian LP dataset, and 1000 synthetic images were generated for detailed analysis. Through manual classification and annotation of the generated images, we performed a thorough study of the model output, such as success rate, character distributions, and type of failures. Our contributions include experimental validation of the efficacy of diffusion models for LP synthesis, along with insights into the characteristics of the generated data. Furthermore, we have prepared a synthetic dataset consisting of 10,000 LP images, publicly available at this https URL. Conducted experiments empirically confirm the usefulness of synthetic data for the LPR task. Despite the initial performance gap between the model trained with real and synthetic data, the expansion of the training data set with pseudolabeled synthetic data leads to an improvement in LPR accuracy by 3% compared to baseline.

Title: Privacy-Preserving Smart Contracts for Permissioned Blockchains: A zk-SNARK-Based Recipe Part-1

• Authors: Aldenio Burgos, Eduardo Alchieri
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.03391
• Pdf URL: https://arxiv.org/pdf/2501.03391
• Copy Paste: [[2501.03391]] Privacy-Preserving Smart Contracts for Permissioned Blockchains: A zk-SNARK-Based Recipe Part-1(https://arxiv.org/abs/2501.03391)
• Keywords: privacy
• Abstract: The Bitcoin white paper introduced blockchain technology, enabling trustful transactions without intermediaries. Smart contracts emerged with Ethereum and blockchains expanded beyond cryptocurrency, applying to auctions, crowdfunding and electronic voting. However, blockchain's transparency raised privacy concerns and initial anonymity measures proved ineffective. Smart contract privacy solutions employed zero-knowledge proofs, homomorphic encryption and trusted execution environments. These approaches have practical drawbacks, such as limited functionality, high computation times and trust on third parties requirements, being not fully decentralized. This work proposes a solution utilizing zk-SNARKs to provide privacy in smart contracts and blockchains. The solution supports both fungible and nonfungible tokens. Additionally, the proposal includes a new type of transactions, called delegated transactions, which enable use cases like Delivery vs Payment (DvP).

Title: Over-the-Air Fair Federated Learning via Multi-Objective Optimization

• Authors: Shayan Mohajer Hamidi, Ali Bereyhi, Saba Asaad, H. Vincent Poor
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03392
• Pdf URL: https://arxiv.org/pdf/2501.03392
• Copy Paste: [[2501.03392]] Over-the-Air Fair Federated Learning via Multi-Objective Optimization(https://arxiv.org/abs/2501.03392)
• Keywords: robust, federate, fair
• Abstract: In federated learning (FL), heterogeneity among the local dataset distributions of clients can result in unsatisfactory performance for some, leading to an unfair model. To address this challenge, we propose an over-the-air fair federated learning algorithm (OTA-FFL), which leverages over-the-air computation to train fair FL models. By formulating FL as a multi-objective minimization problem, we introduce a modified Chebyshev approach to compute adaptive weighting coefficients for gradient aggregation in each communication round. To enable efficient aggregation over the multiple access channel, we derive analytical solutions for the optimal transmit scalars at the clients and the de-noising scalar at the parameter server. Extensive experiments demonstrate the superiority of OTA-FFL in achieving fairness and robust performance compared to existing methods.

Title: DoubleDiffusion: Combining Heat Diffusion with Denoising Diffusion for Generative Learning on 3D Meshes

• Authors: Xuyang Wang, Ziang Cheng, Zhenyu Li, Jiayu Yang, Haorui Ji, Pan Ji, Mehrtash Harandi, Richard Hartley, Hongdong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03397
• Pdf URL: https://arxiv.org/pdf/2501.03397
• Copy Paste: [[2501.03397]] DoubleDiffusion: Combining Heat Diffusion with Denoising Diffusion for Generative Learning on 3D Meshes(https://arxiv.org/abs/2501.03397)
• Keywords: diffusion, generative
• Abstract: This paper proposes DoubleDiffusion, a novel framework that combines heat dissipation diffusion and denoising diffusion for direct generative learning on 3D mesh surfaces. Our approach addresses the challenges of generating continuous signal distributions residing on a curve manifold surface. Unlike previous methods that rely on unrolling 3D meshes into 2D or adopting field representations, DoubleDiffusion leverages the Laplacian-Beltrami operator to process features respecting the mesh structure. This combination enables effective geometry-aware signal diffusion across the underlying geometry. As shown in Fig.~\ref{fig:teaser}, we demonstrate that DoubleDiffusion has the ability to generate RGB signal distributions on complex 3D mesh surfaces and achieves per-category shape-conditioned texture generation across different shape geometry. Our work contributes a new direction in diffusion-based generative modeling on 3D surfaces, with potential applications in the field of 3D asset generation.

Title: BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations

• Authors: Simone Giovannini, Fabio Coppini, Andrea Gemelli, Simone Marinai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03403
• Pdf URL: https://arxiv.org/pdf/2501.03403
• Copy Paste: [[2501.03403]] BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations(https://arxiv.org/abs/2501.03403)
• Keywords: extraction, large language model
• Abstract: We present a unified dataset for document Question-Answering (QA), which is obtained combining several public datasets related to Document AI and visually rich document understanding (VRDU). Our main contribution is twofold: on the one hand we reformulate existing Document AI tasks, such as Information Extraction (IE), into a Question-Answering task, making it a suitable resource for training and evaluating Large Language Models; on the other hand, we release the OCR of all the documents and include the exact position of the answer to be found in the document image as a bounding box. Using this dataset, we explore the impact of different prompting techniques (that might include bounding box information) on the performance of open-weight models, identifying the most effective approaches for document comprehension.

Title: ScaleMAI: Accelerating the Development of Trusted Datasets and AI Models

• Authors: Wenxuan Li, Pedro R. A. S. Bassi, Tianyu Lin, Yu-Cheng Chou, Xinze Zhou, Yucheng Tang, Fabian Isensee, Kang Wang, Qi Chen, Xiaowei Xu, Xiaoxi Chen, Lizhou Wu, Qilong Wu, Yannick Kirchhoff, Maximilian Rokuss, Saikat Roy, Yuxuan Zhao, Dexin Yu, Kai Ding, Constantin Ulrich, Klaus Maier-Hein, Yang Yang, Alan L. Yuille, Zongwei Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03410
• Pdf URL: https://arxiv.org/pdf/2501.03410
• Copy Paste: [[2501.03410]] ScaleMAI: Accelerating the Development of Trusted Datasets and AI Models(https://arxiv.org/abs/2501.03410)
• Keywords: segmentation
• Abstract: Building trusted datasets is critical for transparent and responsible Medical AI (MAI) research, but creating even small, high-quality datasets can take years of effort from multidisciplinary teams. This process often delays AI benefits, as human-centric data creation and AI-centric model development are treated as separate, sequential steps. To overcome this, we propose ScaleMAI, an agent of AI-integrated data curation and annotation, allowing data quality and AI performance to improve in a self-reinforcing cycle and reducing development time from years to months. We adopt pancreatic tumor detection as an example. First, ScaleMAI progressively creates a dataset of 25,362 CT scans, including per-voxel annotations for benign/malignant tumors and 24 anatomical structures. Second, through progressive human-in-the-loop iterations, ScaleMAI provides Flagship AI Model that can approach the proficiency of expert annotators (30-year experience) in detecting pancreatic tumors. Flagship Model significantly outperforms models developed from smaller, fixed-quality datasets, with substantial gains in tumor detection (+14%), segmentation (+5%), and classification (72%) on three prestigious benchmarks. In summary, ScaleMAI transforms the speed, scale, and reliability of medical dataset creation, paving the way for a variety of impactful, data-driven applications.

Title: SALT: Sales Autocompletion Linked Business Tables Dataset

• Authors: Tassilo Klein, Clemens Biehl, Margarida Costa, Andre Sres, Jonas Kolk, Johannes Hoffart
• Subjects: cs.LG, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2501.03413
• Pdf URL: https://arxiv.org/pdf/2501.03413
• Copy Paste: [[2501.03413]] SALT: Sales Autocompletion Linked Business Tables Dataset(https://arxiv.org/abs/2501.03413)
• Keywords: transformer
• Abstract: Foundation models, particularly those that incorporate Transformer architectures, have demonstrated exceptional performance in domains such as natural language processing and image processing. Adapting these models to structured data, like tables, however, introduces significant challenges. These difficulties are even more pronounced when addressing multi-table data linked via foreign key, which is prevalent in the enterprise realm and crucial for empowering business use cases. Despite its substantial impact, research focusing on such linked business tables within enterprise settings remains a significantly important yet underexplored domain. To address this, we introduce a curated dataset sourced from an Enterprise Resource Planning (ERP) system, featuring extensive linked tables. This dataset is specifically designed to support research endeavors in table representation learning. By providing access to authentic enterprise data, our goal is to potentially enhance the effectiveness and applicability of models for real-world business contexts.

Title: SoK: A Review of Cross-Chain Bridge Hacks in 2023

• Authors: Nikita Belenkov, Valerian Callens, Alexandr Murashkin, Kacper Bak, Martin Derka, Jan Gorzny, Sung-Shine Lee
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.03423
• Pdf URL: https://arxiv.org/pdf/2501.03423
• Copy Paste: [[2501.03423]] SoK: A Review of Cross-Chain Bridge Hacks in 2023(https://arxiv.org/abs/2501.03423)
• Keywords: secure, security, attack
• Abstract: Blockchain technology has revolutionized industries by enabling secure and decentralized transactions. However, the isolated nature of blockchain ecosystems hinders the seamless transfer of digital assets across different chains. Cross-chain bridges have emerged as vital web3 infrastructure to address this challenge by facilitating interoperability between distinct blockchains. Cross-chain bridges remain vulnerable to various attacks despite sophisticated designs and security measures. The industry has experienced a surge in bridge attacks, resulting in significant financial losses. The largest hack impacted Axie Infinity Ronin Bridge, with a loss of almost \$600 million USD. This paper analyzes recent cross-chain bridge hacks in 2022 and 2023 and examines the exploited vulnerabilities. By understanding the attack nature and underlying weaknesses, the paper aims to enhance bridge security and propose potential countermeasures. The findings contribute to developing industry-wide standards for bridge security and operational resilience. Addressing the vulnerabilities and weaknesses exploited in recent cross-chain bridge hacks fosters trust and confidence in cross-chain interoperability.

Title: Mixture-of-Experts Graph Transformers for Interpretable Particle Collision Detection

• Authors: Donatella Genovese, Alessandro Sgroi, Alessio Devoto, Samuel Valentine, Lennox Wood, Cristiano Sebastiani, Stefano Giagu, Monica D'Onofrio, Simone Scardapane
• Subjects: cs.LG, hep-ph
• Abstract URL: https://arxiv.org/abs/2501.03432
• Pdf URL: https://arxiv.org/pdf/2501.03432
• Copy Paste: [[2501.03432]] Mixture-of-Experts Graph Transformers for Interpretable Particle Collision Detection(https://arxiv.org/abs/2501.03432)
• Keywords: robust, interpretability, explainability, transformer
• Abstract: The Large Hadron Collider at CERN produces immense volumes of complex data from high-energy particle collisions, demanding sophisticated analytical techniques for effective interpretation. Neural Networks, including Graph Neural Networks, have shown promise in tasks such as event classification and object identification by representing collisions as graphs. However, while Graph Neural Networks excel in predictive accuracy, their "black box" nature often limits their interpretability, making it difficult to trust their decision-making processes. In this paper, we propose a novel approach that combines a Graph Transformer model with Mixture-of-Expert layers to achieve high predictive performance while embedding interpretability into the architecture. By leveraging attention maps and expert specialization, the model offers insights into its internal decision-making, linking predictions to physics-informed features. We evaluate the model on simulated events from the ATLAS experiment, focusing on distinguishing rare Supersymmetric signal events from Standard Model background. Our results highlight that the model achieves competitive classification accuracy while providing interpretable outputs that align with known physics, demonstrating its potential as a robust and transparent tool for high-energy physics data analysis. This approach underscores the importance of explainability in machine learning methods applied to high energy physics, offering a path toward greater trust in AI-driven discoveries.

Title: DAMAGE: Detecting Adversarially Modified AI Generated Text

• Authors: Elyas Masrour, Bradley Emi, Max Spero
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03437
• Pdf URL: https://arxiv.org/pdf/2501.03437
• Copy Paste: [[2501.03437]] DAMAGE: Detecting Adversarially Modified AI Generated Text(https://arxiv.org/abs/2501.03437)
• Keywords: attack, robust
• Abstract: AI humanizers are a new class of online software tools meant to paraphrase and rewrite AI-generated text in a way that allows them to evade AI detection software. We study 19 AI humanizer and paraphrasing tools and qualitatively assess their effects and faithfulness in preserving the meaning of the original text. We show that many existing AI detectors fail to detect humanized text. Finally, we demonstrate a robust model that can detect humanized AI text while maintaining a low false positive rate using a data-centric augmentation approach. We attack our own detector, training our own fine-tuned model optimized against our detector's predictions, and show that our detector's cross-humanizer generalization is sufficient to remain robust to this attack.

Title: Finding A Voice: Evaluating African American Dialect Generation for Chatbot Technology

• Authors: Sarah E. Finch, Ellie S. Paek, Sejung Kwon, Ikseon Choi, Jessica Wells, Rasheeta Chandler, Jinho D. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03441
• Pdf URL: https://arxiv.org/pdf/2501.03441
• Copy Paste: [[2501.03441]] Finding A Voice: Evaluating African American Dialect Generation for Chatbot Technology(https://arxiv.org/abs/2501.03441)
• Keywords: large language model
• Abstract: As chatbots become increasingly integrated into everyday tasks, designing systems that accommodate diverse user populations is crucial for fostering trust, engagement, and inclusivity. This study investigates the ability of contemporary Large Language Models (LLMs) to generate African American Vernacular English (AAVE) and evaluates the impact of AAVE usage on user experiences in chatbot applications. We analyze the performance of three LLM families (Llama, GPT, and Claude) in producing AAVE-like utterances at varying dialect intensities and assess user preferences across multiple domains, including healthcare and education. Despite LLMs' proficiency in generating AAVE-like language, findings indicate that AAVE-speaking users prefer Standard American English (SAE) chatbots, with higher levels of AAVE correlating with lower ratings for a variety of characteristics, including chatbot trustworthiness and role appropriateness. These results highlight the complexities of creating inclusive AI systems and underscore the need for further exploration of diversity to enhance human-computer interactions.

Title: Physics-Constrained Generative Artificial Intelligence for Rapid Takeoff Trajectory Design

• Authors: Samuel Sisk, Xiaosong Du
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03445
• Pdf URL: https://arxiv.org/pdf/2501.03445
• Copy Paste: [[2501.03445]] Physics-Constrained Generative Artificial Intelligence for Rapid Takeoff Trajectory Design(https://arxiv.org/abs/2501.03445)
• Keywords: generative
• Abstract: To aid urban air mobility (UAM), electric vertical takeoff and landing (eVTOL) aircraft are being targeted. Conventional multidisciplinary analysis and optimization (MDAO) can be expensive, while surrogate-based optimization can struggle with challenging physical constraints. This work proposes physics-constrained generative adversarial networks (physicsGAN), to intelligently parameterize the takeoff control profiles of an eVTOL aircraft and to transform the original design space to a feasible space. Specifically, the transformed feasible space refers to a space where all designs directly satisfy all design constraints. The physicsGAN-enabled surrogate-based takeoff trajectory design framework was demonstrated on the Airbus A3 Vahana. The physicsGAN generated only feasible control profiles of power and wing angle in the feasible space with around 98.9% of designs satisfying all constraints. The proposed design framework obtained 99.6% accuracy compared with simulation-based optimal design and took only 2.2 seconds, which reduced the computational time by around 200 times. Meanwhile, data-driven GAN-enabled surrogate-based optimization took 21.9 seconds using a derivative-free optimizer, which was around an order of magnitude slower than the proposed framework. Moreover, the data-driven GAN-based optimization using gradient-based optimizers could not consistently find the optimal design during random trials and got stuck in an infeasible region, which is problematic in real practice. Therefore, the proposed physicsGAN-based design framework outperformed data-driven GAN-based design to the extent of efficiency (2.2 seconds), optimality (99.6% accurate), and feasibility (100% feasible). According to the literature review, this is the first physics-constrained generative artificial intelligence enabled by surrogate models.

Title: Optimizing Value of Learning in Task-Oriented Federated Meta-Learning Systems

• Authors: Bibo Wu, Fang Fang, Xianbin Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03448
• Pdf URL: https://arxiv.org/pdf/2501.03448
• Copy Paste: [[2501.03448]] Optimizing Value of Learning in Task-Oriented Federated Meta-Learning Systems(https://arxiv.org/abs/2501.03448)
• Keywords: privacy, federate, fair
• Abstract: Federated Learning (FL) has gained significant attention in recent years due to its distributed nature and privacy preserving benefits. However, a key limitation of conventional FL is that it learns and distributes a common global model to all participants, which fails to provide customized solutions for diverse task requirements. Federated meta-learning (FML) offers a promising solution to this issue by enabling devices to finetune local models after receiving a shared meta-model from the server. In this paper, we propose a task-oriented FML framework over non-orthogonal multiple access (NOMA) networks. A novel metric, termed value of learning (VoL), is introduced to assess the individual training needs across devices. Moreover, a task-level weight (TLW) metric is defined based on task requirements and fairness considerations, guiding the prioritization of edge devices during FML training. The formulated problem, to maximize the sum of TLW-based VoL across devices, forms a non-convex mixed-integer non-linear programming (MINLP) challenge, addressed here using a parameterized deep Q-network (PDQN) algorithm to handle both discrete and continuous variables. Simulation results demonstrate that our approach significantly outperforms baseline schemes, underscoring the advantages of the proposed framework.

Title: Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction

• Authors: Ying-Ting Yeh, Janghoon Ock, Amir Barati Farimani
• Subjects: cs.CL, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2501.03456
• Pdf URL: https://arxiv.org/pdf/2501.03456
• Copy Paste: [[2501.03456]] Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction(https://arxiv.org/abs/2501.03456)
• Keywords: transformer
• Abstract: In this study, we explore the use of a transformer-based language model as an encoder to predict the band gaps of semiconductor materials directly from their text descriptions. Quantum chemistry simulations, including Density Functional Theory (DFT), are computationally intensive and time-consuming, which limits their practicality for high-throughput material screening, particularly for complex systems. Shallow machine learning (ML) models, while effective, often require extensive data preprocessing to convert non-numerical material properties into numerical inputs. In contrast, our approach leverages textual data directly, bypassing the need for complex feature engineering. We generate material descriptions in two formats: formatted strings combining features and natural language text generated using the ChatGPT API. We demonstrate that the RoBERTa model, pre-trained on natural language processing tasks, performs effectively as an encoder for prediction tasks. With minimal fine-tuning, it achieves a mean absolute error (MAE) of approximately 0.33 eV, performing better than shallow machine learning models such as Support Vector Regression, Random Forest, and XGBoost. Even when only the linear regression head is trained while keeping the RoBERTa encoder layers frozen, the accuracy remains nearly identical to that of the fully trained model. This demonstrates that the pre-trained RoBERTa encoder is highly adaptable for processing domain-specific text related to material properties, such as the band gap, significantly reducing the need for extensive retraining. This study highlights the potential of transformer-based language models to serve as efficient and versatile encoders for semiconductor materials property prediction tasks.

Title: ISSR: Iterative Selection with Self-Review for Vocabulary Test Distractor Generation

• Authors: Yu-Cheng Liu, An-Zi Yen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03462
• Pdf URL: https://arxiv.org/pdf/2501.03462
• Copy Paste: [[2501.03462]] ISSR: Iterative Selection with Self-Review for Vocabulary Test Distractor Generation(https://arxiv.org/abs/2501.03462)
• Keywords: large language model
• Abstract: Vocabulary acquisition is essential to second language learning, as it underpins all core language skills. Accurate vocabulary assessment is particularly important in standardized exams, where test items evaluate learners' comprehension and contextual use of words. Previous research has explored methods for generating distractors to aid in the design of English vocabulary tests. However, current approaches often rely on lexical databases or predefined rules, and frequently produce distractors that risk invalidating the question by introducing multiple correct options. In this study, we focus on English vocabulary questions from Taiwan's university entrance exams. We analyze student response distributions to gain insights into the characteristics of these test items and provide a reference for future research. Additionally, we identify key limitations in how large language models (LLMs) support teachers in generating distractors for vocabulary test design. To address these challenges, we propose the iterative selection with self-review (ISSR) framework, which makes use of a novel LLM-based self-review mechanism to ensure that the distractors remain valid while offering diverse options. Experimental results show that ISSR achieves promising performance in generating plausible distractors, and the self-review mechanism effectively filters out distractors that could invalidate the question.

Title: MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems

• Authors: Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, Marina Danilevsky
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03468
• Pdf URL: https://arxiv.org/pdf/2501.03468
• Copy Paste: [[2501.03468]] MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems(https://arxiv.org/abs/2501.03468)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at this https URL.

Title: Information-Maximized Soft Variable Discretization for Self-Supervised Image Representation Learning

• Authors: Chuang Niu, Wenjun Xia, Hongming Shan, Ge Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03469
• Pdf URL: https://arxiv.org/pdf/2501.03469
• Copy Paste: [[2501.03469]] Information-Maximized Soft Variable Discretization for Self-Supervised Image Representation Learning(https://arxiv.org/abs/2501.03469)
• Keywords: explainability
• Abstract: Self-supervised learning (SSL) has emerged as a crucial technique in image processing, encoding, and understanding, especially for developing today's vision foundation models that utilize large-scale datasets without annotations to enhance various downstream tasks. This study introduces a novel SSL approach, Information-Maximized Soft Variable Discretization (IMSVD), for image representation learning. Specifically, IMSVD softly discretizes each variable in the latent space, enabling the estimation of their probability distributions over training batches and allowing the learning process to be directly guided by information measures. Motivated by the MultiView assumption, we propose an information-theoretic objective function to learn transform-invariant, non-travail, and redundancy-minimized representation features. We then derive a joint-cross entropy loss function for self-supervised image representation learning, which theoretically enjoys superiority over the existing methods in reducing feature redundancy. Notably, our non-contrastive IMSVD method statistically performs contrastive learning. Extensive experimental results demonstrate the effectiveness of IMSVD on various downstream tasks in terms of both accuracy and efficiency. Thanks to our variable discretization, the embedding features optimized by IMSVD offer unique explainability at the variable level. IMSVD has the potential to be adapted to other learning paradigms. Our code is publicly available at this https URL.

Title: Reading with Intent -- Neutralizing Intent

• Authors: Benjamin Reichman, Adar Avsian, Larry Heck
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03475
• Pdf URL: https://arxiv.org/pdf/2501.03475
• Copy Paste: [[2501.03475]] Reading with Intent -- Neutralizing Intent(https://arxiv.org/abs/2501.03475)
• Keywords: large language model
• Abstract: Queries to large language models (LLMs) can be divided into two parts: the instruction/question and the accompanying context. The context for retrieval-augmented generation (RAG) systems in most benchmarks comes from Wikipedia or Wikipedia-like texts which are written in a neutral and factual tone. However, when RAG systems retrieve internet-based content, they encounter text with diverse tones and linguistic styles, introducing challenges for downstream tasks. The Reading with Intent task addresses this issue by evaluating how varying tones in context passages affect model performance. Building on prior work that focused on sarcasm, we extend this paradigm by constructing a dataset where context passages are transformed to $11$ distinct emotions using a better synthetic data generation approach. Using this dataset, we train an emotion translation model to systematically adapt passages to specified emotional tones. The human evaluation shows that the LLM fine-tuned to become the emotion-translator benefited from the synthetically generated data. Finally, the emotion-translator is used in the Reading with Intent task to transform the passages to a neutral tone. By neutralizing the passages, it mitigates the challenges posed by sarcastic passages and improves overall results on this task by about $3\%$.

Title: A study on performance limitations in Federated Learning

• Authors: Karthik Mohan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03477
• Pdf URL: https://arxiv.org/pdf/2501.03477
• Copy Paste: [[2501.03477]] A study on performance limitations in Federated Learning(https://arxiv.org/abs/2501.03477)
• Keywords: privacy, attack, robust, federate
• Abstract: Increasing privacy concerns and unrestricted access to data lead to the development of a novel machine learning paradigm called Federated Learning (FL). FL borrows many of the ideas from distributed machine learning, however, the challenges associated with federated learning makes it an interesting engineering problem since the models are trained on edge devices. It was introduced in 2016 by Google, and since then active research is being carried out in different areas within FL such as federated optimization algorithms, model and update compression, differential privacy, robustness, and attacks, federated GANs and privacy preserved personalization. There are many open challenges in the development of such federated machine learning systems and this project will be focusing on the communication bottleneck and data Non IID-ness, and its effect on the performance of the models. These issues are characterized on a baseline model, model performance is evaluated, and discussions are made to overcome these issues.

Title: VOILA: Complexity-Aware Universal Segmentation of CT images by Voxel Interacting with Language

• Authors: Zishuo Wan, Yu Gao, Wanyuan Pang, Dawei Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03482
• Pdf URL: https://arxiv.org/pdf/2501.03482
• Copy Paste: [[2501.03482]] VOILA: Complexity-Aware Universal Segmentation of CT images by Voxel Interacting with Language(https://arxiv.org/abs/2501.03482)
• Keywords: segmentation
• Abstract: Satisfactory progress has been achieved recently in universal segmentation of CT images. Following the success of vision-language methods, there is a growing trend towards utilizing text prompts and contrastive learning to develop universal segmentation models. However, there exists a significant imbalance in information density between 3D images and text prompts. Moreover, the standard fully connected layer segmentation approach faces significant challenges in handling multiple classes and exhibits poor generalizability. To address these challenges, we propose the VOxel Interacting with LAnguage method (VOILA) for universal CT image segmentation. Initially, we align voxels and language into a shared representation space and classify voxels on the basis of cosine similarity. Subsequently, we develop the Voxel-Language Interaction framework to mitigate the impact of class imbalance caused by foreground-background discrepancies and variations in target volumes. Furthermore, a Complexity-Aware Sampling method is proposed to focus on region hard to segment, achieved by generating pseudo-heatmaps from a trainable Gaussian mixture distribution. Our results indicate the proposed VOILA is capable to achieve improved performance with reduced parameters and computational cost during training. Furthermore, it demonstrates significant generalizability across diverse datasets without additional fine-tuning.

Title: Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment

• Authors: Prashant Trivedi, Souradip Chakraborty, Avinash Reddy, Vaneet Aggarwal, Amrit Singh Bedi, George K. Atia
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03486
• Pdf URL: https://arxiv.org/pdf/2501.03486
• Copy Paste: [[2501.03486]] Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment(https://arxiv.org/abs/2501.03486)
• Keywords: large language model
• Abstract: The alignment of large language models (LLMs) with human values is critical as these models become increasingly integrated into various societal and decision-making processes. Traditional methods, such as reinforcement learning from human feedback (RLHF), achieve alignment by fine-tuning model parameters, but these approaches are often computationally expensive and impractical when models are frozen or inaccessible for parameter modification. In contrast, prompt optimization is a viable alternative to RLHF for LLM alignment. While the existing literature has shown empirical promise of prompt optimization, its theoretical underpinning remains under-explored. We address this gap by formulating prompt optimization as an optimization problem and try to provide theoretical insights into the optimality of such a framework. To analyze the performance of the prompt optimization, we study theoretical suboptimality bounds and provide insights in terms of how prompt optimization depends upon the given prompter and target model. We also provide empirical validation through experiments on various datasets, demonstrating that prompt optimization can effectively align LLMs, even when parameter fine-tuning is not feasible.

Title: Entropy-Guided Attention for Private LLMs

• Authors: Nandan Kumar Jha, Brandon Reagen
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.03489
• Pdf URL: https://arxiv.org/pdf/2501.03489
• Copy Paste: [[2501.03489]] Entropy-Guided Attention for Private LLMs(https://arxiv.org/abs/2501.03489)
• Keywords: privacy, transformer
• Abstract: The pervasiveness of proprietary language models has raised critical privacy concerns, necessitating advancements in private inference (PI), where computations are performed directly on encrypted data without revealing users' sensitive information. While PI offers a promising solution, its practical deployment is hindered by substantial communication and latency overheads, primarily stemming from nonlinear operations. To address this, we introduce an information-theoretic framework to characterize the role of nonlinearities in decoder-only language models, laying a principled foundation for optimizing transformer-architectures tailored to the demands of PI. By leveraging Shannon's entropy as a quantitative measure, we uncover the previously unexplored dual significance of nonlinearities: beyond ensuring training stability, they are crucial for maintaining attention head diversity. Specifically, we find that their removal triggers two critical failure modes: {\em entropy collapse} in deeper layers that destabilizes training, and {\em entropic overload} in earlier layers that leads to under-utilization of Multi-Head Attention's (MHA) representational capacity. We propose an entropy-guided attention mechanism paired with a novel entropy regularization technique to mitigate entropic overload. Additionally, we explore PI-friendly alternatives to layer normalization for preventing entropy collapse and stabilizing the training of LLMs with reduced-nonlinearities. Our study bridges the gap between information theory and architectural design, establishing entropy dynamics as a principled guide for developing efficient PI architectures. The code and implementation are available at \href{this https URL}{entropy-guided-llm}.

Title: SceneBooth: Diffusion-based Framework for Subject-preserved Text-to-Image Generation

• Authors: Shang Chai, Zihang Lin, Min Zhou, Xubin Li, Liansheng Zhuang, Houqiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03490
• Pdf URL: https://arxiv.org/pdf/2501.03490
• Copy Paste: [[2501.03490]] SceneBooth: Diffusion-based Framework for Subject-preserved Text-to-Image Generation(https://arxiv.org/abs/2501.03490)
• Keywords: diffusion
• Abstract: Due to the demand for personalizing image generation, subject-driven text-to-image generation method, which creates novel renditions of an input subject based on text prompts, has received growing research interest. Existing methods often learn subject representation and incorporate it into the prompt embedding to guide image generation, but they struggle with preserving subject fidelity. To solve this issue, this paper approaches a novel framework named SceneBooth for subject-preserved text-to-image generation, which consumes inputs of a subject image, object phrases and text prompts. Instead of learning the subject representation and generating a subject, our SceneBooth fixes the given subject image and generates its background image guided by the text prompts. To this end, our SceneBooth introduces two key components, i.e., a multimodal layout generation module and a background painting module. The former determines the position and scale of the subject by generating appropriate scene layouts that align with text captions, object phrases, and subject visual information. The latter integrates two adapters (ControlNet and Gated Self-Attention) into the latent diffusion model to generate a background that harmonizes with the subject guided by scene layouts and text descriptions. In this manner, our SceneBooth ensures accurate preservation of the subject's appearance in the output. Quantitative and qualitative experimental results demonstrate that SceneBooth significantly outperforms baseline methods in terms of subject preservation, image harmonization and overall quality.

Title: Textualize Visual Prompt for Image Editing via Diffusion Bridge

• Authors: Pengcheng Xu, Qingnan Fan, Fei Kou, Shuai Qin, Hong Gu, Ruoyu Zhao, Charles Ling, Boyu Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03495
• Pdf URL: https://arxiv.org/pdf/2501.03495
• Copy Paste: [[2501.03495]] Textualize Visual Prompt for Image Editing via Diffusion Bridge(https://arxiv.org/abs/2501.03495)
• Keywords: diffusion, generative
• Abstract: Visual prompt, a pair of before-and-after edited images, can convey indescribable imagery transformations and prosper in image editing. However, current visual prompt methods rely on a pretrained text-guided image-to-image generative model that requires a triplet of text, before, and after images for retraining over a text-to-image model. Such crafting triplets and retraining processes limit the scalability and generalization of editing. In this paper, we present a framework based on any single text-to-image model without reliance on the explicit image-to-image model thus enhancing the generalizability and scalability. Specifically, by leveraging the probability-flow ordinary equation, we construct a diffusion bridge to transfer the distribution between before-and-after images under the text guidance. By optimizing the text via the bridge, the framework adaptively textualizes the editing transformation conveyed by visual prompts into text embeddings without other models. Meanwhile, we introduce differential attention control during text optimization, which disentangles the text embedding from the invariance of the before-and-after images and makes it solely capture the delicate transformation and generalize to edit various images. Experiments on real images validate competitive results on the generalization, contextual coherence, and high fidelity for delicate editing with just one image pair as the visual prompt.

Title: An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning

• Authors: Fatemeh Ghofrani, Pooyan Jamshidi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03507
• Pdf URL: https://arxiv.org/pdf/2501.03507
• Copy Paste: [[2501.03507]] An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning(https://arxiv.org/abs/2501.03507)
• Keywords: robust
• Abstract: Self-supervised learning (SSL) has significantly advanced image representation learning, yet efficiency challenges persist, particularly with adversarial training. Many SSL methods require extensive epochs to achieve convergence, a demand further amplified in adversarial settings. To address this inefficiency, we revisit the robust EMP-SSL framework, emphasizing the importance of increasing the number of crops per image to accelerate learning. Unlike traditional contrastive learning, robust EMP-SSL leverages multi-crop sampling, integrates an invariance term and regularization, and reduces training epochs, enhancing time efficiency. Evaluated with both standard linear classifiers and multi-patch embedding aggregation, robust EMP-SSL provides new insights into SSL evaluation strategies. Our results show that robust crop-based EMP-SSL not only accelerates convergence but also achieves a superior balance between clean accuracy and adversarial robustness, outperforming multi-crop embedding aggregation. Additionally, we extend this approach with free adversarial training in Multi-Crop SSL, introducing the Cost-Free Adversarial Multi-Crop Self-Supervised Learning (CF-AMC-SSL) method. CF-AMC-SSL demonstrates the effectiveness of free adversarial training in reducing training time while simultaneously improving clean accuracy and adversarial robustness. These findings underscore the potential of CF-AMC-SSL for practical SSL applications. Our code is publicly available at this https URL.

Title: A Sequential Optimal Learning Approach to Automated Prompt Engineering in Large Language Models

• Authors: Shuyang Wang, Somayeh Moazeni, Diego Klabjan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03508
• Pdf URL: https://arxiv.org/pdf/2501.03508
• Copy Paste: [[2501.03508]] A Sequential Optimal Learning Approach to Automated Prompt Engineering in Large Language Models(https://arxiv.org/abs/2501.03508)
• Keywords: large language model
• Abstract: Designing effective prompts is essential to guiding large language models (LLMs) toward desired responses. Automated prompt engineering aims to reduce reliance on manual effort by streamlining the design, refinement, and optimization of natural language prompts. This paper proposes an optimal learning framework for automated prompt engineering, designed to sequentially identify effective prompt features while efficiently allocating a limited evaluation budget. We introduce a feature-based method to express prompts, which significantly broadens the search space. Bayesian regression is employed to utilize correlations among similar prompts, accelerating the learning process. To efficiently explore the large space of prompt features for a high quality prompt, we adopt the forward-looking Knowledge-Gradient (KG) policy for sequential optimal learning. The KG policy is computed efficiently by solving mixed-integer second-order cone optimization problems, making it scalable and capable of accommodating prompts characterized only through constraints. We demonstrate that our method significantly outperforms a set of benchmark strategies assessed on instruction induction tasks. The results highlight the advantages of using the KG policy for prompt learning given a limited evaluation budget. Our framework provides a solution to deploying automated prompt engineering in a wider range applications where prompt evaluation is costly.

Title: Deep Learning within Tabular Data: Foundations, Challenges, Advances and Future Directions

• Authors: Weijieying Ren, Tianxiang Zhao, Yuqing Huang, Vasant Honavar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03540
• Pdf URL: https://arxiv.org/pdf/2501.03540
• Copy Paste: [[2501.03540]] Deep Learning within Tabular Data: Foundations, Challenges, Advances and Future Directions(https://arxiv.org/abs/2501.03540)
• Keywords: robust, transformer
• Abstract: Tabular data remains one of the most prevalent data types across a wide range of real-world applications, yet effective representation learning for this domain poses unique challenges due to its irregular patterns, heterogeneous feature distributions, and complex inter-column dependencies. This survey provides a comprehensive review of state-of-the-art techniques in tabular data representation learning, structured around three foundational design elements: training data, neural architectures, and learning objectives. Unlike prior surveys that focus primarily on either architecture design or learning strategies, we adopt a holistic perspective that emphasizes the universality and robustness of representation learning methods across diverse downstream tasks. We examine recent advances in data augmentation and generation, specialized neural network architectures tailored to tabular data, and innovative learning objectives that enhance representation quality. Additionally, we highlight the growing influence of self-supervised learning and the adaptation of transformer-based foundation models for tabular data. Our review is based on a systematic literature search using rigorous inclusion criteria, encompassing 127 papers published since 2020 in top-tier conferences and journals. Through detailed analysis and comparison, we identify emerging trends, critical gaps, and promising directions for future research, aiming to guide the development of more generalizable and effective tabular data representation methods.

Title: PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models

• Authors: Lingzhi Yuan, Xinfeng Li, Chejian Xu, Guanhong Tao, Xiaojun Jia, Yihao Huang, Wei Dong, Yang Liu, XiaoFeng Wang, Bo Li
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.03544
• Pdf URL: https://arxiv.org/pdf/2501.03544
• Copy Paste: [[2501.03544]] PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models(https://arxiv.org/abs/2501.03544)
• Keywords: defense, large language model
• Abstract: Text-to-image (T2I) models have been shown to be vulnerable to misuse, particularly in generating not-safe-for-work (NSFW) content, raising serious ethical concerns. In this work, we present PromptGuard, a novel content moderation technique that draws inspiration from the system prompt mechanism in large language models (LLMs) for safety alignment. Unlike LLMs, T2I models lack a direct interface for enforcing behavioral guidelines. Our key idea is to optimize a safety soft prompt that functions as an implicit system prompt within the T2I model's textual embedding space. This universal soft prompt (P*) directly moderates NSFW inputs, enabling safe yet realistic image generation without altering the inference efficiency or requiring proxy models. Extensive experiments across three datasets demonstrate that PromptGuard effectively mitigates NSFW content generation while preserving high-quality benign outputs. PromptGuard achieves 7.8 times faster than prior content moderation methods, surpassing eight state-of-the-art defenses with an optimal unsafe ratio down to 5.84%.

Title: Rethinking Adversarial Attacks in Reinforcement Learning from Policy Distribution Perspective

• Authors: Tianyang Duan, Zongyuan Zhang, Zheng Lin, Yue Gao, Ling Xiong, Yong Cui, Hongbin Liang, Xianhao Chen, Heming Cui, Dong Huang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03562
• Pdf URL: https://arxiv.org/pdf/2501.03562
• Copy Paste: [[2501.03562]] Rethinking Adversarial Attacks in Reinforcement Learning from Policy Distribution Perspective(https://arxiv.org/abs/2501.03562)
• Keywords: attack, robust
• Abstract: Deep Reinforcement Learning (DRL) suffers from uncertainties and inaccuracies in the observation signal in realworld applications. Adversarial attack is an effective method for evaluating the robustness of DRL agents. However, existing attack methods targeting individual sampled actions have limited impacts on the overall policy distribution, particularly in continuous action spaces. To address these limitations, we propose the Distribution-Aware Projected Gradient Descent attack (DAPGD). DAPGD uses distribution similarity as the gradient perturbation input to attack the policy network, which leverages the entire policy distribution rather than relying on individual samples. We utilize the Bhattacharyya distance in DAPGD to measure policy similarity, enabling sensitive detection of subtle but critical differences between probability distributions. Our experiment results demonstrate that DAPGD achieves SOTA results compared to the baselines in three robot navigation tasks, achieving an average 22.03% higher reward drop compared to the best baseline.

Title: Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis

• Authors: Haoran Lai, Zihang Jiang, Qingsong Yao, Rongsheng Wang, Zhiyang He, Xiaodong Tao, Wei Wei, Weifu Lv, S.Kevin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03565
• Pdf URL: https://arxiv.org/pdf/2501.03565
• Copy Paste: [[2501.03565]] Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis(https://arxiv.org/abs/2501.03565)
• Keywords: large language model
• Abstract: 3D medical images such as Computed tomography (CT) are widely used in clinical practice, offering a great potential for automatic diagnosis. Supervised learning-based approaches have achieved significant progress but rely heavily on extensive manual annotations, limited by the availability of training data and the diversity of abnormality types. Vision-language alignment (VLA) offers a promising alternative by enabling zero-shot learning without additional annotations. However, we empirically discover that the visual and textural embeddings after alignment endeavors from existing VLA methods form two well-separated clusters, presenting a wide gap to be bridged. To bridge this gap, we propose a Bridged Semantic Alignment (BrgSA) framework. First, we utilize a large language model to perform semantic summarization of reports, extracting high-level semantic information. Second, we design a Cross-Modal Knowledge Interaction (CMKI) module that leverages a cross-modal knowledge bank as a semantic bridge, facilitating interaction between the two modalities, narrowing the gap, and improving their alignment. To comprehensively evaluate our method, we construct a benchmark dataset that includes 15 underrepresented abnormalities as well as utilize two existing benchmark datasets. Experimental results demonstrate that BrgSA achieves state-of-the-art performances on both public benchmark datasets and our custom-labeled dataset, with significant improvements in zero-shot diagnosis of underrepresented abnormalities.

Title: BASIC: Semi-supervised Multi-organ Segmentation with Balanced Subclass Regularization and Semantic-conflict Penalty

• Authors: Zhenghao Feng, Lu Wen, Yuanyuan Xu, Binyu Yan, Xi Wu, Jiliu Zhou, Yan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03580
• Pdf URL: https://arxiv.org/pdf/2501.03580
• Copy Paste: [[2501.03580]] BASIC: Semi-supervised Multi-organ Segmentation with Balanced Subclass Regularization and Semantic-conflict Penalty(https://arxiv.org/abs/2501.03580)
• Keywords: segmentation
• Abstract: Semi-supervised learning (SSL) has shown notable potential in relieving the heavy demand of dense prediction tasks on large-scale well-annotated datasets, especially for the challenging multi-organ segmentation (MoS). However, the prevailing class-imbalance problem in MoS caused by the substantial variations in organ size exacerbates the learning difficulty of the SSL network. To address this issue, in this paper, we propose an innovative semi-supervised network with BAlanced Subclass regularIzation and semantic-Conflict penalty mechanism (BASIC) to effectively learn the unbiased knowledge for semi-supervised MoS. Concretely, we construct a novel auxiliary subclass segmentation (SCS) task based on priorly generated balanced subclasses, thus deeply excavating the unbiased information for the main MoS task with the fashion of multi-task learning. Additionally, based on a mean teacher framework, we elaborately design a balanced subclass regularization to utilize the teacher predictions of SCS task to supervise the student predictions of MoS task, thus effectively transferring unbiased knowledge to the MoS subnetwork and alleviating the influence of the class-imbalance problem. Considering the similar semantic information inside the subclasses and their corresponding original classes (i.e., parent classes), we devise a semantic-conflict penalty mechanism to give heavier punishments to the conflicting SCS predictions with wrong parent classes and provide a more accurate constraint to the MoS predictions. Extensive experiments conducted on two publicly available datasets, i.e., the WORD dataset and the MICCAI FLARE 2022 dataset, have verified the superior performance of our proposed BASIC compared to other state-of-the-art methods.

Title: ConcealGS: Concealing Invisible Copyright Information in 3D Gaussian Splatting

• Authors: Yifeng Yang, Hengyu Liu, Chenxin Li, Yining Sun, Wuyang Li, Yifan Liu, Yiyang Lin, Yixuan Yuan, Nanyang Ye
• Subjects: cs.CV, cs.MM, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.03605
• Pdf URL: https://arxiv.org/pdf/2501.03605
• Copy Paste: [[2501.03605]] ConcealGS: Concealing Invisible Copyright Information in 3D Gaussian Splatting(https://arxiv.org/abs/2501.03605)
• Keywords: protect, robust
• Abstract: With the rapid development of 3D reconstruction technology, the widespread distribution of 3D data has become a future trend. While traditional visual data (such as images and videos) and NeRF-based formats already have mature techniques for copyright protection, steganographic techniques for the emerging 3D Gaussian Splatting (3D-GS) format have yet to be fully explored. To address this, we propose ConcealGS, an innovative method for embedding implicit information into 3D-GS. By introducing the knowledge distillation and gradient optimization strategy based on 3D-GS, ConcealGS overcomes the limitations of NeRF-based models and enhances the robustness of implicit information and the quality of 3D reconstruction. We evaluate ConcealGS in various potential application scenarios, and experimental results have demonstrated that ConcealGS not only successfully recovers implicit information but also has almost no impact on rendering quality, providing a new approach for embedding invisible and recoverable information into 3D models in the future.

Title: BTMTrack: Robust RGB-T Tracking via Dual-template Bridging and Temporal-Modal Candidate Elimination

• Authors: Zhongxuan Zhang, Bi Zeng, Xinyu Ni, Yimin Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03616
• Pdf URL: https://arxiv.org/pdf/2501.03616
• Copy Paste: [[2501.03616]] BTMTrack: Robust RGB-T Tracking via Dual-template Bridging and Temporal-Modal Candidate Elimination(https://arxiv.org/abs/2501.03616)
• Keywords: robust
• Abstract: RGB-T tracking leverages the complementary strengths of RGB and thermal infrared (TIR) modalities to address challenging scenarios such as low illumination and adverse weather. However, existing methods often fail to effectively integrate temporal information and perform efficient cross-modal interactions, which constrain their adaptability to dynamic targets. In this paper, we propose BTMTrack, a novel framework for RGB-T tracking. The core of our approach lies in the dual-template backbone network and the Temporal-Modal Candidate Elimination (TMCE) strategy. The dual-template backbone effectively integrates temporal information, while the TMCE strategy focuses the model on target-relevant tokens by evaluating temporal and modal correlations, reducing computational overhead and avoiding irrelevant background noise. Building upon this foundation, we propose the Temporal Dual Template Bridging (TDTB) module, which facilitates precise cross-modal fusion through dynamically filtered tokens. This approach further strengthens the interaction between templates and the search region. Extensive experiments conducted on three benchmark datasets demonstrate the effectiveness of BTMTrack. Our method achieves state-of-the-art performance, with a 72.3% precision rate on the LasHeR test set and competitive results on RGBT210 and RGBT234 datasets.

Title: Coupled Hierarchical Structure Learning using Tree-Wasserstein Distance

• Authors: Ya-Wei Eileen Lin, Ronald R. Coifman, Gal Mishne, Ronen Talmon
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.03627
• Pdf URL: https://arxiv.org/pdf/2501.03627
• Copy Paste: [[2501.03627]] Coupled Hierarchical Structure Learning using Tree-Wasserstein Distance(https://arxiv.org/abs/2501.03627)
• Keywords: diffusion
• Abstract: In many applications, both data samples and features have underlying hierarchical structures. However, existing methods for learning these latent structures typically focus on either samples or features, ignoring possible coupling between them. In this paper, we introduce a coupled hierarchical structure learning method using tree-Wasserstein distance (TWD). Our method jointly computes TWDs for samples and features, representing their latent hierarchies as trees. We propose an iterative, unsupervised procedure to build these sample and feature trees based on diffusion geometry, hyperbolic geometry, and wavelet filters. We show that this iterative procedure converges and empirically improves the quality of the constructed trees. The method is also computationally efficient and scales well in high-dimensional settings. Our method can be seamlessly integrated with hyperbolic graph convolutional networks (HGCN). We demonstrate that our method outperforms competing approaches in sparse approximation and unsupervised Wasserstein distance learning on several word-document and single-cell RNA-sequencing datasets. In addition, integrating our method into HGCN enhances performance in link prediction and node classification tasks.

Title: CFFormer: Cross CNN-Transformer Channel Attention and Spatial Feature Fusion for Improved Segmentation of Low Quality Medical Images

• Authors: Jiaxuan Li, Qing Xu, Xiangjian He, Ziyu Liu, Daokun Zhang, Ruili Wang, Rong Qu, Guoping Qiu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03629
• Pdf URL: https://arxiv.org/pdf/2501.03629
• Copy Paste: [[2501.03629]] CFFormer: Cross CNN-Transformer Channel Attention and Spatial Feature Fusion for Improved Segmentation of Low Quality Medical Images(https://arxiv.org/abs/2501.03629)
• Keywords: extraction, transformer, segmentation
• Abstract: Hybrid CNN-Transformer models are designed to combine the advantages of Convolutional Neural Networks (CNNs) and Transformers to efficiently model both local information and long-range dependencies. However, most research tends to focus on integrating the spatial features of CNNs and Transformers, while overlooking the critical importance of channel features. This is particularly significant for model performance in low-quality medical image segmentation. Effective channel feature extraction can significantly enhance the model's ability to capture contextual information and improve its representation capabilities. To address this issue, we propose a hybrid CNN-Transformer model, CFFormer, and introduce two modules: the Cross Feature Channel Attention (CFCA) module and the X-Spatial Feature Fusion (XFF) module. The model incorporates dual encoders, with the CNN encoder focusing on capturing local features and the Transformer encoder modeling global features. The CFCA module filters and facilitates interactions between the channel features from the two encoders, while the XFF module effectively reduces the significant semantic information differences in spatial features, enabling a smooth and cohesive spatial feature fusion. We evaluate our model across eight datasets covering five modalities to test its generalization capability. Experimental results demonstrate that our model outperforms current state-of-the-art (SOTA) methods, with particularly superior performance on datasets characterized by blurry boundaries and low contrast.

Title: MC-VTON: Minimal Control Virtual Try-On Diffusion Transformer

• Authors: Junsheng Luan, Guangyuan Li, Lei Zhao, Wei Xing
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03630
• Pdf URL: https://arxiv.org/pdf/2501.03630
• Copy Paste: [[2501.03630]] MC-VTON: Minimal Control Virtual Try-On Diffusion Transformer(https://arxiv.org/abs/2501.03630)
• Keywords: diffusion, transformer
• Abstract: Virtual try-on methods based on diffusion models achieve realistic try-on effects. They use an extra reference network or an additional image encoder to process multiple conditional image inputs, which results in high training costs. Besides, they require more than 25 inference steps, bringing a long inference time. In this work, with the development of diffusion transformer (DiT), we rethink the necessity of reference network or image encoder, then propose MC-VTON, enabling DiT to integrate minimal conditional try-on inputs by utilizing its intrinsic backbone. Compared to existing methods, the superiority of MC-VTON is demonstrated in four aspects: (1)Superior detail fidelity. Our DiT-based MC-VTON exhibits superior fidelity in preserving fine-grained details. (2)Simplified network and inputs. We remove any extra reference network or image encoder. We also remove unnecessary conditions like the long prompt, pose estimation, human parsing, and depth map. We require only the masked person image and the garment image. (3)Parameter-efficient training. To process the try-on task, we fine-tune the FLUX.1-dev with only 39.7M additional parameters 0.33% of the backbone parameters). (4)Less inference steps. We apply distillation diffusion on MC-VTON and only need 8 steps to generate a realistic try-on image, with only 86.8M additional parameters (0.72% of the backbone parameters). Experiments show that MC-VTON achieves superior qualitative and quantitative results with fewer condition inputs, fewer inference steps, and fewer trainable parameters than baseline methods.

Title: Action Quality Assessment via Hierarchical Pose-guided Multi-stage Contrastive Regression

• Authors: Mengshi Qi, Hao Ye, Jiaxuan Peng, Huadong Ma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03674
• Pdf URL: https://arxiv.org/pdf/2501.03674
• Copy Paste: [[2501.03674]] Action Quality Assessment via Hierarchical Pose-guided Multi-stage Contrastive Regression(https://arxiv.org/abs/2501.03674)
• Keywords: fair, segmentation
• Abstract: Action Quality Assessment (AQA), which aims at automatic and fair evaluation of athletic performance, has gained increasing attention in recent years. However, athletes are often in rapid movement and the corresponding visual appearance variances are subtle, making it challenging to capture fine-grained pose differences and leading to poor estimation performance. Furthermore, most common AQA tasks, such as diving in sports, are usually divided into multiple sub-actions, each of which contains different durations. However, existing methods focus on segmenting the video into fixed frames, which disrupts the temporal continuity of sub-actions resulting in unavoidable prediction errors. To address these challenges, we propose a novel action quality assessment method through hierarchically pose-guided multi-stage contrastive regression. Firstly, we introduce a multi-scale dynamic visual-skeleton encoder to capture fine-grained spatio-temporal visual and skeletal features. Then, a procedure segmentation network is introduced to separate different sub-actions and obtain segmented features. Afterwards, the segmented visual and skeletal features are both fed into a multi-modal fusion module as physics structural priors, to guide the model in learning refined activity similarities and variances. Finally, a multi-stage contrastive learning regression approach is employed to learn discriminative representations and output prediction results. In addition, we introduce a newly-annotated FineDiving-Pose Dataset to improve the current low-quality human pose labels. In experiments, the results on FineDiving and MTL-AQA datasets demonstrate the effectiveness and superiority of our proposed approach. Our source code and dataset are available at this https URL.

Title: SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoning

• Authors: Andrew Li, Rahul Thapa, Rahul Chalamala, Qingyang Wu, Kezhen Chen, James Zou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03675
• Pdf URL: https://arxiv.org/pdf/2501.03675
• Copy Paste: [[2501.03675]] SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoning(https://arxiv.org/abs/2501.03675)
• Keywords: robust
• Abstract: Vision-Language Models (VLMs) have shown strong performance in understanding single images, aided by numerous high-quality instruction datasets. However, multi-image reasoning tasks are still under-explored in the open-source community due to two main challenges: (1) scaling datasets with multiple correlated images and complex reasoning instructions is resource-intensive and maintaining quality is difficult, and (2) there is a lack of robust evaluation benchmarks for multi-image tasks. To address these issues, we introduce SMIR, an efficient synthetic data-generation pipeline for multi-image reasoning, and a high-quality dataset generated using this pipeline. Our pipeline efficiently extracts highly correlated images using multimodal embeddings, combining visual and descriptive information and leverages open-source LLMs to generate quality instructions. Using this pipeline, we generated 160K synthetic training samples, offering a cost-effective alternative to expensive closed-source solutions. Additionally, we present SMIR-BENCH, a novel multi-image reasoning evaluation benchmark comprising 200 diverse examples across 7 complex multi-image reasoning tasks. SMIR-BENCH is multi-turn and utilizes a VLM judge to evaluate free-form responses, providing a comprehensive assessment of model expressiveness and reasoning capability across modalities. We demonstrate the effectiveness of SMIR dataset by fine-tuning several open-source VLMs and evaluating their performance on SMIR-BENCH. Our results show that models trained on our dataset outperform baseline models in multi-image reasoning tasks up to 8% with a much more scalable data pipeline.

Title: SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment

• Authors: Yuchun Fan, Yongyu Mu, Yilin Wang, Lei Huang, Junhao Ruan, Bei Li, Tong Xiao, Shujian Huang, Xiaocheng Feng, Jingbo Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03681
• Pdf URL: https://arxiv.org/pdf/2501.03681
• Copy Paste: [[2501.03681]] SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment(https://arxiv.org/abs/2501.03681)
• Keywords: large language model
• Abstract: Despite the significant improvements achieved by large language models (LLMs) in English reasoning tasks, these models continue to struggle with multilingual reasoning. Recent studies leverage a full-parameter and two-stage training paradigm to teach models to first understand non-English questions and then reason. However, this method suffers from both substantial computational resource computing and catastrophic forgetting. The fundamental cause is that, with the primary goal of enhancing multilingual comprehension, an excessive number of irrelevant layers and parameters are tuned during the first stage. Given our findings that the representation learning of languages is merely conducted in lower-level layers, we propose an efficient multilingual reasoning alignment approach that precisely identifies and fine-tunes the layers responsible for handling multilingualism. Experimental results show that our method, SLAM, only tunes 6 layers' feed-forward sub-layers including 6.5-8% of all parameters within 7B and 13B LLMs, achieving superior average performance than all strong baselines across 10 languages. Meanwhile, SLAM only involves one training stage, reducing training time by 4.1-11.9 compared to the two-stage method.

Title: Exploring Molecule Generation Using Latent Space Graph Diffusion

• Authors: Prashanth Pombala, Gerrit Grossmann, Verena Wolf
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03696
• Pdf URL: https://arxiv.org/pdf/2501.03696
• Copy Paste: [[2501.03696]] Exploring Molecule Generation Using Latent Space Graph Diffusion(https://arxiv.org/abs/2501.03696)
• Keywords: diffusion, generative
• Abstract: Generating molecular graphs is a challenging task due to their discrete nature and the competitive objectives involved. Diffusion models have emerged as SOTA approaches in data generation across various modalities. For molecular graphs, graph neural networks (GNNs) as a diffusion backbone have achieved impressive results. Latent space diffusion, where diffusion occurs in a low-dimensional space via an autoencoder, has demonstrated computational efficiency. However, the literature on latent space diffusion for molecular graphs is scarce, and no commonly accepted best practices exist. In this work, we explore different approaches and hyperparameters, contrasting generative flow models (denoising diffusion, flow matching, heat dissipation) and architectures (GNNs and E(3)-equivariant GNNs). Our experiments reveal a high sensitivity to the choice of approach and design decisions. Code is made available at this http URL.

Title: Motion-Aware Generative Frame Interpolation

• Authors: Guozhen Zhang, Yuhan Zhu, Yutao Cui, Xiaotong Zhao, Kai Ma, Limin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03699
• Pdf URL: https://arxiv.org/pdf/2501.03699
• Copy Paste: [[2501.03699]] Motion-Aware Generative Frame Interpolation(https://arxiv.org/abs/2501.03699)
• Keywords: generative
• Abstract: Generative frame interpolation, empowered by large-scale pre-trained video generation models, has demonstrated remarkable advantages in complex scenes. However, existing methods heavily rely on the generative model to independently infer the correspondences between input frames, an ability that is inadequately developed during pre-training. In this work, we propose a novel framework, termed Motion-aware Generative frame interpolation (MoG), to significantly enhance the model's motion awareness by integrating explicit motion guidance. Specifically we investigate two key questions: what can serve as an effective motion guidance, and how we can seamlessly embed this guidance into the generative model. For the first question, we reveal that the intermediate flow from flow-based interpolation models could efficiently provide task-oriented motion guidance. Regarding the second, we first obtain guidance-based representations of intermediate frames by warping input frames' representations using guidance, and then integrate them into the model at both latent and feature levels. To demonstrate the versatility of our method, we train MoG on both real-world and animation datasets. Comprehensive evaluations show that our MoG significantly outperforms the existing methods in both domains, achieving superior video quality and improved fidelity.

Title: AuxDepthNet: Real-Time Monocular 3D Object Detection with Depth-Sensitive Features

• Authors: Ruochen Zhang, Hyeung-Sik Choi, Dongwook Jung, Phan Huy Nam Anh, Sang-Ki Jeong, Zihao Zhu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03700
• Pdf URL: https://arxiv.org/pdf/2501.03700
• Copy Paste: [[2501.03700]] AuxDepthNet: Real-Time Monocular 3D Object Detection with Depth-Sensitive Features(https://arxiv.org/abs/2501.03700)
• Keywords: robust, transformer
• Abstract: Monocular 3D object detection is a challenging task in autonomous systems due to the lack of explicit depth information in single-view images. Existing methods often depend on external depth estimators or expensive sensors, which increase computational complexity and hinder real-time performance. To overcome these limitations, we propose AuxDepthNet, an efficient framework for real-time monocular 3D object detection that eliminates the reliance on external depth maps or pre-trained depth models. AuxDepthNet introduces two key components: the Auxiliary Depth Feature (ADF) module, which implicitly learns depth-sensitive features to improve spatial reasoning and computational efficiency, and the Depth Position Mapping (DPM) module, which embeds depth positional information directly into the detection process to enable accurate object localization and 3D bounding box regression. Leveraging the DepthFusion Transformer architecture, AuxDepthNet globally integrates visual and depth-sensitive features through depth-guided interactions, ensuring robust and efficient detection. Extensive experiments on the KITTI dataset show that AuxDepthNet achieves state-of-the-art performance, with $\text{AP}_{3D}$ scores of 24.72\% (Easy), 18.63\% (Moderate), and 15.31\% (Hard), and $\text{AP}_{\text{BEV}}$ scores of 34.11\% (Easy), 25.18\% (Moderate), and 21.90\% (Hard) at an IoU threshold of 0.7.

Title: Unsupervised Speech Segmentation: A General Approach Using Speech Language Models

• Authors: Avishai Elmakies, Omri Abend, Yossi Adi
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.03711
• Pdf URL: https://arxiv.org/pdf/2501.03711
• Copy Paste: [[2501.03711]] Unsupervised Speech Segmentation: A General Approach Using Speech Language Models(https://arxiv.org/abs/2501.03711)
• Keywords: segmentation
• Abstract: In this paper, we introduce an unsupervised approach for Speech Segmentation, which builds on previously researched approaches, e.g., Speaker Diarization, while being applicable to an inclusive set of acoustic-semantic distinctions, paving a path towards a general Unsupervised Speech Segmentation approach. Unlike traditional speech and audio segmentation, which mainly focuses on spectral changes in the input signal, e.g., phone segmentation, our approach tries to segment the spoken utterance into chunks with differing acoustic-semantic styles, focusing on acoustic-semantic information that does not translate well into text, e.g., emotion or speaker. While most Speech Segmentation tasks only handle one style change, e.g., emotion diarization, our approach tries to handle multiple acoustic-semantic style changes. Leveraging recent advances in Speech Language Models (SLMs), we propose a simple unsupervised method to segment a given speech utterance. We empirically demonstrate the effectiveness of the proposed approach by considering several setups. Results suggest that the proposed method is superior to the evaluated baselines on boundary detection, segment purity, and over-segmentation. Code is available at this https URL.

Title: Materialist: Physically Based Editing Using Single-Image Inverse Rendering

• Authors: Lezhong Wang, Duc Minh Tran, Ruiqi Cui, Thomson TG, Manmohan Chandraker, Jeppe Revall Frisvad
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.03717
• Pdf URL: https://arxiv.org/pdf/2501.03717
• Copy Paste: [[2501.03717]] Materialist: Physically Based Editing Using Single-Image Inverse Rendering(https://arxiv.org/abs/2501.03717)
• Keywords: interpretability, diffusion
• Abstract: To perform image editing based on single-view, inverse physically based rendering, we present a method combining a learning-based approach with progressive differentiable rendering. Given an image, our method leverages neural networks to predict initial material properties. Progressive differentiable rendering is then used to optimize the environment map and refine the material properties with the goal of closely matching the rendered result to the input image. We require only a single image while other inverse rendering methods based on the rendering equation require multiple views. In comparison to single-view methods that rely on neural renderers, our approach achieves more realistic light material interactions, accurate shadows, and global illumination. Furthermore, with optimized material properties and illumination, our method enables a variety of tasks, including physically based material editing, object insertion, and relighting. We also propose a method for material transparency editing that operates effectively without requiring full scene geometry. Compared with methods based on Stable Diffusion, our approach offers stronger interpretability and more realistic light refraction based on empirical results.

Title: Self-adaptive vision-language model for 3D segmentation of pulmonary artery and vein

• Authors: Xiaotong Guo, Deqian Yang, Dan Wang, Haochen Zhao, Yuan Li, Zhilin Sui, Tao Zhou, Lijun Zhang, Yanda Meng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03722
• Pdf URL: https://arxiv.org/pdf/2501.03722
• Copy Paste: [[2501.03722]] Self-adaptive vision-language model for 3D segmentation of pulmonary artery and vein(https://arxiv.org/abs/2501.03722)
• Keywords: segmentation
• Abstract: Accurate segmentation of pulmonary structures iscrucial in clinical diagnosis, disease study, and treatment planning. Significant progress has been made in deep learning-based segmentation techniques, but most require much labeled data for training. Consequently, developing precise segmentation methods that demand fewer labeled datasets is paramount in medical image analysis. The emergence of pre-trained vision-language foundation models, such as CLIP, recently opened the door for universal computer vision tasks. Exploiting the generalization ability of these pre-trained foundation models on downstream tasks, such as segmentation, leads to unexpected performance with a relatively small amount of labeled data. However, exploring these models for pulmonary artery-vein segmentation is still limited. This paper proposes a novel framework called Language-guided self-adaptive Cross-Attention Fusion Framework. Our method adopts pre-trained CLIP as a strong feature extractor for generating the segmentation of 3D CT scans, while adaptively aggregating the cross-modality of text and image representations. We propose a s pecially designed adapter module to fine-tune pre-trained CLIP with a self-adaptive learning strategy to effectively fuse the two modalities of embeddings. We extensively validate our method on a local dataset, which is the largest pulmonary artery-vein CT dataset to date and consists of 718 labeled data in total. The experiments show that our method outperformed other state-of-the-art methods by a large margin. Our data and code will be made publicly available upon acceptance.

Title: Realistic Test-Time Adaptation of Vision-Language Models

• Authors: Maxime Zanella, Clément Fuchs, Christophe De Vleeschouwer, Ismail Ben Ayed
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03729
• Pdf URL: https://arxiv.org/pdf/2501.03729
• Copy Paste: [[2501.03729]] Realistic Test-Time Adaptation of Vision-Language Models(https://arxiv.org/abs/2501.03729)
• Keywords: robust
• Abstract: The zero-shot capabilities of Vision-Language Models (VLMs) have been widely leveraged to improve predictive performance. However, previous works on transductive or test-time adaptation (TTA) often make strong assumptions about the data distribution, such as the presence of all classes. Our work challenges these favorable deployment scenarios, and introduces a more realistic evaluation framework, including: (i) a variable number of effective classes for adaptation within a single batch, and (ii) non-i.i.d. batches of test samples in online adaptation settings. We provide comprehensive evaluations, comparisons, and ablation studies that demonstrate how current transductive or TTA methods for VLMs systematically compromise the models' initial zero-shot robustness across various realistic scenarios, favoring performance gains under advantageous assumptions about the test samples' distributions. Furthermore, we introduce StatA, a versatile method that could handle a wide range of deployment scenarios, including those with a variable number of effective classes at test time. Our approach incorporates a novel regularization term designed specifically for VLMs, which acts as a statistical anchor preserving the initial text-encoder knowledge, particularly in low-data regimes. Code available at this https URL.

Title: A Multimodal Lightweight Approach to Fault Diagnosis of Induction Motors in High-Dimensional Dataset

• Authors: Usman Ali
• Subjects: cs.LG, eess.SP, eess.SY
• Abstract URL: https://arxiv.org/abs/2501.03746
• Pdf URL: https://arxiv.org/pdf/2501.03746
• Copy Paste: [[2501.03746]] A Multimodal Lightweight Approach to Fault Diagnosis of Induction Motors in High-Dimensional Dataset(https://arxiv.org/abs/2501.03746)
• Keywords: robust
• Abstract: An accurate AI-based diagnostic system for induction motors (IMs) holds the potential to enhance proactive maintenance, mitigating unplanned downtime and curbing overall maintenance costs within an industrial environment. Notably, among the prevalent faults in IMs, a Broken Rotor Bar (BRB) fault is frequently encountered. Researchers have proposed various fault diagnosis approaches using signal processing (SP), machine learning (ML), deep learning (DL), and hybrid architectures for BRB faults. One limitation in the existing literature is the training of these architectures on relatively small datasets, risking overfitting when implementing such systems in industrial environments. This paper addresses this limitation by implementing large-scale data of BRB faults by using a transfer-learning-based lightweight DL model named ShuffleNetV2 for diagnosing one, two, three, and four BRB faults using current and vibration signal data. Spectral images for training and testing are generated using a Short-Time Fourier Transform (STFT). The dataset comprises 57,500 images, with 47,500 used for training and 10,000 for testing. Remarkably, the ShuffleNetV2 model exhibited superior performance, in less computational cost as well as accurately classifying 98.856% of spectral images. To further enhance the visualization of harmonic sidebands resulting from broken bars, Fast Fourier Transform (FFT) is applied to current and vibration data. The paper also provides insights into the training and testing times for each model, contributing to a comprehensive understanding of the proposed fault diagnosis methodology. The findings of our research provide valuable insights into the performance and efficiency of different ML and DL models, offering a foundation for the development of robust fault diagnosis systems for induction motors in industrial settings.

Title: Context-Alignment: Activating and Enhancing LLM Capabilities in Time Series

• Authors: Yuxiao Hu, Qian Li, Dongxiao Zhang, Jinyue Yan, Yuntian Chen
• Subjects: cs.LG, cs.CL, stat.AP
• Abstract URL: https://arxiv.org/abs/2501.03747
• Pdf URL: https://arxiv.org/pdf/2501.03747
• Copy Paste: [[2501.03747]] Context-Alignment: Activating and Enhancing LLM Capabilities in Time Series(https://arxiv.org/abs/2501.03747)
• Keywords: large language model
• Abstract: Recently, leveraging pre-trained Large Language Models (LLMs) for time series (TS) tasks has gained increasing attention, which involves activating and enhancing LLMs' capabilities. Many methods aim to activate LLMs' capabilities based on token-level alignment but overlook LLMs' inherent strength on natural language processing -- their deep understanding of linguistic logic and structure rather than superficial embedding processing. We propose Context-Alignment, a new paradigm that aligns TS with a linguistic component in the language environments familiar to LLMs to enable LLMs to contextualize and comprehend TS data, thereby activating their capabilities. Specifically, such context-level alignment comprises structural alignment and logical alignment, which is achieved by a Dual-Scale Context-Alignment GNNs (DSCA-GNNs) applied to TS-language multimodal inputs. Structural alignment utilizes dual-scale nodes to describe hierarchical structure in TS-language, enabling LLMs treat long TS data as a whole linguistic component while preserving intrinsic token features. Logical alignment uses directed edges to guide logical relationships, ensuring coherence in the contextual semantics. Demonstration examples prompt are employed to construct Demonstration Examples based Context-Alignment (DECA) following DSCA-GNNs framework. DECA can be flexibly and repeatedly integrated into various layers of pre-trained LLMs to improve awareness of logic and structure, thereby enhancing performance. Extensive experiments show the effectiveness of DECA and the importance of Context-Alignment across tasks, particularly in few-shot and zero-shot forecasting, confirming that Context-Alignment provide powerful prior knowledge on context.

Title: Image Segmentation: Inducing graph-based learning

• Authors: Aryan Singh, Pepijn Van de Ven, Ciarán Eising, Patrick Denny
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.03765
• Pdf URL: https://arxiv.org/pdf/2501.03765
• Copy Paste: [[2501.03765]] Image Segmentation: Inducing graph-based learning(https://arxiv.org/abs/2501.03765)
• Keywords: transformer, segmentation
• Abstract: This study explores the potential of graph neural networks (GNNs) to enhance semantic segmentation across diverse image modalities. We evaluate the effectiveness of a novel GNN-based U-Net architecture on three distinct datasets: PascalVOC, a standard benchmark for natural image segmentation, WoodScape, a challenging dataset of fisheye images commonly used in autonomous driving, introducing significant geometric distortions; and ISIC2016, a dataset of dermoscopic images for skin lesion segmentation. We compare our proposed UNet-GNN model against established convolutional neural networks (CNNs) based segmentation models, including U-Net and U-Net++, as well as the transformer-based SwinUNet. Unlike these methods, which primarily rely on local convolutional operations or global self-attention, GNNs explicitly model relationships between image regions by constructing and operating on a graph representation of the image features. This approach allows the model to capture long-range dependencies and complex spatial relationships, which we hypothesize will be particularly beneficial for handling geometric distortions present in fisheye imagery and capturing intricate boundaries in medical images. Our analysis demonstrates the versatility of GNNs in addressing diverse segmentation challenges and highlights their potential to improve segmentation accuracy in various applications, including autonomous driving and medical image analysis.

Title: AutoFish: Dataset and Benchmark for Fine-grained Analysis of Fish

• Authors: Stefan Hein Bengtson, Daniel Lehotský, Vasiliki Ismiroglou, Niels Madsen, Thomas B. Moeslund, Malte Pedersen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03767
• Pdf URL: https://arxiv.org/pdf/2501.03767
• Copy Paste: [[2501.03767]] AutoFish: Dataset and Benchmark for Fine-grained Analysis of Fish(https://arxiv.org/abs/2501.03767)
• Keywords: segmentation
• Abstract: Automated fish documentation processes are in the near future expected to play an essential role in sustainable fisheries management and for addressing challenges of overfishing. In this paper, we present a novel and publicly available dataset named AutoFish designed for fine-grained fish analysis. The dataset comprises 1,500 images of 454 specimens of visually similar fish placed in various constellations on a white conveyor belt and annotated with instance segmentation masks, IDs, and length measurements. The data was collected in a controlled environment using an RGB camera. The annotation procedure involved manual point annotations, initial segmentation masks proposed by the Segment Anything Model (SAM), and subsequent manual correction of the masks. We establish baseline instance segmentation results using two variations of the Mask2Former architecture, with the best performing model reaching an mAP of 89.15%. Additionally, we present two baseline length estimation methods, the best performing being a custom MobileNetV2-based regression model reaching an MAE of 0.62cm in images with no occlusion and 1.38cm in images with occlusion. Link to project page: this https URL.

Title: Strip R-CNN: Large Strip Convolution for Remote Sensing Object Detection

• Authors: Xinbin Yuan, ZhaoHui Zheng, Yuxuan Li, Xialei Liu, Li Liu, Xiang Li, Qibin Hou, Ming-Ming Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03775
• Pdf URL: https://arxiv.org/pdf/2501.03775
• Copy Paste: [[2501.03775]] Strip R-CNN: Large Strip Convolution for Remote Sensing Object Detection(https://arxiv.org/abs/2501.03775)
• Keywords: fair
• Abstract: While witnessed with rapid development, remote sensing object detection remains challenging for detecting high aspect ratio objects. This paper shows that large strip convolutions are good feature representation learners for remote sensing object detection and can detect objects of various aspect ratios well. Based on large strip convolutions, we build a new network architecture called Strip R-CNN, which is simple, efficient, and powerful. Unlike recent remote sensing object detectors that leverage large-kernel convolutions with square shapes, our Strip R-CNN takes advantage of sequential orthogonal large strip convolutions to capture spatial information. In addition, we enhance the localization capability of remote-sensing object detectors by decoupling the detection heads and equipping the localization head with strip convolutions to better localize the target objects. Extensive experiments on several benchmarks, e.g., DOTA, FAIR1M, HRSC2016, and DIOR, show that our Strip R-CNN can largely improve previous works. Notably, our 30M model achieves 82.75% mAP on DOTA-v1.0, setting a new state-of-the-art this http URL is available at this https URL.

Title: Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights

• Authors: Sy-Tuyen Ho, Tuan Van Vo, Somayeh Ebrahimkhani, Ngai-Man Cheung
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03782
• Pdf URL: https://arxiv.org/pdf/2501.03782
• Copy Paste: [[2501.03782]] Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights(https://arxiv.org/abs/2501.03782)
• Keywords: robust, transformer
• Abstract: While ViTs have achieved across machine learning tasks, deploying them in real-world scenarios faces a critical challenge: generalizing under OoD shifts. A crucial research gap exists in understanding how to design ViT architectures, both manually and automatically, for better OoD generalization. To this end, we introduce OoD-ViT-NAS, the first systematic benchmark for ViTs NAS focused on OoD generalization. This benchmark includes 3000 ViT architectures of varying computational budgets evaluated on 8 common OoD datasets. Using this benchmark, we analyze factors contributing to OoD generalization. Our findings reveal key insights. First, ViT architecture designs significantly affect OoD generalization. Second, ID accuracy is often a poor indicator of OoD accuracy, highlighting the risk of optimizing ViT architectures solely for ID performance. Third, we perform the first study of NAS for ViTs OoD robustness, analyzing 9 Training-free NAS methods. We find that existing Training-free NAS methods are largely ineffective in predicting OoD accuracy despite excelling at ID accuracy. Simple proxies like Param or Flop surprisingly outperform complex Training-free NAS methods in predicting OoD accuracy. Finally, we study how ViT architectural attributes impact OoD generalization and discover that increasing embedding dimensions generally enhances performance. Our benchmark shows that ViT architectures exhibit a wide range of OoD accuracy, with up to 11.85% improvement for some OoD shifts. This underscores the importance of studying ViT architecture design for OoD. We believe OoD-ViT-NAS can catalyze further research into how ViT designs influence OoD generalization.

Title: KAnoCLIP: Zero-Shot Anomaly Detection through Knowledge-Driven Prompt Learning and Enhanced Cross-Modal Integration

• Authors: Chengyuan Li, Suyang Zhou, Jieping Kong, Lei Qi, Hui Xue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03786
• Pdf URL: https://arxiv.org/pdf/2501.03786
• Copy Paste: [[2501.03786]] KAnoCLIP: Zero-Shot Anomaly Detection through Knowledge-Driven Prompt Learning and Enhanced Cross-Modal Integration(https://arxiv.org/abs/2501.03786)
• Keywords: privacy, large language model, segmentation
• Abstract: Zero-shot anomaly detection (ZSAD) identifies anomalies without needing training samples from the target dataset, essential for scenarios with privacy concerns or limited data. Vision-language models like CLIP show potential in ZSAD but have limitations: relying on manually crafted fixed textual descriptions or anomaly prompts is time-consuming and prone to semantic ambiguity, and CLIP struggles with pixel-level anomaly segmentation, focusing more on global semantics than local details. To address these limitations, We introduce KAnoCLIP, a novel ZSAD framework that leverages vision-language models. KAnoCLIP combines general knowledge from a Large Language Model (GPT-3.5) and fine-grained, image-specific knowledge from a Visual Question Answering system (Llama3) via Knowledge-Driven Prompt Learning (KnPL). KnPL uses a knowledge-driven (KD) loss function to create learnable anomaly prompts, removing the need for fixed text prompts and enhancing generalization. KAnoCLIP includes the CLIP visual encoder with V-V attention (CLIP-VV), Bi-Directional Cross-Attention for Multi-Level Cross-Modal Interaction (Bi-CMCI), and Conv-Adapter. These components preserve local visual semantics, improve local cross-modal fusion, and align global visual features with textual information, enhancing pixel-level anomaly detection. KAnoCLIP achieves state-of-the-art performance in ZSAD across 12 industrial and medical datasets, demonstrating superior generalization compared to existing methods.

Title: MADation: Face Morphing Attack Detection with Foundation Models

• Authors: Eduarda Caldeira, Guray Ozgur, Tahar Chettaoui, Marija Ivanovska, Fadi Boutros, Vitomir Struc, Naser Damer
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.03800
• Pdf URL: https://arxiv.org/pdf/2501.03800
• Copy Paste: [[2501.03800]] MADation: Face Morphing Attack Detection with Foundation Models(https://arxiv.org/abs/2501.03800)
• Keywords: secure, attack, transformer
• Abstract: Despite the considerable performance improvements of face recognition algorithms in recent years, the same scientific advances responsible for this progress can also be used to create efficient ways to attack them, posing a threat to their secure deployment. Morphing attack detection (MAD) systems aim to detect a specific type of threat, morphing attacks, at an early stage, preventing them from being considered for verification in critical processes. Foundation models (FM) learn from extensive amounts of unlabeled data, achieving remarkable zero-shot generalization to unseen domains. Although this generalization capacity might be weak when dealing with domain-specific downstream tasks such as MAD, FMs can easily adapt to these settings while retaining the built-in knowledge acquired during pre-training. In this work, we recognize the potential of FMs to perform well in the MAD task when properly adapted to its specificities. To this end, we adapt FM CLIP architectures with LoRA weights while simultaneously training a classification header. The proposed framework, MADation surpasses our alternative FM and transformer-based frameworks and constitutes the first adaption of FMs to the MAD task. MADation presents competitive results with current MAD solutions in the literature and even surpasses them in several evaluation scenarios. To encourage reproducibility and facilitate further research in MAD, we publicly release the implementation of MADation at https: //github.com/gurayozgur/MADation

Title: Private, Auditable, and Distributed Ledger for Financial Institutes

• Authors: Shaltiel Eloul, Yash Satsangi, Yeoh Wei Zhu, Omar Amer, Georgios Papadopoulos, Marco Pistoia
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.03808
• Pdf URL: https://arxiv.org/pdf/2501.03808
• Copy Paste: [[2501.03808]] Private, Auditable, and Distributed Ledger for Financial Institutes(https://arxiv.org/abs/2501.03808)
• Keywords: secure, privacy
• Abstract: Distributed ledger technology offers several advantages for banking and finance industry, including efficient transaction processing and cross-party transaction reconciliation. The key challenges for adoption of this technology in financial institutes are (a) the building of a privacy-preserving ledger, (b) supporting auditing and regulatory requirements, and (c) flexibility to adapt to complex use-cases with multiple digital assets and actors. This paper proposes a framework for a private, audit-able, and distributed ledger (PADL) that adapts easily to fundamental use-cases within financial institutes. PADL employs widely-used cryptography schemes combined with zero-knowledge proofs to propose a transaction scheme for a `table' like ledger. It enables fast confidential peer-to-peer multi-asset transactions, and transaction graph anonymity, in a no-trust setup, but with customized privacy. We prove that integrity and anonymity of PADL is secured against a strong threat model. Furthermore, we showcase three fundamental real-life use-cases, namely, an assets exchange ledger, a settlement ledger, and a bond market ledger. Based on these use-cases we show that PADL supports smooth-lined inter-assets auditing while preserving privacy of the participants. For example, we show how a bank can be audited for its liquidity or credit risk without violation of privacy of itself or any other party, or how can PADL ensures honest coupon rate payment in bond market without sharing investors values. Finally, our evaluation shows PADL's advantage in performance against previous relevant schemes.

Title: Three-dimensional attention Transformer for state evaluation in real-time strategy games

• Authors: Yanqing Ye, Weilong Yang, Kai Qiu, Jie Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03832
• Pdf URL: https://arxiv.org/pdf/2501.03832
• Copy Paste: [[2501.03832]] Three-dimensional attention Transformer for state evaluation in real-time strategy games(https://arxiv.org/abs/2501.03832)
• Keywords: transformer
• Abstract: Situation assessment in Real-Time Strategy (RTS) games is crucial for understanding decision-making in complex adversarial environments. However, existing methods remain limited in processing multi-dimensional feature information and temporal dependencies. Here we propose a tri-dimensional Space-Time-Feature Transformer (TSTF Transformer) architecture, which efficiently models battlefield situations through three independent but cascaded modules: spatial attention, temporal attention, and feature attention. On a dataset comprising 3,150 adversarial experiments, the 8-layer TSTF Transformer demonstrates superior performance: achieving 58.7% accuracy in the early game (~4% progress), significantly outperforming the conventional Timesformer's 41.8%; reaching 97.6% accuracy in the mid-game (~40% progress) while maintaining low performance variation (standard deviation 0.114). Meanwhile, this architecture requires fewer parameters (4.75M) compared to the baseline model (5.54M). Our study not only provides new insights into situation assessment in RTS games but also presents an innovative paradigm for Transformer-based multi-dimensional temporal modeling.

Title: LM-Net: A Light-weight and Multi-scale Network for Medical Image Segmentation

• Authors: Zhenkun Lu, Chaoyin She, Wei Wang, Qinghua Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03838
• Pdf URL: https://arxiv.org/pdf/2501.03838
• Copy Paste: [[2501.03838]] LM-Net: A Light-weight and Multi-scale Network for Medical Image Segmentation(https://arxiv.org/abs/2501.03838)
• Keywords: transformer, segmentation
• Abstract: Current medical image segmentation approaches have limitations in deeply exploring multi-scale information and effectively combining local detail textures with global contextual semantic information. This results in over-segmentation, under-segmentation, and blurred segmentation boundaries. To tackle these challenges, we explore multi-scale feature representations from different perspectives, proposing a novel, lightweight, and multi-scale architecture (LM-Net) that integrates advantages of both Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) to enhance segmentation accuracy. LM-Net employs a lightweight multi-branch module to capture multi-scale features at the same level. Furthermore, we introduce two modules to concurrently capture local detail textures and global semantics with multi-scale features at different levels: the Local Feature Transformer (LFT) and Global Feature Transformer (GFT). The LFT integrates local window self-attention to capture local detail textures, while the GFT leverages global self-attention to capture global contextual semantics. By combining these modules, our model achieves complementarity between local and global representations, alleviating the problem of blurred segmentation boundaries in medical image segmentation. To evaluate the feasibility of LM-Net, extensive experiments have been conducted on three publicly available datasets with different modalities. Our proposed model achieves state-of-the-art results, surpassing previous methods, while only requiring 4.66G FLOPs and 5.4M parameters. These state-of-the-art results on three datasets with different modalities demonstrate the effectiveness and adaptability of our proposed LM-Net for various medical image segmentation tasks.

Title: Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control

• Authors: Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, Wenping Wang, Yuan Liu
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.03847
• Pdf URL: https://arxiv.org/pdf/2501.03847
• Copy Paste: [[2501.03847]] Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control(https://arxiv.org/abs/2501.03847)
• Keywords: diffusion
• Abstract: Diffusion models have demonstrated impressive performance in generating high-quality videos from text prompts or images. However, precise control over the video generation process, such as camera manipulation or content editing, remains a significant challenge. Existing methods for controlled video generation are typically limited to a single control type, lacking the flexibility to handle diverse control demands. In this paper, we introduce Diffusion as Shader (DaS), a novel approach that supports multiple video control tasks within a unified architecture. Our key insight is that achieving versatile video control necessitates leveraging 3D control signals, as videos are fundamentally 2D renderings of dynamic 3D content. Unlike prior methods limited to 2D control signals, DaS leverages 3D tracking videos as control inputs, making the video diffusion process inherently 3D-aware. This innovation allows DaS to achieve a wide range of video controls by simply manipulating the 3D tracking videos. A further advantage of using 3D tracking videos is their ability to effectively link frames, significantly enhancing the temporal consistency of the generated videos. With just 3 days of fine-tuning on 8 H800 GPUs using less than 10k videos, DaS demonstrates strong control capabilities across diverse tasks, including mesh-to-video generation, camera control, motion transfer, and object manipulation.

Title: Progressive Document-level Text Simplification via Large Language Models

• Authors: Dengzhao Fang, Jipeng Qiang, Yi Zhu, Yunhao Yuan, Wei Li, Yan Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03857
• Pdf URL: https://arxiv.org/pdf/2501.03857
• Copy Paste: [[2501.03857]] Progressive Document-level Text Simplification via Large Language Models(https://arxiv.org/abs/2501.03857)
• Keywords: large language model
• Abstract: Research on text simplification has primarily focused on lexical and sentence-level changes. Long document-level simplification (DS) is still relatively unexplored. Large Language Models (LLMs), like ChatGPT, have excelled in many natural language processing tasks. However, their performance on DS tasks is unsatisfactory, as they often treat DS as merely document summarization. For the DS task, the generated long sequences not only must maintain consistency with the original document throughout, but complete moderate simplification operations encompassing discourses, sentences, and word-level simplifications. Human editors employ a hierarchical complexity simplification strategy to simplify documents. This study delves into simulating this strategy through the utilization of a multi-stage collaboration using LLMs. We propose a progressive simplification method (ProgDS) by hierarchically decomposing the task, including the discourse-level, topic-level, and lexical-level simplification. Experimental results demonstrate that ProgDS significantly outperforms existing smaller models or direct prompting with LLMs, advancing the state-of-the-art in the document simplification task.

Title: Improving Dialectal Slot and Intent Detection with Auxiliary Tasks: A Multi-Dialectal Bavarian Case Study

• Authors: Xaver Maria Krückl, Verena Blaschke, Barbara Plank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03863
• Pdf URL: https://arxiv.org/pdf/2501.03863
• Copy Paste: [[2501.03863]] Improving Dialectal Slot and Intent Detection with Auxiliary Tasks: A Multi-Dialectal Bavarian Case Study(https://arxiv.org/abs/2501.03863)
• Keywords: transformer
• Abstract: Reliable slot and intent detection (SID) is crucial in natural language understanding for applications like digital assistants. Encoder-only transformer models fine-tuned on high-resource languages generally perform well on SID. However, they struggle with dialectal data, where no standardized form exists and training data is scarce and costly to produce. We explore zero-shot transfer learning for SID, focusing on multiple Bavarian dialects, for which we release a new dataset for the Munich dialect. We evaluate models trained on auxiliary tasks in Bavarian, and compare joint multi-task learning with intermediate-task training. We also compare three types of auxiliary tasks: token-level syntactic tasks, named entity recognition (NER), and language modelling. We find that the included auxiliary tasks have a more positive effect on slot filling than intent classification (with NER having the most positive effect), and that intermediate-task training yields more consistent performance gains. Our best-performing approach improves intent classification performance on Bavarian dialects by 5.1 and slot filling F1 by 8.4 percentage points.

Title: Add Noise, Tasks, or Layers? MaiNLP at the VarDial 2025 Shared Task on Norwegian Dialectal Slot and Intent Detection

• Authors: Verena Blaschke, Felicia Körner, Barbara Plank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03870
• Pdf URL: https://arxiv.org/pdf/2501.03870
• Copy Paste: [[2501.03870]] Add Noise, Tasks, or Layers? MaiNLP at the VarDial 2025 Shared Task on Norwegian Dialectal Slot and Intent Detection(https://arxiv.org/abs/2501.03870)
• Keywords: robust
• Abstract: Slot and intent detection (SID) is a classic natural language understanding task. Despite this, research has only more recently begun focusing on SID for dialectal and colloquial varieties. Many approaches for low-resource scenarios have not yet been applied to dialectal SID data, or compared to each other on the same datasets. We participate in the VarDial 2025 shared task on slot and intent detection in Norwegian varieties, and compare multiple set-ups: varying the training data (English, Norwegian, or dialectal Norwegian), injecting character-level noise, training on auxiliary tasks, and applying Layer Swapping, a technique in which layers of models fine-tuned on different datasets are assembled into a model. We find noise injection to be beneficial while the effects of auxiliary tasks are mixed. Though some experimentation was required to successfully assemble a model from layers, it worked surprisingly well; a combination of models trained on English and small amounts of dialectal data produced the most robust slot predictions. Our best models achieve 97.6% intent accuracy and 85.6% slot F1 in the shared task.

Title: ZDySS -- Zero-Shot Dynamic Scene Stylization using Gaussian Splatting

• Authors: Abhishek Saroha, Florian Hofherr, Mariia Gladkova, Cecilia Curreli, Or Litany, Daniel Cremers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03875
• Pdf URL: https://arxiv.org/pdf/2501.03875
• Copy Paste: [[2501.03875]] ZDySS -- Zero-Shot Dynamic Scene Stylization using Gaussian Splatting(https://arxiv.org/abs/2501.03875)
• Keywords: robust
• Abstract: Stylizing a dynamic scene based on an exemplar image is critical for various real-world applications, including gaming, filmmaking, and augmented and virtual reality. However, achieving consistent stylization across both spatial and temporal dimensions remains a significant challenge. Most existing methods are designed for static scenes and often require an optimization process for each style image, limiting their adaptability. We introduce ZDySS, a zero-shot stylization framework for dynamic scenes, allowing our model to generalize to previously unseen style images at inference. Our approach employs Gaussian splatting for scene representation, linking each Gaussian to a learned feature vector that renders a feature map for any given view and timestamp. By applying style transfer on the learned feature vectors instead of the rendered feature map, we enhance spatio-temporal consistency across frames. Our method demonstrates superior performance and coherence over state-of-the-art baselines in tests on real-world dynamic scenes, making it a robust solution for practical applications.

Title: CL3DOR: Contrastive Learning for 3D Large Multimodal Models via Odds Ratio on High-Resolution Point Clouds

• Authors: Keonwoo Kim, Yeongjae Cho, Taebaek Hwang, Minsoo Jo, Sangdo Han
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03879
• Pdf URL: https://arxiv.org/pdf/2501.03879
• Copy Paste: [[2501.03879]] CL3DOR: Contrastive Learning for 3D Large Multimodal Models via Odds Ratio on High-Resolution Point Clouds(https://arxiv.org/abs/2501.03879)
• Keywords: large language model
• Abstract: Recent research has demonstrated that Large Language Models (LLMs) are not limited to text-only tasks but can also function as multimodal models across various modalities, including audio, images, and videos. In particular, research on 3D Large Multimodal Models (3D LMMs) is making notable strides, driven by the potential of processing higher-dimensional data like point clouds. However, upon closer examination, we find that the visual and textual content within each sample of existing training datasets lacks both high informational granularity and clarity, which serve as a bottleneck for precise cross-modal understanding. To address these issues, we propose CL3DOR, Contrastive Learning for 3D large multimodal models via Odds ratio on high-Resolution point clouds, designed to ensure greater specificity and clarity in both visual and textual content. Specifically, we increase the density of point clouds per object and construct informative hard negative responses in the training dataset to penalize unwanted responses. To leverage hard negative responses, we incorporate the odds ratio as an auxiliary term for contrastive learning into the conventional language modeling loss. CL3DOR achieves state-of-the-art performance in 3D scene understanding and reasoning benchmarks. Additionally, we demonstrate the effectiveness of CL3DOR's key components through extensive experiments.

Title: AlphaPO -- Reward shape matters for LLM alignment

• Authors: Aman Gupta, Shao Tang, Qingquan Song, Sirou Zhu, Jiwoo Hong, Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Jason Zhu, Natesh Pillai, S. Sathiya Keerthi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03884
• Pdf URL: https://arxiv.org/pdf/2501.03884
• Copy Paste: [[2501.03884]] AlphaPO -- Reward shape matters for LLM alignment(https://arxiv.org/abs/2501.03884)
• Keywords: large language model
• Abstract: Reinforcement Learning with Human Feedback (RLHF) and its variants have made huge strides toward the effective alignment of large language models (LLMs) to follow instructions and reflect human values. More recently, Direct Alignment Algorithms (DAAs) have emerged in which the reward modeling stage of RLHF is skipped by characterizing the reward directly as a function of the policy being learned. Examples include Direct Preference Optimization (DPO) and Simple Preference Optimization (SimPO). These methods often suffer from likelihood displacement, a phenomenon by which the probabilities of preferred responses are often reduced undesirably. In this paper, we argue that, for DAAs the reward (function) shape matters. We introduce AlphaPO, a new DAA method that leverages an $\alpha$-parameter to help change the shape of the reward function beyond the standard log reward. AlphaPO helps maintain fine-grained control over likelihood displacement and over-optimization. Compared to SimPO, one of the best performing DAAs, AlphaPO leads to about 7\% to 10\% relative improvement in alignment performance for the instruct versions of Mistral-7B and Llama3-8B. The analysis and results presented highlight the importance of the reward shape, and how one can systematically change it to affect training dynamics, as well as improve alignment performance.

Title: Superpixel Boundary Correction for Weakly-Supervised Semantic Segmentation on Histopathology Images

• Authors: Hongyi Wu, Hong Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03891
• Pdf URL: https://arxiv.org/pdf/2501.03891
• Copy Paste: [[2501.03891]] Superpixel Boundary Correction for Weakly-Supervised Semantic Segmentation on Histopathology Images(https://arxiv.org/abs/2501.03891)
• Keywords: segmentation
• Abstract: With the rapid advancement of deep learning, computational pathology has made significant progress in cancer diagnosis and subtyping. Tissue segmentation is a core challenge, essential for prognosis and treatment decisions. Weakly supervised semantic segmentation (WSSS) reduces the annotation requirement by using image-level labels instead of pixel-level ones. However, Class Activation Map (CAM)-based methods still suffer from low spatial resolution and unclear boundaries. To address these issues, we propose a multi-level superpixel correction algorithm that refines CAM boundaries using superpixel clustering and floodfill. Experimental results show that our method achieves great performance on breast cancer segmentation dataset with mIoU of 71.08%, significantly improving tumor microenvironment boundary delineation.

Title: LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token

• Authors: Shaolei Zhang, Qingkai Fang, Zhe Yang, Yang Feng
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03895
• Pdf URL: https://arxiv.org/pdf/2501.03895
• Copy Paste: [[2501.03895]] LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token(https://arxiv.org/abs/2501.03895)
• Keywords: large language model
• Abstract: The advent of real-time large multimodal models (LMMs) like GPT-4o has sparked considerable interest in efficient LMMs. LMM frameworks typically encode visual inputs into vision tokens (continuous representations) and integrate them and textual instructions into the context of large language models (LLMs), where large-scale parameters and numerous context tokens (predominantly vision tokens) result in substantial computational overhead. Previous efforts towards efficient LMMs always focus on replacing the LLM backbone with smaller models, while neglecting the crucial issue of token quantity. In this paper, we introduce LLaVA-Mini, an efficient LMM with minimal vision tokens. To achieve a high compression ratio of vision tokens while preserving visual information, we first analyze how LMMs understand vision tokens and find that most vision tokens only play a crucial role in the early layers of LLM backbone, where they mainly fuse visual information into text tokens. Building on this finding, LLaVA-Mini introduces modality pre-fusion to fuse visual information into text tokens in advance, thereby facilitating the extreme compression of vision tokens fed to LLM backbone into one token. LLaVA-Mini is a unified large multimodal model that can support the understanding of images, high-resolution images, and videos in an efficient manner. Experiments across 11 image-based and 7 video-based benchmarks demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by 77%, deliver low-latency responses within 40 milliseconds, and process over 10,000 frames of video on the GPU hardware with 24GB of memory.

Title: SPECTRE: A Hybrid System for an Adaptative and Optimised Cyber Threats Detection, Response and Investigation in Volatile Memory

• Authors: Arslan Tariq Syed, Mohamed Chahine Ghanem, Elhadj Benkhelifa, Fauzia Idrees Abro
• Subjects: cs.CR, cs.NI, cs.OS
• Abstract URL: https://arxiv.org/abs/2501.03898
• Pdf URL: https://arxiv.org/pdf/2501.03898
• Copy Paste: [[2501.03898]] SPECTRE: A Hybrid System for an Adaptative and Optimised Cyber Threats Detection, Response and Investigation in Volatile Memory(https://arxiv.org/abs/2501.03898)
• Keywords: attack, robust
• Abstract: The increasing sophistication of modern cyber threats, particularly file-less malware relying on living-off-the-land techniques, poses significant challenges to traditional detection mechanisms. Memory forensics has emerged as a crucial method for uncovering such threats by analysing dynamic changes in memory. This research introduces SPECTRE (Snapshot Processing, Emulation, Comparison, and Threat Reporting Engine), a modular Cyber Incident Response System designed to enhance threat detection, investigation, and visualization. By adopting Volatility JSON format as an intermediate output, SPECTRE ensures compatibility with widely used DFIR tools, minimizing manual data transformations and enabling seamless integration into established workflows. Its emulation capabilities safely replicate realistic attack scenarios, such as credential dumping and malicious process injections, for controlled experimentation and validation. The anomaly detection module addresses critical attack vectors, including RunDLL32 abuse and malicious IP detection, while the IP forensics module enhances threat intelligence by integrating tools like Virus Total and geolocation APIs. SPECTRE advanced visualization techniques transform raw memory data into actionable insights, aiding Red, Blue and Purple teams in refining strategies and responding effectively to threats. Bridging gaps between memory and network forensics, SPECTRE offers a scalable, robust platform for advancing threat detection, team training, and forensic research in combating sophisticated cyber threats.

Title: Explainable Reinforcement Learning via Temporal Policy Decomposition

• Authors: Franco Ruggeri, Alessio Russo, Rafia Inam, Karl Henrik Johansson
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03902
• Pdf URL: https://arxiv.org/pdf/2501.03902
• Copy Paste: [[2501.03902]] Explainable Reinforcement Learning via Temporal Policy Decomposition(https://arxiv.org/abs/2501.03902)
• Keywords: interpretability, explainability
• Abstract: We investigate the explainability of Reinforcement Learning (RL) policies from a temporal perspective, focusing on the sequence of future outcomes associated with individual actions. In RL, value functions compress information about rewards collected across multiple trajectories and over an infinite horizon, allowing a compact form of knowledge representation. However, this compression obscures the temporal details inherent in sequential decision-making, presenting a key challenge for interpretability. We present Temporal Policy Decomposition (TPD), a novel explainability approach that explains individual RL actions in terms of their Expected Future Outcome (EFO). These explanations decompose generalized value functions into a sequence of EFOs, one for each time step up to a prediction horizon of interest, revealing insights into when specific outcomes are expected to occur. We leverage fixed-horizon temporal difference learning to devise an off-policy method for learning EFOs for both optimal and suboptimal actions, enabling contrastive explanations consisting of EFOs for different state-action pairs. Our experiments demonstrate that TPD generates accurate explanations that (i) clarify the policy's future strategy and anticipated trajectory for a given action and (ii) improve understanding of the reward composition, facilitating fine-tuning of the reward function to align with human expectations.

Title: Exploring the Potential of Large Language Models in Public Transportation: San Antonio Case Study

• Authors: Ramya Jonnala, Gongbo Liang, Jeong Yang, Izzat Alsmadi
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.03904
• Pdf URL: https://arxiv.org/pdf/2501.03904
• Copy Paste: [[2501.03904]] Exploring the Potential of Large Language Models in Public Transportation: San Antonio Case Study(https://arxiv.org/abs/2501.03904)
• Keywords: large language model
• Abstract: The integration of large language models (LLMs) into public transit systems presents a transformative opportunity to enhance urban mobility. This study explores the potential of LLMs to revolutionize public transportation management within the context of San Antonio's transit system. Leveraging the capabilities of LLMs in natural language processing and data analysis, we investigate their capabilities to optimize route planning, reduce wait times, and provide personalized travel assistance. By utilizing the General Transit Feed Specification (GTFS) and other relevant data, this research aims to demonstrate how LLMs can potentially improve resource allocation, elevate passenger satisfaction, and inform data-driven decision-making in transit operations. A comparative analysis of different ChatGPT models was conducted to assess their ability to understand transportation information, retrieve relevant data, and provide comprehensive responses. Findings from this study suggest that while LLMs hold immense promise for public transit, careful engineering and fine-tuning are essential to realizing their full potential. San Antonio serves as a case study to inform the development of LLM-powered transit systems in other urban environments.

Title: HYB-VITON: A Hybrid Approach to Virtual Try-On Combining Explicit and Implicit Warping

• Authors: Kosuke Takemoto, Takafumi Koshinaka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03910
• Pdf URL: https://arxiv.org/pdf/2501.03910
• Copy Paste: [[2501.03910]] HYB-VITON: A Hybrid Approach to Virtual Try-On Combining Explicit and Implicit Warping(https://arxiv.org/abs/2501.03910)
• Keywords: diffusion
• Abstract: Virtual try-on systems have significant potential in e-commerce, allowing customers to visualize garments on themselves. Existing image-based methods fall into two categories: those that directly warp garment-images onto person-images (explicit warping), and those using cross-attention to reconstruct given garments (implicit warping). Explicit warping preserves garment details but often produces unrealistic output, while implicit warping achieves natural reconstruction but struggles with fine details. We propose HYB-VITON, a novel approach that combines the advantages of each method and includes both a preprocessing pipeline for warped garments and a novel training option. These components allow us to utilize beneficial regions of explicitly warped garments while leveraging the natural reconstruction of implicit warping. A series of experiments demonstrates that HYB-VITON preserves garment details more faithfully than recent diffusion-based methods, while producing more realistic results than a state-of-the-art explicit warping method.

Title: Magic Mirror: ID-Preserved Video Generation in Video Diffusion Transformers

• Authors: Yuechen Zhang, Yaoyang Liu, Bin Xia, Bohao Peng, Zexin Yan, Eric Lo, Jiaya Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03931
• Pdf URL: https://arxiv.org/pdf/2501.03931
• Copy Paste: [[2501.03931]] Magic Mirror: ID-Preserved Video Generation in Video Diffusion Transformers(https://arxiv.org/abs/2501.03931)
• Keywords: diffusion, transformer
• Abstract: We present Magic Mirror, a framework for generating identity-preserved videos with cinematic-level quality and dynamic motion. While recent advances in video diffusion models have shown impressive capabilities in text-to-video generation, maintaining consistent identity while producing natural motion remains challenging. Previous methods either require person-specific fine-tuning or struggle to balance identity preservation with motion diversity. Built upon Video Diffusion Transformers, our method introduces three key components: (1) a dual-branch facial feature extractor that captures both identity and structural features, (2) a lightweight cross-modal adapter with Conditioned Adaptive Normalization for efficient identity integration, and (3) a two-stage training strategy combining synthetic identity pairs with video data. Extensive experiments demonstrate that Magic Mirror effectively balances identity consistency with natural motion, outperforming existing methods across multiple metrics while requiring minimal parameters added. The code and model will be made publicly available at: this https URL

Title: A precise asymptotic analysis of learning diffusion models: theory and insights

• Authors: Hugo Cui, Cengiz Pehlevan, Yue M. Lu
• Subjects: cs.LG, cond-mat.dis-nn
• Abstract URL: https://arxiv.org/abs/2501.03937
• Pdf URL: https://arxiv.org/pdf/2501.03937
• Copy Paste: [[2501.03937]] A precise asymptotic analysis of learning diffusion models: theory and insights(https://arxiv.org/abs/2501.03937)
• Keywords: diffusion, generative
• Abstract: In this manuscript, we consider the problem of learning a flow or diffusion-based generative model parametrized by a two-layer auto-encoder, trained with online stochastic gradient descent, on a high-dimensional target density with an underlying low-dimensional manifold structure. We derive a tight asymptotic characterization of low-dimensional projections of the distribution of samples generated by the learned model, ascertaining in particular its dependence on the number of training samples. Building on this analysis, we discuss how mode collapse can arise, and lead to model collapse when the generative model is re-trained on generated synthetic data.

Title: Visual question answering: from early developments to recent advances -- a survey

• Authors: Ngoc Dung Huynh, Mohamed Reda Bouadjenek, Sunil Aryal, Imran Razzak, Hakim Hacid
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2501.03939
• Pdf URL: https://arxiv.org/pdf/2501.03939
• Copy Paste: [[2501.03939]] Visual question answering: from early developments to recent advances -- a survey(https://arxiv.org/abs/2501.03939)
• Keywords: extraction
• Abstract: Visual Question Answering (VQA) is an evolving research field aimed at enabling machines to answer questions about visual content by integrating image and language processing techniques such as feature extraction, object detection, text embedding, natural language understanding, and language generation. With the growth of multimodal data research, VQA has gained significant attention due to its broad applications, including interactive educational tools, medical image diagnosis, customer service, entertainment, and social media captioning. Additionally, VQA plays a vital role in assisting visually impaired individuals by generating descriptive content from images. This survey introduces a taxonomy of VQA architectures, categorizing them based on design choices and key components to facilitate comparative analysis and evaluation. We review major VQA approaches, focusing on deep learning-based methods, and explore the emerging field of Large Visual Language Models (LVLMs) that have demonstrated success in multimodal tasks like VQA. The paper further examines available datasets and evaluation metrics essential for measuring VQA system performance, followed by an exploration of real-world VQA applications. Finally, we highlight ongoing challenges and future directions in VQA research, presenting open questions and potential areas for further development. This survey serves as a comprehensive resource for researchers and practitioners interested in the latest advancements and future

Title: Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection

• Authors: Pablo Miralles-González, Javier Huertas-Tato, Alejandro Martín, David Camacho
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03940
• Pdf URL: https://arxiv.org/pdf/2501.03940
• Copy Paste: [[2501.03940]] Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection(https://arxiv.org/abs/2501.03940)
• Keywords: attack, robust, large language model
• Abstract: The rapid advancement in large language models (LLMs) has significantly enhanced their ability to generate coherent and contextually relevant text, raising concerns about the misuse of AI-generated content and making it critical to detect it. However, the task remains challenging, particularly in unseen domains or with unfamiliar LLMs. Leveraging LLM next-token distribution outputs offers a theoretically appealing approach for detection, as they encapsulate insights from the models' extensive pre-training on diverse corpora. Despite its promise, zero-shot methods that attempt to operationalize these outputs have met with limited success. We hypothesize that one of the problems is that they use the mean to aggregate next-token distribution metrics across tokens, when some tokens are naturally easier or harder to predict and should be weighted differently. Based on this idea, we propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length. Although not zero-shot, our method allows us to cache the last hidden states and next-token distribution metrics on disk, greatly reducing the training resource requirements. PAWN shows competitive and even better performance in-distribution than the strongest baselines (fine-tuned LMs) with a fraction of their trainable parameters. Our model also generalizes better to unseen domains and source models, with smaller variability in the decision boundary across distribution shifts. It is also more robust to adversarial attacks, and if the backbone has multilingual capabilities, it presents decent generalization to languages not seen during supervised training, with LLaMA3-1B reaching a mean macro-averaged F1 score of 81.46% in cross-validation with nine languages.

Title: Synthetic Data Privacy Metrics

• Authors: Amy Steier, Lipika Ramaswamy, Andre Manoel, Alexa Haushalter
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03941
• Pdf URL: https://arxiv.org/pdf/2501.03941
• Copy Paste: [[2501.03941]] Synthetic Data Privacy Metrics(https://arxiv.org/abs/2501.03941)
• Keywords: privacy, attack, generative
• Abstract: Recent advancements in generative AI have made it possible to create synthetic datasets that can be as accurate as real-world data for training AI models, powering statistical insights, and fostering collaboration with sensitive datasets while offering strong privacy guarantees. Effectively measuring the empirical privacy of synthetic data is an important step in the process. However, while there is a multitude of new privacy metrics being published every day, there currently is no standardization. In this paper, we review the pros and cons of popular metrics that include simulations of adversarial attacks. We also review current best practices for amending generative models to enhance the privacy of the data they create (e.g. differential privacy).

Title: Localizing AI: Evaluating Open-Weight Language Models for Languages of Baltic States

• Authors: Jurgita Kapočiūtė-Dzikienė, Toms Bergmanis, Mārcis Pinnis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03952
• Pdf URL: https://arxiv.org/pdf/2501.03952
• Copy Paste: [[2501.03952]] Localizing AI: Evaluating Open-Weight Language Models for Languages of Baltic States(https://arxiv.org/abs/2501.03952)
• Keywords: privacy, large language model
• Abstract: Although large language models (LLMs) have transformed our expectations of modern language technologies, concerns over data privacy often restrict the use of commercially available LLMs hosted outside of EU jurisdictions. This limits their application in governmental, defence, and other data-sensitive sectors. In this work, we evaluate the extent to which locally deployable open-weight LLMs support lesser-spoken languages such as Lithuanian, Latvian, and Estonian. We examine various size and precision variants of the top-performing multilingual open-weight models, Llama~3, Gemma~2, Phi, and NeMo, on machine translation, multiple-choice question answering, and free-form text generation. The results indicate that while certain models like Gemma~2 perform close to the top commercially available models, many LLMs struggle with these languages. Most surprisingly, however, we find that these models, while showing close to state-of-the-art translation performance, are still prone to lexical hallucinations with errors in at least 1 in 20 words for all open-weight multilingual LLMs.

Title: Cyber Spectrum Intelligence: Security Applications, Challenges and Road Ahead

• Authors: Savio Sciancalepore, Gabriele Oligeri
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.03977
• Pdf URL: https://arxiv.org/pdf/2501.03977
• Copy Paste: [[2501.03977]] Cyber Spectrum Intelligence: Security Applications, Challenges and Road Ahead(https://arxiv.org/abs/2501.03977)
• Keywords: security
• Abstract: Cyber Spectrum Intelligence (SpecInt) is emerging as a concept that extends beyond basic {\em spectrum sensing} and {\em signal intelligence} to encompass a broader set of capabilities and technologies aimed at monitoring the use of the radio spectrum and extracting information. SpecInt merges traditional spectrum sensing techniques with Artificial Intelligence (AI) and parallel processing to enhance the ability to extract and correlate simultaneous events occurring on various frequencies, allowing for a new wave of intelligence applications. This paper provides an overview of the emerging SpecInt research area, characterizing the system architecture and the most relevant applications for cyber-physical security. We identify five subcategories of spectrum intelligence for cyber-physical security, encompassing Device Intelligence, Channel Intelligence, Location Intelligence, Communication Intelligence, and Ambient Intelligence. We also provide preliminary results based on an experimental testbed showing the viability, feasibility, and potential of this emerging application area. Finally, we point out current research challenges and future directions paving the way for further research in this domain.

Title: Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles

• Authors: Yuxi Xia, Pedro Henrique Luz de Araujo, Klim Zaporojets, Benjamin Roth
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03991
• Pdf URL: https://arxiv.org/pdf/2501.03991
• Copy Paste: [[2501.03991]] Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles(https://arxiv.org/abs/2501.03991)
• Keywords: robust, large language model
• Abstract: Calibration, the alignment between model confidence and prediction accuracy, is critical for the reliable deployment of large language models (LLMs). Existing works neglect to measure the generalization of their methods to other prompt styles and different sizes of LLMs. To address this, we define a controlled experimental setting covering 12 LLMs and four prompt styles. We additionally investigate if incorporating the response agreement of multiple LLMs and an appropriate loss function can improve calibration performance. Concretely, we build Calib-n, a novel framework that trains an auxiliary model for confidence estimation that aggregates responses from multiple LLMs to capture inter-model agreement. To optimize calibration, we integrate focal and AUC surrogate losses alongside binary cross-entropy. Experiments across four datasets demonstrate that both response agreement and focal loss improve calibration from baselines. We find that few-shot prompts are the most effective for auxiliary model-based methods, and auxiliary models demonstrate robust calibration performance across accuracy variations, outperforming LLMs' internal probabilities and verbalized confidences. These insights deepen the understanding of influence factors in LLM calibration, supporting their reliable deployment in diverse applications.

Title: NeuralSVG: An Implicit Representation for Text-to-Vector Generation

• Authors: Sagi Polaczek, Yuval Alaluf, Elad Richardson, Yael Vinker, Daniel Cohen-Or
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03992
• Pdf URL: https://arxiv.org/pdf/2501.03992
• Copy Paste: [[2501.03992]] NeuralSVG: An Implicit Representation for Text-to-Vector Generation(https://arxiv.org/abs/2501.03992)
• Keywords: diffusion
• Abstract: Vector graphics are essential in design, providing artists with a versatile medium for creating resolution-independent and highly editable visual content. Recent advancements in vision-language and diffusion models have fueled interest in text-to-vector graphics generation. However, existing approaches often suffer from over-parameterized outputs or treat the layered structure - a core feature of vector graphics - as a secondary goal, diminishing their practical use. Recognizing the importance of layered SVG representations, we propose NeuralSVG, an implicit neural representation for generating vector graphics from text prompts. Inspired by Neural Radiance Fields (NeRFs), NeuralSVG encodes the entire scene into the weights of a small MLP network, optimized using Score Distillation Sampling (SDS). To encourage a layered structure in the generated SVG, we introduce a dropout-based regularization technique that strengthens the standalone meaning of each shape. We additionally demonstrate that utilizing a neural representation provides an added benefit of inference-time control, enabling users to dynamically adapt the generated SVG based on user-provided inputs, all with a single learned representation. Through extensive qualitative and quantitative evaluations, we demonstrate that NeuralSVG outperforms existing methods in generating structured and flexible SVG.

Title: RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance

• Authors: Matin Mortaheb, Mohammad A. Amir Khojastepour, Srimat T. Chakradhar, Sennur Ulukus
• Subjects: cs.LG, cs.CV, cs.IR, cs.IT
• Abstract URL: https://arxiv.org/abs/2501.03995
• Pdf URL: https://arxiv.org/pdf/2501.03995
• Copy Paste: [[2501.03995]] RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance(https://arxiv.org/abs/2501.03995)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) improves large language models (LLMs) by using external knowledge to guide response generation, reducing hallucinations. However, RAG, particularly multi-modal RAG, can introduce new hallucination sources: (i) the retrieval process may select irrelevant pieces (e.g., documents, images) as raw context from the database, and (ii) retrieved images are processed into text-based context via vision-language models (VLMs) or directly used by multi-modal language models (MLLMs) like GPT-4o, which may hallucinate. To address this, we propose a novel framework to evaluate the reliability of multi-modal RAG using two performance measures: (i) the relevancy score (RS), assessing the relevance of retrieved entries to the query, and (ii) the correctness score (CS), evaluating the accuracy of the generated response. We train RS and CS models using a ChatGPT-derived database and human evaluator samples. Results show that both models achieve ~88% accuracy on test data. Additionally, we construct a 5000-sample human-annotated database evaluating the relevancy of retrieved pieces and the correctness of response statements. Our RS model aligns with human preferences 20% more often than CLIP in retrieval, and our CS model matches human preferences ~91% of the time. Finally, we assess various RAG systems' selection and generation performances using RS and CS.

Title: WAPTS: A Weighted Allocation Probability Adjusted Thompson Sampling Algorithm for High-Dimensional and Sparse Experiment Settings

• Authors: Haochen Song, Ilya Musabirov, Ananya Bhattacharjee, Audrey Durand, Meredith Franklin, Anna Rafferty, Joseph Jay Williams
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.03999
• Pdf URL: https://arxiv.org/pdf/2501.03999
• Copy Paste: [[2501.03999]] WAPTS: A Weighted Allocation Probability Adjusted Thompson Sampling Algorithm for High-Dimensional and Sparse Experiment Settings(https://arxiv.org/abs/2501.03999)
• Keywords: segmentation
• Abstract: Aiming for more effective experiment design, such as in video content advertising where different content options compete for user engagement, these scenarios can be modeled as multi-arm bandit problems. In cases where limited interactions are available due to external factors, such as the cost of conducting experiments, recommenders often face constraints due to the small number of user interactions. In addition, there is a trade-off between selecting the best treatment and the ability to personalize and contextualize based on individual factors. A popular solution to this dilemma is the Contextual Bandit framework. It aims to maximize outcomes while incorporating personalization (contextual) factors, customizing treatments such as a user's profile to individual preferences. Despite their advantages, Contextual Bandit algorithms face challenges like measurement bias and the 'curse of dimensionality.' These issues complicate the management of numerous interventions and often lead to data sparsity through participant segmentation. To address these problems, we introduce the Weighted Allocation Probability Adjusted Thompson Sampling (WAPTS) algorithm. WAPTS builds on the contextual Thompson Sampling method by using a dynamic weighting parameter. This improves the allocation process for interventions and enables rapid optimization in data-sparse environments. We demonstrate the performance of our approach on different numbers of arms and effect sizes.

Title: A Survey on Federated Learning in Human Sensing

• Authors: Mohan Li, Martin Gjoreski, Pietro Barbiero, Gašper Slapničar, Mitja Luštrek, Nicholas D. Lane, Marc Langheinrich
• Subjects: cs.LG, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.04000
• Pdf URL: https://arxiv.org/pdf/2501.04000
• Copy Paste: [[2501.04000]] A Survey on Federated Learning in Human Sensing(https://arxiv.org/abs/2501.04000)
• Keywords: security, privacy, federate
• Abstract: Human Sensing, a field that leverages technology to monitor human activities, psycho-physiological states, and interactions with the environment, enhances our understanding of human behavior and drives the development of advanced services that improve overall quality of life. However, its reliance on detailed and often privacy-sensitive data as the basis for its machine learning (ML) models raises significant legal and ethical concerns. The recently proposed ML approach of Federated Learning (FL) promises to alleviate many of these concerns, as it is able to create accurate ML models without sending raw user data to a central server. While FL has demonstrated its usefulness across a variety of areas, such as text prediction and cyber security, its benefits in Human Sensing are under-explored, given the particular challenges in this domain. This survey conducts a comprehensive analysis of the current state-of-the-art studies on FL in Human Sensing, and proposes a taxonomy and an eight-dimensional assessment for FL approaches. Through the eight-dimensional assessment, we then evaluate whether the surveyed studies consider a specific FL-in-Human-Sensing challenge or not. Finally, based on the overall analysis, we discuss open challenges and highlight five research aspects related to FL in Human Sensing that require urgent research attention. Our work provides a comprehensive corpus of FL studies and aims to assist FL practitioners in developing and evaluating solutions that effectively address the real-world complexities of Human Sensing.

Title: Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos

• Authors: Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04001
• Pdf URL: https://arxiv.org/pdf/2501.04001
• Copy Paste: [[2501.04001]] Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos(https://arxiv.org/abs/2501.04001)
• Keywords: large language model, segmentation
• Abstract: This work presents Sa2VA, the first unified model for dense grounded understanding of both images and videos. Unlike existing multi-modal large language models, which are often limited to specific modalities and tasks, Sa2VA supports a wide range of image and video tasks, including referring segmentation and conversation, with minimal one-shot instruction tuning. Sa2VA combines SAM-2, a foundation video segmentation model, with LLaVA, an advanced vision-language model, and unifies text, image, and video into a shared LLM token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2 in producing precise masks, enabling a grounded, multi-modal understanding of both static and dynamic visual content. Additionally, we introduce Ref-SAV, an auto-labeled dataset containing over 72k object expressions in complex video scenes, designed to boost model performance. We also manually validate 2k video objects in the Ref-SAV datasets to benchmark referring video object segmentation in complex environments. Experiments show that Sa2VA achieves state-of-the-art across multiple tasks, particularly in referring video object segmentation, highlighting its potential for complex real-world applications.

Title: Extraction Of Cumulative Blobs From Dynamic Gestures

• Authors: Rishabh Naulakha, Shubham Gaur, Dhairya Lodha, Mehek Tulsyan, Utsav Kotecha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04002
• Pdf URL: https://arxiv.org/pdf/2501.04002
• Copy Paste: [[2501.04002]] Extraction Of Cumulative Blobs From Dynamic Gestures(https://arxiv.org/abs/2501.04002)
• Keywords: extraction
• Abstract: Gesture recognition is a perceptual user interface, which is based on CV technology that allows the computer to interpret human motions as commands, allowing users to communicate with a computer without the use of hands, thus making the mouse and keyboard superfluous. Gesture recognition's main weakness is a light condition because gesture control is based on computer vision, which heavily relies on cameras. These cameras are used to interpret gestures in 2D and 3D, so the extracted information can vary depending on the source of light. The limitation of the system cannot work in a dark environment. A simple night vision camera can be used as our camera for motion capture as they also blast out infrared light which is not visible to humans but can be clearly seen with a camera that has no infrared filter this majorly overcomes the limitation of systems which cannot work in a dark environment. So, the video stream from the camera is fed into a Raspberry Pi which has a Python program running OpenCV module which is used for detecting, isolating and tracking the path of dynamic gesture, then we use an algorithm of machine learning to recognize the pattern drawn and accordingly control the GPIOs of the raspberry pi to perform some activities.

Title: Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives

• Authors: Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, Liang Pan
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.04003
• Pdf URL: https://arxiv.org/pdf/2501.04003
• Copy Paste: [[2501.04003]] Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives(https://arxiv.org/abs/2501.04003)
• Keywords: robust
• Abstract: Recent advancements in Vision-Language Models (VLMs) have sparked interest in their use for autonomous driving, particularly in generating interpretable driving decisions through natural language. However, the assumption that VLMs inherently provide visually grounded, reliable, and interpretable explanations for driving remains largely unexamined. To address this gap, we introduce DriveBench, a benchmark dataset designed to evaluate VLM reliability across 17 settings (clean, corrupted, and text-only inputs), encompassing 19,200 frames, 20,498 question-answer pairs, three question types, four mainstream driving tasks, and a total of 12 popular VLMs. Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding, especially under degraded or missing visual inputs. This behavior, concealed by dataset imbalances and insufficient evaluation metrics, poses significant risks in safety-critical scenarios like autonomous driving. We further observe that VLMs struggle with multi-modal reasoning and display heightened sensitivity to input corruptions, leading to inconsistencies in performance. To address these challenges, we propose refined evaluation metrics that prioritize robust visual grounding and multi-modal understanding. Additionally, we highlight the potential of leveraging VLMs' awareness of corruptions to enhance their reliability, offering a roadmap for developing more trustworthy and interpretable decision-making systems in real-world autonomous driving contexts. The benchmark toolkit is publicly accessible.

Title: LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenes

• Authors: Xiang Xu, Lingdong Kong, Hui Shuai, Liang Pan, Ziwei Liu, Qingshan Liu
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.04004
• Pdf URL: https://arxiv.org/pdf/2501.04004
• Copy Paste: [[2501.04004]] LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenes(https://arxiv.org/abs/2501.04004)
• Keywords: segmentation
• Abstract: LiDAR data pretraining offers a promising approach to leveraging large-scale, readily available datasets for enhanced data utilization. However, existing methods predominantly focus on sparse voxel representation, overlooking the complementary attributes provided by other LiDAR representations. In this work, we propose LiMoE, a framework that integrates the Mixture of Experts (MoE) paradigm into LiDAR data representation learning to synergistically combine multiple representations, such as range images, sparse voxels, and raw points. Our approach consists of three stages: i) Image-to-LiDAR Pretraining, which transfers prior knowledge from images to point clouds across different representations; ii) Contrastive Mixture Learning (CML), which uses MoE to adaptively activate relevant attributes from each representation and distills these mixed features into a unified 3D network; iii) Semantic Mixture Supervision (SMS), which combines semantic logits from multiple representations to boost downstream segmentation performance. Extensive experiments across 11 large-scale LiDAR datasets demonstrate our effectiveness and superiority. The code and model checkpoints have been made publicly accessible.

Title: LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving

• Authors: Lingdong Kong, Xiang Xu, Youquan Liu, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, Ziwei Liu
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.04005
• Pdf URL: https://arxiv.org/pdf/2501.04005
• Copy Paste: [[2501.04005]] LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving(https://arxiv.org/abs/2501.04005)
• Keywords: robust, segmentation
• Abstract: Recent advancements in vision foundation models (VFMs) have revolutionized visual perception in 2D, yet their potential for 3D scene understanding, particularly in autonomous driving applications, remains underexplored. In this paper, we introduce LargeAD, a versatile and scalable framework designed for large-scale 3D pretraining across diverse real-world driving datasets. Our framework leverages VFMs to extract semantically rich superpixels from 2D images, which are aligned with LiDAR point clouds to generate high-quality contrastive samples. This alignment facilitates cross-modal representation learning, enhancing the semantic consistency between 2D and 3D data. We introduce several key innovations: i) VFM-driven superpixel generation for detailed semantic representation, ii) a VFM-assisted contrastive learning strategy to align multimodal features, iii) superpoint temporal consistency to maintain stable representations across time, and iv) multi-source data pretraining to generalize across various LiDAR configurations. Our approach delivers significant performance improvements over state-of-the-art methods in both linear probing and fine-tuning tasks for both LiDAR-based segmentation and object detection. Extensive experiments on eleven large-scale multi-modal datasets highlight our superior performance, demonstrating the adaptability, efficiency, and robustness in real-world autonomous driving scenarios.