2025-01-07

Title: INFELM: In-depth Fairness Evaluation of Large Text-To-Image Models

• Authors: Di Jin, Xing Liu, Yu Liu, Jia Qing Yap, Andrea Wong, Adriana Crespo, Qi Lin, Zhiyuan Yin, Qiang Yan, Ryan Ye
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.01973
• Pdf URL: https://arxiv.org/pdf/2501.01973
• Copy Paste: [[2501.01973]] INFELM: In-depth Fairness Evaluation of Large Text-To-Image Models(https://arxiv.org/abs/2501.01973)
• Keywords: robust, fair, large language model
• Abstract: The rapid development of large language models (LLMs) and large vision models (LVMs) have propelled the evolution of multi-modal AI systems, which have demonstrated the remarkable potential for industrial applications by emulating human-like cognition. However, they also pose significant ethical challenges, including amplifying harmful content and reinforcing societal biases. For instance, biases in some industrial image generation models highlighted the urgent need for robust fairness assessments. Most existing evaluation frameworks focus on the comprehensiveness of various aspects of the models, but they exhibit critical limitations, including insufficient attention to content generation alignment and social bias-sensitive domains. More importantly, their reliance on pixel-detection techniques is prone to inaccuracies. To address these issues, this paper presents INFELM, an in-depth fairness evaluation on widely-used text-to-image models. Our key contributions are: (1) an advanced skintone classifier incorporating facial topology and refined skin pixel representation to enhance classification precision by at least 16.04%, (2) a bias-sensitive content alignment measurement for understanding societal impacts, (3) a generalizable representation bias evaluation for diverse demographic groups, and (4) extensive experiments analyzing large-scale text-to-image model outputs across six social-bias-sensitive domains. We find that existing models in the study generally do not meet the empirical fairness criteria, and representation bias is generally more pronounced than alignment errors. INFELM establishes a robust benchmark for fairness assessment, supporting the development of multi-modal AI systems that align with ethical and human-centric principles.

Title: Optical Character Recognition using Convolutional Neural Networks for Ashokan Brahmi Inscriptions

• Authors: Yash Agrawal, Srinidhi Balasubramanian, Rahul Meena, Rohail Alam, Himanshu Malviya, Rohini P
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.01981
• Pdf URL: https://arxiv.org/pdf/2501.01981
• Copy Paste: [[2501.01981]] Optical Character Recognition using Convolutional Neural Networks for Ashokan Brahmi Inscriptions(https://arxiv.org/abs/2501.01981)
• Keywords: segmentation
• Abstract: This research paper delves into the development of an Optical Character Recognition (OCR) system for the recognition of Ashokan Brahmi characters using Convolutional Neural Networks. It utilizes a comprehensive dataset of character images to train the models, along with data augmentation techniques to optimize the training process. Furthermore, the paper incorporates image preprocessing to remove noise, as well as image segmentation to facilitate line and character segmentation. The study mainly focuses on three pre-trained CNNs, namely LeNet, VGG-16, and MobileNet and compares their accuracy. Transfer learning was employed to adapt the pre-trained models to the Ashokan Brahmi character dataset. The findings reveal that MobileNet outperforms the other two models in terms of accuracy, achieving a validation accuracy of 95.94% and validation loss of 0.129. The paper provides an in-depth analysis of the implementation process using MobileNet and discusses the implications of the findings. The use of OCR for character recognition is of significant importance in the field of epigraphy, specifically for the preservation and digitization of ancient scripts. The results of this research paper demonstrate the effectiveness of using pre-trained CNNs for the recognition of Ashokan Brahmi characters.

Title: ECG-guided individual identification via PPG

• Authors: Riling Wei, Hanjie Chen, Kelu Yao, Chuanguang Yang, Jun Wang, Chao Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01983
• Pdf URL: https://arxiv.org/pdf/2501.01983
• Copy Paste: [[2501.01983]] ECG-guided individual identification via PPG(https://arxiv.org/abs/2501.01983)
• Keywords: security
• Abstract: Photoplethsmography (PPG)-based individual identification aiming at recognizing humans via intrinsic cardiovascular activities has raised extensive attention due to its high security and resistance to mimicry. However, this kind of technology witnesses unpromising results due to the limitation of low information density. To this end, electrocardiogram (ECG) signals have been introduced as a novel modality to enhance the density of input information. Specifically, a novel cross-modal knowledge distillation framework is implemented to propagate discriminate knowledge from ECG modality to PPG modality without incurring additional computational demands at the inference phase. Furthermore, to ensure efficient knowledge propagation, Contrastive Language-Image Pre-training (CLIP)-based knowledge alignment and cross-knowledge assessment modules are proposed respectively. Comprehensive experiments are conducted and results show our framework outperforms the baseline model with the improvement of 2.8% and 3.0% in terms of overall accuracy on seen- and unseen individual recognitions.

Title: Fall Detection in Passenger Elevators using Intelligent Surveillance Camera Systems: An Application with YoloV8 Nano Model

• Authors: Pinar Yozgatli, Yavuz Acar, Mehmet Tulumen, Selman Minga, Salih Selamet, Beytullah Nalbant, Mustafa Talha Toru, Berna Koca, Tevfik Keles, Mehmet Selcok
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01985
• Pdf URL: https://arxiv.org/pdf/2501.01985
• Copy Paste: [[2501.01985]] Fall Detection in Passenger Elevators using Intelligent Surveillance Camera Systems: An Application with YoloV8 Nano Model(https://arxiv.org/abs/2501.01985)
• Keywords: robust
• Abstract: Computer vision technology, which involves analyzing images and videos captured by cameras through deep learning algorithms, has significantly advanced the field of human fall detection. This study focuses on the application of the YoloV8 Nano model in identifying fall incidents within passenger elevators, a context that presents unique challenges due to the enclosed environment and varying lighting conditions. By training the model on a robust dataset comprising over 10,000 images across diverse elevator types, we aim to enhance the detection precision and recall rates. The model's performance, with an 85% precision and 82% recall in fall detection, underscores its potential for integration into existing elevator safety systems to enable rapid intervention.

Title: Towards Sustainable Large Language Model Serving

• Authors: Sophia Nguyen, Beihao Zhou, Yi Ding, Sihang Liu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.01990
• Pdf URL: https://arxiv.org/pdf/2501.01990
• Copy Paste: [[2501.01990]] Towards Sustainable Large Language Model Serving(https://arxiv.org/abs/2501.01990)
• Keywords: large language model
• Abstract: In this work, we study LLMs from a carbon emission perspective, addressing both operational and embodied emissions, and paving the way for sustainable LLM serving. We characterize the performance and energy of LLaMA with 1B, 3B, and 7B parameters using two Nvidia GPU types, a latest-generation RTX6000 Ada and an older-generation T4. We analytically model operational carbon emissions based on energy consumption and carbon intensities from three grid regions -- each representing a different energy source mix, and embodied carbon emissions based on chip area and memory size. Our characterization and modeling provide us with an in-depth understanding of the performance, energy, and carbon emissions of LLM serving. Our findings highlight the potential for optimizing sustainable LLM serving systems by considering both operational and embodied carbon emissions simultaneously.

Title: A Hybrid Deep Learning and Model-Checking Framework for Accurate Brain Tumor Detection and Validation

• Authors: Lahcen El Fatimi, Elhoucine Elfatimi, Hanifa Bouchaneb
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01991
• Pdf URL: https://arxiv.org/pdf/2501.01991
• Copy Paste: [[2501.01991]] A Hybrid Deep Learning and Model-Checking Framework for Accurate Brain Tumor Detection and Validation(https://arxiv.org/abs/2501.01991)
• Keywords: robust, extraction, segmentation
• Abstract: Model checking, a formal verification technique, ensures systems meet predefined requirements, playing a crucial role in minimizing errors and enhancing quality during development. This paper introduces a novel hybrid framework integrating model checking with deep learning for brain tumor detection and validation in medical imaging. By combining model-checking principles with CNN-based feature extraction and K-FCM clustering for segmentation, the proposed approach enhances the reliability of tumor detection and segmentation. Experimental results highlight the framework's effectiveness, achieving 98\% accuracy, 96.15\% precision, and 100\% recall, demonstrating its potential as a robust tool for advanced medical image analysis.

Title: SmartSpatial: Enhancing the 3D Spatial Arrangement Capabilities of Stable Diffusion Models and Introducing a Novel 3D Spatial Evaluation Framework

• Authors: Mao Xun Huang, Hen-Hsen Huang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01998
• Pdf URL: https://arxiv.org/pdf/2501.01998
• Copy Paste: [[2501.01998]] SmartSpatial: Enhancing the 3D Spatial Arrangement Capabilities of Stable Diffusion Models and Introducing a Novel 3D Spatial Evaluation Framework(https://arxiv.org/abs/2501.01998)
• Keywords: diffusion
• Abstract: Stable Diffusion models have made remarkable strides in generating photorealistic images from text prompts but often falter when tasked with accurately representing complex spatial arrangements, particularly involving intricate 3D relationships. To address this limitation, we introduce SmartSpatial, an innovative approach that enhances the spatial arrangement capabilities of Stable Diffusion models through 3D-aware conditioning and attention-guided mechanisms. SmartSpatial incorporates depth information and employs cross-attention control to ensure precise object placement, delivering notable improvements in spatial accuracy metrics. In conjunction with SmartSpatial, we present SmartSpatialEval, a comprehensive evaluation framework designed to assess spatial relationships. This framework utilizes vision-language models and graph-based dependency parsing for performance analysis. Experimental results on the COCO and SpatialPrompts datasets show that SmartSpatial significantly outperforms existing methods, setting new benchmarks for spatial arrangement accuracy in image generation.

Title: On the Utility of Equivariance and Symmetry Breaking in Deep Learning Architectures on Point Clouds

• Authors: Sharvaree Vadgama, Mohammad Mohaiminul Islam, Domas Buracus, Christian Shewmake, Erik Bekkers
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01999
• Pdf URL: https://arxiv.org/pdf/2501.01999
• Copy Paste: [[2501.01999]] On the Utility of Equivariance and Symmetry Breaking in Deep Learning Architectures on Point Clouds(https://arxiv.org/abs/2501.01999)
• Keywords: segmentation
• Abstract: This paper explores the key factors that influence the performance of models working with point clouds, across different tasks of varying geometric complexity. In this work, we explore the trade-offs between flexibility and weight-sharing introduced by equivariant layers, assessing when equivariance boosts or detracts from performance. It is often argued that providing more information as input improves a model's performance. However, if this additional information breaks certain properties, such as $\SE(3)$ equivariance, does it remain beneficial? We identify the key aspects of equivariant and non-equivariant architectures that drive success in different tasks by benchmarking them on segmentation, regression, and generation tasks across multiple datasets with increasing complexity. We observe a positive impact of equivariance, which becomes more pronounced with increasing task complexity, even when strict equivariance is not required.

Title: HMM-LSTM Fusion Model for Economic Forecasting

• Authors: Guhan Sivakumar
• Subjects: cs.LG, econ.EM, stat.ME
• Abstract URL: https://arxiv.org/abs/2501.02002
• Pdf URL: https://arxiv.org/pdf/2501.02002
• Copy Paste: [[2501.02002]] HMM-LSTM Fusion Model for Economic Forecasting(https://arxiv.org/abs/2501.02002)
• Keywords: interpretability
• Abstract: This paper explores the application of Hidden Markov Models (HMM) and Long Short-Term Memory (LSTM) neural networks for economic forecasting, focusing on predicting CPI inflation rates. The study explores a new approach that integrates HMM-derived hidden states and means as additional features for LSTM modeling, aiming to enhance the interpretability and predictive performance of the models. The research begins with data collection and preprocessing, followed by the implementation of the HMM to identify hidden states representing distinct economic conditions. Subsequently, LSTM models are trained using the original and augmented data sets, allowing for comparative analysis and evaluation. The results demonstrate that incorporating HMM-derived data improves the predictive accuracy of LSTM models, particularly in capturing complex temporal patterns and mitigating the impact of volatile economic conditions. Additionally, the paper discusses the implementation of Integrated Gradients for model interpretability and provides insights into the economic dynamics reflected in the forecasting outcomes.

Title: Multi-Task Semantic Communication With Graph Attention-Based Feature Correlation Extraction

• Authors: Xi Yu, Tiejun Lv, Weicai Li, Wei Ni, Dusit Niyato, Ekram Hossain
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02006
• Pdf URL: https://arxiv.org/pdf/2501.02006
• Copy Paste: [[2501.02006]] Multi-Task Semantic Communication With Graph Attention-Based Feature Correlation Extraction(https://arxiv.org/abs/2501.02006)
• Keywords: extraction
• Abstract: Multi-task semantic communication can serve multiple learning tasks using a shared encoder model. Existing models have overlooked the intricate relationships between features extracted during an encoding process of tasks. This paper presents a new graph attention inter-block (GAI) module to the encoder/transmitter of a multi-task semantic communication system, which enriches the features for multiple tasks by embedding the intermediate outputs of encoding in the features, compared to the existing techniques. The key idea is that we interpret the outputs of the intermediate feature extraction blocks of the encoder as the nodes of a graph to capture the correlations of the intermediate features. Another important aspect is that we refine the node representation using a graph attention mechanism to extract the correlations and a multi-layer perceptron network to associate the node representations with different tasks. Consequently, the intermediate features are weighted and embedded into the features transmitted for executing multiple tasks at the receiver. Experiments demonstrate that the proposed model surpasses the most competitive and publicly available models by 11.4% on the CityScapes 2Task dataset and outperforms the established state-of-the-art by 3.97% on the NYU V2 3Task dataset, respectively, when the bandwidth ratio of the communication channel (i.e., compression level for transmission over the channel) is as constrained as 1 12 .

Title: TART: Token-based Architecture Transformer for Neural Network Performance Prediction

• Authors: Yannis Y. He
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02007
• Pdf URL: https://arxiv.org/pdf/2501.02007
• Copy Paste: [[2501.02007]] TART: Token-based Architecture Transformer for Neural Network Performance Prediction(https://arxiv.org/abs/2501.02007)
• Keywords: transformer
• Abstract: In the realm of neural architecture design, achieving high performance is largely reliant on the manual expertise of researchers. Despite the emergence of Neural Architecture Search (NAS) as a promising technique for automating this process, current NAS methods still require human input to expand the search space and cannot generate new architectures. This paper explores the potential of Transformers in comprehending neural architectures and their performance, with the objective of establishing the foundation for utilizing Transformers to generate novel networks. We propose the Token-based Architecture Transformer (TART), which predicts neural network performance without the need to train candidate networks. TART attains state-of-the-art performance on the DeepNets-1M dataset for performance prediction tasks without edge information, indicating the potential of Transformers to aid in discovering novel and high-performing neural architectures.

Title: Cross-model Transferability among Large Language Models on the Platonic Representations of Concepts

• Authors: Youcheng Huang, Chen Huang, Duanyu Feng, Wenqiang Lei, Jiancheng Lv
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02009
• Pdf URL: https://arxiv.org/pdf/2501.02009
• Copy Paste: [[2501.02009]] Cross-model Transferability among Large Language Models on the Platonic Representations of Concepts(https://arxiv.org/abs/2501.02009)
• Keywords: large language model
• Abstract: Understanding the inner workings of Large Language Models (LLMs) is a critical research frontier. Prior research has shown that a single LLM's concept representations can be captured as steering vectors (SVs), enabling the control of LLM behavior (e.g., towards generating harmful content). Our work takes a novel approach by exploring the intricate relationships between concept representations across different LLMs, drawing an intriguing parallel to Plato's Allegory of the Cave. In particular, we introduce a linear transformation method to bridge these representations and present three key findings: 1) Concept representations across different LLMs can be effectively aligned using simple linear transformations, enabling efficient cross-model transfer and behavioral control via SVs. 2) This linear transformation generalizes across concepts, facilitating alignment and control of SVs representing different concepts across LLMs. 3) A weak-to-strong transferability exists between LLM concept representations, whereby SVs extracted from smaller LLMs can effectively control the behavior of larger LLMs.

Title: Explainable Neural Networks with Guarantees: A Sparse Estimation Approach

• Authors: Antoine Ledent, Peng Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02010
• Pdf URL: https://arxiv.org/pdf/2501.02010
• Copy Paste: [[2501.02010]] Explainable Neural Networks with Guarantees: A Sparse Estimation Approach(https://arxiv.org/abs/2501.02010)
• Keywords: interpretability, explainability
• Abstract: Balancing predictive power and interpretability has long been a challenging research area, particularly in powerful yet complex models like neural networks, where nonlinearity obstructs direct interpretation. This paper introduces a novel approach to constructing an explainable neural network that harmonizes predictiveness and explainability. Our model, termed SparXnet, is designed as a linear combination of a sparse set of jointly learned features, each derived from a different trainable function applied to a single 1-dimensional input feature. Leveraging the ability to learn arbitrarily complex relationships, our neural network architecture enables automatic selection of a sparse set of important features, with the final prediction being a linear combination of rescaled versions of these features. We demonstrate the ability to select significant features while maintaining comparable predictive performance and direct interpretability through extensive experiments on synthetic and real-world datasets. We also provide theoretical analysis on the generalization bounds of our framework, which is favorably linear in the number of selected features and only logarithmic in the number of input features. We further lift any dependence of sample complexity on the number of parameters or the architectural details under very mild conditions. Our research paves the way for further research on sparse and explainable neural networks with guarantee.

Title: Anti-counterfeiting tags with camouflaged QR codes on nanocavities, using polymer-dispersed-liquid-crystals

• Authors: Giuseppe Nicoletta, Mauro Daniel Luigi Bruno, Peng Yu, Zhiming Wang, Maria Penelope De Santo, Roberto Caputo, Antonio Ferraro
• Subjects: cs.CR, cond-mat.soft, cs.ET
• Abstract URL: https://arxiv.org/abs/2501.02011
• Pdf URL: https://arxiv.org/pdf/2501.02011
• Copy Paste: [[2501.02011]] Anti-counterfeiting tags with camouflaged QR codes on nanocavities, using polymer-dispersed-liquid-crystals(https://arxiv.org/abs/2501.02011)
• Keywords: security
• Abstract: Counterfeiting poses an evergrowing challenge, driving the need for innovative and sophisticated anti-counterfeiting strategies and technologies. Many solutions focus on tags characterized by optical features that are partially or completely camouflaged to the human eye, thus discouraging scammers. In this paper, a QR code is laser printed on a thin plastic foil previously coated by a specific nanocavity consisting of a metal/insulator/metal/insulator (MIMI) multilayer. This metamaterial possesses unique features in terms of light transmission that are due to the specific design. A thin layer of polymer dispersed liquid crystals, fabricated incorporating specific nematic liquid crystals in a polymer matrix, is able to camouflage the QR code that becomes, then, readable only under specific thermal conditions. Three anti-counterfeiting tags were fabricated, each using a distinct LC with its own nematic-isotropic transition temperature. The peculiar combination of the unique optical properties of nematic liquid crystals and optical nanocavities results in the creation of a novel type of tags showing two different encoding levels. Stress tests including water immersion, bending test, and prolonged heating have been performed ensuring the long-term stability of the tags. The realized two security-level anti-counterfeiting tags are cost-effective, straightforward to manufacture and, thanks to their flexibility, can be easily integrated into packaging and products.

Title: Information Subtraction: Learning Representations for Conditional Entropy

• Authors: Keng Hou Leong, Yuxuan Xiu, Wai Kin (Victor)Chan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02012
• Pdf URL: https://arxiv.org/pdf/2501.02012
• Copy Paste: [[2501.02012]] Information Subtraction: Learning Representations for Conditional Entropy(https://arxiv.org/abs/2501.02012)
• Keywords: fair, generative
• Abstract: The representations of conditional entropy and conditional mutual information are significant in explaining the unique effects among variables. While previous studies based on conditional contrastive sampling have effectively removed information regarding discrete sensitive variables, they have not yet extended their scope to continuous cases. This paper introduces Information Subtraction, a framework designed to generate representations that preserve desired information while eliminating the undesired. We implement a generative-based architecture that outputs these representations by simultaneously maximizing an information term and minimizing another. With its flexibility in disentangling information, we can iteratively apply Information Subtraction to represent arbitrary information components between continuous variables, thereby explaining the various relationships that exist between them. Our results highlight the representations' ability to provide semantic features of conditional entropy. By subtracting sensitive and domain-specific information, our framework demonstrates effective performance in fair learning and domain generalization. The code for this paper is available at this https URL

Title: Machine Learning-Based Differential Diagnosis of Parkinson's Disease Using Kinematic Feature Extraction and Selection

• Authors: Masahiro Matsumoto, Abu Saleh Musa Miah, Nobuyoshi Asai, Jungpil Shin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02014
• Pdf URL: https://arxiv.org/pdf/2501.02014
• Copy Paste: [[2501.02014]] Machine Learning-Based Differential Diagnosis of Parkinson's Disease Using Kinematic Feature Extraction and Selection(https://arxiv.org/abs/2501.02014)
• Keywords: extraction, generative
• Abstract: Parkinson's disease (PD), the second most common neurodegenerative disorder, is characterized by dopaminergic neuron loss and the accumulation of abnormal synuclein. PD presents both motor and non-motor symptoms that progressively impair daily functioning. The severity of these symptoms is typically assessed using the MDS-UPDRS rating scale, which is subjective and dependent on the physician's experience. Additionally, PD shares symptoms with other neurodegenerative diseases, such as progressive supranuclear palsy (PSP) and multiple system atrophy (MSA), complicating accurate diagnosis. To address these diagnostic challenges, we propose a machine learning-based system for differential diagnosis of PD, PSP, MSA, and healthy controls (HC). This system utilizes a kinematic feature-based hierarchical feature extraction and selection approach. Initially, 18 kinematic features are extracted, including two newly proposed features: Thumb-to-index vector velocity and acceleration, which provide insights into motor control patterns. In addition, 41 statistical features were extracted here from each kinematic feature, including some new approaches such as Average Absolute Change, Rhythm, Amplitude, Frequency, Standard Deviation of Frequency, and Slope. Feature selection is performed using One-way ANOVA to rank features, followed by Sequential Forward Floating Selection (SFFS) to identify the most relevant ones, aiming to reduce the computational complexity. The final feature set is used for classification, achieving a classification accuracy of 66.67% for each dataset and 88.89% for each patient, with particularly high performance for the MSA and HC groups using the SVM algorithm. This system shows potential as a rapid and accurate diagnostic tool in clinical practice, though further data collection and refinement are needed to enhance its reliability.

Title: KANS: Knowledge Discovery Graph Attention Network for Soft Sensing in Multivariate Industrial Processes

• Authors: Hwa Hui Tew, Gaoxuan Li, Fan Ding, Xuewen Luo, Junn Yong Loo, Chee-Ming Ting, Ze Yang Ding, Chee Pin Tan
• Subjects: cs.LG, cs.AI, eess.SP, eess.SY
• Abstract URL: https://arxiv.org/abs/2501.02015
• Pdf URL: https://arxiv.org/pdf/2501.02015
• Copy Paste: [[2501.02015]] KANS: Knowledge Discovery Graph Attention Network for Soft Sensing in Multivariate Industrial Processes(https://arxiv.org/abs/2501.02015)
• Keywords: interpretability
• Abstract: Soft sensing of hard-to-measure variables is often crucial in industrial processes. Current practices rely heavily on conventional modeling techniques that show success in improving accuracy. However, they overlook the non-linear nature, dynamics characteristics, and non-Euclidean dependencies between complex process variables. To tackle these challenges, we present a framework known as a Knowledge discovery graph Attention Network for effective Soft sensing (KANS). Unlike the existing deep learning soft sensor models, KANS can discover the intrinsic correlations and irregular relationships between the multivariate industrial processes without a predefined topology. First, an unsupervised graph structure learning method is introduced, incorporating the cosine similarity between different sensor embedding to capture the correlations between sensors. Next, we present a graph attention-based representation learning that can compute the multivariate data parallelly to enhance the model in learning complex sensor nodes and edges. To fully explore KANS, knowledge discovery analysis has also been conducted to demonstrate the interpretability of the model. Experimental results demonstrate that KANS significantly outperforms all the baselines and state-of-the-art methods in soft sensing performance. Furthermore, the analysis shows that KANS can find sensors closely related to different process variables without domain knowledge, significantly improving soft sensing accuracy.

Title: Safeguarding Large Language Models in Real-time with Tunable Safety-Performance Trade-offs

• Authors: Joao Fonseca, Andrew Bell, Julia Stoyanovich
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02018
• Pdf URL: https://arxiv.org/pdf/2501.02018
• Copy Paste: [[2501.02018]] Safeguarding Large Language Models in Real-time with Tunable Safety-Performance Trade-offs(https://arxiv.org/abs/2501.02018)
• Keywords: attack, transformer, large language model
• Abstract: Large Language Models (LLMs) have been shown to be susceptible to jailbreak attacks, or adversarial attacks used to illicit high risk behavior from a model. Jailbreaks have been exploited by cybercriminals and blackhat actors to cause significant harm, highlighting the critical need to safeguard widely-deployed models. Safeguarding approaches, which include fine-tuning models or having LLMs "self-reflect", may lengthen the inference time of a model, incur a computational penalty, reduce the semantic fluency of an output, and restrict ``normal'' model behavior. Importantly, these Safety-Performance Trade-offs (SPTs) remain an understudied area. In this work, we introduce a novel safeguard, called SafeNudge, that combines Controlled Text Generation with "nudging", or using text interventions to change the behavior of a model. SafeNudge triggers during text-generation while a jailbreak attack is being executed, and can reduce successful jailbreak attempts by 30% by guiding the LLM towards a safe responses. It adds minimal latency to inference and has a negligible impact on the semantic fluency of outputs. Further, we allow for tunable SPTs. SafeNudge is open-source and available through this https URL, and is compatible with models loaded with the Hugging Face "transformers" library.

Title: Enhancing Uncertainty Modeling with Semantic Graph for Hallucination Detection

• Authors: Kedi Chen, Qin Chen, Jie Zhou, Xinqi Tao, Bowen Ding, Jingwen Xie, Mingchen Xie, Peilong Li, Feng Zheng, Liang He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02020
• Pdf URL: https://arxiv.org/pdf/2501.02020
• Copy Paste: [[2501.02020]] Enhancing Uncertainty Modeling with Semantic Graph for Hallucination Detection(https://arxiv.org/abs/2501.02020)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are prone to hallucination with non-factual or unfaithful statements, which undermines the applications in real-world scenarios. Recent researches focus on uncertainty-based hallucination detection, which utilizes the output probability of LLMs for uncertainty calculation and does not rely on external knowledge or frequent sampling from LLMs. Whereas, most approaches merely consider the uncertainty of each independent token, while the intricate semantic relations among tokens and sentences are not well studied, which limits the detection of hallucination that spans over multiple tokens and sentences in the passage. In this paper, we propose a method to enhance uncertainty modeling with semantic graph for hallucination detection. Specifically, we first construct a semantic graph that well captures the relations among entity tokens and sentences. Then, we incorporate the relations between two entities for uncertainty propagation to enhance sentence-level hallucination detection. Given that hallucination occurs due to the conflict between sentences, we further present a graph-based uncertainty calibration method that integrates the contradiction probability of the sentence with its neighbors in the semantic graph for uncertainty calculation. Extensive experiments on two datasets show the great advantages of our proposed approach. In particular, we obtain substantial improvements with 19.78% in passage-level hallucination detection.

Title: Weakly Supervised Learning on Large Graphs

• Authors: Aditya Prakash
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02021
• Pdf URL: https://arxiv.org/pdf/2501.02021
• Copy Paste: [[2501.02021]] Weakly Supervised Learning on Large Graphs(https://arxiv.org/abs/2501.02021)
• Keywords: extraction
• Abstract: Graph classification plays a pivotal role in various domains, including pathology, where images can be represented as this http URL this domain, images can be represented as graphs, where nodes might represent individual nuclei, and edges capture the spatial or functional relationships between them. Often, the overall label of the graph, such as a cancer type or disease state, is determined by patterns within smaller, localized regions of the image. This work introduces a weakly-supervised graph classification framework leveraging two subgraph extraction techniques: (1) Sliding-window approach (2) BFS-based approach. Subgraphs are processed using a Graph Attention Network (GAT), which employs attention mechanisms to identify the most informative subgraphs for classification. Weak supervision is achieved by propagating graph-level labels to subgraphs, eliminating the need for detailed subgraph annotations.

Title: Model Checking in Medical Imaging for Tumor Detection and Segmentation

• Authors: Elhoucine Elfatimi, Lahcen El fatimi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02024
• Pdf URL: https://arxiv.org/pdf/2501.02024
• Copy Paste: [[2501.02024]] Model Checking in Medical Imaging for Tumor Detection and Segmentation(https://arxiv.org/abs/2501.02024)
• Keywords: robust, segmentation
• Abstract: Recent advancements in model checking have demonstrated significant potential across diverse applications, particularly in signal and image analysis. Medical imaging stands out as a critical domain where model checking can be effectively applied to design and evaluate robust frameworks. These frameworks facilitate automatic and semi-automatic delineation of regions of interest within images, aiding in accurate segmentation. This paper provides a comprehensive analysis of recent works leveraging spatial logic to develop operators and tools for identifying regions of interest, including tumorous and non-tumorous areas. Additionally, we examine the challenges inherent to spatial model-checking techniques, such as variability in ground truth data and the need for streamlined procedures suitable for routine clinical practice.

Title: RealDiffFusionNet: Neural Controlled Differential Equation Informed Multi-Head Attention Fusion Networks for Disease Progression Modeling Using Real-World Data

• Authors: Aashish Cheruvu, Nathaniel Rigoni
• Subjects: cs.LG, cs.CV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2501.02025
• Pdf URL: https://arxiv.org/pdf/2501.02025
• Copy Paste: [[2501.02025]] RealDiffFusionNet: Neural Controlled Differential Equation Informed Multi-Head Attention Fusion Networks for Disease Progression Modeling Using Real-World Data(https://arxiv.org/abs/2501.02025)
• Keywords: robust
• Abstract: This paper presents a novel deep learning-based approach named RealDiffFusionNet incorporating Neural Controlled Differential Equations (Neural CDE) - time series models that are robust in handling irregularly sampled data - and multi-head attention to align relevant multimodal context (image data, time invariant data, etc.) at each time point. Long short-term memory (LSTM) models were also used as a baseline. Two different datasets were used: a data from the Open-Source Imaging Consortium (OSIC) containing structured time series data of demographics and lung function with a baseline CT scan of the lungs and the second from the Alzheimer's Disease Neuroimaging Initiative (ADNI) containing a series of MRI scans along with demographics, physical examinations, and cognitive assessment data. An ablation study was performed to understand the role of CDEs, multimodal data, attention fusion, and interpolation strategies on model performance. When the baseline models were evaluated, the use of multimodal data resulted in an improvement in Neural CDE performance, with a lower test RMSE. Additionally, the performance of multimodal Neural CDE was also superior to multimodal LSTM. In the attention-based architectures, fusion through concatenation and rectilinear interpolation were found to improve model performance. The performance of the proposed RealDiffFusionNet was found to be superior (0.2570) to all models. For the ADNI dataset, between the Neural-CDE and LSTM models trained only on the structured data, the test RMSE were comparable (0.471 for LSTM vs. 0.4581 Neural-CDE). Furthermore, the addition of image features from patients' MRI series resulted in an improvement in performance, with a lower test RMSE (0.4372 with multimodal vs 0.4581 with structured data). RealDiffFusionNet has shown promise in utilizing CDEs and multimodal data to accurately predict disease progression.

Title: Recursive Decomposition of Logical Thoughts: Framework for Superior Reasoning and Knowledge Propagation in Large Language Models

• Authors: Kaleem Ullah Qasim, Jiashu Zhang, Tariq Alsahfi, Ateeq Ur Rehman Butt
• Subjects: cs.CL, cs.AI, cs.LG, cs.LO
• Abstract URL: https://arxiv.org/abs/2501.02026
• Pdf URL: https://arxiv.org/pdf/2501.02026
• Copy Paste: [[2501.02026]] Recursive Decomposition of Logical Thoughts: Framework for Superior Reasoning and Knowledge Propagation in Large Language Models(https://arxiv.org/abs/2501.02026)
• Keywords: large language model
• Abstract: Enhancing the reasoning capabilities of Large Language Models remains a critical challenge in artificial intelligence. We introduce RDoLT, Recursive Decomposition of Logical Thought prompting, a novel framework that significantly boosts LLM reasoning performance. RDoLT is built on three key innovations: (1) recursively breaking down complex reasoning tasks into sub-tasks of progressive complexity; (2) employing an advanced selection and scoring mechanism to identify the most promising reasoning thoughts; and (3) integrating a knowledge propagation module that mimics human learning by keeping track of strong and weak thoughts for information propagation. Our approach was evaluated across multiple benchmarks, including GSM8K, SVAMP, MultiArith, LastLetterConcatenation, and Gaokao2023 Math. The results demonstrate that RDoLT consistently outperforms existing state-of-the-art techniques, achieving a 90.98 percent accuracy on GSM8K with ChatGPT-4, surpassing state-of-the-art techniques by 6.28 percent. Similar improvements were observed on other benchmarks, with accuracy gains ranging from 5.5 percent to 6.75 percent. These findings highlight RDoLT's potential to advance prompt engineering, offering a more effective and generalizable approach to complex reasoning tasks.

Title: Spot Risks Before Speaking! Unraveling Safety Attention Heads in Large Vision-Language Models

• Authors: Ziwei Zheng, Junyao Zhao, Le Yang, Lijun He, Fan Li
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02029
• Pdf URL: https://arxiv.org/pdf/2501.02029
• Copy Paste: [[2501.02029]] Spot Risks Before Speaking! Unraveling Safety Attention Heads in Large Vision-Language Models(https://arxiv.org/abs/2501.02029)
• Keywords: protect, attack
• Abstract: With the integration of an additional modality, large vision-language models (LVLMs) exhibit greater vulnerability to safety risks (e.g., jailbreaking) compared to their language-only predecessors. Although recent studies have devoted considerable effort to the post-hoc alignment of LVLMs, the inner safety mechanisms remain largely unexplored. In this paper, we discover that internal activations of LVLMs during the first token generation can effectively identify malicious prompts across different attacks. This inherent safety perception is governed by sparse attention heads, which we term ``safety heads." Further analysis reveals that these heads act as specialized shields against malicious prompts; ablating them leads to higher attack success rates, while the model's utility remains unaffected. By locating these safety heads and concatenating their activations, we construct a straightforward but powerful malicious prompt detector that integrates seamlessly into the generation process with minimal extra inference overhead. Despite its simple structure of a logistic regression model, the detector surprisingly exhibits strong zero-shot generalization capabilities. Experiments across various prompt-based attacks confirm the effectiveness of leveraging safety heads to protect LVLMs. Code is available at \url{this https URL}.

Title: CarbonChat: Large Language Model-Based Corporate Carbon Emission Analysis and Climate Knowledge Q&A System

• Authors: Zhixuan Cao, Ming Han, Jingtao Wang, Meng Jia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02031
• Pdf URL: https://arxiv.org/pdf/2501.02031
• Copy Paste: [[2501.02031]] CarbonChat: Large Language Model-Based Corporate Carbon Emission Analysis and Climate Knowledge Q&A System(https://arxiv.org/abs/2501.02031)
• Keywords: extraction, large language model, segmentation
• Abstract: As the impact of global climate change intensifies, corporate carbon emissions have become a focal point of global attention. In response to issues such as the lag in climate change knowledge updates within large language models, the lack of specialization and accuracy in traditional augmented generation architectures for complex problems, and the high cost and time consumption of sustainability report analysis, this paper proposes CarbonChat: Large Language Model-based corporate carbon emission analysis and climate knowledge Q&A system, aimed at achieving precise carbon emission analysis and policy this http URL, a diversified index module construction method is proposed to handle the segmentation of rule-based and long-text documents, as well as the extraction of structured data, thereby optimizing the parsing of key this http URL, an enhanced self-prompt retrieval-augmented generation architecture is designed, integrating intent recognition, structured reasoning chains, hybrid retrieval, and Text2SQL, improving the efficiency of semantic understanding and query this http URL, based on the greenhouse gas accounting framework, 14 dimensions are established for carbon emission analysis, enabling report summarization, relevance evaluation, and customized this http URL, through a multi-layer chunking mechanism, timestamps, and hallucination detection features, the accuracy and verifiability of the analysis results are ensured, reducing hallucination rates and enhancing the precision of the responses.

Title: Dynamic Feature Fusion: Combining Global Graph Structures and Local Semantics for Blockchain Fraud Detection

• Authors: Zhang Sheng, Liangliang Song, Yanbin Wang
• Subjects: cs.CR, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.02032
• Pdf URL: https://arxiv.org/pdf/2501.02032
• Copy Paste: [[2501.02032]] Dynamic Feature Fusion: Combining Global Graph Structures and Local Semantics for Blockchain Fraud Detection(https://arxiv.org/abs/2501.02032)
• Keywords: robust, extraction
• Abstract: The advent of blockchain technology has facilitated the widespread adoption of smart contracts in the financial sector. However, current fraud detection methodologies exhibit limitations in capturing both global structural patterns within transaction networks and local semantic relationships embedded in transaction data. Most existing models focus on either structural information or semantic features individually, leading to suboptimal performance in detecting complex fraud this http URL this paper, we propose a dynamic feature fusion model that combines graph-based representation learning and semantic feature extraction for blockchain fraud detection. Specifically, we construct global graph representations to model account relationships and extract local contextual features from transaction data. A dynamic multimodal fusion mechanism is introduced to adaptively integrate these features, enabling the model to capture both structural and semantic fraud patterns effectively. We further develop a comprehensive data processing pipeline, including graph construction, temporal feature enhancement, and text preprocessing. Experimental results on large-scale real-world blockchain datasets demonstrate that our method outperforms existing benchmarks across accuracy, F1 score, and recall metrics. This work highlights the importance of integrating structural relationships and semantic similarities for robust fraud detection and offers a scalable solution for securing blockchain systems.

Title: An Investigation into Value Misalignment in LLM-Generated Texts for Cultural Heritage

• Authors: Fan Bu, Zheng Wang, Siyi Wang, Ziyao Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02039
• Pdf URL: https://arxiv.org/pdf/2501.02039
• Copy Paste: [[2501.02039]] An Investigation into Value Misalignment in LLM-Generated Texts for Cultural Heritage(https://arxiv.org/abs/2501.02039)
• Keywords: large language model
• Abstract: As Large Language Models (LLMs) become increasingly prevalent in tasks related to cultural heritage, such as generating descriptions of historical monuments, translating ancient texts, preserving oral traditions, and creating educational content, their ability to produce accurate and culturally aligned texts is being increasingly relied upon by users and researchers. However, cultural value misalignments may exist in generated texts, such as the misrepresentation of historical facts, the erosion of cultural identity, and the oversimplification of complex cultural narratives, which may lead to severe consequences. Therefore, investigating value misalignment in the context of LLM for cultural heritage is crucial for mitigating these risks, yet there has been a significant lack of systematic and comprehensive study and investigation in this area. To fill this gap, we systematically assess the reliability of LLMs in generating culturally aligned texts for cultural heritage-related tasks. We conduct a comprehensive evaluation by compiling an extensive set of 1066 query tasks covering 5 widely recognized categories with 17 aspects within the knowledge framework of cultural heritage across 5 open-source LLMs, and examine both the type and rate of cultural value misalignments in the generated texts. Using both automated and manual approaches, we effectively detect and analyze the cultural value misalignments in LLM-generated texts. Our findings are concerning: over 65% of the generated texts exhibit notable cultural misalignments, with certain tasks demonstrating almost complete misalignment with key cultural values. Beyond these findings, this paper introduces a benchmark dataset and a comprehensive evaluation workflow that can serve as a valuable resource for future research aimed at enhancing the cultural sensitivity and reliability of LLMs.

Title: A Separable Self-attention Inspired by the State Space Model for Computer Vision

• Authors: Juntao Zhang, Shaogeng Liu, Kun Bian, You Zhou, Pei Zhang, Jianning Liu, Jun Zhou, Bingyan Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02040
• Pdf URL: https://arxiv.org/pdf/2501.02040
• Copy Paste: [[2501.02040]] A Separable Self-attention Inspired by the State Space Model for Computer Vision(https://arxiv.org/abs/2501.02040)
• Keywords: fair, transformer
• Abstract: Mamba is an efficient State Space Model (SSM) with linear computational complexity. Although SSMs are not suitable for handling non-causal data, Vision Mamba (ViM) methods still demonstrate good performance in tasks such as image classification and object detection. Recent studies have shown that there is a rich theoretical connection between state space models and attention variants. We propose a novel separable self attention method, for the first time introducing some excellent design concepts of Mamba into separable self-attention. To ensure a fair comparison with ViMs, we introduce VMINet, a simple yet powerful prototype architecture, constructed solely by stacking our novel attention modules with the most basic down-sampling layers. Notably, VMINet differs significantly from the conventional Transformer architecture. Our experiments demonstrate that VMINet has achieved competitive results on image classification and high-resolution dense prediction this http URL is available at: \url{this https URL}.

Title: MRG: A Multi-Robot Manufacturing Digital Scene Generation Method Using Multi-Instance Point Cloud Registration

• Authors: Songjie Han, Yinhua Liu, Yanzheng Li, Hua Chen, Dongmei Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02041
• Pdf URL: https://arxiv.org/pdf/2501.02041
• Copy Paste: [[2501.02041]] MRG: A Multi-Robot Manufacturing Digital Scene Generation Method Using Multi-Instance Point Cloud Registration(https://arxiv.org/abs/2501.02041)
• Keywords: transformer, segmentation
• Abstract: A high-fidelity digital simulation environment is crucial for accurately replicating physical operational processes. However, inconsistencies between simulation and physical environments result in low confidence in simulation outcomes, limiting their effectiveness in guiding real-world production. Unlike the traditional step-by-step point cloud "segmentation-registration" generation method, this paper introduces, for the first time, a novel Multi-Robot Manufacturing Digital Scene Generation (MRG) method that leverages multi-instance point cloud registration, specifically within manufacturing scenes. Tailored to the characteristics of industrial robots and manufacturing settings, an instance-focused transformer module is developed to delineate instance boundaries and capture correlations between local regions. Additionally, a hypothesis generation module is proposed to extract target instances while preserving key features. Finally, an efficient screening and optimization algorithm is designed to refine the final registration results. Experimental evaluations on the Scan2CAD and Welding-Station datasets demonstrate that: (1) the proposed method outperforms existing multi-instance point cloud registration techniques; (2) compared to state-of-the-art methods, the Scan2CAD dataset achieves improvements in MR and MP by 12.15% and 17.79%, respectively; and (3) on the Welding-Station dataset, MR and MP are enhanced by 16.95% and 24.15%, respectively. This work marks the first application of multi-instance point cloud registration in manufacturing scenes, significantly advancing the precision and reliability of digital simulation environments for industrial applications.

Title: Towards Robust and Accurate Stability Estimation of Local Surrogate Models in Text-based Explainable AI

• Authors: Christopher Burger, Charles Walter, Thai Le, Lingwei Chen
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.02042
• Pdf URL: https://arxiv.org/pdf/2501.02042
• Copy Paste: [[2501.02042]] Towards Robust and Accurate Stability Estimation of Local Surrogate Models in Text-based Explainable AI(https://arxiv.org/abs/2501.02042)
• Keywords: attack, robust
• Abstract: Recent work has investigated the concept of adversarial attacks on explainable AI (XAI) in the NLP domain with a focus on examining the vulnerability of local surrogate methods such as Lime to adversarial perturbations or small changes on the input of a machine learning (ML) model. In such attacks, the generated explanation is manipulated while the meaning and structure of the original input remain similar under the ML model. Such attacks are especially alarming when XAI is used as a basis for decision making (e.g., prescribing drugs based on AI medical predictors) or for legal action (e.g., legal dispute involving AI software). Although weaknesses across many XAI methods have been shown to exist, the reasons behind why remain little explored. Central to this XAI manipulation is the similarity measure used to calculate how one explanation differs from another. A poor choice of similarity measure can lead to erroneous conclusions about the stability or adversarial robustness of an XAI method. Therefore, this work investigates a variety of similarity measures designed for text-based ranked lists referenced in related work to determine their comparative suitability for use. We find that many measures are overly sensitive, resulting in erroneous estimates of stability. We then propose a weighting scheme for text-based data that incorporates the synonymity between the features within an explanation, providing more accurate estimates of the actual weakness of XAI methods to adversarial examples.

Title: DreamMask: Boosting Open-vocabulary Panoptic Segmentation with Synthetic Data

• Authors: Yuanpeng Tu, Xi Chen, Ser-Nam Lim, Hengshuang Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02048
• Pdf URL: https://arxiv.org/pdf/2501.02048
• Copy Paste: [[2501.02048]] DreamMask: Boosting Open-vocabulary Panoptic Segmentation with Synthetic Data(https://arxiv.org/abs/2501.02048)
• Keywords: robust, segmentation
• Abstract: Open-vocabulary panoptic segmentation has received significant attention due to its applicability in the real world. Despite claims of robust generalization, we find that the advancements of previous works are attributed mainly on trained categories, exposing a lack of generalization to novel classes. In this paper, we explore boosting existing models from a data-centric perspective. We propose DreamMask, which systematically explores how to generate training data in the open-vocabulary setting, and how to train the model with both real and synthetic data. For the first part, we propose an automatic data generation pipeline with off-the-shelf models. We propose crucial designs for vocabulary expansion, layout arrangement, data filtering, etc. Equipped with these techniques, our generated data could significantly outperform the manually collected web data. To train the model with generated data, a synthetic-real alignment loss is designed to bridge the representation gap, bringing noticeable improvements across multiple benchmarks. In general, DreamMask significantly simplifies the collection of large-scale training data, serving as a plug-and-play enhancement for existing methods. For instance, when trained on COCO and tested on ADE20K, the model equipped with DreamMask outperforms the previous state-of-the-art by a substantial margin of 2.1% mIoU.

Title: Active Learning Enables Extrapolation in Molecular Generative Models

• Authors: Evan R. Antoniuk, Peggy Li, Nathan Keilbart, Stephen Weitzner, Bhavya Kailkhura, Anna M. Hiszpanski
• Subjects: cs.LG, cond-mat.mtrl-sci, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2501.02059
• Pdf URL: https://arxiv.org/pdf/2501.02059
• Copy Paste: [[2501.02059]] Active Learning Enables Extrapolation in Molecular Generative Models(https://arxiv.org/abs/2501.02059)
• Keywords: generative
• Abstract: Although generative models hold promise for discovering molecules with optimized desired properties, they often fail to suggest synthesizable molecules that improve upon the known molecules seen in training. We find that a key limitation is not in the molecule generation process itself, but in the poor generalization capabilities of molecular property predictors. We tackle this challenge by creating an active-learning, closed-loop molecule generation pipeline, whereby molecular generative models are iteratively refined on feedback from quantum chemical simulations to improve generalization to new chemical space. Compared against other generative model approaches, only our active learning approach generates molecules with properties that extrapolate beyond the training data (reaching up to 0.44 standard deviations beyond the training data range) and out-of-distribution molecule classification accuracy is improved by 79%. By conditioning molecular generation on thermodynamic stability data from the active-learning loop, the proportion of stable molecules generated is 3.5x higher than the next-best model.

Title: AGGA: A Dataset of Academic Guidelines for Generative AI and Large Language Models

• Authors: Junfeng Jiao, Saleh Afroogh, Kevin Chen, David Atkinson, Amit Dhurandhar
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.02063
• Pdf URL: https://arxiv.org/pdf/2501.02063
• Copy Paste: [[2501.02063]] AGGA: A Dataset of Academic Guidelines for Generative AI and Large Language Models(https://arxiv.org/abs/2501.02063)
• Keywords: generative, large language model
• Abstract: This study introduces AGGA, a dataset comprising 80 academic guidelines for the use of Generative AIs (GAIs) and Large Language Models (LLMs) in academic settings, meticulously collected from official university websites. The dataset contains 188,674 words and serves as a valuable resource for natural language processing tasks commonly applied in requirements engineering, such as model synthesis, abstraction identification, and document structure assessment. Additionally, AGGA can be further annotated to function as a benchmark for various tasks, including ambiguity detection, requirements categorization, and the identification of equivalent requirements. Our methodologically rigorous approach ensured a thorough examination, with a selection of universities that represent a diverse range of global institutions, including top-ranked universities across six continents. The dataset captures perspectives from a variety of academic fields, including humanities, technology, and both public and private institutions, offering a broad spectrum of insights into the integration of GAIs and LLMs in academia.

Title: ArtCrafter: Text-Image Aligning Style Transfer via Embedding Reframing

• Authors: Nisha Huang, Kaer Huang, Yifan Pu, Jiangshan Wang, Jie Guo, Yiqiang Yan, Xiu Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02064
• Pdf URL: https://arxiv.org/pdf/2501.02064
• Copy Paste: [[2501.02064]] ArtCrafter: Text-Image Aligning Style Transfer via Embedding Reframing(https://arxiv.org/abs/2501.02064)
• Keywords: extraction, diffusion
• Abstract: Recent years have witnessed significant advancements in text-guided style transfer, primarily attributed to innovations in diffusion models. These models excel in conditional guidance, utilizing text or images to direct the sampling process. However, despite their capabilities, direct conditional guidance approaches often face challenges in balancing the expressiveness of textual semantics with the diversity of output results while capturing stylistic features. To address these challenges, we introduce ArtCrafter, a novel framework for text-to-image style transfer. Specifically, we introduce an attention-based style extraction module, meticulously engineered to capture the subtle stylistic elements within an image. This module features a multi-layer architecture that leverages the capabilities of perceiver attention mechanisms to integrate fine-grained information. Additionally, we present a novel text-image aligning augmentation component that adeptly balances control over both modalities, enabling the model to efficiently map image and text embeddings into a shared feature space. We achieve this through attention operations that enable smooth information flow between modalities. Lastly, we incorporate an explicit modulation that seamlessly blends multimodal enhanced embeddings with original embeddings through an embedding reframing design, empowering the model to generate diverse outputs. Extensive experiments demonstrate that ArtCrafter yields impressive results in visual stylization, exhibiting exceptional levels of stylistic intensity, controllability, and diversity.

Title: Counterfactual Explanation for Auto-Encoder Based Time-Series Anomaly Detection

• Authors: Abhishek Srinivasan, Varun Singapuri Ravi, Juan Carlos Andresen, Anders Holst
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02069
• Pdf URL: https://arxiv.org/pdf/2501.02069
• Copy Paste: [[2501.02069]] Counterfactual Explanation for Auto-Encoder Based Time-Series Anomaly Detection(https://arxiv.org/abs/2501.02069)
• Keywords: interpretability
• Abstract: The complexity of modern electro-mechanical systems require the development of sophisticated diagnostic methods like anomaly detection capable of detecting deviations. Conventional anomaly detection approaches like signal processing and statistical modelling often struggle to effectively handle the intricacies of complex systems, particularly when dealing with multi-variate signals. In contrast, neural network-based anomaly detection methods, especially Auto-Encoders, have emerged as a compelling alternative, demonstrating remarkable performance. However, Auto-Encoders exhibit inherent opaqueness in their decision-making processes, hindering their practical implementation at scale. Addressing this opacity is essential for enhancing the interpretability and trustworthiness of anomaly detection models. In this work, we address this challenge by employing a feature selector to select features and counterfactual explanations to give a context to the model output. We tested this approach on the SKAB benchmark dataset and an industrial time-series dataset. The gradient based counterfactual explanation approach was evaluated via validity, sparsity and distance measures. Our experimental findings illustrate that our proposed counterfactual approach can offer meaningful and valuable insights into the model decision-making process, by explaining fewer signals compared to conventional approaches. These insights enhance the trustworthiness and interpretability of anomaly detection models.

Title: AI-Powered Cow Detection in Complex Farm Environments

• Authors: Voncarlos, Ines, Thomas, Sebastien, Elsa, Marjorie, Abdoulaye
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02080
• Pdf URL: https://arxiv.org/pdf/2501.02080
• Copy Paste: [[2501.02080]] AI-Powered Cow Detection in Complex Farm Environments(https://arxiv.org/abs/2501.02080)
• Keywords: robust
• Abstract: Animal welfare has become a critical issue in contemporary society, emphasizing our ethical responsibilities toward animals, particularly within livestock farming. The advent of Artificial Intelligence (AI) technologies, specifically computer vision, offers an innovative approach to monitoring and enhancing animal welfare. Cows, as essential contributors to sustainable agriculture, are central to this effort. However, existing cow detection algorithms face challenges in real-world farming environments, such as complex lighting, occlusions, pose variations, and background interference, hindering detection. Model generalization is crucial for adaptation across contexts beyond the training dataset. This study addresses these challenges using a diverse cow dataset from six environments, including indoor and outdoor scenarios. We propose a detection model combining YOLOv8 with the CBAM (Convolutional Block Attention Module) and assess its performance against baseline models, including Mask R-CNN, YOLOv5, and YOLOv8. Our findings show baseline models degrade in complex conditions, while our approach improves using CBAM. YOLOv8-CBAM outperformed YOLOv8 by 2.3% in mAP, achieving 95.2% precision and an mAP@0.5:0.95 of 82.6%, demonstrating superior accuracy. Contributions include (1) analyzing detection limitations, (2) proposing a robust model, and (3) benchmarking state-of-the-art algorithms. Applications include health monitoring, behavioral analysis, and tracking in smart farms, enabling precise detection in challenging settings. This study advances AI-driven livestock monitoring, improving animal welfare and smart agriculture.

Title: Instruction-Following Pruning for Large Language Models

• Authors: Bairu Hou, Qibin Chen, Jianyu Wang, Guoli Yin, Chong Wang, Nan Du, Ruoming Pang, Shiyu Chang, Tao Lei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02086
• Pdf URL: https://arxiv.org/pdf/2501.02086
• Copy Paste: [[2501.02086]] Instruction-Following Pruning for Large Language Models(https://arxiv.org/abs/2501.02086)
• Keywords: large language model
• Abstract: With the rapid scaling of large language models (LLMs), structured pruning has become a widely used technique to learn efficient, smaller models from larger ones, delivering superior performance compared to training similarly sized models from scratch. In this paper, we move beyond the traditional static pruning approach of determining a fixed pruning mask for a model, and propose a dynamic approach to structured pruning. In our method, the pruning mask is input-dependent and adapts dynamically based on the information described in a user instruction. Our approach, termed "instruction-following pruning", introduces a sparse mask predictor that takes the user instruction as input and dynamically selects the most relevant model parameters for the given task. To identify and activate effective parameters, we jointly optimize the sparse mask predictor and the LLM, leveraging both instruction-following data and the pre-training corpus. Experimental results demonstrate the effectiveness of our approach on a wide range of evaluation benchmarks. For example, our 3B activated model improves over the 3B dense model by 5-8 points of absolute margin on domains such as math and coding, and rivals the performance of a 9B model.

Title: PriveShield: Enhancing User Privacy Using Automatic Isolated Profiles in Browsers

• Authors: Seyed Ali Akhavani, Engin Kirda, Amin Kharraz
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.02091
• Pdf URL: https://arxiv.org/pdf/2501.02091
• Copy Paste: [[2501.02091]] PriveShield: Enhancing User Privacy Using Automatic Isolated Profiles in Browsers(https://arxiv.org/abs/2501.02091)
• Keywords: security, privacy
• Abstract: Online tracking is a widespread practice on the web with questionable ethics, security, and privacy concerns. While web tracking can offer personalized and curated content to Internet users, it operates as a sophisticated surveillance mechanism to gather extensive user information. This paper introduces PriveShield, a light-weight privacy mechanism that disrupts the information gathering cycle while offering more control to Internet users to maintain their privacy. PriveShield is implemented as a browser extension that offers an adjustable privacy feature to surf the web with multiple identities or accounts simultaneously without any changes to underlying browser code or services. When necessary, multiple factors are automatically analyzed on the client side to isolate cookies and other information that are the basis of online tracking. PriveShield creates isolated profiles for clients based on their browsing history, interactions with websites, and the amount of time they spend on specific websites. This allows the users to easily prevent unwanted browsing information from being shared with third parties and ad exchanges without the need for manual configuration. Our evaluation results from 54 real-world scenarios show that our extension is effective in preventing retargeted ads in 91% of those scenarios.

Title: How Your Location Relates to Health: Variable Importance and Interpretable Machine Learning for Environmental and Sociodemographic Data

• Authors: Ishaan Maitra, Raymond Lin, Eric Chen, Jon Donnelly, Sanja Šćepanović, Cynthia Rudin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02111
• Pdf URL: https://arxiv.org/pdf/2501.02111
• Copy Paste: [[2501.02111]] How Your Location Relates to Health: Variable Importance and Interpretable Machine Learning for Environmental and Sociodemographic Data(https://arxiv.org/abs/2501.02111)
• Keywords: robust
• Abstract: Health outcomes depend on complex environmental and sociodemographic factors whose effects change over location and time. Only recently has fine-grained spatial and temporal data become available to study these effects, namely the MEDSAT dataset of English health, environmental, and sociodemographic information. Leveraging this new resource, we use a variety of variable importance techniques to robustly identify the most informative predictors across multiple health outcomes. We then develop an interpretable machine learning framework based on Generalized Additive Models (GAMs) and Multiscale Geographically Weighted Regression (MGWR) to analyze both local and global spatial dependencies of each variable on various health outcomes. Our findings identify NO2 as a global predictor for asthma, hypertension, and anxiety, alongside other outcome-specific predictors related to occupation, marriage, and vegetation. Regional analyses reveal local variations with air pollution and solar radiation, with notable shifts during COVID. This comprehensive approach provides actionable insights for addressing health disparities, and advocates for the integration of interpretable machine learning in public health.

Title: K-Gate Lock: Multi-Key Logic Locking Using Input Encoding Against Oracle-Guided Attacks

• Authors: Kevin Lopez, Amin Rezaei
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.02118
• Pdf URL: https://arxiv.org/pdf/2501.02118
• Copy Paste: [[2501.02118]] K-Gate Lock: Multi-Key Logic Locking Using Input Encoding Against Oracle-Guided Attacks(https://arxiv.org/abs/2501.02118)
• Keywords: secure, protect, attack
• Abstract: Logic locking has emerged to prevent piracy and overproduction of integrated circuits ever since the split of the design house and manufacturing foundry was established. While there has been a lot of research using a single global key to lock the circuit, even the most sophisticated single-key locking methods have been shown to be vulnerable to powerful SAT-based oracle-guided attacks that can extract the correct key with the help of an activated chip bought off the market and the locked netlist leaked from the untrusted foundry. To address this challenge, we propose, implement, and evaluate a novel logic locking method called K-Gate Lock that encodes input patterns using multiple keys that are applied to one set of key inputs at different operational times. Our comprehensive experimental results confirm that using multiple keys will make the circuit secure against oracle-guided attacks and increase attacker efforts to an exponentially time-consuming brute force search. K-Gate Lock has reasonable power and performance overheads, making it a practical solution for real-world hardware intellectual property protection.

Title: AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs

• Authors: Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Yaoting Wang, Mohamed Elhoseiny, Ruohan Gao, Dinesh Manocha
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02135
• Pdf URL: https://arxiv.org/pdf/2501.02135
• Copy Paste: [[2501.02135]] AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs(https://arxiv.org/abs/2501.02135)
• Keywords: attack, robust, large language model
• Abstract: With the rapid advancement of Multi-modal Large Language Models (MLLMs), several diagnostic benchmarks have recently been developed to assess these models' multi-modal reasoning proficiency. However, these benchmarks are restricted to assessing primarily the visual aspect and do not examine the holistic audio-visual (AV) understanding. Moreover, currently, there are no benchmarks that investigate the capabilities of AVLLMs to calibrate their responses when presented with perturbed inputs. To this end, we introduce Audio-Visual Trustworthiness assessment Benchmark (AVTrustBench), comprising 600K samples spanning over 9 meticulously crafted tasks, evaluating the capabilities of AVLLMs across three distinct dimensions: Adversarial attack, Compositional reasoning, and Modality-specific dependency. Using our benchmark we extensively evaluate 13 state-of-the-art AVLLMs. The findings reveal that the majority of existing models fall significantly short of achieving human-like comprehension, offering valuable insights for future research directions. To alleviate the limitations in the existing approaches, we further propose a robust, model-agnostic calibrated audio-visual preference optimization based training strategy CAVPref, obtaining a gain up to 30.19% across all 9 tasks. We will publicly release our code and benchmark to facilitate future research in this direction.

Title: Plasma-CycleGAN: Plasma Biomarker-Guided MRI to PET Cross-modality Translation Using Conditional CycleGAN

• Authors: Yanxi Chen, Yi Su, Celine Dumitrascu, Kewei Chen, David Weidman, Richard J Caselli, Nicholas Ashton, Eric M Reiman, Yalin Wang
• Subjects: cs.CV, cs.AI, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2501.02146
• Pdf URL: https://arxiv.org/pdf/2501.02146
• Copy Paste: [[2501.02146]] Plasma-CycleGAN: Plasma Biomarker-Guided MRI to PET Cross-modality Translation Using Conditional CycleGAN(https://arxiv.org/abs/2501.02146)
• Keywords: generative
• Abstract: Cross-modality translation between MRI and PET imaging is challenging due to the distinct mechanisms underlying these modalities. Blood-based biomarkers (BBBMs) are revolutionizing Alzheimer's disease (AD) detection by identifying patients and quantifying brain amyloid levels. However, the potential of BBBMs to enhance PET image synthesis remains unexplored. In this paper, we performed a thorough study on the effect of incorporating BBBM into deep generative models. By evaluating three widely used cross-modality translation models, we found that BBBMs integration consistently enhances the generative quality across all models. By visual inspection of the generated results, we observed that PET images generated by CycleGAN exhibit the best visual fidelity. Based on these findings, we propose Plasma-CycleGAN, a novel generative model based on CycleGAN, to synthesize PET images from MRI using BBBMs as conditions. This is the first approach to integrate BBBMs in conditional cross-modality translation between MRI and PET.

Title: Exploring Secure Machine Learning Through Payload Injection and FGSM Attacks on ResNet-50

• Authors: Umesh Yadav, Suman Niraula, Gaurav Kumar Gupta, Bicky Yadav
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02147
• Pdf URL: https://arxiv.org/pdf/2501.02147
• Copy Paste: [[2501.02147]] Exploring Secure Machine Learning Through Payload Injection and FGSM Attacks on ResNet-50(https://arxiv.org/abs/2501.02147)
• Keywords: secure, security, defense, attack, robust, steal
• Abstract: This paper investigates the resilience of a ResNet-50 image classification model under two prominent security threats: Fast Gradient Sign Method (FGSM) adversarial attacks and malicious payload injection. Initially, the model attains a 53.33% accuracy on clean images. When subjected to FGSM perturbations, its overall accuracy remains unchanged; however, the model's confidence in incorrect predictions notably increases. Concurrently, a payload injection scheme is successfully executed in 93.33% of the tested samples, revealing how stealthy attacks can manipulate model predictions without degrading visual quality. These findings underscore the vulnerability of even high-performing neural networks and highlight the urgency of developing more robust defense mechanisms for security-critical applications.

Title: Personalized Graph-Based Retrieval for Large Language Models

• Authors: Steven Au, Cameron J. Dimacali, Ojasmitha Pedirappagari, Namyong Park, Franck Dernoncourt, Yu Wang, Nikos Kanakaris, Hanieh Deilamsalehy, Ryan A. Rossi, Nesreen K. Ahmed
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02157
• Pdf URL: https://arxiv.org/pdf/2501.02157
• Copy Paste: [[2501.02157]] Personalized Graph-Based Retrieval for Large Language Models(https://arxiv.org/abs/2501.02157)
• Keywords: large language model
• Abstract: As large language models (LLMs) evolve, their ability to deliver personalized and context-aware responses offers transformative potential for improving user experiences. Existing personalization approaches, however, often rely solely on user history to augment the prompt, limiting their effectiveness in generating tailored outputs, especially in cold-start scenarios with sparse data. To address these limitations, we propose Personalized Graph-based Retrieval-Augmented Generation (PGraphRAG), a framework that leverages user-centric knowledge graphs to enrich personalization. By directly integrating structured user knowledge into the retrieval process and augmenting prompts with user-relevant context, PGraphRAG enhances contextual understanding and output quality. We also introduce the Personalized Graph-based Benchmark for Text Generation, designed to evaluate personalized text generation tasks in real-world settings where user history is sparse or unavailable. Experimental results show that PGraphRAG significantly outperforms state-of-the-art personalization methods across diverse tasks, demonstrating the unique advantages of graph-based retrieval for personalization.

Title: Generating Multimodal Images with GAN: Integrating Text, Image, and Style

• Authors: Chaoyi Tan, Wenqing Zhang, Zhen Qi, Kowei Shih, Xinshi Li, Ao Xiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02167
• Pdf URL: https://arxiv.org/pdf/2501.02167
• Copy Paste: [[2501.02167]] Generating Multimodal Images with GAN: Integrating Text, Image, and Style(https://arxiv.org/abs/2501.02167)
• Keywords: generative
• Abstract: In the field of computer vision, multimodal image generation has become a research hotspot, especially the task of integrating text, image, and style. In this study, we propose a multimodal image generation method based on Generative Adversarial Networks (GAN), capable of effectively combining text descriptions, reference images, and style information to generate images that meet multimodal requirements. This method involves the design of a text encoder, an image feature extractor, and a style integration module, ensuring that the generated images maintain high quality in terms of visual content and style consistency. We also introduce multiple loss functions, including adversarial loss, text-image consistency loss, and style matching loss, to optimize the generation process. Experimental results show that our method produces images with high clarity and consistency across multiple public datasets, demonstrating significant performance improvements compared to existing methods. The outcomes of this study provide new insights into multimodal image generation and present broad application prospects.

Title: AdaMixup: A Dynamic Defense Framework for Membership Inference Attack Mitigation

• Authors: Ying Chen, Jiajing Chen, Yijie Weng, ChiaHua Chang, Dezhi Yu, Guanbiao Lin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02182
• Pdf URL: https://arxiv.org/pdf/2501.02182
• Copy Paste: [[2501.02182]] AdaMixup: A Dynamic Defense Framework for Membership Inference Attack Mitigation(https://arxiv.org/abs/2501.02182)
• Keywords: privacy, protect, defense, attack, robust, membership infer
• Abstract: Membership inference attacks have emerged as a significant privacy concern in the training of deep learning models, where attackers can infer whether a data point was part of the training set based on the model's outputs. To address this challenge, we propose a novel defense mechanism, AdaMixup. AdaMixup employs adaptive mixup techniques to enhance the model's robustness against membership inference attacks by dynamically adjusting the mixup strategy during training. This method not only improves the model's privacy protection but also maintains high performance. Experimental results across multiple datasets demonstrate that AdaMixup significantly reduces the risk of membership inference attacks while achieving a favorable trade-off between defensive efficiency and model accuracy. This research provides an effective solution for data privacy protection and lays the groundwork for future advancements in mixup training methods.

Title: Benchmark Evaluations, Applications, and Challenges of Large Vision Language Models: A Survey

• Authors: Zongxia Li, Xiyang Wu, Hongyang Du, Huy Nghiem, Guangyao Shi
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.02189
• Pdf URL: https://arxiv.org/pdf/2501.02189
• Copy Paste: [[2501.02189]] Benchmark Evaluations, Applications, and Challenges of Large Vision Language Models: A Survey(https://arxiv.org/abs/2501.02189)
• Keywords: fair
• Abstract: Multimodal Vision Language Models (VLMs) have emerged as a transformative technology at the intersection of computer vision and natural language processing, enabling machines to perceive and reason about the world through both visual and textual modalities. For example, models such as CLIP, Claude, and GPT-4V demonstrate strong reasoning and understanding abilities on visual and textual data and beat classical single modality vision models on zero-shot classification. Despite their rapid advancements in research and growing popularity in applications, a comprehensive survey of existing studies on VLMs is notably lacking, particularly for researchers aiming to leverage VLMs in their specific domains. To this end, we provide a systematic overview of VLMs in the following aspects: model information of the major VLMs developed over the past five years (2019-2024); the main architectures and training methods of these VLMs; summary and categorization of the popular benchmarks and evaluation metrics of VLMs; the applications of VLMs including embodied agents, robotics, and video generation; the challenges and issues faced by current VLMs such as hallucination, fairness, and safety. Detailed collections including papers and model repository links are listed in this https URL.

Title: On LLM-Enhanced Mixed-Type Data Imputation with High-Order Message Passing

• Authors: Jianwei Wang, Kai Wang, Ying Zhang, Wenjie Zhang, Xiwei Xu, Xuemin Lin
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.02191
• Pdf URL: https://arxiv.org/pdf/2501.02191
• Copy Paste: [[2501.02191]] On LLM-Enhanced Mixed-Type Data Imputation with High-Order Message Passing(https://arxiv.org/abs/2501.02191)
• Keywords: large language model
• Abstract: Missing data imputation, which aims to impute the missing values in the raw datasets to achieve the completeness of datasets, is crucial for modern data-driven models like large language models (LLMs) and has attracted increasing interest over the past decades. Despite its importance, existing solutions for missing data imputation either 1) only support numerical and categorical data or 2) show an unsatisfactory performance due to their design prioritizing text data and the lack of key properties for tabular data imputation. In this paper, we propose UnIMP, a Unified IMPutation framework that leverages LLM and high-order message passing to enhance the imputation of mixed-type data including numerical, categorical, and text data. Specifically, we first introduce a cell-oriented hypergraph to model the table. We then propose BiHMP, an efficient Bidirectional High-order Message-Passing network to aggregate global-local information and high-order relationships on the constructed hypergraph while capturing the inter-column heterogeneity and intra-column homogeneity. To effectively and efficiently align the capacity of the LLM with the information aggregated by BiHMP, we introduce Xfusion, which, together with BiHMP, acts as adapters for the LLM. We follow a pre-training and fine-tuning pipeline to train UnIMP, integrating two optimizations: chunking technique, which divides tables into smaller chunks to enhance efficiency; and progressive masking technique, which gradually adapts the model to learn more complex data patterns. Both theoretical proofs and empirical experiments on 10 real world datasets highlight the superiority of UnIMP over existing techniques.

Title: CPTuning: Contrastive Prompt Tuning for Generative Relation Extraction

• Authors: Jiaxin Duan, Fengyu Lu, Junfei Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02196
• Pdf URL: https://arxiv.org/pdf/2501.02196
• Copy Paste: [[2501.02196]] CPTuning: Contrastive Prompt Tuning for Generative Relation Extraction(https://arxiv.org/abs/2501.02196)
• Keywords: extraction, generative
• Abstract: Generative relation extraction (RE) commonly involves first reformulating RE as a linguistic modeling problem easily tackled with pre-trained language models (PLM) and then fine-tuning a PLM with supervised cross-entropy loss. Although having achieved promising performance, existing approaches assume only one deterministic relation between each pair of entities without considering real scenarios where multiple relations may be valid, i.e., entity pair overlap, causing their limited applications. To address this problem, we introduce a novel contrastive prompt tuning method for RE, CPTuning, which learns to associate a candidate relation between two in-context entities with a probability mass above or below a threshold, corresponding to whether the relation exists. Beyond learning schema, CPTuning also organizes RE as a verbalized relation generation task and uses Trie-constrained decoding to ensure a model generates valid relations. It adaptively picks out the generated candidate relations with a high estimated likelihood in inference, thereby achieving multi-relation extraction. We conduct extensive experiments on four widely used datasets to validate our method. Results show that T5-large fine-tuned with CPTuning significantly outperforms previous methods, regardless of single or multiple relations extraction.

Title: Accounting for Focus Ambiguity in Visual Questions

• Authors: Chongyan Chen, Yu-Yun Tseng, Zhuoheng Li, Anush Venkatesh, Danna Gurari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02201
• Pdf URL: https://arxiv.org/pdf/2501.02201
• Copy Paste: [[2501.02201]] Accounting for Focus Ambiguity in Visual Questions(https://arxiv.org/abs/2501.02201)
• Keywords: segmentation
• Abstract: No existing work on visual question answering explicitly accounts for ambiguity regarding where the content described in the question is located in the image. To fill this gap, we introduce VQ-FocusAmbiguity, the first VQA dataset that visually grounds each region described in the question that is necessary to arrive at the answer. We then provide an analysis showing how our dataset for visually grounding `questions' is distinct from visually grounding `answers', and characterize the properties of the questions and segmentations provided in our dataset. Finally, we benchmark modern models for two novel tasks: recognizing whether a visual question has focus ambiguity and localizing all plausible focus regions within the image. Results show that the dataset is challenging for modern models. To facilitate future progress on these tasks, we publicly share the dataset with an evaluation server at this https URL.

Title: Secure IAM on AWS with Multi-Account Strategy

• Authors: Sungchan Yi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.02203
• Pdf URL: https://arxiv.org/pdf/2501.02203
• Copy Paste: [[2501.02203]] Secure IAM on AWS with Multi-Account Strategy(https://arxiv.org/abs/2501.02203)
• Keywords: secure, security, attack
• Abstract: Many recent IT companies use cloud services for deploying their products, mainly because of their convenience. As such, cloud assets have become a new attack surface, and the concept of cloud security has emerged. However, cloud security is not emphasized enough compared to on-premise security, resulting in many insecure cloud architectures. In particular, small organizations often don't have enough human resources to design a secure architecture, leaving them vulnerable to cloud security breaches. We suggest the multi-account strategy for securing the cloud architecture. This strategy cost-effectively improves security by separating assets and reducing management overheads on the cloud infrastructure. When implemented, it automatically provides access restriction within the boundary of an account and eliminates redundancies in policy management. Since access control is a critical objective for constructing secure architectures, this practical method successfully enhances security even in small companies. In this paper, we analyze the benefits of multi-accounts compared to single accounts and explain how to deploy multiple accounts effortlessly using the services provided by AWS. Then, we present possible design choices for multi-account structures with a concrete example. Finally, we illustrate two techniques for operational excellence on multi-account structures. We take an incremental approach to secure policy management with the principle of least privilege and introduce methods for auditing multiple accounts.

Title: Examining the Robustness of Homogeneity Bias to Hyperparameter Adjustments in GPT-4

• Authors: Messi H.J. Lee
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02211
• Pdf URL: https://arxiv.org/pdf/2501.02211
• Copy Paste: [[2501.02211]] Examining the Robustness of Homogeneity Bias to Hyperparameter Adjustments in GPT-4(https://arxiv.org/abs/2501.02211)
• Keywords: robust
• Abstract: Vision-Language Models trained on massive collections of human-generated data often reproduce and amplify societal stereotypes. One critical form of stereotyping reproduced by these models is homogeneity bias-the tendency to represent certain groups as more homogeneous than others. We investigate how this bias responds to hyperparameter adjustments in GPT-4, specifically examining sampling temperature and top p which control the randomness of model outputs. By generating stories about individuals from different racial and gender groups and comparing their similarities using vector representations, we assess both bias robustness and its relationship with hyperparameter values. We find that (1) homogeneity bias persists across most hyperparameter configurations, with Black Americans and women being represented more homogeneously than White Americans and men, (2) the relationship between hyperparameters and group representations shows unexpected non-linear patterns, particularly at extreme values, and (3) hyperparameter adjustments affect racial and gender homogeneity bias differently-while increasing temperature or decreasing top p can reduce racial homogeneity bias, these changes show different effects on gender homogeneity bias. Our findings suggest that while hyperparameter tuning may mitigate certain biases to some extent, it cannot serve as a universal solution for addressing homogeneity bias across different social group dimensions.

Title: Diffusion Model-Based Data Synthesis Aided Federated Semi-Supervised Learning

• Authors: Zhongwei Wang, Tong Wu, Zhiyong Chen, Liang Qian, Yin Xu, Meixia Tao
• Subjects: cs.LG, cs.AI, cs.IT
• Abstract URL: https://arxiv.org/abs/2501.02219
• Pdf URL: https://arxiv.org/pdf/2501.02219
• Copy Paste: [[2501.02219]] Diffusion Model-Based Data Synthesis Aided Federated Semi-Supervised Learning(https://arxiv.org/abs/2501.02219)
• Keywords: federate, diffusion
• Abstract: Federated semi-supervised learning (FSSL) is primarily challenged by two factors: the scarcity of labeled data across clients and the non-independent and identically distribution (non-IID) nature of data among clients. In this paper, we propose a novel approach, diffusion model-based data synthesis aided FSSL (DDSA-FSSL), which utilizes a diffusion model (DM) to generate synthetic data, bridging the gap between heterogeneous local data distributions and the global data distribution. In DDSA-FSSL, clients address the challenge of the scarcity of labeled data by employing a federated learning-trained classifier to perform pseudo labeling for unlabeled data. The DM is then collaboratively trained using both labeled and precision-optimized pseudo-labeled data, enabling clients to generate synthetic samples for classes that are absent in their labeled datasets. This process allows clients to generate more comprehensive synthetic datasets aligned with the global distribution. Extensive experiments conducted on multiple datasets and varying non-IID distributions demonstrate the effectiveness of DDSA-FSSL, e.g., it improves accuracy from 38.46% to 52.14% on CIFAR-10 datasets with 10% labeled data.

Title: Leveraging Large Language Models and Machine Learning for Smart Contract Vulnerability Detection

• Authors: S M Mostaq Hossain, Amani Altarawneh, Jesse Roberts
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.02229
• Pdf URL: https://arxiv.org/pdf/2501.02229
• Copy Paste: [[2501.02229]] Leveraging Large Language Models and Machine Learning for Smart Contract Vulnerability Detection(https://arxiv.org/abs/2501.02229)
• Keywords: secure, security, robust, large language model
• Abstract: As blockchain technology and smart contracts become widely adopted, securing them throughout every stage of the transaction process is essential. The concern of improved security for smart contracts is to find and detect vulnerabilities using classical Machine Learning (ML) models and fine-tuned Large Language Models (LLM). The robustness of such work rests on a labeled smart contract dataset that includes annotated vulnerabilities on which several LLMs alongside various traditional machine learning algorithms such as DistilBERT model is trained and tested. We train and test machine learning algorithms to classify smart contract codes according to vulnerability types in order to compare model performance. Having fine-tuned the LLMs specifically for smart contract code classification should help in getting better results when detecting several types of well-known vulnerabilities, such as Reentrancy, Integer Overflow, Timestamp Dependency and Dangerous Delegatecall. From our initial experimental results, it can be seen that our fine-tuned LLM surpasses the accuracy of any other model by achieving an accuracy of over 90%, and this advances the existing vulnerability detection benchmarks. Such performance provides a great deal of evidence for LLMs ability to describe the subtle patterns in the code that traditional ML models could miss. Thus, we compared each of the ML and LLM models to give a good overview of each models strengths, from which we can choose the most effective one for real-world applications in smart contract security. Our research combines machine learning and large language models to provide a rich and interpretable framework for detecting different smart contract vulnerabilities, which lays a foundation for a more secure blockchain ecosystem.

Title: Distillation-Enhanced Physical Adversarial Attacks

• Authors: Wei Liu, Yonglin Wu, Chaoqun Li, Zhuodong Liu, Huanqian Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02232
• Pdf URL: https://arxiv.org/pdf/2501.02232
• Copy Paste: [[2501.02232]] Distillation-Enhanced Physical Adversarial Attacks(https://arxiv.org/abs/2501.02232)
• Keywords: attack, robust, steal
• Abstract: The study of physical adversarial patches is crucial for identifying vulnerabilities in AI-based recognition systems and developing more robust deep learning models. While recent research has focused on improving patch stealthiness for greater practical applicability, achieving an effective balance between stealth and attack performance remains a significant challenge. To address this issue, we propose a novel physical adversarial attack method that leverages knowledge distillation. Specifically, we first define a stealthy color space tailored to the target environment to ensure smooth blending. Then, we optimize an adversarial patch in an unconstrained color space, which serves as the 'teacher' patch. Finally, we use an adversarial knowledge distillation module to transfer the teacher patch's knowledge to the 'student' patch, guiding the optimization of the stealthy patch. Experimental results show that our approach improves attack performance by 20%, while maintaining stealth, highlighting its practical value.

Title: Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends

• Authors: Camille Barboule, Benjamin Piwowarski, Yoan Chabot
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02235
• Pdf URL: https://arxiv.org/pdf/2501.02235
• Copy Paste: [[2501.02235]] Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends(https://arxiv.org/abs/2501.02235)
• Keywords: large language model
• Abstract: Using Large Language Models (LLMs) for Visually-rich Document Understanding (VrDU) has significantly improved performance on tasks requiring both comprehension and generation, such as question answering, albeit introducing new challenges. This survey explains how VrDU models enhanced by LLMs function, covering methods for integrating VrD features into LLMs and highlighting key challenges.

Title: Financial Named Entity Recognition: How Far Can LLM Go?

• Authors: Yi-Te Lu, Yintong Huo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02237
• Pdf URL: https://arxiv.org/pdf/2501.02237
• Copy Paste: [[2501.02237]] Financial Named Entity Recognition: How Far Can LLM Go?(https://arxiv.org/abs/2501.02237)
• Keywords: extraction, large language model
• Abstract: The surge of large language models (LLMs) has revolutionized the extraction and analysis of crucial information from a growing volume of financial statements, announcements, and business news. Recognition for named entities to construct structured data poses a significant challenge in analyzing financial documents and is a foundational task for intelligent financial analytics. However, how effective are these generic LLMs and their performance under various prompts are yet need a better understanding. To fill in the blank, we present a systematic evaluation of state-of-the-art LLMs and prompting methods in the financial Named Entity Recognition (NER) problem. Specifically, our experimental results highlight their strengths and limitations, identify five representative failure types, and provide insights into their potential and challenges for domain-specific tasks.

Title: MagicFace: High-Fidelity Facial Expression Editing with Action-Unit Control

• Authors: Mengting Wei, Tuomas Varanka, Xingxun Jiang, Huai-Qian Khor, Guoying Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02260
• Pdf URL: https://arxiv.org/pdf/2501.02260
• Copy Paste: [[2501.02260]] MagicFace: High-Fidelity Facial Expression Editing with Action-Unit Control(https://arxiv.org/abs/2501.02260)
• Keywords: diffusion
• Abstract: We address the problem of facial expression editing by controling the relative variation of facial action-unit (AU) from the same person. This enables us to edit this specific person's expression in a fine-grained, continuous and interpretable manner, while preserving their identity, pose, background and detailed facial attributes. Key to our model, which we dub MagicFace, is a diffusion model conditioned on AU variations and an ID encoder to preserve facial details of high consistency. Specifically, to preserve the facial details with the input identity, we leverage the power of pretrained Stable-Diffusion models and design an ID encoder to merge appearance features through self-attention. To keep background and pose consistency, we introduce an efficient Attribute Controller by explicitly informing the model of current background and pose of the target. By injecting AU variations into a denoising UNet, our model can animate arbitrary identities with various AU combinations, yielding superior results in high-fidelity expression editing compared to other facial expression editing works. Code is publicly available at this https URL.

Title: Unsupervised Class Generation to Expand Semantic Segmentation Datasets

• Authors: Javier Montalvo, Álvaro García-Martín, Pablo Carballeira, Juan C. SanMiguel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02264
• Pdf URL: https://arxiv.org/pdf/2501.02264
• Copy Paste: [[2501.02264]] Unsupervised Class Generation to Expand Semantic Segmentation Datasets(https://arxiv.org/abs/2501.02264)
• Keywords: diffusion, generative, segmentation
• Abstract: Semantic segmentation is a computer vision task where classification is performed at a pixel level. Due to this, the process of labeling images for semantic segmentation is time-consuming and expensive. To mitigate this cost there has been a surge in the use of synthetically generated data -- usually created using simulators or videogames -- which, in combination with domain adaptation methods, can effectively learn how to segment real data. Still, these datasets have a particular limitation: due to their closed-set nature, it is not possible to include novel classes without modifying the tool used to generate them, which is often not public. Concurrently, generative models have made remarkable progress, particularly with the introduction of diffusion models, enabling the creation of high-quality images from text prompts without additional supervision. In this work, we propose an unsupervised pipeline that leverages Stable Diffusion and Segment Anything Module to generate class examples with an associated segmentation mask, and a method to integrate generated cutouts for novel classes in semantic segmentation datasets, all with minimal user input. Our approach aims to improve the performance of unsupervised domain adaptation methods by introducing novel samples into the training data without modifications to the underlying algorithms. With our methods, we show how models can not only effectively learn how to segment novel classes, with an average performance of 51% IoU, but also reduce errors for other, already existing classes, reaching a higher performance level overall.

Title: What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph

• Authors: Yutao Jiang, Qiong Wu, Wenhao Lin, Wei Yu, Yiyi Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02268
• Pdf URL: https://arxiv.org/pdf/2501.02268
• Copy Paste: [[2501.02268]] What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph(https://arxiv.org/abs/2501.02268)
• Keywords: large language model
• Abstract: Recent Multimodal Large Language Models(MLLMs) often use a large number of visual tokens to compensate their visual shortcoming, leading to excessive computation and obvious visual redundancy. In this paper, we investigate what kind of visual tokens are needed for MLLMs, and reveal that both foreground and background tokens are critical for MLLMs given the varying difficulties of examples. Based on this observation, we propose a graph-based method towards training-free visual token pruning, termed this http URL particular, G-Prune regards visual tokens as nodes, and construct their connections based on their semantic similarities. Afterwards, the information flow is propagated via weighted links, and the most important tokens after iterations are kept for MLLMs, which can be front or this http URL validate G-Prune, we apply it to a recent MLLM called LLaVA-NeXT, and conduct extensive experiments on a set of this http URL experiment results show that G-Prune can greatly reduce computation overhead while retaining high performance on both coarse- and fine-grained tasks. For instance, G-Prune can reduce 63.57\% FLOPs of LLaVA-NeXT on VQA2.0 and TextVQA with only 0.95\% and 2.34\% accuracy drops, respectively.

Title: TDM: Temporally-Consistent Diffusion Model for All-in-One Real-World Video Restoration

• Authors: Yizhou Li, Zihua Liu, Yusuke Monno, Masatoshi Okutomi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02269
• Pdf URL: https://arxiv.org/pdf/2501.02269
• Copy Paste: [[2501.02269]] TDM: Temporally-Consistent Diffusion Model for All-in-One Real-World Video Restoration(https://arxiv.org/abs/2501.02269)
• Keywords: diffusion
• Abstract: In this paper, we propose the first diffusion-based all-in-one video restoration method that utilizes the power of a pre-trained Stable Diffusion and a fine-tuned ControlNet. Our method can restore various types of video degradation with a single unified model, overcoming the limitation of standard methods that require specific models for each restoration task. Our contributions include an efficient training strategy with Task Prompt Guidance (TPG) for diverse restoration tasks, an inference strategy that combines Denoising Diffusion Implicit Models~(DDIM) inversion with a novel Sliding Window Cross-Frame Attention (SW-CFA) mechanism for enhanced content preservation and temporal consistency, and a scalable pipeline that makes our method all-in-one to adapt to different video restoration tasks. Through extensive experiments on five video restoration tasks, we demonstrate the superiority of our method in generalization capability to real-world videos and temporal consistency preservation over existing state-of-the-art methods. Our method advances the video restoration task by providing a unified solution that enhances video quality across multiple applications.

Title: Post-Quantum Key Agreement Protocols Based on Modified Matrix-Power Functions over Singular Random Integer Matrix Semirings

• Authors: Juan Pedro Hecht, Hugo Daniel Scolnik
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.02292
• Pdf URL: https://arxiv.org/pdf/2501.02292
• Copy Paste: [[2501.02292]] Post-Quantum Key Agreement Protocols Based on Modified Matrix-Power Functions over Singular Random Integer Matrix Semirings(https://arxiv.org/abs/2501.02292)
• Keywords: secure, security, attack
• Abstract: Post-quantum cryptography is essential for securing digital communications against threats posed by quantum computers. Re-searchers have focused on developing algorithms that can withstand attacks from both classical and quantum computers, thereby ensuring the security of data transmissions over public networks. A critical component of this security is the key agreement protocol, which allows two parties to establish a shared secret key over an insecure channel. This paper introduces two novel post-quantum key agreement protocols that can be easily implemented on standard computers using rectangular or rank-deficient matrices, exploiting the generalizations of the matrix power function, which is a generator of NP-hard problems. We provide basic concepts and proofs, pseudocodes, examples, and a discussion of complexity.

Title: Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection

• Authors: Yachao Zhao, Bo Wang, Yan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02295
• Pdf URL: https://arxiv.org/pdf/2501.02295
• Copy Paste: [[2501.02295]] Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection(https://arxiv.org/abs/2501.02295)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have been shown to exhibit various biases and stereotypes in their generated content. While extensive research has investigated bias in LLMs, prior work has predominantly focused on explicit bias, leaving the more nuanced implicit biases largely unexplored. This paper presents a systematic framework grounded in social psychology theories to investigate and compare explicit and implicit biases in LLMs. We propose a novel "self-reflection" based evaluation framework that operates in two phases: first measuring implicit bias through simulated psychological assessment methods, then evaluating explicit bias by prompting LLMs to analyze their own generated content. Through extensive experiments on state-of-the-art LLMs across multiple social dimensions, we demonstrate that LLMs exhibit a substantial inconsistency between explicit and implicit biases, where explicit biases manifest as mild stereotypes while implicit biases show strong stereotypes. Furthermore, we investigate the underlying factors contributing to this explicit-implicit bias inconsistency. Our experiments examine the effects of training data scale, model parameters, and alignment techniques. Results indicate that while explicit bias diminishes with increased training data and model size, implicit bias exhibits a contrasting upward trend. Notably, contemporary alignment methods (e.g., RLHF, DPO) effectively suppress explicit bias but show limited efficacy in mitigating implicit bias. These findings suggest that while scaling up models and alignment training can address explicit bias, the challenge of implicit bias requires novel approaches beyond current methodologies.

Title: DiffGraph: Heterogeneous Graph Diffusion Model

• Authors: Zongwei Li, Lianghao Xia, Hua Hua, Shijie Zhang, Shuangyang Wang, Chao Huang
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.02313
• Pdf URL: https://arxiv.org/pdf/2501.02313
• Copy Paste: [[2501.02313]] DiffGraph: Heterogeneous Graph Diffusion Model(https://arxiv.org/abs/2501.02313)
• Keywords: robust, diffusion
• Abstract: Recent advances in Graph Neural Networks (GNNs) have revolutionized graph-structured data modeling, yet traditional GNNs struggle with complex heterogeneous structures prevalent in real-world scenarios. Despite progress in handling heterogeneous interactions, two fundamental challenges persist: noisy data significantly compromising embedding quality and learning performance, and existing methods' inability to capture intricate semantic transitions among heterogeneous relations, which impacts downstream predictions. To address these fundamental issues, we present the Heterogeneous Graph Diffusion Model (DiffGraph), a pioneering framework that introduces an innovative cross-view denoising strategy. This advanced approach transforms auxiliary heterogeneous data into target semantic spaces, enabling precise distillation of task-relevant information. At its core, DiffGraph features a sophisticated latent heterogeneous graph diffusion mechanism, implementing a novel forward and backward diffusion process for superior noise management. This methodology achieves simultaneous heterogeneous graph denoising and cross-type transition, while significantly simplifying graph generation through its latent-space diffusion capabilities. Through rigorous experimental validation on both public and industrial datasets, we demonstrate that DiffGraph consistently surpasses existing methods in link prediction and node classification tasks, establishing new benchmarks for robustness and efficiency in heterogeneous graph processing. The model implementation is publicly available at: this https URL.

Title: SR-Reward: Taking The Path More Traveled

• Authors: Seyed Mahdi B. Azad, Zahra Padar, Gabriel Kalweit, Joschka Boedecker
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02330
• Pdf URL: https://arxiv.org/pdf/2501.02330
• Copy Paste: [[2501.02330]] SR-Reward: Taking The Path More Traveled(https://arxiv.org/abs/2501.02330)
• Keywords: robust
• Abstract: In this paper, we propose a novel method for learning reward functions directly from offline demonstrations. Unlike traditional inverse reinforcement learning (IRL), our approach decouples the reward function from the learner's policy, eliminating the adversarial interaction typically required between the two. This results in a more stable and efficient training process. Our reward function, called \textit{SR-Reward}, leverages successor representation (SR) to encode a state based on expected future states' visitation under the demonstration policy and transition dynamics. By utilizing the Bellman equation, SR-Reward can be learned concurrently with most reinforcement learning (RL) algorithms without altering the existing training pipeline. We also introduce a negative sampling strategy to mitigate overestimation errors by reducing rewards for out-of-distribution data, thereby enhancing robustness. This strategy inherently introduces a conservative bias into RL algorithms that employ the learned reward. We evaluate our method on the D4RL benchmark, achieving competitive results compared to offline RL algorithms with access to true rewards and imitation learning (IL) techniques like behavioral cloning. Moreover, our ablation studies on data size and quality reveal the advantages and limitations of SR-Reward as a proxy for true rewards.

Title: Validity Arguments For Constructed Response Scoring Using Generative Artificial Intelligence Applications

• Authors: Jodi M. Casabianca, Daniel F. McCaffrey, Matthew S. Johnson, Naim Alper, Vladimir Zubenko
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.02334
• Pdf URL: https://arxiv.org/pdf/2501.02334
• Copy Paste: [[2501.02334]] Validity Arguments For Constructed Response Scoring Using Generative Artificial Intelligence Applications(https://arxiv.org/abs/2501.02334)
• Keywords: generative, large language model
• Abstract: The rapid advancements in large language models and generative artificial intelligence (AI) capabilities are making their broad application in the high-stakes testing context more likely. Use of generative AI in the scoring of constructed responses is particularly appealing because it reduces the effort required for handcrafting features in traditional AI scoring and might even outperform those methods. The purpose of this paper is to highlight the differences in the feature-based and generative AI applications in constructed response scoring systems and propose a set of best practices for the collection of validity evidence to support the use and interpretation of constructed response scores from scoring systems using generative AI. We compare the validity evidence needed in scoring systems using human ratings, feature-based natural language processing AI scoring engines, and generative AI. The evidence needed in the generative AI context is more extensive than in the feature-based NLP scoring context because of the lack of transparency and other concerns unique to generative AI such as consistency. Constructed response score data from standardized tests demonstrate the collection of validity evidence for different types of scoring systems and highlights the numerous complexities and considerations when making a validity argument for these scores. In addition, we discuss how the evaluation of AI scores might include a consideration of how a contributory scoring approach combining multiple AI scores (from different sources) will cover more of the construct in the absence of human ratings.

Title: AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference

• Authors: Zhuomin He, Yizhen Yao, Pengfei Zuo, Bin Gao, Qinya Li, Zhenzhe Zheng, Fan Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02336
• Pdf URL: https://arxiv.org/pdf/2501.02336
• Copy Paste: [[2501.02336]] AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference(https://arxiv.org/abs/2501.02336)
• Keywords: large language model
• Abstract: Long-context large language models (LLMs) inference is increasingly critical, motivating a number of studies devoted to alleviating the substantial storage and computational costs in such scenarios. Layer-wise skipping methods are promising optimizations but rarely explored in long-context inference. We observe that existing layer-wise skipping strategies have several limitations when applied in long-context inference, including the inability to adapt to model and context variability, disregard for sublayer significance, and inapplicability for the prefilling phase. This paper proposes \sysname, an adaptive sublayer skipping method specifically designed for long-context inference. \sysname adaptively identifies less important layers by leveraging on-the-fly similarity information, enables sublayer-wise skipping, and accelerates both the prefilling and decoding phases. The effectiveness of \sysname is demonstrated through extensive experiments on various long-context benchmarks and models, showcasing its superior inference performance over existing baselines.

Title: Optimizing Small Language Models for In-Vehicle Function-Calling

• Authors: Yahya Sowti Khiabani, Farris Atif, Chieh Hsu, Sven Stahlmann, Tobias Michels, Sebastian Kramer, Benedikt Heidrich, M. Saquib Sarfraz, Julian Merten, Faezeh Tafazzoli
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.02342
• Pdf URL: https://arxiv.org/pdf/2501.02342
• Copy Paste: [[2501.02342]] Optimizing Small Language Models for In-Vehicle Function-Calling(https://arxiv.org/abs/2501.02342)
• Keywords: robust
• Abstract: We propose a holistic approach for deploying Small Language Models (SLMs) as function-calling agents within vehicles as edge devices, offering a more flexible and robust alternative to traditional rule-based systems. By leveraging SLMs, we simplify vehicle control mechanisms and enhance the user experience. Given the in-vehicle hardware constraints, we apply state-of-the-art model compression techniques, including structured pruning, healing, and quantization, ensuring that the model fits within the resource limitations while maintaining acceptable performance. Our work focuses on optimizing a representative SLM, Microsoft's Phi-3 mini, and outlines best practices for enabling embedded models, including compression, task-specific fine-tuning, and vehicle integration. We demonstrate that, despite significant reduction in model size which removes up to 2 billion parameters from the original model, our approach preserves the model's ability to handle complex in-vehicle tasks accurately and efficiently. Furthermore, by executing the model in a lightweight runtime environment, we achieve a generation speed of 11 tokens per second, making real-time, on-device inference feasible without hardware acceleration. Our results demonstrate the potential of SLMs to transform vehicle control systems, enabling more intuitive interactions between users and their vehicles for an enhanced driving experience.

Title: Thinking with Many Minds: Using Large Language Models for Multi-Perspective Problem-Solving

• Authors: Sanghyun Park, Boris Maciejovsky, Phanish Puranam
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.02348
• Pdf URL: https://arxiv.org/pdf/2501.02348
• Copy Paste: [[2501.02348]] Thinking with Many Minds: Using Large Language Models for Multi-Perspective Problem-Solving(https://arxiv.org/abs/2501.02348)
• Keywords: large language model
• Abstract: Complex problem-solving requires cognitive flexibility--the capacity to entertain multiple perspectives while preserving their distinctiveness. This flexibility replicates the "wisdom of crowds" within a single individual, allowing them to "think with many minds." While mental simulation enables imagined deliberation, cognitive constraints limit its effectiveness. We propose synthetic deliberation, a Large Language Model (LLM)-based method that simulates discourse between agents embodying diverse perspectives, as a solution. Using a custom GPT-based model, we showcase its benefits: concurrent processing of multiple viewpoints without cognitive degradation, parallel exploration of perspectives, and precise control over viewpoint synthesis. By externalizing the deliberative process and distributing cognitive labor between parallel search and integration, synthetic deliberation transcends mental simulation's limitations. This approach shows promise for strategic planning, policymaking, and conflict resolution.

Title: PM-Dedup: Secure Deduplication with Partial Migration from Cloud to Edge Servers

• Authors: Zhaokang Ke, Haoyu Gong, David H.C. Du
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.02350
• Pdf URL: https://arxiv.org/pdf/2501.02350
• Copy Paste: [[2501.02350]] PM-Dedup: Secure Deduplication with Partial Migration from Cloud to Edge Servers(https://arxiv.org/abs/2501.02350)
• Keywords: secure, security, attack
• Abstract: Currently, an increasing number of users and enterprises are storing their data in the cloud but do not fully trust cloud providers with their data in plaintext form. To address this concern, they encrypt their data before uploading it to the cloud. However, encryption with different keys means that even identical data will become different ciphertexts, making deduplication less effective. Encrypted deduplication avoids this issue by ensuring that identical data chunks generate the same ciphertext with content-based keys, enabling the cloud to efficiently identify and remove duplicates even in encrypted form. Current encrypted data deduplication work can be classified into two types: target-based and source-based. Target-based encrypted deduplication requires clients to upload all encrypted chunks (the basic unit of deduplication) to the cloud with high network bandwidth overhead. Source-based deduplication involves clients uploading fingerprints (hashes) of encrypted chunks for duplicate checking and only uploading unique encrypted chunks, which reduces network transfer but introduces high latency and potential side-channel attacks, which need to be mitigated by Proof of Ownership (PoW), and high computing overhead of the cloud. So, reducing the latency and the overheads of network and cloud while ensuring security has become a significant challenge for secure data deduplication in cloud storage. In response to this challenge, we present PM-Dedup, a novel secure source-based deduplication approach that relocates a portion of the deduplication checking process and PoW tasks from the cloud to the trusted execution environments (TEEs) in the client-side edge servers. We also propose various designs to enhance the security and efficiency of data deduplication.

Title: GNSS/GPS Spoofing and Jamming Identification Using Machine Learning and Deep Learning

• Authors: Ali Ghanbarzade, Hossein Soleimani
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02352
• Pdf URL: https://arxiv.org/pdf/2501.02352
• Copy Paste: [[2501.02352]] GNSS/GPS Spoofing and Jamming Identification Using Machine Learning and Deep Learning(https://arxiv.org/abs/2501.02352)
• Keywords: security, attack
• Abstract: The increasing reliance on Global Navigation Satellite Systems (GNSS), particularly the Global Positioning System (GPS), underscores the urgent need to safeguard these technologies against malicious threats such as spoofing and jamming. As the backbone for positioning, navigation, and timing (PNT) across various applications including transportation, telecommunications, and emergency services GNSS is vulnerable to deliberate interference that poses significant risks. Spoofing attacks, which involve transmitting counterfeit GNSS signals to mislead receivers into calculating incorrect positions, can result in serious consequences, from navigational errors in civilian aviation to security breaches in military operations. Furthermore, the lack of inherent security measures within GNSS systems makes them attractive targets for adversaries. While GNSS/GPS jamming and spoofing systems consist of numerous components, the ability to distinguish authentic signals from malicious ones is essential for maintaining system integrity. Recent advancements in machine learning and deep learning provide promising avenues for enhancing detection and mitigation strategies against these threats. This paper addresses both spoofing and jamming by tackling real-world challenges through machine learning, deep learning, and computer vision techniques. Through extensive experiments on two real-world datasets related to spoofing and jamming detection using advanced algorithms, we achieved state of the art results. In the GNSS/GPS jamming detection task, we attained approximately 99% accuracy, improving performance by around 5% compared to previous studies. Additionally, we addressed a challenging tasks related to spoofing detection, yielding results that underscore the potential of machine learning and deep learning in this domain.

Title: CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models

• Authors: Kuan-Hung Liu, Cheng-Kun Yang, Min-Hung Chen, Yu-Lun Liu, Yen-Yu Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02355
• Pdf URL: https://arxiv.org/pdf/2501.02355
• Copy Paste: [[2501.02355]] CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models(https://arxiv.org/abs/2501.02355)
• Keywords: diffusion
• Abstract: In the task of reference-based image inpainting, an additional reference image is provided to restore a damaged target image to its original state. The advancement of diffusion models, particularly Stable Diffusion, allows for simple formulations in this task. However, existing diffusion-based methods often lack explicit constraints on the correlation between the reference and damaged images, resulting in lower faithfulness to the reference images in the inpainting results. In this work, we propose CorrFill, a training-free module designed to enhance the awareness of geometric correlations between the reference and target images. This enhancement is achieved by guiding the inpainting process with correspondence constraints estimated during inpainting, utilizing attention masking in self-attention layers and an objective function to update the input tensor according to the constraints. Experimental results demonstrate that CorrFill significantly enhances the performance of multiple baseline diffusion-based methods, including state-of-the-art approaches, by emphasizing faithfulness to the reference images.

Title: Easing Optimization Paths: a Circuit Perspective

• Authors: Ambroise Odonnat, Wassim Bouaziz, Vivien Cabannes
• Subjects: cs.LG, eess.SP, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.02362
• Pdf URL: https://arxiv.org/pdf/2501.02362
• Copy Paste: [[2501.02362]] Easing Optimization Paths: a Circuit Perspective(https://arxiv.org/abs/2501.02362)
• Keywords: interpretability
• Abstract: Gradient descent is the method of choice for training large artificial intelligence systems. As these systems become larger, a better understanding of the mechanisms behind gradient training would allow us to alleviate compute costs and help steer these systems away from harmful behaviors. To that end, we suggest utilizing the circuit perspective brought forward by mechanistic interpretability. After laying out our intuition, we illustrate how it enables us to design a curriculum for efficient learning in a controlled setting. The code is available at \url{this https URL}.

Title: V2X-DGPE: Addressing Domain Gaps and Pose Errors for Robust Collaborative 3D Object Detection

• Authors: Sichao Wang, Chuang Zhang, Ming Yuan, Qing Xu, Lei He, Jianqiang Wang
• Subjects: cs.CV, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.02363
• Pdf URL: https://arxiv.org/pdf/2501.02363
• Copy Paste: [[2501.02363]] V2X-DGPE: Addressing Domain Gaps and Pose Errors for Robust Collaborative 3D Object Detection(https://arxiv.org/abs/2501.02363)
• Keywords: robust
• Abstract: In V2X collaborative perception, the domain gaps between heterogeneous nodes pose a significant challenge for effective information fusion. Pose errors arising from latency and GPS localization noise further exacerbate the issue by leading to feature misalignment. To overcome these challenges, we propose V2X-DGPE, a high-accuracy and robust V2X feature-level collaborative perception framework. V2X-DGPE employs a Knowledge Distillation Framework and a Feature Compensation Module to learn domain-invariant representations from multi-source data, effectively reducing the feature distribution gap between vehicles and roadside infrastructure. Historical information is utilized to provide the model with a more comprehensive understanding of the current scene. Furthermore, a Collaborative Fusion Module leverages a heterogeneous self-attention mechanism to extract and integrate heterogeneous representations from vehicles and infrastructure. To address pose errors, V2X-DGPE introduces a deformable attention mechanism, enabling the model to adaptively focus on critical parts of the input features by dynamically offsetting sampling points. Extensive experiments on the real-world DAIR-V2X dataset demonstrate that the proposed method outperforms existing approaches, achieving state-of-the-art detection performance. The code is available at this https URL.

Title: Understanding How Nonlinear Layers Create Linearly Separable Features for Low-Dimensional Data

• Authors: Alec S. Xu, Can Yaras, Peng Wang, Qing Qu
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.02364
• Pdf URL: https://arxiv.org/pdf/2501.02364
• Copy Paste: [[2501.02364]] Understanding How Nonlinear Layers Create Linearly Separable Features for Low-Dimensional Data(https://arxiv.org/abs/2501.02364)
• Keywords: interpretability
• Abstract: Deep neural networks have attained remarkable success across diverse classification tasks. Recent empirical studies have shown that deep networks learn features that are linearly separable across classes. However, these findings often lack rigorous justifications, even under relatively simple settings. In this work, we address this gap by examining the linear separation capabilities of shallow nonlinear networks. Specifically, inspired by the low intrinsic dimensionality of image data, we model inputs as a union of low-dimensional subspaces (UoS) and demonstrate that a single nonlinear layer can transform such data into linearly separable sets. Theoretically, we show that this transformation occurs with high probability when using random weights and quadratic activations. Notably, we prove this can be achieved when the network width scales polynomially with the intrinsic dimension of the data rather than the ambient dimension. Experimental results corroborate these theoretical findings and demonstrate that similar linear separation properties hold in practical scenarios beyond our analytical scope. This work bridges the gap between empirical observations and theoretical understanding of the separation capacity of nonlinear networks, offering deeper insights into model interpretability and generalization.

Title: Predicting two-dimensional spatiotemporal chaotic patterns with optimized high-dimensional hybrid reservoir computing

• Authors: Tamon Nakano Sebastian Baur Christoph Räth
• Subjects: cs.LG, nlin.CD
• Abstract URL: https://arxiv.org/abs/2501.02369
• Pdf URL: https://arxiv.org/pdf/2501.02369
• Copy Paste: [[2501.02369]] Predicting two-dimensional spatiotemporal chaotic patterns with optimized high-dimensional hybrid reservoir computing(https://arxiv.org/abs/2501.02369)
• Keywords: robust, interpretability
• Abstract: As an alternative approach for predicting complex dynamical systems where physics-based models are no longer reliable, reservoir computing (RC) has gained popularity. The hybrid approach is considered an interesting option for improving the prediction performance of RC. The idea is to combine a knowledge-based model (KBM) to support the fully data-driven RC prediction. There are three types of hybridization for RC, namely full hybrid (FH), input hybrid (IH) and output hybrid (OH), where it was shown that the latter one is superior in terms of the accuracy and the robustness for the prediction of low-dimensional chaotic systems. Here, we extend the formalism to the prediction of spatiotemporal patterns in two dimensions. To overcome the curse of dimensionality for this very high-dimensional case we employ the local states ansatz, where only a few locally adjacent time series are utilized for the RC-based prediction. Using simulation data from the Barkley model describing chaotic electrical wave propagation in cardiac tissue, we outline the formalism of high-dimensional hybrid RC and assess the performance of the different hybridization schemes. We find that all three methods (FH, IH and OH) perform better than reservoir only, where improvements are small when the model is very inaccurate. For small model errors and small reservoirs FH and OH perform nearly equally well and better than IH. Given the smaller CPU needs for OH and especially the better interpretability of it, OH is to be favored. For large reservoirs the performance of OH drops below that of FH and IH. Generally, it maybe advisable to test the three setups for a given application and select the best suited one that optimizes between the counteracting factors of prediction performance and CPU needs.

Title: Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison

• Authors: Tsz Kin Lam, Marco Gaido, Sara Papi, Luisa Bentivogli, Barry Haddow
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.02370
• Pdf URL: https://arxiv.org/pdf/2501.02370
• Copy Paste: [[2501.02370]] Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison(https://arxiv.org/abs/2501.02370)
• Keywords: large language model
• Abstract: Following the remarkable success of Large Language Models (LLMs) in NLP tasks, there is increasing interest in extending their capabilities to speech -- the most common form in communication. To integrate speech into LLMs, one promising approach is dense feature prepending (DFP) which prepends the projected speech representations to the textual representations, allowing end-to-end training with the speech encoder. However, DFP typically requires connecting a text decoder to a speech encoder. This raises questions about the importance of having a sophisticated speech encoder for DFP, and how its performance compares with a standard encoder-decoder (i.e. cross-attention) architecture. In order to perform a controlled architectural comparison, we train all models from scratch, rather than using large pretrained models, and use comparable data and parameter settings, testing speech-to-text recognition (ASR) and translation (ST) on MuST-C v1.0 and CoVoST2 datasets. We study the influence of a speech encoder in DFP. More importantly, we compare DFP and cross-attention under a variety of configurations, such as CTC compression, sequence-level knowledge distillation, generation speed and GPU memory footprint on monolingual, bilingual and multilingual models. Despite the prevalence of DFP over cross-attention, our overall results do not indicate a clear advantage of DFP.

Title: BADTV: Unveiling Backdoor Threats in Third-Party Task Vectors

• Authors: Chia-Yi Hsu, Yu-Lin Tsai, Yu Zhe, Yan-Lun Chen, Chih-Hsun Lin, Chia-Mu Yu, Yang Zhang, Chun-Ying Huang, Jun Sakuma
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.02373
• Pdf URL: https://arxiv.org/pdf/2501.02373
• Copy Paste: [[2501.02373]] BADTV: Unveiling Backdoor Threats in Third-Party Task Vectors(https://arxiv.org/abs/2501.02373)
• Keywords: secure, security, defense, attack, robust
• Abstract: Task arithmetic in large-scale pre-trained models enables flexible adaptation to diverse downstream tasks without extensive re-training. By leveraging task vectors (TVs), users can perform modular updates to pre-trained models through simple arithmetic operations like addition and subtraction. However, this flexibility introduces new security vulnerabilities. In this paper, we identify and evaluate the susceptibility of TVs to backdoor attacks, demonstrating how malicious actors can exploit TVs to compromise model integrity. By developing composite backdoors and eliminating redudant clean tasks, we introduce BadTV, a novel backdoor attack specifically designed to remain effective under task learning, forgetting, and analogies operations. Our extensive experiments reveal that BadTV achieves near-perfect attack success rates across various scenarios, significantly impacting the security of models using task arithmetic. We also explore existing defenses, showing that current methods fail to detect or mitigate BadTV. Our findings highlight the need for robust defense mechanisms to secure TVs in real-world applications, especially as TV services become more popular in machine-learning ecosystems.

Title: Generalizable Origin Identification for Text-Guided Image-to-Image Diffusion Models

• Authors: Wenhao Wang, Yifan Sun, Zongxin Yang, Zhentao Tan, Zhengdong Hu, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02376
• Pdf URL: https://arxiv.org/pdf/2501.02376
• Copy Paste: [[2501.02376]] Generalizable Origin Identification for Text-Guided Image-to-Image Diffusion Models(https://arxiv.org/abs/2501.02376)
• Keywords: diffusion
• Abstract: Text-guided image-to-image diffusion models excel in translating images based on textual prompts, allowing for precise and creative visual modifications. However, such a powerful technique can be misused for spreading misinformation, infringing on copyrights, and evading content tracing. This motivates us to introduce the task of origin IDentification for text-guided Image-to-image Diffusion models (ID$^2$), aiming to retrieve the original image of a given translated query. A straightforward solution to ID$^2$ involves training a specialized deep embedding model to extract and compare features from both query and reference images. However, due to visual discrepancy across generations produced by different diffusion models, this similarity-based approach fails when training on images from one model and testing on those from another, limiting its effectiveness in real-world applications. To solve this challenge of the proposed ID$^2$ task, we contribute the first dataset and a theoretically guaranteed method, both emphasizing generalizability. The curated dataset, OriPID, contains abundant Origins and guided Prompts, which can be used to train and test potential IDentification models across various diffusion models. In the method section, we first prove the existence of a linear transformation that minimizes the distance between the pre-trained Variational Autoencoder (VAE) embeddings of generated samples and their origins. Subsequently, it is demonstrated that such a simple linear transformation can be generalized across different diffusion models. Experimental results show that the proposed method achieves satisfying generalization performance, significantly surpassing similarity-based methods ($+31.6\%$ mAP), even those with generalization designs.

Title: Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations

• Authors: Kangyu Zhu, Ziyuan Qin, Huahui Yi, Zekun Jiang, Qicheng Lao, Shaoting Zhang, Kang Li
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02385
• Pdf URL: https://arxiv.org/pdf/2501.02385
• Copy Paste: [[2501.02385]] Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations(https://arxiv.org/abs/2501.02385)
• Keywords: large language model
• Abstract: With the recent advancements in vision-language models (VLMs) driven by large language models (LLMs), many researchers have focused on models that comprised of an image encoder, an image-to-language projection layer, and a text decoder architectures, leading to the emergence of works like LLava-Med. However, these works primarily operate at the whole-image level, aligning general information from 2D medical images without attending to finer details. As a result, these models often provide irrelevant or non-clinically valuable information while missing critical details. Medical vision-language tasks differ significantly from general images, particularly in their focus on fine-grained details, while excluding irrelevant content. General domain VLMs tend to prioritize global information due to their design, which compresses the entire image into a multi-token representation that is passed into the LLM decoder. Therefore, current VLMs all lack the capability to restrict their attention to particular areas. To address this critical issue in the medical domain, we introduce MedVP, an visual prompt generation and fine-tuning framework, which involves extract medical entities, generate visual prompts, and adapt datasets for visual prompt guided fine-tuning. To the best of our knowledge, this is the first work to explicitly introduce visual prompt into medical VLMs, and we successfully outperform recent state-of-the-art large models across multiple medical VQA datasets. Extensive experiments are conducted to analyze the impact of different visual prompt forms and how they contribute to performance improvement. The results demonstrate both the effectiveness and clinical significance of our approach

Title: Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformers

• Authors: Markus J. Buehler
• Subjects: cs.LG, cond-mat.mes-hall, cond-mat.mtrl-sci, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02393
• Pdf URL: https://arxiv.org/pdf/2501.02393
• Copy Paste: [[2501.02393]] Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformers(https://arxiv.org/abs/2501.02393)
• Keywords: transformer
• Abstract: We present an approach to modifying Transformer architectures by integrating graph-aware relational reasoning into the attention mechanism, merging concepts from graph neural networks and language modeling. Building on the inherent connection between attention and graph theory, we reformulate the Transformer's attention mechanism as a graph operation and propose Graph-Aware Isomorphic Attention. This method leverages advanced graph modeling strategies, including Graph Isomorphism Networks (GIN) and Principal Neighborhood Aggregation (PNA), to enrich the representation of relational structures. Our approach captures complex dependencies and generalizes across tasks, as evidenced by a reduced generalization gap and improved learning performance. Additionally, we expand the concept of graph-aware attention to introduce Sparse GIN-Attention, a fine-tuning approach that employs sparse GINs. By interpreting attention matrices as sparse adjacency graphs, this technique enhances the adaptability of pre-trained foundational models with minimal computational overhead, endowing them with graph-aware capabilities. Sparse GIN-Attention fine-tuning achieves improved training dynamics and better generalization compared to alternative methods like low-rank adaption (LoRA). We discuss latent graph-like structures within traditional attention mechanisms, offering a new lens through which Transformers can be understood. By evolving Transformers as hierarchical GIN models for relational reasoning. This perspective suggests profound implications for foundational model development, enabling the design of architectures that dynamically adapt to both local and global dependencies. Applications in bioinformatics, materials science, language modeling, and beyond could benefit from this synthesis of relational and sequential data modeling, setting the stage for interpretable and generalizable modeling strategies.

Title: Anonymization by Design of Language Modeling

• Authors: Antoine Boutet, Zakaria El Kazdam, Lucas Magnana, Helain Zimmermann
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02407
• Pdf URL: https://arxiv.org/pdf/2501.02407
• Copy Paste: [[2501.02407]] Anonymization by Design of Language Modeling(https://arxiv.org/abs/2501.02407)
• Keywords: privacy
• Abstract: Rapid advances in Natural Language Processing (NLP) have revolutionized many fields, including healthcare. However, these advances raise significant privacy concerns, especially when models specialized on sensitive data can memorize and then expose and regurgitate confidential information. This paper presents a privacy-by-design language modeling approach to address the problem of language models anonymization, and thus promote their sharing. Specifically, we propose both a Masking Language Modeling (MLM) methodology to specialize a BERT-like language model, and a Causal Language Modeling (CLM) methodology to specialize a GPT-like model that avoids the model from memorizing direct and indirect identifying information present in the training data. We have comprehensively evaluated our approaches using medical datasets and compared them against different baselines. Our results indicate that by avoiding memorizing both direct and indirect identifiers during model specialization, our masking and causal language modeling schemes offer the best tradeoff for maintaining high privacy while retaining high utility.

Title: Journey into Automation: Image-Derived Pavement Texture Extraction and Evaluation

• Authors: Bingjie Lu (1), Han-Cheng Dan (1), Yichen Zhang (1), Zhetao Huang (1) ((1) Central South University)
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02414
• Pdf URL: https://arxiv.org/pdf/2501.02414
• Copy Paste: [[2501.02414]] Journey into Automation: Image-Derived Pavement Texture Extraction and Evaluation(https://arxiv.org/abs/2501.02414)
• Keywords: extraction
• Abstract: Mean texture depth (MTD) is pivotal in assessing the skid resistance of asphalt pavements and ensuring road safety. This study focuses on developing an automated system for extracting texture features and evaluating MTD based on pavement images. The contributions of this work are threefold: firstly, it proposes an economical method to acquire three-dimensional (3D) pavement texture data; secondly, it enhances 3D image processing techniques and formulates features that represent various aspects of texture; thirdly, it establishes multivariate prediction models that link these features with MTD values. Validation results demonstrate that the Gradient Boosting Tree (GBT) model achieves remarkable prediction stability and accuracy (R2 = 0.9858), and field tests indicate the superiority of the proposed method over other techniques, with relative errors below 10%. This method offers a comprehensive end-to-end solution for pavement quality evaluation, from images input to MTD predictions output.

Title: FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance

• Authors: Haicheng Wang, Zhemeng Yu, Gabriele Spadaro, Chen Ju, Victor Quétu, Enzo Tartaglione
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02430
• Pdf URL: https://arxiv.org/pdf/2501.02430
• Copy Paste: [[2501.02430]] FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance(https://arxiv.org/abs/2501.02430)
• Keywords: large language model
• Abstract: Recently, Multi-modal Large Language Models (MLLMs) have shown remarkable effectiveness for multi-modal tasks due to their abilities to generate and understand cross-modal data. However, processing long sequences of visual tokens extracted from visual backbones poses a challenge for deployment in real-time applications. To address this issue, we introduce FOLDER, a simple yet effective plug-and-play module designed to reduce the length of the visual token sequence, mitigating both computational and memory demands during training and inference. Through a comprehensive analysis of the token reduction process, we analyze the information loss introduced by different reduction strategies and develop FOLDER to preserve key information while removing visual redundancy. We showcase the effectiveness of FOLDER by integrating it into the visual backbone of several MLLMs, significantly accelerating the inference phase. Furthermore, we evaluate its utility as a training accelerator or even performance booster for MLLMs. In both contexts, FOLDER achieves comparable or even better performance than the original models, while dramatically reducing complexity by removing up to 70% of visual tokens.

Title: An Analysis Framework for Understanding Deep Neural Networks Based on Network Dynamics

• Authors: Yuchen Lin, Yong Zhang, Sihan Feng, Hong Zhao
• Subjects: cs.LG, nlin.CD, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.02436
• Pdf URL: https://arxiv.org/pdf/2501.02436
• Copy Paste: [[2501.02436]] An Analysis Framework for Understanding Deep Neural Networks Based on Network Dynamics(https://arxiv.org/abs/2501.02436)
• Keywords: extraction
• Abstract: Advancing artificial intelligence demands a deeper understanding of the mechanisms underlying deep learning. Here, we propose a straightforward analysis framework based on the dynamics of learning models. Neurons are categorized into two modes based on whether their transformation functions preserve order. This categorization reveals how deep neural networks (DNNs) maximize information extraction by rationally allocating the proportion of neurons in different modes across deep layers. We further introduce the attraction basins of the training samples in both the sample vector space and the weight vector space to characterize the generalization ability of DNNs. This framework allows us to identify optimal depth and width configurations, providing a unified explanation for fundamental DNN behaviors such as the "flat minima effect," "grokking," and double descent phenomena. Our analysis extends to networks with depths up to 100 layers.

Title: Efficient Deployment of Large Language Models on Resource-constrained Devices

• Authors: Zhiwei Yao, Yang Xu, Hongli Xu, Yunming Liao, Zuan Xie
• Subjects: cs.LG, cs.AI, cs.CL, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.02438
• Pdf URL: https://arxiv.org/pdf/2501.02438
• Copy Paste: [[2501.02438]] Efficient Deployment of Large Language Models on Resource-constrained Devices(https://arxiv.org/abs/2501.02438)
• Keywords: privacy, federate, large language model
• Abstract: Deploying Large Language Models (LLMs) on resource-constrained (or weak) devices presents significant challenges due to limited resources and heterogeneous data distribution. To address the data concern, it is necessary to fine-tune LLMs using on-device private data for various downstream tasks. While Federated Learning (FL) offers a promising privacy-preserving solution, existing fine-tuning methods retain the original LLM size, leaving issues of high inference latency and excessive memory demands unresolved. Hence, we design FedSpine, an FL framework that combines Parameter- Efficient Fine-Tuning (PEFT) with structured pruning for efficient deployment of LLMs on resource-constrained devices. Specifically, FedSpine introduces an iterative process to prune and tune the parameters of LLMs. To mitigate the impact of device heterogeneity, an online Multi-Armed Bandit (MAB) algorithm is employed to adaptively determine different pruning ratios and LoRA ranks for heterogeneous devices without any prior knowledge of their computing and communication capabilities. As a result, FedSpine maintains higher inference accuracy while improving fine-tuning efficiency. Experimental results conducted on a physical platform with 80 devices demonstrate that FedSpine can speed up fine-tuning by 1.4$\times$-6.9$\times$ and improve final accuracy by 0.4%-4.5% under the same sparsity level compared to other baselines.

Title: Unsupervised Search for Ethnic Minorities' Medical Segmentation Training Set

• Authors: Yixiao Chen, Yue Yao, Ruining Yang, Md Zakir Hossain, Ashu Gupta, Tom Gedeon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02442
• Pdf URL: https://arxiv.org/pdf/2501.02442
• Copy Paste: [[2501.02442]] Unsupervised Search for Ethnic Minorities' Medical Segmentation Training Set(https://arxiv.org/abs/2501.02442)
• Keywords: segmentation
• Abstract: This article investigates the critical issue of dataset bias in medical imaging, with a particular emphasis on racial disparities caused by uneven population distribution in dataset collection. Our analysis reveals that medical segmentation datasets are significantly biased, primarily influenced by the demographic composition of their collection sites. For instance, Scanning Laser Ophthalmoscopy (SLO) fundus datasets collected in the United States predominantly feature images of White individuals, with minority racial groups underrepresented. This imbalance can result in biased model performance and inequitable clinical outcomes, particularly for minority populations. To address this challenge, we propose a novel training set search strategy aimed at reducing these biases by focusing on underrepresented racial groups. Our approach utilizes existing datasets and employs a simple greedy algorithm to identify source images that closely match the target domain distribution. By selecting training data that aligns more closely with the characteristics of minority populations, our strategy improves the accuracy of medical segmentation models on specific minorities, i.e., Black. Our experimental results demonstrate the effectiveness of this approach in mitigating bias. We also discuss the broader societal implications, highlighting how addressing these disparities can contribute to more equitable healthcare outcomes.

Title: RTLMarker: Protecting LLM-Generated RTL Copyright via a Hardware Watermarking Framework

• Authors: Kun Wang, Kaiyan Chang, Mengdi Wang, Xinqi Zou, Haobo Xu, Yinhe Han, Ying Wang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02446
• Pdf URL: https://arxiv.org/pdf/2501.02446
• Copy Paste: [[2501.02446]] RTLMarker: Protecting LLM-Generated RTL Copyright via a Hardware Watermarking Framework(https://arxiv.org/abs/2501.02446)
• Keywords: security, protect, watermark, large language model
• Abstract: Recent advances of large language models in the field of Verilog generation have raised several ethical and security concerns, such as code copyright protection and dissemination of malicious code. Researchers have employed watermarking techniques to identify codes generated by large language models. However, the existing watermarking works fail to protect RTL code copyright due to the significant syntactic and semantic differences between RTL code and software code in languages such as Python. This paper proposes a hardware watermarking framework RTLMarker that embeds watermarks into RTL code and deeper into the synthesized netlist. We propose a set of rule-based Verilog code transformations , ensuring the watermarked RTL code's syntactic and semantic correctness. In addition, we consider an inherent tradeoff between watermark transparency and watermark effectiveness and jointly optimize them. The results demonstrate RTLMarker's superiority over the baseline in RTL code watermarking.

Title: MedSegDiffNCA: Diffusion Models With Neural Cellular Automata for Skin Lesion Segmentation

• Authors: Avni Mittal, John Kalkhof, Anirban Mukhopadhyay, Arnav Bhavsar
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.02447
• Pdf URL: https://arxiv.org/pdf/2501.02447
• Copy Paste: [[2501.02447]] MedSegDiffNCA: Diffusion Models With Neural Cellular Automata for Skin Lesion Segmentation(https://arxiv.org/abs/2501.02447)
• Keywords: diffusion, segmentation
• Abstract: Denoising Diffusion Models (DDMs) are widely used for high-quality image generation and medical image segmentation but often rely on Unet-based architectures, leading to high computational overhead, especially with high-resolution images. This work proposes three NCA-based improvements for diffusion-based medical image segmentation. First, Multi-MedSegDiffNCA uses a multilevel NCA framework to refine rough noise estimates generated by lower level NCA models. Second, CBAM-MedSegDiffNCA incorporates channel and spatial attention for improved segmentation. Third, MultiCBAM-MedSegDiffNCA combines these methods with a new RGB channel loss for semantic guidance. Evaluations on Lesion segmentation show that MultiCBAM-MedSegDiffNCA matches Unet-based model performance with dice score of 87.84% while using 60-110 times fewer parameters, offering a more efficient solution for low resource medical settings.

Title: Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap

• Authors: Hyunwoo Ko, Guijin Son, Dasol Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02448
• Pdf URL: https://arxiv.org/pdf/2501.02448
• Copy Paste: [[2501.02448]] Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap(https://arxiv.org/abs/2501.02448)
• Keywords: large language model
• Abstract: Large language models (LLMs) demonstrate exceptional performance on complex reasoning tasks. However, despite their strong reasoning capabilities in high-resource languages (e.g., English and Chinese), a significant performance gap persists in other languages. To investigate this gap in Korean, we introduce HRM8K, a benchmark comprising 8,011 English-Korean parallel bilingual math problems. Through systematic analysis of model behaviors, we identify a key finding: these performance disparities stem primarily from difficulties in comprehending non-English inputs, rather than limitations in reasoning capabilities. Based on these findings, we propose UST (Understand, Solve, and Translate), a method that strategically uses English as an anchor for reasoning and solution generation. By fine-tuning the model on 130k synthetically generated data points, UST achieves a 10.91% improvement on the HRM8K benchmark and reduces the multilingual performance gap from 11.6% to 0.7%. Additionally, we show that improvements from UST generalize effectively to different Korean domains, demonstrating that capabilities acquired from machine-verifiable content can be generalized to other areas. We publicly release the benchmark, training dataset, and models.

Title: GCP: Guarded Collaborative Perception with Spatial-Temporal Aware Malicious Agent Detection

• Authors: Yihang Tao, Senkang Hu, Yue Hu, Haonan An, Hangcheng Cao, Yuguang Fang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02450
• Pdf URL: https://arxiv.org/pdf/2501.02450
• Copy Paste: [[2501.02450]] GCP: Guarded Collaborative Perception with Spatial-Temporal Aware Malicious Agent Detection(https://arxiv.org/abs/2501.02450)
• Keywords: defense, attack
• Abstract: Collaborative perception significantly enhances autonomous driving safety by extending each vehicle's perception range through message sharing among connected and autonomous vehicles. Unfortunately, it is also vulnerable to adversarial message attacks from malicious agents, resulting in severe performance degradation. While existing defenses employ hypothesis-and-verification frameworks to detect malicious agents based on single-shot outliers, they overlook temporal message correlations, which can be circumvented by subtle yet harmful perturbations in model input and output spaces. This paper reveals a novel blind area confusion (BAC) attack that compromises existing single-shot outlier-based detection methods. As a countermeasure, we propose GCP, a Guarded Collaborative Perception framework based on spatial-temporal aware malicious agent detection, which maintains single-shot spatial consistency through a confidence-scaled spatial concordance loss, while simultaneously examining temporal anomalies by reconstructing historical bird's eye view motion flows in low-confidence regions. We also employ a joint spatial-temporal Benjamini-Hochberg test to synthesize dual-domain anomaly results for reliable malicious agent detection. Extensive experiments demonstrate GCP's superior performance under diverse attack scenarios, achieving up to 34.69% improvements in AP@0.5 compared to the state-of-the-art CP defense strategies under BAC attacks, while maintaining consistent 5-8% improvements under other typical attacks. Code will be released at this https URL.

Title: Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applications

• Authors: Zhe Chen, Yusheng Liao, Shuyang Jiang, Pingjie Wang, Yiqiu Guo, Yanfeng Wang, Yu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02460
• Pdf URL: https://arxiv.org/pdf/2501.02460
• Copy Paste: [[2501.02460]] Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applications(https://arxiv.org/abs/2501.02460)
• Keywords: large language model
• Abstract: Large language models (LLMs) hold promise for addressing healthcare challenges but often generate hallucinations due to limited integration of medical knowledge. Incorporating external medical knowledge is therefore critical, especially considering the breadth and complexity of medical content, which necessitates effective multi-source knowledge acquisition. We address this challenge by framing it as a source planning problem, where the task is to formulate context-appropriate queries tailored to the attributes of diverse knowledge sources. Existing approaches either overlook source planning or fail to achieve it effectively due to misalignment between the model's expectation of the sources and their actual content. To bridge this gap, we present MedOmniKB, a comprehensive repository comprising multigenre and multi-structured medical knowledge sources. Leveraging these sources, we propose the Source Planning Optimisation (SPO) method, which enhances multi-source utilisation through explicit planning optimisation. Our approach involves enabling an expert model to explore and evaluate potential plans while training a smaller model to learn source alignment using positive and negative planning samples. Experimental results demonstrate that our method substantially improves multi-source planning performance, enabling the optimised small model to achieve state-of-the-art results in leveraging diverse medical knowledge sources.

Title: FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models

• Authors: Hui Lin, Chao Zhang, Danfeng Hong, Kexin Dong, Congcong Wen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02461
• Pdf URL: https://arxiv.org/pdf/2501.02461
• Copy Paste: [[2501.02461]] FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models(https://arxiv.org/abs/2501.02461)
• Keywords: privacy, federate
• Abstract: Remote sensing data is often distributed across multiple institutions, and due to privacy concerns and data-sharing restrictions, leveraging large-scale datasets in a centralized training framework is challenging. Federated learning offers a promising solution by enabling collaborative model training across distributed data sources without requiring data centralization. However, current Vision-Language Models (VLMs), which typically contain billions of parameters, pose significant communication challenges for traditional federated learning approaches based on model parameter updates, as they would incur substantial communication costs. In this paper, we propose FedRSCLIP, the first federated learning framework designed for remote sensing image classification based on a VLM, specifically CLIP. FedRSCLIP addresses the challenges of data heterogeneity and large-scale model transmission in federated environments by introducing Prompt Learning, which optimizes only a small set of tunable parameters. The framework introduces a dual-prompt mechanism, comprising Shared Prompts for global knowledge sharing and Private Prompts for client-specific adaptation. To maintain semantic coherence between shared and private prompts, we propose the Dual Prompt Alignment Constraint to balance global consistency and local adaptability across diverse client distributions. Additionally, to enhance cross-modal representation learning, we introduce the Cross-Modal Feature Alignment Constraint to align multimodal features between text and image prompts. To validate the effectiveness of our proposed model, we construct a Fed-RSIC dataset based on three existing remote sensing image classification datasets, specifically designed to simulate various federated learning configurations. Experimental results demonstrate the effectiveness and superiority of FedRSCLIP in remote sensing image classification.

Title: Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera

• Authors: Yuliang Guo, Sparsh Garg, S. Mahdi H. Miangoleh, Xinyu Huang, Liu Ren
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.02464
• Pdf URL: https://arxiv.org/pdf/2501.02464
• Copy Paste: [[2501.02464]] Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera(https://arxiv.org/abs/2501.02464)
• Keywords: robust
• Abstract: While recent depth estimation methods exhibit strong zero-shot generalization, achieving accurate metric depth across diverse camera types-particularly those with large fields of view (FoV) such as fisheye and 360-degree cameras-remains a significant challenge. This paper presents Depth Any Camera (DAC), a powerful zero-shot metric depth estimation framework that extends a perspective-trained model to effectively handle cameras with varying FoVs. The framework is designed to ensure that all existing 3D data can be leveraged, regardless of the specific camera types used in new applications. Remarkably, DAC is trained exclusively on perspective images but generalizes seamlessly to fisheye and 360-degree cameras without the need for specialized training data. DAC employs Equi-Rectangular Projection (ERP) as a unified image representation, enabling consistent processing of images with diverse FoVs. Its key components include a pitch-aware Image-to-ERP conversion for efficient online augmentation in ERP space, a FoV alignment operation to support effective training across a wide range of FoVs, and multi-resolution data augmentation to address resolution disparities between training and testing. DAC achieves state-of-the-art zero-shot metric depth estimation, improving delta-1 ($\delta_1$) accuracy by up to 50% on multiple fisheye and 360-degree datasets compared to prior metric depth foundation models, demonstrating robust generalization across camera types.

Title: DeTrack: In-model Latent Denoising Learning for Visual Object Tracking

• Authors: Xinyu Zhou, Jinglun Li, Lingyi Hong, Kaixun Jiang, Pinxue Guo, Weifeng Ge, Wenqiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02467
• Pdf URL: https://arxiv.org/pdf/2501.02467
• Copy Paste: [[2501.02467]] DeTrack: In-model Latent Denoising Learning for Visual Object Tracking(https://arxiv.org/abs/2501.02467)
• Keywords: robust, diffusion, transformer
• Abstract: Previous visual object tracking methods employ image-feature regression models or coordinate autoregression models for bounding box prediction. Image-feature regression methods heavily depend on matching results and do not utilize positional prior, while the autoregressive approach can only be trained using bounding boxes available in the training set, potentially resulting in suboptimal performance during testing with unseen data. Inspired by the diffusion model, denoising learning enhances the model's robustness to unseen data. Therefore, We introduce noise to bounding boxes, generating noisy boxes for training, thus enhancing model robustness on testing data. We propose a new paradigm to formulate the visual object tracking problem as a denoising learning process. However, tracking algorithms are usually asked to run in real-time, directly applying the diffusion model to object tracking would severely impair tracking speed. Therefore, we decompose the denoising learning process into every denoising block within a model, not by running the model multiple times, and thus we summarize the proposed paradigm as an in-model latent denoising learning process. Specifically, we propose a denoising Vision Transformer (ViT), which is composed of multiple denoising blocks. In the denoising block, template and search embeddings are projected into every denoising block as conditions. A denoising block is responsible for removing the noise in a predicted bounding box, and multiple stacked denoising blocks cooperate to accomplish the whole denoising process. Subsequently, we utilize image features and trajectory information to refine the denoised bounding box. Besides, we also utilize trajectory memory and visual memory to improve tracking stability. Experimental results validate the effectiveness of our approach, achieving competitive performance on several challenging datasets.

Title: Hengqin-RA-v1: Advanced Large Language Model for Diagnosis and Treatment of Rheumatoid Arthritis with Dataset based Traditional Chinese Medicine

• Authors: Yishen Liu, Shengda Luo, Zishao Zhong, Tongtong Wu, Jianguo Zhang, Peiyao Ou, Yong Liang, Liang Liu, Hudan Pan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02471
• Pdf URL: https://arxiv.org/pdf/2501.02471
• Copy Paste: [[2501.02471]] Hengqin-RA-v1: Advanced Large Language Model for Diagnosis and Treatment of Rheumatoid Arthritis with Dataset based Traditional Chinese Medicine(https://arxiv.org/abs/2501.02471)
• Keywords: large language model
• Abstract: Large language models (LLMs) primarily trained on English texts, often face biases and inaccuracies in Chinese contexts. Their limitations are pronounced in fields like Traditional Chinese Medicine (TCM), where cultural and clinical subtleties are vital, further hindered by a lack of domain-specific data, such as rheumatoid arthritis (RA). To address these issues, this paper introduces Hengqin-RA-v1, the first large language model specifically tailored for TCM with a focus on diagnosing and treating RA. We also present HQ-GCM-RA-C1, a comprehensive RA-specific dataset curated from ancient Chinese medical literature, classical texts, and modern clinical studies. This dataset empowers Hengqin-RA-v1 to deliver accurate and culturally informed responses, effectively bridging the gaps left by general-purpose models. Extensive experiments demonstrate that Hengqin-RA-v1 outperforms state-of-the-art models, even surpassing the diagnostic accuracy of TCM practitioners in certain cases.

Title: A Deep Positive-Negative Prototype Approach to Integrated Prototypical Discriminative Learning

• Authors: Ramin Zarei-Sabzevar, Ahad Harati
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02477
• Pdf URL: https://arxiv.org/pdf/2501.02477
• Copy Paste: [[2501.02477]] A Deep Positive-Negative Prototype Approach to Integrated Prototypical Discriminative Learning(https://arxiv.org/abs/2501.02477)
• Keywords: interpretability
• Abstract: This paper proposes a novel Deep Positive-Negative Prototype (DPNP) model that combines prototype-based learning (PbL) with discriminative methods to improve class compactness and separability in deep neural networks. While PbL traditionally emphasizes interpretability by classifying samples based on their similarity to representative prototypes, it struggles with creating optimal decision boundaries in complex scenarios. Conversely, discriminative methods effectively separate classes but often lack intuitive interpretability. Toward exploiting advantages of these two approaches, the suggested DPNP model bridges between them by unifying class prototypes with weight vectors, thereby establishing a structured latent space that enables accurate classification using interpretable prototypes alongside a properly learned feature representation. Based on this central idea of unified prototype-weight representation, Deep Positive Prototype (DPP) is formed in the latent space as a representative for each class using off-the-shelf deep networks as feature extractors. Then, rival neighboring class DPPs are treated as implicit negative prototypes with repulsive force in DPNP, which push away DPPs from each other. This helps to enhance inter-class separation without the need for any extra parameters. Hence, through a novel loss function that integrates cross-entropy, prototype alignment, and separation terms, DPNP achieves well-organized feature space geometry, maximizing intra-class compactness and inter-class margins. We show that DPNP can organize prototypes in nearly regular positions within feature space, such that it is possible to achieve competitive classification accuracy even in much lower-dimensional feature spaces. Experimental results on several datasets demonstrate that DPNP outperforms state-of-the-art models, while using smaller networks.

Title: Decoding News Bias: Multi Bias Detection in News Articles

• Authors: Bhushan Santosh Shah, Deven Santosh Shah, Vahida Attar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02482
• Pdf URL: https://arxiv.org/pdf/2501.02482
• Copy Paste: [[2501.02482]] Decoding News Bias: Multi Bias Detection in News Articles(https://arxiv.org/abs/2501.02482)
• Keywords: large language model
• Abstract: News Articles provides crucial information about various events happening in the society but they unfortunately come with different kind of biases. These biases can significantly distort public opinion and trust in the media, making it essential to develop techniques to detect and address them. Previous works have majorly worked towards identifying biases in particular domains e.g., Political, gender biases. However, more comprehensive studies are needed to detect biases across diverse domains. Large language models (LLMs) offer a powerful way to analyze and understand natural language, making them ideal for constructing datasets and detecting these biases. In this work, we have explored various biases present in the news articles, built a dataset using LLMs and present results obtained using multiple detection techniques. Our approach highlights the importance of broad-spectrum bias detection and offers new insights for improving the integrity of news articles.

Title: ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling

• Authors: Chaojie Mao, Jingfeng Zhang, Yulin Pan, Zeyinzi Jiang, Zhen Han, Yu Liu, Jingren Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02487
• Pdf URL: https://arxiv.org/pdf/2501.02487
• Copy Paste: [[2501.02487]] ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling(https://arxiv.org/abs/2501.02487)
• Keywords: diffusion, generative
• Abstract: We report ACE++, an instruction-based diffusion framework that tackles various image generation and editing tasks. Inspired by the input format for the inpainting task proposed by FLUX.1-Fill-dev, we improve the Long-context Condition Unit (LCU) introduced in ACE and extend this input paradigm to any editing and generation tasks. To take full advantage of image generative priors, we develop a two-stage training scheme to minimize the efforts of finetuning powerful text-to-image diffusion models like FLUX.1-dev. In the first stage, we pre-train the model using task data with the 0-ref tasks from the text-to-image model. There are many models in the community based on the post-training of text-to-image foundational models that meet this training paradigm of the first stage. For example, FLUX.1-Fill-dev deals primarily with painting tasks and can be used as an initialization to accelerate the training process. In the second stage, we finetune the above model to support the general instructions using all tasks defined in ACE. To promote the widespread application of ACE++ in different scenarios, we provide a comprehensive set of models that cover both full finetuning and lightweight finetuning, while considering general applicability and applicability in vertical scenarios. The qualitative analysis showcases the superiority of ACE++ in terms of generating image quality and prompt following ability.

Title: Predicting Vulnerability to Malware Using Machine Learning Models: A Study on Microsoft Windows Machines

• Authors: Marzieh Esnaashari, Nima Moradi
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02493
• Pdf URL: https://arxiv.org/pdf/2501.02493
• Copy Paste: [[2501.02493]] Predicting Vulnerability to Malware Using Machine Learning Models: A Study on Microsoft Windows Machines(https://arxiv.org/abs/2501.02493)
• Keywords: security
• Abstract: In an era of escalating cyber threats, malware poses significant risks to individuals and organizations, potentially leading to data breaches, system failures, and substantial financial losses. This study addresses the urgent need for effective malware detection strategies by leveraging Machine Learning (ML) techniques on extensive datasets collected from Microsoft Windows Defender. Our research aims to develop an advanced ML model that accurately predicts malware vulnerabilities based on the specific conditions of individual machines. Moving beyond traditional signature-based detection methods, we incorporate historical data and innovative feature engineering to enhance detection capabilities. This study makes several contributions: first, it advances existing malware detection techniques by employing sophisticated ML algorithms; second, it utilizes a large-scale, real-world dataset to ensure the applicability of findings; third, it highlights the importance of feature analysis in identifying key indicators of malware infections; and fourth, it proposes models that can be adapted for enterprise environments, offering a proactive approach to safeguarding extensive networks against emerging threats. We aim to improve cybersecurity resilience, providing critical insights for practitioners in the field and addressing the evolving challenges posed by malware in a digital landscape. Finally, discussions on results, insights, and conclusions are presented.

Title: ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use

• Authors: Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, Jiechao Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02506
• Pdf URL: https://arxiv.org/pdf/2501.02506
• Copy Paste: [[2501.02506]] ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use(https://arxiv.org/abs/2501.02506)
• Keywords: large language model
• Abstract: Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models (LLMs). However, progress has been hindered by a lack of reliable evaluation datasets. To address this, we present ToolHop, a dataset comprising 995 user queries and 3,912 associated tools, specifically designed for rigorous evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach that includes tool creation, document refinement, and code generation. We evaluate 14 LLMs across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and GPT), uncovering significant challenges in handling multi-hop tool-use scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%, underscoring substantial room for improvement. Further analysis reveals variations in tool-use strategies for various families, offering actionable insights to guide the development of more effective approaches. Code and data can be found in this https URL.

Title: CHAIR-Classifier of Hallucination as Improver

• Authors: Ao Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02518
• Pdf URL: https://arxiv.org/pdf/2501.02518
• Copy Paste: [[2501.02518]] CHAIR-Classifier of Hallucination as Improver(https://arxiv.org/abs/2501.02518)
• Keywords: large language model
• Abstract: This paper presents a supervised method for detecting hallucinations in large language models. By analyzing token scores (logitis) across layers of the LLaMA model, we derive a small set, aiming to reduce overfitting, of features-including maximum, minimum, mean, standard deviation, and slope. We use logistic regression for classification and validate the model on the TruthfulQA and MMLU datasets. The results demonstrate significant performance gains, especially in zero-shot scenarios, highlighting the effectiveness and potential for generalization.

Title: Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors

• Authors: Minglin Chen, Longguang Wang, Sheng Ao, Ye Zhang, Kai Xu, Yulan Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02519
• Pdf URL: https://arxiv.org/pdf/2501.02519
• Copy Paste: [[2501.02519]] Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors(https://arxiv.org/abs/2501.02519)
• Keywords: diffusion
• Abstract: 3D scene generation conditioned on text prompts has significantly progressed due to the development of 2D diffusion generation models. However, the textual description of 3D scenes is inherently inaccurate and lacks fine-grained control during training, leading to implausible scene generation. As an intuitive and feasible solution, the 3D layout allows for precise specification of object locations within the scene. To this end, we present a text-to-scene generation method (namely, Layout2Scene) using additional semantic layout as the prompt to inject precise control of 3D object positions. Specifically, we first introduce a scene hybrid representation to decouple objects and backgrounds, which is initialized via a pre-trained text-to-3D model. Then, we propose a two-stage scheme to optimize the geometry and appearance of the initialized scene separately. To fully leverage 2D diffusion priors in geometry and appearance generation, we introduce a semantic-guided geometry diffusion model and a semantic-geometry guided diffusion model which are finetuned on a scene dataset. Extensive experiments demonstrate that our method can generate more plausible and realistic scenes as compared to state-of-the-art approaches. Furthermore, the generated scene allows for flexible yet precise editing, thereby facilitating multiple downstream applications.

Title: Predicting IoT Device Vulnerability Fix Times with Survival and Failure Time Models

• Authors: Carlos A Rivera A, Xinzhang Chen, Arash Shaghaghi, Gustavo Batista, Salil Kanhere
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.02520
• Pdf URL: https://arxiv.org/pdf/2501.02520
• Copy Paste: [[2501.02520]] Predicting IoT Device Vulnerability Fix Times with Survival and Failure Time Models(https://arxiv.org/abs/2501.02520)
• Keywords: security
• Abstract: The rapid integration of Internet of Things (IoT) devices into enterprise environments presents significant security challenges. Many IoT devices are released to the market with minimal security measures, often harbouring an average of 25 vulnerabilities per device. To enhance cybersecurity measures and aid system administrators in managing IoT patches more effectively, we propose an innovative framework that predicts the time it will take for a vulnerable IoT device to receive a fix or patch. We developed a survival analysis model based on the Accelerated Failure Time (AFT) approach, implemented using the XGBoost ensemble regression model, to predict when vulnerable IoT devices will receive fixes or patches. By constructing a comprehensive IoT vulnerabilities database that combines public and private sources, we provide insights into affected devices, vulnerability detection dates, published CVEs, patch release dates, and associated Twitter activity trends. We conducted thorough experiments evaluating different combinations of features, including fundamental device and vulnerability data, National Vulnerability Database (NVD) information such as CVE, CWE, and CVSS scores, transformed textual descriptions into sentence vectors, and the frequency of Twitter trends related to CVEs. Our experiments demonstrate that the proposed model accurately predicts the time to fix for IoT vulnerabilities, with data from VulDB and NVD proving particularly effective. Incorporating Twitter trend data offered minimal additional benefit. This framework provides a practical tool for organisations to anticipate vulnerability resolutions, improve IoT patch management, and strengthen their cybersecurity posture against potential threats.

Title: Face-MakeUp: Multimodal Facial Prompts for Text-to-Image Generation

• Authors: Dawei Dai, Mingming Jia, Yinxiu Zhou, Hang Xing, Chenghang Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02523
• Pdf URL: https://arxiv.org/pdf/2501.02523
• Copy Paste: [[2501.02523]] Face-MakeUp: Multimodal Facial Prompts for Text-to-Image Generation(https://arxiv.org/abs/2501.02523)
• Keywords: diffusion
• Abstract: Facial images have extensive practical applications. Although the current large-scale text-image diffusion models exhibit strong generation capabilities, it is challenging to generate the desired facial images using only text prompt. Image prompts are a logical choice. However, current methods of this type generally focus on general domain. In this paper, we aim to optimize image makeup techniques to generate the desired facial images. Specifically, (1) we built a dataset of 4 million high-quality face image-text pairs (FaceCaptionHQ-4M) based on LAION-Face to train our Face-MakeUp model; (2) to maintain consistency with the reference facial image, we extract/learn multi-scale content features and pose features for the facial image, integrating these into the diffusion model to enhance the preservation of facial identity features for diffusion models. Validation on two face-related test datasets demonstrates that our Face-MakeUp can achieve the best comprehensive this http URL codes are available at:this https URL

Title: Vision-Driven Prompt Optimization for Large Language Models in Multimodal Generative Tasks

• Authors: Leo Franklin, Apiradee Boonmee, Kritsada Wongsuwan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02527
• Pdf URL: https://arxiv.org/pdf/2501.02527
• Copy Paste: [[2501.02527]] Vision-Driven Prompt Optimization for Large Language Models in Multimodal Generative Tasks(https://arxiv.org/abs/2501.02527)
• Keywords: robust, generative, large language model
• Abstract: Vision generation remains a challenging frontier in artificial intelligence, requiring seamless integration of visual understanding and generative capabilities. In this paper, we propose a novel framework, Vision-Driven Prompt Optimization (VDPO), that leverages Large Language Models (LLMs) to dynamically generate textual prompts from visual inputs, guiding high-fidelity image synthesis. VDPO combines a visual embedding prompt tuner, a textual instruction generator, and a vision generation module to achieve state-of-the-art performance in diverse vision generation tasks. Extensive experiments on benchmarks such as COCO and Sketchy demonstrate that VDPO consistently outperforms existing methods, achieving significant improvements in FID, LPIPS, and BLEU/CIDEr scores. Additional analyses reveal the scalability, robustness, and generalization capabilities of VDPO, making it a versatile solution for in-domain and out-of-domain tasks. Human evaluations further validate the practical superiority of VDPO in generating visually appealing and semantically coherent outputs.

Title: Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasm

• Authors: Ljubisa Bojic, Olga Zagovora, Asta Zelenkauskaite, Vuk Vukovic, Milan Cabarkapa, Selma Veseljević Jerkovic, Ana Jovančevic
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.02532
• Pdf URL: https://arxiv.org/pdf/2501.02532
• Copy Paste: [[2501.02532]] Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasm(https://arxiv.org/abs/2501.02532)
• Keywords: large language model
• Abstract: In the era of rapid digital communication, vast amounts of textual data are generated daily, demanding efficient methods for latent content analysis to extract meaningful insights. Large Language Models (LLMs) offer potential for automating this process, yet comprehensive assessments comparing their performance to human annotators across multiple dimensions are lacking. This study evaluates the reliability, consistency, and quality of seven state-of-the-art LLMs, including variants of OpenAI's GPT-4, Gemini, Llama, and Mixtral, relative to human annotators in analyzing sentiment, political leaning, emotional intensity, and sarcasm detection. A total of 33 human annotators and eight LLM variants assessed 100 curated textual items, generating 3,300 human and 19,200 LLM annotations, with LLMs evaluated across three time points to examine temporal consistency. Inter-rater reliability was measured using Krippendorff's alpha, and intra-class correlation coefficients assessed consistency over time. The results reveal that both humans and LLMs exhibit high reliability in sentiment analysis and political leaning assessments, with LLMs demonstrating higher internal consistency than humans. In emotional intensity, LLMs displayed higher agreement compared to humans, though humans rated emotional intensity significantly higher. Both groups struggled with sarcasm detection, evidenced by low agreement. LLMs showed excellent temporal consistency across all dimensions, indicating stable performance over time. This research concludes that LLMs, especially GPT-4, can effectively replicate human analysis in sentiment and political leaning, although human expertise remains essential for emotional intensity interpretation. The findings demonstrate the potential of LLMs for consistent and high-quality performance in certain areas of latent content analysis.

Title: A completely uniform transformer for parity

• Authors: Alexander Kozachinskiy, Tomasz Steifer
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02535
• Pdf URL: https://arxiv.org/pdf/2501.02535
• Copy Paste: [[2501.02535]] A completely uniform transformer for parity(https://arxiv.org/abs/2501.02535)
• Keywords: transformer
• Abstract: We construct a 3-layer constant-dimension transformer, recognizing the parity language, where neither parameter matrices nor the positional encoding depend on the input length. This improves upon a construction of Chiang and Cholak who use a positional encoding, depending on the input length (but their construction has 2 layers).

Title: Multi-LLM Collaborative Caption Generation in Scientific Documents

• Authors: Jaeyoung Kim, Jongho Lee, Hong-Jun Choi, Ting-Yao Hsu, Chieh-Yang Huang, Sungchul Kim, Ryan Rossi, Tong Yu, Clyde Lee Giles, Ting-Hao 'Kenneth' Huang, Sungchul Choi
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02552
• Pdf URL: https://arxiv.org/pdf/2501.02552
• Copy Paste: [[2501.02552]] Multi-LLM Collaborative Caption Generation in Scientific Documents(https://arxiv.org/abs/2501.02552)
• Keywords: large language model
• Abstract: Scientific figure captioning is a complex task that requires generating contextually appropriate descriptions of visual content. However, existing methods often fall short by utilizing incomplete information, treating the task solely as either an image-to-text or text summarization problem. This limitation hinders the generation of high-quality captions that fully capture the necessary details. Moreover, existing data sourced from arXiv papers contain low-quality captions, posing significant challenges for training large language models (LLMs). In this paper, we introduce a framework called Multi-LLM Collaborative Figure Caption Generation (MLBCAP) to address these challenges by leveraging specialized LLMs for distinct sub-tasks. Our approach unfolds in three key modules: (Quality Assessment) We utilize multimodal LLMs to assess the quality of training data, enabling the filtration of low-quality captions. (Diverse Caption Generation) We then employ a strategy of fine-tuning/prompting multiple LLMs on the captioning task to generate candidate captions. (Judgment) Lastly, we prompt a prominent LLM to select the highest quality caption from the candidates, followed by refining any remaining inaccuracies. Human evaluations demonstrate that informative captions produced by our approach rank better than human-written captions, highlighting its effectiveness. Our code is available at this https URL

Title: Decoding fMRI Data into Captions using Prefix Language Modeling

• Authors: Vyacheslav Shen, Kassymzhomart Kunanbayev, Dae-Shik Kim
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02570
• Pdf URL: https://arxiv.org/pdf/2501.02570
• Copy Paste: [[2501.02570]] Decoding fMRI Data into Captions using Prefix Language Modeling(https://arxiv.org/abs/2501.02570)
• Keywords: diffusion
• Abstract: With the advancements in Large Language and Latent Diffusion models, brain decoding has achieved remarkable results in recent years. The works on the NSD dataset, with stimuli images from the COCO dataset, leverage the embeddings from the CLIP model for image reconstruction and GIT for captioning. However, the current captioning approach introduces the challenge of potential data contamination given that the GIT model was trained on the COCO dataset. In this work, we present an alternative method for decoding brain signals into image captions by predicting a DINOv2 model's embedding of an image from the corresponding fMRI signal and then providing its [CLS] token as the prefix to the GPT-2 language model which decreases computational requirements considerably. Additionally, instead of commonly used Linear Regression, we explore 3D Convolutional Neural Network mapping of fMRI signals to image embedding space for better accounting positional information of voxels.

Title: LeetDecoding: A PyTorch Library for Exponentially Decaying Causal Linear Attention with CUDA Implementations

• Authors: Jiaping Wang, Simiao Zhang, Qiao-Chu He, Yifan Chen
• Subjects: cs.LG, cs.CL, cs.MS
• Abstract URL: https://arxiv.org/abs/2501.02573
• Pdf URL: https://arxiv.org/pdf/2501.02573
• Copy Paste: [[2501.02573]] LeetDecoding: A PyTorch Library for Exponentially Decaying Causal Linear Attention with CUDA Implementations(https://arxiv.org/abs/2501.02573)
• Keywords: transformer, generative, large language model
• Abstract: The machine learning and data science community has made significant while dispersive progress in accelerating transformer-based large language models (LLMs), and one promising approach is to replace the original causal attention in a generative pre-trained transformer (GPT) with \emph{exponentially decaying causal linear attention}. In this paper, we present LeetDecoding, which is the first Python package that provides a large set of computation routines for this fundamental operator. The launch of LeetDecoding was motivated by the current lack of (1) clear understanding of the complexity regarding this operator, (2) a comprehensive collection of existing computation methods (usually spread in seemingly unrelated fields), and (3) CUDA implementations for fast inference on GPU. LeetDecoding's design is easy to integrate with existing linear-attention LLMs, and allows for researchers to benchmark and evaluate new computation methods for exponentially decaying causal linear attention. The usage of LeetDecoding does not require any knowledge of GPU programming and the underlying complexity analysis, intentionally making LeetDecoding accessible to LLM practitioners. The source code of LeetDecoding is provided at \href{this https URL}{this GitHub repository}, and users can simply install LeetDecoding by the command \texttt{pip install leet-decoding}.

Title: DepthMaster: Taming Diffusion Models for Monocular Depth Estimation

• Authors: Ziyang Song, Zerong Wang, Bo Li, Hao Zhang, Ruijie Zhu, Li Liu, Peng-Tao Jiang, Tianzhu Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02576
• Pdf URL: https://arxiv.org/pdf/2501.02576
• Copy Paste: [[2501.02576]] DepthMaster: Taming Diffusion Models for Monocular Depth Estimation(https://arxiv.org/abs/2501.02576)
• Keywords: diffusion, generative
• Abstract: Monocular depth estimation within the diffusion-denoising paradigm demonstrates impressive generalization ability but suffers from low inference speed. Recent methods adopt a single-step deterministic paradigm to improve inference efficiency while maintaining comparable performance. However, they overlook the gap between generative and discriminative features, leading to suboptimal results. In this work, we propose DepthMaster, a single-step diffusion model designed to adapt generative features for the discriminative depth estimation task. First, to mitigate overfitting to texture details introduced by generative features, we propose a Feature Alignment module, which incorporates high-quality semantic features to enhance the denoising network's representation capability. Second, to address the lack of fine-grained details in the single-step deterministic framework, we propose a Fourier Enhancement module to adaptively balance low-frequency structure and high-frequency details. We adopt a two-stage training strategy to fully leverage the potential of the two modules. In the first stage, we focus on learning the global scene structure with the Feature Alignment module, while in the second stage, we exploit the Fourier Enhancement module to improve the visual quality. Through these efforts, our model achieves state-of-the-art performance in terms of generalization and detail preservation, outperforming other diffusion-based methods across various datasets. Our project page can be found at this https URL.

Title: GIT-CXR: End-to-End Transformer for Chest X-Ray Report Generation

• Authors: Iustin Sîrbu, Iulia-Renata Sîrbu, Jasmina Bogojeska, Traian Rebedea
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02598
• Pdf URL: https://arxiv.org/pdf/2501.02598
• Copy Paste: [[2501.02598]] GIT-CXR: End-to-End Transformer for Chest X-Ray Report Generation(https://arxiv.org/abs/2501.02598)
• Keywords: transformer
• Abstract: Medical imaging is crucial for diagnosing, monitoring, and treating medical conditions. The medical reports of radiology images are the primary medium through which medical professionals attest their findings, but their writing is time consuming and requires specialized clinical expertise. The automated generation of radiography reports has thus the potential to improve and standardize patient care and significantly reduce clinicians workload. Through our work, we have designed and evaluated an end-to-end transformer-based method to generate accurate and factually complete radiology reports for X-ray images. Additionally, we are the first to introduce curriculum learning for end-to-end transformers in medical imaging and demonstrate its impact in obtaining improved performance. The experiments have been conducted using the MIMIC-CXR-JPG database, the largest available chest X-ray dataset. The results obtained are comparable with the current state-of-the-art on the natural language generation (NLG) metrics BLEU and ROUGE-L, while setting new state-of-the-art results on F1 examples-averaged, F1-macro and F1-micro metrics for clinical accuracy and on the METEOR metric widely used for NLG.

Title: Empowering Bengali Education with AI: Solving Bengali Math Word Problems through Transformer Models

• Authors: Jalisha Jashim Era, Bidyarthi Paul, Tahmid Sattar Aothoi, Mirazur Rahman Zim, Faisal Muhammad Shah
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02599
• Pdf URL: https://arxiv.org/pdf/2501.02599
• Copy Paste: [[2501.02599]] Empowering Bengali Education with AI: Solving Bengali Math Word Problems through Transformer Models(https://arxiv.org/abs/2501.02599)
• Keywords: transformer
• Abstract: Mathematical word problems (MWPs) involve the task of converting textual descriptions into mathematical equations. This poses a significant challenge in natural language processing, particularly for low-resource languages such as Bengali. This paper addresses this challenge by developing an innovative approach to solving Bengali MWPs using transformer-based models, including Basic Transformer, mT5, BanglaT5, and mBART50. To support this effort, the "PatiGonit" dataset was introduced, containing 10,000 Bengali math problems, and these models were fine-tuned to translate the word problems into equations accurately. The evaluation revealed that the mT5 model achieved the highest accuracy of 97.30%, demonstrating the effectiveness of transformer models in this domain. This research marks a significant step forward in Bengali natural language processing, offering valuable methodologies and resources for educational AI tools. By improving math education, it also supports the development of advanced problem-solving skills for Bengali-speaking students.

Title: Chameleon2++: An Efficient Chameleon2 Clustering with Approximate Nearest Neighbors

• Authors: Priyanshu Singh, Kapil Ahuja
• Subjects: cs.LG, cs.DS
• Abstract URL: https://arxiv.org/abs/2501.02612
• Pdf URL: https://arxiv.org/pdf/2501.02612
• Copy Paste: [[2501.02612]] Chameleon2++: An Efficient Chameleon2 Clustering with Approximate Nearest Neighbors(https://arxiv.org/abs/2501.02612)
• Keywords: robust
• Abstract: Clustering algorithms are fundamental tools in data analysis, with hierarchical methods being particularly valuable for their flexibility. Chameleon is a widely used hierarchical clustering algorithm that excels at identifying high-quality clusters of arbitrary shapes, sizes, and densities. Chameleon2 is the most recent variant that has demonstrated significant improvements, but suffers from critical failings and there are certain improvements that can be made. The first failure we address is that the complexity of Chameleon2 is claimed to be $O(n^2)$, while we demonstrate that it is actually $O(n^2\log{n})$, with $n$ being the number of data points. Furthermore, we suggest improvements to Chameleon2 that ensure that the complexity remains $O(n^2)$ with minimal to no loss of performance. The second failing of Chameleon2 is that it lacks transparency and it does not provide the fine-tuned algorithm parameters used to obtain the claimed results. We meticulously provide all such parameter values to enhance replicability. The improvement which we make in Chameleon2 is that we replace the exact $k$-NN search with an approximate $k$-NN search. This further reduces the algorithmic complexity down to $O(n\log{n})$ without any performance loss. Here, we primarily configure three approximate nearest neighbor search algorithms (Annoy, FLANN and NMSLIB) to align with the overarching Chameleon2 clustering framework. Experimental evaluations on standard benchmark datasets demonstrate that the proposed Chameleon2++ algorithm is more efficient, robust, and computationally optimal.

Title: HALO: Hadamard-Assisted Lossless Optimization for Efficient Low-Precision LLM Training and Fine-Tuning

• Authors: Saleh Ashkboos, Mahdi Nikdan, Soroush Tabesh, Roberto L. Castro, Torsten Hoefler, Dan Alistarh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02625
• Pdf URL: https://arxiv.org/pdf/2501.02625
• Copy Paste: [[2501.02625]] HALO: Hadamard-Assisted Lossless Optimization for Efficient Low-Precision LLM Training and Fine-Tuning(https://arxiv.org/abs/2501.02625)
• Keywords: transformer, large language model
• Abstract: Quantized training of Large Language Models (LLMs) remains an open challenge, as maintaining accuracy while performing all matrix multiplications in low precision has proven difficult. This is particularly the case when fine-tuning pre-trained models, which often already have large weight and activation outlier values that render quantized optimization difficult. We present HALO, a novel quantization-aware training approach for Transformers that enables accurate and efficient low-precision training by combining 1) strategic placement of Hadamard rotations in both forward and backward passes, to mitigate outliers during the low-precision computation, 2) FSDP integration for low-precision communication, and 3) high-performance kernel support. Our approach ensures that all large matrix multiplications during the forward and backward passes are executed in lower precision. Applied to LLAMA-family models, HALO achieves near-full-precision-equivalent results during fine-tuning on various tasks, while delivering up to 1.31x end-to-end speedup for full fine-tuning on RTX 4090 GPUs. Our method supports both standard and parameter-efficient fine-tuning (PEFT) methods, both backed by efficient kernel implementations. Our results demonstrate the first practical approach to fully quantized LLM fine-tuning that maintains accuracy in FP8 precision, while delivering performance benefits.

Title: Layer-Level Self-Exposure and Patch: Affirmative Token Mitigation for Jailbreak Attack Defense

• Authors: Yang Ouyang, Hengrui Gu, Shuhang Lin, Wenyue Hua, Jie Peng, Bhavya Kailkhura, Tianlong Chen, Kaixiong Zhou
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02629
• Pdf URL: https://arxiv.org/pdf/2501.02629
• Copy Paste: [[2501.02629]] Layer-Level Self-Exposure and Patch: Affirmative Token Mitigation for Jailbreak Attack Defense(https://arxiv.org/abs/2501.02629)
• Keywords: defense, attack, large language model
• Abstract: As large language models (LLMs) are increasingly deployed in diverse applications, including chatbot assistants and code generation, aligning their behavior with safety and ethical standards has become paramount. However, jailbreak attacks, which exploit vulnerabilities to elicit unintended or harmful outputs, threaten LLMs' safety significantly. In this paper, we introduce Layer-AdvPatcher, a novel methodology designed to defend against jailbreak attacks by utilizing an unlearning strategy to patch specific layers within LLMs through self-augmented datasets. Our insight is that certain layer(s), tend to produce affirmative tokens when faced with harmful prompts. By identifying these layers and adversarially exposing them to generate more harmful data, one can understand their inherent and diverse vulnerabilities to attacks. With these exposures, we then "unlearn" these issues, reducing the impact of affirmative tokens and hence minimizing jailbreak risks while keeping the model's responses to safe queries intact. We conduct extensive experiments on two models, four benchmark datasets, and multiple state-of-the-art jailbreak benchmarks to demonstrate the efficacy of our approach. Results indicate that our framework reduces the harmfulness and attack success rate of jailbreak attacks without compromising utility for benign queries compared to recent defense methods.

Title: Trust and Dependability in Blockchain & AI Based MedIoT Applications: Research Challenges and Future Directions

• Authors: Ellis Solaiman, Christa Awad
• Subjects: cs.CR, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.02647
• Pdf URL: https://arxiv.org/pdf/2501.02647
• Copy Paste: [[2501.02647]] Trust and Dependability in Blockchain & AI Based MedIoT Applications: Research Challenges and Future Directions(https://arxiv.org/abs/2501.02647)
• Keywords: secure, security, privacy
• Abstract: This paper critically reviews the integration of Artificial Intelligence (AI) and blockchain technologies in the context of Medical Internet of Things (MedIoT) applications, where they collectively promise to revolutionize healthcare delivery. By examining current research, we underscore AI's potential in advancing diagnostics and patient care, alongside blockchain's capacity to bolster data security and patient privacy. We focus particularly on the imperative to cultivate trust and ensure reliability within these systems. Our review highlights innovative solutions for managing healthcare data and challenges such as ensuring scalability, maintaining privacy, and promoting ethical practices within the MedIoT domain. We present a vision for integrating AI-driven insights with blockchain security in healthcare, offering a comprehensive review of current research and future directions. We conclude with a set of identified research gaps and propose that addressing these is crucial for achieving the dependable, secure, and patient -centric MedIoT applications of tomorrow.

Title: Representation Learning of Lab Values via Masked AutoEncoder

• Authors: David Restrepo, Chenwei Wu, Yueran Jia, Jaden K. Sun, Jack Gallifant, Catherine G. Bielick, Yugang Jia, Leo A. Celi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02648
• Pdf URL: https://arxiv.org/pdf/2501.02648
• Copy Paste: [[2501.02648]] Representation Learning of Lab Values via Masked AutoEncoder(https://arxiv.org/abs/2501.02648)
• Keywords: robust, fair, transformer
• Abstract: Accurate imputation of missing laboratory values in electronic health records (EHRs) is critical to enable robust clinical predictions and reduce biases in AI systems in healthcare. Existing methods, such as variational autoencoders (VAEs) and decision tree-based approaches such as XGBoost, struggle to model the complex temporal and contextual dependencies in EHR data, mainly in underrepresented groups. In this work, we propose Lab-MAE, a novel transformer-based masked autoencoder framework that leverages self-supervised learning for the imputation of continuous sequential lab values. Lab-MAE introduces a structured encoding scheme that jointly models laboratory test values and their corresponding timestamps, enabling explicit capturing temporal dependencies. Empirical evaluation on the MIMIC-IV dataset demonstrates that Lab-MAE significantly outperforms the state-of-the-art baselines such as XGBoost across multiple metrics, including root mean square error (RMSE), R-squared (R2), and Wasserstein distance (WD). Notably, Lab-MAE achieves equitable performance across demographic groups of patients, advancing fairness in clinical predictions. We further investigate the role of follow-up laboratory values as potential shortcut features, revealing Lab-MAE's robustness in scenarios where such data is unavailable. The findings suggest that our transformer-based architecture, adapted to the characteristics of the EHR data, offers a foundation model for more accurate and fair clinical imputation models. In addition, we measure and compare the carbon footprint of Lab-MAE with the baseline XGBoost model, highlighting its environmental requirements.

Title: Tighnari: Multi-modal Plant Species Prediction Based on Hierarchical Cross-Attention Using Graph-Based and Vision Backbone-Extracted Features

• Authors: Haixu Liu, Penghao Jiang, Zerui Tao, Muyan Wan, Qiuzhuang Sun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02649
• Pdf URL: https://arxiv.org/pdf/2501.02649
• Copy Paste: [[2501.02649]] Tighnari: Multi-modal Plant Species Prediction Based on Hierarchical Cross-Attention Using Graph-Based and Vision Backbone-Extracted Features(https://arxiv.org/abs/2501.02649)
• Keywords: robust, extraction, transformer
• Abstract: Predicting plant species composition in specific spatiotemporal contexts plays an important role in biodiversity management and conservation, as well as in improving species identification tools. Our work utilizes 88,987 plant survey records conducted in specific spatiotemporal contexts across Europe. We also use the corresponding satellite images, time series data, climate time series, and other rasterized environmental data such as land cover, human footprint, bioclimatic, and soil variables as training data to train the model to predict the outcomes of 4,716 plant surveys. We propose a feature construction and result correction method based on the graph structure. Through comparative experiments, we select the best-performing backbone networks for feature extraction in both temporal and image modalities. In this process, we built a backbone network based on the Swin-Transformer Block for extracting temporal Cubes features. We then design a hierarchical cross-attention mechanism capable of robustly fusing features from multiple modalities. During training, we adopt a 10-fold cross-fusion method based on fine-tuning and use a Threshold Top-K method for post-processing. Ablation experiments demonstrate the improvements in model performance brought by our proposed solution pipeline.

Title: A New Interpretation of the Certainty-Equivalence Approach for PAC Reinforcement Learning with a Generative Model

• Authors: Shivaram Kalyanakrishnan, Sheel Shah, Santhosh Kumar Guguloth
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.02652
• Pdf URL: https://arxiv.org/pdf/2501.02652
• Copy Paste: [[2501.02652]] A New Interpretation of the Certainty-Equivalence Approach for PAC Reinforcement Learning with a Generative Model(https://arxiv.org/abs/2501.02652)
• Keywords: generative
• Abstract: Reinforcement learning (RL) enables an agent interacting with an unknown MDP $M$ to optimise its behaviour by observing transitions sampled from $M$. A natural entity that emerges in the agent's reasoning is $\widehat{M}$, the maximum likelihood estimate of $M$ based on the observed transitions. The well-known \textit{certainty-equivalence} method (CEM) dictates that the agent update its behaviour to $\widehat{\pi}$, which is an optimal policy for $\widehat{M}$. Not only is CEM intuitive, it has been shown to enjoy minimax-optimal sample complexity in some regions of the parameter space for PAC RL with a generative model~\citep{Agarwal2020GenModel}. A seemingly unrelated algorithm is the ``trajectory tree method'' (TTM)~\citep{Kearns+MN:1999}, originally developed for efficient decision-time planning in large POMDPs. This paper presents a theoretical investigation that stems from the surprising finding that CEM may indeed be viewed as an application of TTM. The qualitative benefits of this view are (1) new and simple proofs of sample complexity upper bounds for CEM, in fact under a (2) weaker assumption on the rewards than is prevalent in the current literature. Our analysis applies to both non-stationary and stationary MDPs. Quantitatively, we obtain (3) improvements in the sample-complexity upper bounds for CEM both for non-stationary and stationary MDPs, in the regime that the ``mistake probability'' $\delta$ is small. Additionally, we show (4) a lower bound on the sample complexity for finite-horizon MDPs, which establishes the minimax-optimality of our upper bound for non-stationary MDPs in the small-$\delta$ regime.

Title: Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarks

• Authors: Yang Wang, Chenghua Lin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02654
• Pdf URL: https://arxiv.org/pdf/2501.02654
• Copy Paste: [[2501.02654]] Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarks(https://arxiv.org/abs/2501.02654)
• Keywords: attack, robust
• Abstract: vulnerability of deep learning models to adversarial attacks. While various defence mechanisms have been proposed, there is a lack of comprehensive benchmarks that evaluate these defences across diverse datasets, models, and tasks. In this work, we address this gap by presenting an extensive benchmark for textual adversarial defence that significantly expands upon previous work. Our benchmark incorporates a wide range of datasets, evaluates state-of-the-art defence mechanisms, and extends the assessment to include critical tasks such as single-sentence classification, similarity and paraphrase identification, natural language inference, and commonsense reasoning. This work not only serves as a valuable resource for researchers and practitioners in the field of adversarial robustness but also identifies key areas for future research in textual adversarial defence. By establishing a new standard for benchmarking in this domain, we aim to accelerate progress towards more robust and reliable natural language processing systems.

Title: Incentive-Compatible Federated Learning with Stackelberg Game Modeling

• Authors: Simin Javaherian, Bryce Turney, Li Chen, Nian-Feng Tzeng
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.02662
• Pdf URL: https://arxiv.org/pdf/2501.02662
• Copy Paste: [[2501.02662]] Incentive-Compatible Federated Learning with Stackelberg Game Modeling(https://arxiv.org/abs/2501.02662)
• Keywords: privacy, federate, fair
• Abstract: Federated Learning (FL) has gained prominence as a decentralized machine learning paradigm, allowing clients to collaboratively train a global model while preserving data privacy. Despite its potential, FL faces significant challenges in heterogeneous environments, where varying client resources and capabilities can undermine overall system performance. Existing approaches primarily focus on maximizing global model accuracy, often at the expense of unfairness among clients and suboptimal system efficiency, particularly in non-IID (non-Independent and Identically Distributed) settings. In this paper, we introduce FLamma, a novel Federated Learning framework based on adaptive gamma-based Stackelberg game, designed to address the aforementioned limitations and promote fairness. Our approach allows the server to act as the leader, dynamically adjusting a decay factor while clients, acting as followers, optimally select their number of local epochs to maximize their utility. Over time, the server incrementally balances client influence, initially rewarding higher-contributing clients and gradually leveling their impact, driving the system toward a Stackelberg Equilibrium. Extensive simulations on both IID and non-IID datasets show that our method significantly improves fairness in accuracy distribution without compromising overall model performance or convergence speed, outperforming traditional FL baselines.

Title: From Superficial Patterns to Semantic Understanding: Fine-Tuning Language Models on Contrast Sets

• Authors: Daniel Petrov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02683
• Pdf URL: https://arxiv.org/pdf/2501.02683
• Copy Paste: [[2501.02683]] From Superficial Patterns to Semantic Understanding: Fine-Tuning Language Models on Contrast Sets(https://arxiv.org/abs/2501.02683)
• Keywords: robust
• Abstract: Large scale pretrained language models have demonstrated high performance on standard datasets for natural language inference (NLI) tasks. Unfortunately, these evaluations can be misleading, as although the models can perform well on in-distribution data, they perform poorly on out-of-distribution test sets, such as contrast sets. Contrast sets consist of perturbed instances of data that have very minor, but meaningful, changes to the input that alter the gold label, revealing how models can learn superficial patterns in the training data rather than learning more sophisticated language nuances. As an example, the ELECTRA-small language model achieves nearly 90% accuracy on an SNLI dataset but drops to 75% when tested on an out-of-distribution contrast set. The research performed in this study explores how a language models' robustness can be improved by exposing it to small amounts of more complex contrast sets during training to help it better learn language patterns. With this approach, the model regains performance and achieves nearly 90% accuracy on contrast sets, highlighting the importance of diverse and challenging training data.

Title: Decoding specialised feature neurons in LLMs with the final projection layer

• Authors: Harry J Davies
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02688
• Pdf URL: https://arxiv.org/pdf/2501.02688
• Copy Paste: [[2501.02688]] Decoding specialised feature neurons in LLMs with the final projection layer(https://arxiv.org/abs/2501.02688)
• Keywords: interpretability, large language model
• Abstract: Large Language Models (LLMs) typically have billions of parameters and are thus often difficult to interpret in their operation. Such black-box models can pose a significant risk to safety when trusted to make important decisions. The lack of interpretability of LLMs is more related to their sheer size, rather than the complexity of their individual components. The TARS method for knowledge removal (Davies et al 2024) provides strong evidence for the hypothesis that that linear layer weights which act directly on the residual stream may have high correlation with different concepts encoded in the residual stream. Building upon this, we attempt to decode neuron weights directly into token probabilities through the final projection layer of the model (the LM-head). Firstly, we show that with Llama 3.1 8B we can utilise the LM-head to decode specialised feature neurons that respond strongly to certain concepts, with examples such as "dog" and "California". This is then confirmed by demonstrating that these neurons can be clamped to affect the probability of the concept in the output. This extends to the fine-tuned assistant Llama 3.1 8B instruct model, where we find that over 75% of neurons in the up-projection layers have the same top associated token compared to the pretrained model. Finally, we demonstrate that clamping the "dog" neuron leads the instruct model to always discuss dogs when asked about its favourite animal. Through our method, it is possible to map the entirety of Llama 3.1 8B's up-projection neurons in less than 15 minutes with no parallelization.

Title: GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Tracking

• Authors: Weikang Bian, Zhaoyang Huang, Xiaoyu Shi, Yijin Li, Fu-Yun Wang, Hongsheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02690
• Pdf URL: https://arxiv.org/pdf/2501.02690
• Copy Paste: [[2501.02690]] GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Tracking(https://arxiv.org/abs/2501.02690)
• Keywords: diffusion, transformer
• Abstract: 4D video control is essential in video generation as it enables the use of sophisticated lens techniques, such as multi-camera shooting and dolly zoom, which are currently unsupported by existing methods. Training a video Diffusion Transformer (DiT) directly to control 4D content requires expensive multi-view videos. Inspired by Monocular Dynamic novel View Synthesis (MDVS) that optimizes a 4D representation and renders videos according to different 4D elements, such as camera pose and object motion editing, we bring pseudo 4D Gaussian fields to video generation. Specifically, we propose a novel framework that constructs a pseudo 4D Gaussian field with dense 3D point tracking and renders the Gaussian field for all video frames. Then we finetune a pretrained DiT to generate videos following the guidance of the rendered video, dubbed as GS-DiT. To boost the training of the GS-DiT, we also propose an efficient Dense 3D Point Tracking (D3D-PT) method for the pseudo 4D Gaussian field construction. Our D3D-PT outperforms SpatialTracker, the state-of-the-art sparse 3D point tracking method, in accuracy and accelerates the inference speed by two orders of magnitude. During the inference stage, GS-DiT can generate videos with the same dynamic content while adhering to different camera parameters, addressing a significant limitation of current video generation models. GS-DiT demonstrates strong generalization capabilities and extends the 4D controllability of Gaussian splatting to video generation beyond just camera poses. It supports advanced cinematic effects through the manipulation of the Gaussian field and camera intrinsics, making it a powerful tool for creative video production. Demos are available at this https URL.

Title: EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models

• Authors: Andrés Villa, Juan León Alcázar, Motasem Alfarra, Vladimir Araujo, Alvaro Soto, Bernard Ghanem
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02699
• Pdf URL: https://arxiv.org/pdf/2501.02699
• Copy Paste: [[2501.02699]] EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models(https://arxiv.org/abs/2501.02699)
• Keywords: transformer, large language model
• Abstract: Large language models and vision transformers have demonstrated impressive zero-shot capabilities, enabling significant transferability in downstream tasks. The fusion of these models has resulted in multi-modal architectures with enhanced instructional capabilities. Despite incorporating vast image and language pre-training, these multi-modal architectures often generate responses that deviate from the ground truth in the image data. These failure cases are known as hallucinations. Current methods for mitigating hallucinations generally focus on regularizing the language component, improving the fusion module, or ensembling multiple visual encoders to improve visual representation. In this paper, we address the hallucination issue by directly enhancing the capabilities of the visual component. Our approach, named EAGLE, is fully agnostic to the LLM or fusion module and works as a post-pretraining approach that improves the grounding and language alignment of the visual encoder. We show that a straightforward reformulation of the original contrastive pre-training task results in an improved visual encoder that can be incorporated into the instructional multi-modal architecture without additional instructional training. As a result, EAGLE achieves a significant reduction in hallucinations across multiple challenging benchmarks and tasks.

Title: QuIM-RAG: Advancing Retrieval-Augmented Generation with Inverted Question Matching for Enhanced QA Performance

• Authors: Binita Saha, Utsha Saha, Muhammad Zubair Malik
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02702
• Pdf URL: https://arxiv.org/pdf/2501.02702
• Copy Paste: [[2501.02702]] QuIM-RAG: Advancing Retrieval-Augmented Generation with Inverted Question Matching for Enhanced QA Performance(https://arxiv.org/abs/2501.02702)
• Keywords: large language model
• Abstract: This work presents a novel architecture for building Retrieval-Augmented Generation (RAG) systems to improve Question Answering (QA) tasks from a target corpus. Large Language Models (LLMs) have revolutionized the analyzing and generation of human-like text. These models rely on pre-trained data and lack real-time updates unless integrated with live data tools. RAG enhances LLMs by integrating online resources and databases to generate contextually appropriate responses. However, traditional RAG still encounters challenges like information dilution and hallucinations when handling vast amounts of data. Our approach addresses these challenges by converting corpora into a domain-specific dataset and RAG architecture is constructed to generate responses from the target document. We introduce QuIM-RAG (Question-to-question Inverted Index Matching), a novel approach for the retrieval mechanism in our system. This strategy generates potential questions from document chunks and matches these with user queries to identify the most relevant text chunks for generating accurate answers. We have implemented our RAG system on top of the open-source Meta-LLaMA3-8B-instruct model by Meta Inc. that is available on Hugging Face. We constructed a custom corpus of 500+ pages from a high-traffic website accessed thousands of times daily for answering complex questions, along with manually prepared ground truth QA for evaluation. We compared our approach with traditional RAG models using BERT-Score and RAGAS, state-of-the-art metrics for evaluating LLM applications. Our evaluation demonstrates that our approach outperforms traditional RAG architectures on both metrics.

Title: Persistence of Backdoor-based Watermarks for Neural Networks: A Comprehensive Evaluation

• Authors: Anh Tu Ngo, Chuan Song Heng, Nandish Chattopadhyay, Anupam Chattopadhyay
• Subjects: cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2501.02704
• Pdf URL: https://arxiv.org/pdf/2501.02704
• Copy Paste: [[2501.02704]] Persistence of Backdoor-based Watermarks for Neural Networks: A Comprehensive Evaluation(https://arxiv.org/abs/2501.02704)
• Keywords: attack, robust, watermark
• Abstract: Deep Neural Networks (DNNs) have gained considerable traction in recent years due to the unparalleled results they gathered. However, the cost behind training such sophisticated models is resource intensive, resulting in many to consider DNNs to be intellectual property (IP) to model owners. In this era of cloud computing, high-performance DNNs are often deployed all over the internet so that people can access them publicly. As such, DNN watermarking schemes, especially backdoor-based watermarks, have been actively developed in recent years to preserve proprietary rights. Nonetheless, there lies much uncertainty on the robustness of existing backdoor watermark schemes, towards both adversarial attacks and unintended means such as fine-tuning neural network models. One reason for this is that no complete guarantee of robustness can be assured in the context of backdoor-based watermark. In this paper, we extensively evaluate the persistence of recent backdoor-based watermarks within neural networks in the scenario of fine-tuning, we propose/develop a novel data-driven idea to restore watermark after fine-tuning without exposing the trigger set. Our empirical results show that by solely introducing training data after fine-tuning, the watermark can be restored if model parameters do not shift dramatically during fine-tuning. Depending on the types of trigger samples used, trigger accuracy can be reinstated to up to 100%. Our study further explores how the restoration process works using loss landscape visualization, as well as the idea of introducing training data in fine-tuning stage to alleviate watermark vanishing.

Title: Knowledge Distillation with Adapted Weight

• Authors: Sirong Wu, Xi Luo, Junjie Liu, Yuhui Deng
• Subjects: cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2501.02705
• Pdf URL: https://arxiv.org/pdf/2501.02705
• Copy Paste: [[2501.02705]] Knowledge Distillation with Adapted Weight(https://arxiv.org/abs/2501.02705)
• Keywords: robust, fair, interpretability, explainability
• Abstract: Although large models have shown a strong capacity to solve large-scale problems in many areas including natural language and computer vision, their voluminous parameters are hard to deploy in a real-time system due to computational and energy constraints. Addressing this, knowledge distillation through Teacher-Student architecture offers a sustainable pathway to compress the knowledge of large models into more manageable sizes without significantly compromising performance. To enhance the robustness and interpretability of this framework, it is critical to understand how individual training data impact model performance, which is an area that remains underexplored. We propose the \textbf{Knowledge Distillation with Adaptive Influence Weight (KD-AIF)} framework which leverages influence functions from robust statistics to assign weights to training data, grounded in the four key SAFE principles: Sustainability, Accuracy, Fairness, and Explainability. This novel approach not only optimizes distillation but also increases transparency by revealing the significance of different data. The exploration of various update mechanisms within the KD-AIF framework further elucidates its potential to significantly improve learning efficiency and generalization in student models, marking a step toward more explainable and deployable Large Models. KD-AIF is effective in knowledge distillation while also showing exceptional performance in semi-supervised learning with outperforms existing baselines and methods in multiple benchmarks (CIFAR-100, CIFAR-10-4k, SVHN-1k, and GLUE).

Title: Multilevel Semantic-Aware Model for AI-Generated Video Quality Assessment

• Authors: Jiaze Li, Haoran Xu, Shiding Zhu, Junwei He, Haozhao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02706
• Pdf URL: https://arxiv.org/pdf/2501.02706
• Copy Paste: [[2501.02706]] Multilevel Semantic-Aware Model for AI-Generated Video Quality Assessment(https://arxiv.org/abs/2501.02706)
• Keywords: diffusion
• Abstract: The rapid development of diffusion models has greatly advanced AI-generated videos in terms of length and consistency recently, yet assessing AI-generated videos still remains challenging. Previous approaches have often focused on User-Generated Content(UGC), but few have targeted AI-Generated Video Quality Assessment methods. In this work, we introduce MSA-VQA, a Multilevel Semantic-Aware Model for AI-Generated Video Quality Assessment, which leverages CLIP-based semantic supervision and cross-attention mechanisms. Our hierarchical framework analyzes video content at three levels: frame, segment, and video. We propose a Prompt Semantic Supervision Module using text encoder of CLIP to ensure semantic consistency between videos and conditional prompts. Additionally, we propose the Semantic Mutation-aware Module to capture subtle variations between frames. Extensive experiments demonstrate our method achieves state-of-the-art results.

Title: OpenGU: A Comprehensive Benchmark for Graph Unlearning

• Authors: Bowen Fan, Yuming Ai, Xunkai Li, Zhilin Guo, Rong-Hua Li, Guoren Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02728
• Pdf URL: https://arxiv.org/pdf/2501.02728
• Copy Paste: [[2501.02728]] OpenGU: A Comprehensive Benchmark for Graph Unlearning(https://arxiv.org/abs/2501.02728)
• Keywords: privacy, fair
• Abstract: Graph Machine Learning is essential for understanding and analyzing relational data. However, privacy-sensitive applications demand the ability to efficiently remove sensitive information from trained graph neural networks (GNNs), avoiding the unnecessary time and space overhead caused by retraining models from scratch. To address this issue, Graph Unlearning (GU) has emerged as a critical solution, with the potential to support dynamic graph updates in data management systems and enable scalable unlearning in distributed data systems while ensuring privacy compliance. Unlike machine unlearning in computer vision or other fields, GU faces unique difficulties due to the non-Euclidean nature of graph data and the recursive message-passing mechanism of GNNs. Additionally, the diversity of downstream tasks and the complexity of unlearning requests further amplify these challenges. Despite the proliferation of diverse GU strategies, the absence of a benchmark providing fair comparisons for GU, and the limited flexibility in combining downstream tasks and unlearning requests, have yielded inconsistencies in evaluations, hindering the development of this domain. To fill this gap, we present OpenGU, the first GU benchmark, where 16 SOTA GU algorithms and 37 multi-domain datasets are integrated, enabling various downstream tasks with 13 GNN backbones when responding to flexible unlearning requests. Based on this unified benchmark framework, we are able to provide a comprehensive and fair evaluation for GU. Through extensive experimentation, we have drawn $8$ crucial conclusions about existing GU methods, while also gaining valuable insights into their limitations, shedding light on potential avenues for future research.

Title: AFed: Algorithmic Fair Federated Learning

• Authors: Huiqiang Chen, Tianqing Zhu, Wanlei Zhou, Wei Zhao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02732
• Pdf URL: https://arxiv.org/pdf/2501.02732
• Copy Paste: [[2501.02732]] AFed: Algorithmic Fair Federated Learning(https://arxiv.org/abs/2501.02732)
• Keywords: privacy, federate, fair
• Abstract: Federated Learning (FL) has gained significant attention as it facilitates collaborative machine learning among multiple clients without centralizing their data on a server. FL ensures the privacy of participating clients by locally storing their data, which creates new challenges in fairness. Traditional debiasing methods assume centralized access to sensitive information, rendering them impractical for the FL setting. Additionally, FL is more susceptible to fairness issues than centralized machine learning due to the diverse client data sources that may be associated with group information. Therefore, training a fair model in FL without access to client local data is important and challenging. This paper presents AFed, a straightforward yet effective framework for promoting group fairness in FL. The core idea is to circumvent restricted data access by learning the global data distribution. This paper proposes two approaches: AFed-G, which uses a conditional generator trained on the server side, and AFed-GAN, which improves upon AFed-G by training a conditional GAN on the client side. We augment the client data with the generated samples to help remove bias. Our theoretical analysis justifies the proposed methods, and empirical results on multiple real-world datasets demonstrate a substantial improvement in AFed over several baselines.

Title: Sequence Complementor: Complementing Transformers For Time Series Forecasting with Learnable Sequences

• Authors: Xiwen Chen, Peijie Qiu, Wenhui Zhu, Huayu Li, Hao Wang, Aristeidis Sotiras, Yalin Wang, Abolfazl Razi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02735
• Pdf URL: https://arxiv.org/pdf/2501.02735
• Copy Paste: [[2501.02735]] Sequence Complementor: Complementing Transformers For Time Series Forecasting with Learnable Sequences(https://arxiv.org/abs/2501.02735)
• Keywords: transformer
• Abstract: Since its introduction, the transformer has shifted the development trajectory away from traditional models (e.g., RNN, MLP) in time series forecasting, which is attributed to its ability to capture global dependencies within temporal tokens. Follow-up studies have largely involved altering the tokenization and self-attention modules to better adapt Transformers for addressing special challenges like non-stationarity, channel-wise dependency, and variable correlation in time series. However, we found that the expressive capability of sequence representation is a key factor influencing Transformer performance in time forecasting after investigating several representative methods, where there is an almost linear relationship between sequence representation entropy and mean square error, with more diverse representations performing better. In this paper, we propose a novel attention mechanism with Sequence Complementors and prove feasible from an information theory perspective, where these learnable sequences are able to provide complementary information beyond current input to feed attention. We further enhance the Sequence Complementors via a diversification loss that is theoretically covered. The empirical evaluation of both long-term and short-term forecasting has confirmed its superiority over the recent state-of-the-art methods.

Title: Holistic Semantic Representation for Navigational Trajectory Generation

• Authors: Ji Cao, Tongya Zheng, Qinghong Guo, Yu Wang, Junshu Dai, Shunyu Liu, Jie Yang, Jie Song, Mingli Song
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02737
• Pdf URL: https://arxiv.org/pdf/2501.02737
• Copy Paste: [[2501.02737]] Holistic Semantic Representation for Navigational Trajectory Generation(https://arxiv.org/abs/2501.02737)
• Keywords: privacy
• Abstract: Trajectory generation has garnered significant attention from researchers in the field of spatio-temporal analysis, as it can generate substantial synthesized human mobility trajectories that enhance user privacy and alleviate data scarcity. However, existing trajectory generation methods often focus on improving trajectory generation quality from a singular perspective, lacking a comprehensive semantic understanding across various scales. Consequently, we are inspired to develop a HOlistic SEmantic Representation (HOSER) framework for navigational trajectory generation. Given an origin-and-destination (OD) pair and the starting time point of a latent trajectory, we first propose a Road Network Encoder to expand the receptive field of road- and zone-level semantics. Second, we design a Multi-Granularity Trajectory Encoder to integrate the spatio-temporal semantics of the generated trajectory at both the point and trajectory levels. Finally, we employ a Destination-Oriented Navigator to seamlessly integrate destination-oriented guidance. Extensive experiments on three real-world datasets demonstrate that HOSER outperforms state-of-the-art baselines by a significant margin. Moreover, the model's performance in few-shot learning and zero-shot learning scenarios further verifies the effectiveness of our holistic semantic representation.

Title: Interpretable Recognition of Fused Magnesium Furnace Working Conditions with Deep Convolutional Stochastic Configuration Networks

• Authors: Li Weitao, Zhang Xinru, Wang Dianhui, Tong Qianqian, Chai Tianyou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02740
• Pdf URL: https://arxiv.org/pdf/2501.02740
• Copy Paste: [[2501.02740]] Interpretable Recognition of Fused Magnesium Furnace Working Conditions with Deep Convolutional Stochastic Configuration Networks(https://arxiv.org/abs/2501.02740)
• Keywords: interpretability
• Abstract: To address the issues of a weak generalization capability and interpretability in working condition recognition model of a fused magnesium furnace, this paper proposes an interpretable working condition recognition method based on deep convolutional stochastic configuration networks (DCSCNs). Firstly, a supervised learning mechanism is employed to generate physically meaningful Gaussian differential convolution kernels. An incremental method is utilized to construct a DCSCNs model, ensuring the convergence of recognition errors in a hierarchical manner and avoiding the iterative optimization process of convolutional kernel parameters using the widely used backpropagation algorithm. The independent coefficient of channel feature maps is defined to obtain the visualization results of feature class activation maps for the fused magnesium furnace. A joint reward function is constructed based on the recognition accuracy, the interpretable trustworthiness evaluation metrics, and the model parameter quantity. Reinforcement learning (RL) is applied to adaptively prune the convolutional kernels of the DCSCNs model, aiming to build a compact, highly performed and interpretable network. The experimental results demonstrate that the proposed method outperforms the other deep learning approaches in terms of recognition accuracy and interpretability.

Title: Brick-Diffusion: Generating Long Videos with Brick-to-Wall Denoising

• Authors: Yunlong Yuan, Yuanfan Guo, Chunwei Wang, Hang Xu, Li Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02741
• Pdf URL: https://arxiv.org/pdf/2501.02741
• Copy Paste: [[2501.02741]] Brick-Diffusion: Generating Long Videos with Brick-to-Wall Denoising(https://arxiv.org/abs/2501.02741)
• Keywords: diffusion
• Abstract: Recent advances in diffusion models have greatly improved text-driven video generation. However, training models for long video generation demands significant computational power and extensive data, leading most video diffusion models to be limited to a small number of frames. Existing training-free methods that attempt to generate long videos using pre-trained short video diffusion models often struggle with issues such as insufficient motion dynamics and degraded video fidelity. In this paper, we present Brick-Diffusion, a novel, training-free approach capable of generating long videos of arbitrary length. Our method introduces a brick-to-wall denoising strategy, where the latent is denoised in segments, with a stride applied in subsequent iterations. This process mimics the construction of a staggered brick wall, where each brick represents a denoised segment, enabling communication between frames and improving overall video quality. Through quantitative and qualitative evaluations, we demonstrate that Brick-Diffusion outperforms existing baseline methods in generating high-fidelity videos.

Title: MBTSAD: Mitigating Backdoors in Language Models Based on Token Splitting and Attention Distillation

• Authors: Yidong Ding, Jiafei Niu, Ping Yi
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02754
• Pdf URL: https://arxiv.org/pdf/2501.02754
• Copy Paste: [[2501.02754]] MBTSAD: Mitigating Backdoors in Language Models Based on Token Splitting and Attention Distillation(https://arxiv.org/abs/2501.02754)
• Keywords: attack
• Abstract: In recent years, attention-based models have excelled across various domains but remain vulnerable to backdoor attacks, often from downloading or fine-tuning on poisoned datasets. Many current methods to mitigate backdoors in NLP models rely on the pre-trained (unfine-tuned) weights, but these methods fail in scenarios where the pre-trained weights are not available. In this work, we propose MBTSAD, which can mitigate backdoors in the language model by utilizing only a small subset of clean data and does not require pre-trained weights. Specifically, MBTSAD retrains the backdoored model on a dataset generated by token splitting. Then MBTSAD leverages attention distillation, the retrained model is the teacher model, and the original backdoored model is the student model. Experimental results demonstrate that MBTSAD achieves comparable backdoor mitigation performance as the methods based on pre-trained weights while maintaining the performance on clean data. MBTSAD does not rely on pre-trained weights, enhancing its utility in scenarios where pre-trained weights are inaccessible. In addition, we simplify the min-max problem of adversarial training and visualize text representations to discover that the token splitting method in MBTSAD's first step generates Out-of-Distribution (OOD) data, leading the model to learn more generalized features and eliminate backdoor patterns.

Title: Visual Large Language Models for Generalized and Specialized Applications

• Authors: Yifan Li, Zhixin Lai, Wentao Bao, Zhen Tan, Anh Dao, Kewei Sui, Jiayi Shen, Dong Liu, Huan Liu, Yu Kong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02765
• Pdf URL: https://arxiv.org/pdf/2501.02765
• Copy Paste: [[2501.02765]] Visual Large Language Models for Generalized and Specialized Applications(https://arxiv.org/abs/2501.02765)
• Keywords: large language model
• Abstract: Visual-language models (VLM) have emerged as a powerful tool for learning a unified embedding space for vision and language. Inspired by large language models, which have demonstrated strong reasoning and multi-task capabilities, visual large language models (VLLMs) are gaining increasing attention for building general-purpose VLMs. Despite the significant progress made in VLLMs, the related literature remains limited, particularly from a comprehensive application perspective, encompassing generalized and specialized applications across vision (image, video, depth), action, and language modalities. In this survey, we focus on the diverse applications of VLLMs, examining their using scenarios, identifying ethics consideration and challenges, and discussing future directions for their development. By synthesizing these contents, we aim to provide a comprehensive guide that will pave the way for future innovations and broader applications of VLLMs. The paper list repository is available: this https URL.

Title: Unsupervised Domain Adaptation for Occlusion Resilient Human Pose Estimation

• Authors: Arindam Dutta, Sarosij Bose, Saketh Bachu, Calvin-Khang Ta, Konstantinos Karydis, Amit K. Roy-Chowdhury
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02773
• Pdf URL: https://arxiv.org/pdf/2501.02773
• Copy Paste: [[2501.02773]] Unsupervised Domain Adaptation for Occlusion Resilient Human Pose Estimation(https://arxiv.org/abs/2501.02773)
• Keywords: robust
• Abstract: Occlusions are a significant challenge to human pose estimation algorithms, often resulting in inaccurate and anatomically implausible poses. Although current occlusion-robust human pose estimation algorithms exhibit impressive performance on existing datasets, their success is largely attributed to supervised training and the availability of additional information, such as multiple views or temporal continuity. Furthermore, these algorithms typically suffer from performance degradation under distribution shifts. While existing domain adaptive human pose estimation algorithms address this bottleneck, they tend to perform suboptimally when the target domain images are occluded, a common occurrence in real-life scenarios. To address these challenges, we propose OR-POSE: Unsupervised Domain Adaptation for Occlusion Resilient Human POSE Estimation. OR-POSE is an innovative unsupervised domain adaptation algorithm which effectively mitigates domain shifts and overcomes occlusion challenges by employing the mean teacher framework for iterative pseudo-label refinement. Additionally, OR-POSE reinforces realistic pose prediction by leveraging a learned human pose prior which incorporates the anatomical constraints of humans in the adaptation process. Lastly, OR-POSE avoids overfitting to inaccurate pseudo labels generated from heavily occluded images by employing a novel visibility-based curriculum learning approach. This enables the model to gradually transition from training samples with relatively less occlusion to more challenging, heavily occluded samples. Extensive experiments show that OR-POSE outperforms existing analogous state-of-the-art algorithms by $\sim$ 7% on challenging occluded human pose estimation datasets.

Title: From Dense to Sparse: Event Response for Enhanced Residential Load Forecasting

• Authors: Xin Cao, Qinghua Tao, Yingjie Zhou, Lu Zhang, Le Zhang, Dongjin Song, Dapeng Oliver Wu, Ce Zhu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02781
• Pdf URL: https://arxiv.org/pdf/2501.02781
• Copy Paste: [[2501.02781]] From Dense to Sparse: Event Response for Enhanced Residential Load Forecasting(https://arxiv.org/abs/2501.02781)
• Keywords: extraction
• Abstract: Residential load forecasting (RLF) is crucial for resource scheduling in power systems. Most existing methods utilize all given load records (dense data) to indiscriminately extract the dependencies between historical and future time series. However, there exist important regular patterns residing in the event-related associations among different appliances (sparse knowledge), which have yet been this http URL this paper, we propose an Event-Response Knowledge Guided approach (ERKG) for RLF by incorporating the estimation of electricity usage events for different appliances, mining event-related sparse knowledge from the load series. With ERKG, the event-response estimation enables portraying the electricity consumption behaviors of residents, revealing regular variations in appliance operational this http URL be specific, ERKG consists of knowledge extraction and guidance: i) a forecasting model is designed for the electricity usage events by estimating appliance operational states, aiming to extract the event-related sparse knowledge; ii) a novel knowledge-guided mechanism is established by fusing such state estimates of the appliance events into the RLF model, which can give particular focuses on the patterns of users' electricity consumption this http URL, ERKG can flexibly serve as a plug-in module to boost the capability of existing forecasting models by leveraging event response. In numerical experiments, extensive comparisons and ablation studies have verified the effectiveness of our ERKG, e.g., over 8% MAE can be reduced on the tested state-of-the-art forecasting models. The source code will be available at this https URL.

Title: GLoG-CSUnet: Enhancing Vision Transformers with Adaptable Radiomic Features for Medical Image Segmentation

• Authors: Niloufar Eghbali, Hassan Bagher-Ebadian, Tuka Alhanai, Mohammad M. Ghassemi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02788
• Pdf URL: https://arxiv.org/pdf/2501.02788
• Copy Paste: [[2501.02788]] GLoG-CSUnet: Enhancing Vision Transformers with Adaptable Radiomic Features for Medical Image Segmentation(https://arxiv.org/abs/2501.02788)
• Keywords: extraction, transformer, segmentation
• Abstract: Vision Transformers (ViTs) have shown promise in medical image semantic segmentation (MISS) by capturing long-range correlations. However, ViTs often struggle to model local spatial information effectively, which is essential for accurately segmenting fine anatomical details, particularly when applied to small datasets without extensive pre-training. We introduce Gabor and Laplacian of Gaussian Convolutional Swin Network (GLoG-CSUnet), a novel architecture enhancing Transformer-based models by incorporating learnable radiomic features. This approach integrates dynamically adaptive Gabor and Laplacian of Gaussian (LoG) filters to capture texture, edge, and boundary information, enhancing the feature representation processed by the Transformer model. Our method uniquely combines the long-range dependency modeling of Transformers with the texture analysis capabilities of Gabor and LoG features. Evaluated on the Synapse multi-organ and ACDC cardiac segmentation datasets, GLoG-CSUnet demonstrates significant improvements over state-of-the-art models, achieving a 1.14\% increase in Dice score for Synapse and 0.99\% for ACDC, with minimal computational overhead (only 15 and 30 additional parameters, respectively). GLoG-CSUnet's flexible design allows integration with various base models, offering a promising approach for incorporating radiomics-inspired feature extraction in Transformer architectures for medical image analysis. The code implementation is available on GitHub at: this https URL.

Title: Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model

• Authors: Yueqin Yin, Shentao Yang, Yujia Xie, Ziyi Yang, Yuting Sun, Hany Awadalla, Weizhu Chen, Mingyuan Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02790
• Pdf URL: https://arxiv.org/pdf/2501.02790
• Copy Paste: [[2501.02790]] Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model(https://arxiv.org/abs/2501.02790)
• Keywords: segmentation
• Abstract: Reinforcement learning from human feedback (RLHF) has been widely adopted to align language models (LMs) with human preference. Prior RLHF works typically take a bandit formulation, which, though intuitive, ignores the sequential nature of LM generation and can suffer from the sparse reward issue. While recent works propose dense token-level RLHF, treating each token as an action may be oversubtle to proper reward assignment. In this paper, we seek to get the best of both by training and utilizing a segment-level reward model, which assigns a reward to each semantically complete text segment that spans over a short sequence of tokens. For reward learning, our method allows dynamic text segmentation and compatibility with standard sequence-preference datasets. For effective RL-based LM training against segment reward, we generalize the classical scalar bandit reward normalizers into location-aware normalizer functions and interpolate the segment reward for further densification. With these designs, our method performs competitively on three popular RLHF benchmarks for LM policy: AlpacaEval 2.0, Arena-Hard, and MT-Bench. Ablation studies are conducted to further demonstrate our method.

Title: InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion

• Authors: Zhaoyi Yan, Zhijie Sang, Yiming Zhang, Yuhao Fu, Baoyi He, Qi Zhou, Yining Di, Chunlin Ji, Shengyu Zhang, Fei Wu, Hongxia Yang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02795
• Pdf URL: https://arxiv.org/pdf/2501.02795
• Copy Paste: [[2501.02795]] InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion(https://arxiv.org/abs/2501.02795)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated strong performance across various reasoning tasks, yet building a single model that consistently excels across all domains remains challenging. This paper addresses this problem by exploring strategies to integrate multiple domain-specialized models into an efficient pivot this http URL propose two fusion strategies to combine the strengths of multiple LLMs: (1) a pairwise, multi-step fusion approach that sequentially distills each source model into the pivot model, followed by a weight merging step to integrate the distilled models into the final model. This method achieves strong performance but requires substantial training effort; and (2) a unified fusion approach that aggregates all source models' outputs this http URL improve the fusion process, we introduce a novel Rate-Skewness Adaptive Fusion (RSAF) technique, which dynamically adjusts top-K ratios during parameter merging for enhanced flexibility and this http URL, we propose an uncertainty-based weighting method for the unified approach, which dynamically balances the contributions of source models and outperforms other logits/distribution ensemble this http URL achieved accuracy improvements of 9.27%, 8.80%, and 8.89% on the GSM8K, MATH, and HumanEval tasks, respectively.

Title: GraphDART: Graph Distillation for Efficient Advanced Persistent Threat Detection

• Authors: Saba Fathi Rabooki, Bowen Li, Falih Gozi Febrinanto, Ciyuan Peng, Elham Naghizade, Fengling Han, Feng Xia
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02796
• Pdf URL: https://arxiv.org/pdf/2501.02796
• Copy Paste: [[2501.02796]] GraphDART: Graph Distillation for Efficient Advanced Persistent Threat Detection(https://arxiv.org/abs/2501.02796)
• Keywords: security, robust
• Abstract: Cyber-physical-social systems (CPSSs) have emerged in many applications over recent decades, requiring increased attention to security concerns. The rise of sophisticated threats like Advanced Persistent Threats (APTs) makes ensuring security in CPSSs particularly challenging. Provenance graph analysis has proven effective for tracing and detecting anomalies within systems, but the sheer size and complexity of these graphs hinder the efficiency of existing methods, especially those relying on graph neural networks (GNNs). To address these challenges, we present GraphDART, a modular framework designed to distill provenance graphs into compact yet informative representations, enabling scalable and effective anomaly detection. GraphDART can take advantage of diverse graph distillation techniques, including classic and modern graph distillation methods, to condense large provenance graphs while preserving essential structural and contextual information. This approach significantly reduces computational overhead, allowing GNNs to learn from distilled graphs efficiently and enhance detection performance. Extensive evaluations on benchmark datasets demonstrate the robustness of GraphDART in detecting malicious activities across cyber-physical-social systems. By optimizing computational efficiency, GraphDART provides a scalable and practical solution to safeguard interconnected environments against APTs.

Title: COph100: A comprehensive fundus image registration dataset from infants constituting the "RIDIRP" database

• Authors: Yan Hu, Mingdao Gong, Zhongxi Qiu, Jiabao Liu, Hongli Shen, Mingzhen Yuan, Xiaoqing Zhang, Heng Li, Hai Lu, Jiang Liu
• Subjects: cs.CV, cs.CE
• Abstract URL: https://arxiv.org/abs/2501.02800
• Pdf URL: https://arxiv.org/pdf/2501.02800
• Copy Paste: [[2501.02800]] COph100: A comprehensive fundus image registration dataset from infants constituting the "RIDIRP" database(https://arxiv.org/abs/2501.02800)
• Keywords: robust, segmentation
• Abstract: Retinal image registration is vital for diagnostic therapeutic applications within the field of ophthalmology. Existing public datasets, focusing on adult retinal pathologies with high-quality images, have limited number of image pairs and neglect clinical challenges. To address this gap, we introduce COph100, a novel and challenging dataset known as the Comprehensive Ophthalmology Retinal Image Registration dataset for infants with a wide range of image quality issues constituting the public "RIDIRP" database. COph100 consists of 100 eyes, each with 2 to 9 examination sessions, amounting to a total of 491 image pairs carefully selected from the publicly available dataset. We manually labeled the corresponding ground truth image points and provided automatic vessel segmentation masks for each image. We have assessed COph100 in terms of image quality and registration outcomes using state-of-the-art algorithms. This resource enables a robust comparison of retinal registration methodologies and aids in the analysis of disease progression in infants, thereby deepening our understanding of pediatric ophthalmic conditions.

Title: AE-NeRF: Augmenting Event-Based Neural Radiance Fields for Non-ideal Conditions and Larger Scene

• Authors: Chaoran Feng, Wangbo Yu, Xinhua Cheng, Zhenyu Tang, Junwu Zhang, Li Yuan, Yonghong Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02807
• Pdf URL: https://arxiv.org/pdf/2501.02807
• Copy Paste: [[2501.02807]] AE-NeRF: Augmenting Event-Based Neural Radiance Fields for Non-ideal Conditions and Larger Scene(https://arxiv.org/abs/2501.02807)
• Keywords: robust
• Abstract: Compared to frame-based methods, computational neuromorphic imaging using event cameras offers significant advantages, such as minimal motion blur, enhanced temporal resolution, and high dynamic range. The multi-view consistency of Neural Radiance Fields combined with the unique benefits of event cameras, has spurred recent research into reconstructing NeRF from data captured by moving event cameras. While showing impressive performance, existing methods rely on ideal conditions with the availability of uniform and high-quality event sequences and accurate camera poses, and mainly focus on the object level reconstruction, thus limiting their practical applications. In this work, we propose AE-NeRF to address the challenges of learning event-based NeRF from non-ideal conditions, including non-uniform event sequences, noisy poses, and various scales of scenes. Our method exploits the density of event streams and jointly learn a pose correction module with an event-based NeRF (e-NeRF) framework for robust 3D reconstruction from inaccurate camera poses. To generalize to larger scenes, we propose hierarchical event distillation with a proposal e-NeRF network and a vanilla e-NeRF network to resample and refine the reconstruction process. We further propose an event reconstruction loss and a temporal loss to improve the view consistency of the reconstructed scene. We established a comprehensive benchmark that includes large-scale scenes to simulate practical non-ideal conditions, incorporating both synthetic and challenging real-world event datasets. The experimental results show that our method achieves a new state-of-the-art in event-based 3D reconstruction.

Title: First-place Solution for Streetscape Shop Sign Recognition Competition

• Authors: Bin Wang, Li Jing
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02811
• Pdf URL: https://arxiv.org/pdf/2501.02811
• Copy Paste: [[2501.02811]] First-place Solution for Streetscape Shop Sign Recognition Competition(https://arxiv.org/abs/2501.02811)
• Keywords: transformer
• Abstract: Text recognition technology applied to street-view storefront signs is increasingly utilized across various practical domains, including map navigation, smart city planning analysis, and business value assessments in commercial districts. This technology holds significant research and commercial potential. Nevertheless, it faces numerous challenges. Street view images often contain signboards with complex designs and diverse text styles, complicating the text recognition process. A notable advancement in this field was introduced by our team in a recent competition. We developed a novel multistage approach that integrates multimodal feature fusion, extensive self-supervised training, and a Transformer-based large model. Furthermore, innovative techniques such as BoxDQN, which relies on reinforcement learning, and text rectification methods were employed, leading to impressive outcomes. Comprehensive experiments have validated the effectiveness of these methods, showcasing our potential to enhance text recognition capabilities in complex urban environments.

Title: InpDiffusion: Image Inpainting Localization via Conditional Diffusion Models

• Authors: Kai Wang, Shaozhang Niu, Qixian Hao, Jiwei Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02816
• Pdf URL: https://arxiv.org/pdf/2501.02816
• Copy Paste: [[2501.02816]] InpDiffusion: Image Inpainting Localization via Conditional Diffusion Models(https://arxiv.org/abs/2501.02816)
• Keywords: robust, diffusion
• Abstract: As artificial intelligence advances rapidly, particularly with the advent of GANs and diffusion models, the accuracy of Image Inpainting Localization (IIL) has become increasingly challenging. Current IIL methods face two main challenges: a tendency towards overconfidence, leading to incorrect predictions; and difficulty in detecting subtle tampering boundaries in inpainted images. In response, we propose a new paradigm that treats IIL as a conditional mask generation task utilizing diffusion models. Our method, InpDiffusion, utilizes the denoising process enhanced by the integration of image semantic conditions to progressively refine predictions. During denoising, we employ edge conditions and introduce a novel edge supervision strategy to enhance the model's perception of edge details in inpainted objects. Balancing the diffusion model's stochastic sampling with edge supervision of tampered image regions mitigates the risk of incorrect predictions from overconfidence and prevents the loss of subtle boundaries that can result from overly stochastic processes. Furthermore, we propose an innovative Dual-stream Multi-scale Feature Extractor (DMFE) for extracting multi-scale features, enhancing feature representation by considering both semantic and edge conditions of the inpainted images. Extensive experiments across challenging datasets demonstrate that the InpDiffusion significantly outperforms existing state-of-the-art methods in IIL tasks, while also showcasing excellent generalization capabilities and robustness.

Title: Randomly Sampled Language Reasoning Problems Reveal Limits of LLMs

• Authors: Kavi Gupta, Kate Sanders, Armando Solar-Lezama
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02825
• Pdf URL: https://arxiv.org/pdf/2501.02825
• Copy Paste: [[2501.02825]] Randomly Sampled Language Reasoning Problems Reveal Limits of LLMs(https://arxiv.org/abs/2501.02825)
• Keywords: fair
• Abstract: Can LLMs pick up language structure from examples? Evidence in prior work seems to indicate yes, as pretrained models repeatedly demonstrate the ability to adapt to new language structures and vocabularies. However, this line of research typically considers languages that are present within common pretraining datasets, or otherwise share notable similarities with these seen languages. In contrast, in this work we attempt to measure models' language understanding capacity while circumventing the risk of dataset recall. We parameterize large families of language tasks recognized by deterministic finite automata (DFAs), and can thus sample novel language reasoning problems to fairly evaulate LLMs regardless of training data. We find that, even in the strikingly simple setting of 3-state DFAs, LLMs underperform unparameterized ngram models on both language recognition and synthesis tasks. These results suggest that LLMs struggle to match the ability of basic language models in recognizing and reasoning over languages that are sufficiently distinct from the ones they see at training time, underscoring the distinction between learning individual languages and possessing a general theory of language.

Title: Samba-asr state-of-the-art speech recognition leveraging structured state-space models

• Authors: Syed Abdul Gaffar Shakhadri, Kruthika KR, Kartik Basavaraj Angadi
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.02832
• Pdf URL: https://arxiv.org/pdf/2501.02832
• Copy Paste: [[2501.02832]] Samba-asr state-of-the-art speech recognition leveraging structured state-space models(https://arxiv.org/abs/2501.02832)
• Keywords: robust, transformer
• Abstract: We propose Samba ASR, the first state-of-the-art Automatic Speech Recognition (ASR) model leveraging the novel Mamba architecture as both encoder and decoder, built on the foundation of state-space models (SSMs). Unlike transformer-based ASR models, which rely on self-attention mechanisms to capture dependencies, Samba ASR effectively models both local and global temporal dependencies using efficient state-space dynamics, achieving remarkable performance gains. By addressing the limitations of transformers, such as quadratic scaling with input length and difficulty in handling long-range dependencies, Samba ASR achieves superior accuracy and efficiency. Experimental results demonstrate that Samba ASR surpasses existing open-source transformer-based ASR models across various standard benchmarks, establishing it as the new state of the art in ASR. Extensive evaluations on benchmark datasets show significant improvements in Word Error Rate (WER), with competitive performance even in low-resource scenarios. Furthermore, the computational efficiency and parameter optimization of the Mamba architecture make Samba ASR a scalable and robust solution for diverse ASR tasks. Our contributions include: A new Samba ASR architecture demonstrating the superiority of SSMs over transformer-based models for speech sequence processing. A comprehensive evaluation on public benchmarks showcasing state-of-the-art performance. An analysis of computational efficiency, robustness to noise, and sequence generalization. This work highlights the viability of Mamba SSMs as a transformer-free alternative for efficient and accurate ASR. By leveraging state-space modeling advancements, Samba ASR sets a new benchmark for ASR performance and future research.

Title: Graph-based Retrieval Augmented Generation for Dynamic Few-shot Text Classification

• Authors: Yubo Wang, Haoyang Li, Fei Teng, Lei Chen
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02844
• Pdf URL: https://arxiv.org/pdf/2501.02844
• Copy Paste: [[2501.02844]] Graph-based Retrieval Augmented Generation for Dynamic Few-shot Text Classification(https://arxiv.org/abs/2501.02844)
• Keywords: large language model
• Abstract: Text classification is a fundamental task in natural language processing, pivotal to various applications such as query optimization, data integration, and schema matching. While neural network-based models, such as CNN and BERT, have demonstrated remarkable performance in text classification, their effectiveness heavily relies on abundant labeled training data. This dependency makes these models less effective in dynamic few-shot text classification, where labeled data is scarce, and target labels frequently evolve based on application needs. Recently, large language models (LLMs) have shown promise due to their extensive pretraining and contextual understanding. Current approaches provide LLMs with text inputs, candidate labels, and additional side information (e.g., descriptions) to predict text labels. However, their effectiveness is hindered by the increased input size and the noise introduced through side information processing. To address these limitations, we propose a graph-based online retrieval-augmented generation framework, namely GORAG, for dynamic few-shot text classification. GORAG constructs and maintains an adaptive information graph by extracting side information across all target texts, rather than treating each input independently. It employs a weighted edge mechanism to prioritize the importance and reliability of extracted information and dynamically retrieves relevant context using a minimum-cost spanning tree tailored for each text input. Empirical evaluations demonstrate that GORAG outperforms existing approaches by providing more comprehensive and accurate contextual information.

Title: Large Language Models for Video Surveillance Applications

• Authors: Ulindu De Silva, Leon Fernando, Billy Lau Pik Lik, Zann Koh, Sam Conrad Joyce, Belinda Yuen, Chau Yuen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02850
• Pdf URL: https://arxiv.org/pdf/2501.02850
• Copy Paste: [[2501.02850]] Large Language Models for Video Surveillance Applications(https://arxiv.org/abs/2501.02850)
• Keywords: robust, generative, large language model
• Abstract: The rapid increase in video content production has resulted in enormous data volumes, creating significant challenges for efficient analysis and resource management. To address this, robust video analysis tools are essential. This paper presents an innovative proof of concept using Generative Artificial Intelligence (GenAI) in the form of Vision Language Models to enhance the downstream video analysis process. Our tool generates customized textual summaries based on user-defined queries, providing focused insights within extensive video datasets. Unlike traditional methods that offer generic summaries or limited action recognition, our approach utilizes Vision Language Models to extract relevant information, improving analysis precision and efficiency. The proposed method produces textual summaries from extensive CCTV footage, which can then be stored for an indefinite time in a very small storage space compared to videos, allowing users to quickly navigate and verify significant events without exhaustive manual review. Qualitative evaluations result in 80% and 70% accuracy in temporal and spatial quality and consistency of the pipeline respectively.

Title: Synthetic Fungi Datasets: A Time-Aligned Approach

• Authors: A. Rani, D. O. Arroyo, P. Durdevic
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02855
• Pdf URL: https://arxiv.org/pdf/2501.02855
• Copy Paste: [[2501.02855]] Synthetic Fungi Datasets: A Time-Aligned Approach(https://arxiv.org/abs/2501.02855)
• Keywords: robust
• Abstract: Fungi undergo dynamic morphological transformations throughout their lifecycle, forming intricate networks as they transition from spores to mature mycelium structures. To support the study of these time-dependent processes, we present a synthetic, time-aligned image dataset that models key stages of fungal growth. This dataset systematically captures phenomena such as spore size reduction, branching dynamics, and the emergence of complex mycelium networks. The controlled generation process ensures temporal consistency, scalability, and structural alignment, addressing the limitations of real-world fungal datasets. Optimized for deep learning (DL) applications, this dataset facilitates the development of models for classifying growth stages, predicting fungal development, and analyzing morphological patterns over time. With applications spanning agriculture, medicine, and industrial mycology, this resource provides a robust foundation for automating fungal analysis, enhancing disease monitoring, and advancing fungal biology research through artificial intelligence.

Title: A Novel Vision Transformer for Camera-LiDAR Fusion based Traffic Object Segmentation

• Authors: Toomas Tahves, Junyi Gu, Mauro Bellone, Raivo Sell
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02858
• Pdf URL: https://arxiv.org/pdf/2501.02858
• Copy Paste: [[2501.02858]] A Novel Vision Transformer for Camera-LiDAR Fusion based Traffic Object Segmentation(https://arxiv.org/abs/2501.02858)
• Keywords: transformer, segmentation
• Abstract: This paper presents Camera-LiDAR Fusion Transformer (CLFT) models for traffic object segmentation, which leverage the fusion of camera and LiDAR data using vision transformers. Building on the methodology of visual transformers that exploit the self-attention mechanism, we extend segmentation capabilities with additional classification options to a diverse class of objects including cyclists, traffic signs, and pedestrians across diverse weather conditions. Despite good performance, the models face challenges under adverse conditions which underscores the need for further optimization to enhance performance in darkness and rain. In summary, the CLFT models offer a compelling solution for autonomous driving perception, advancing the state-of-the-art in multimodal fusion and object segmentation, with ongoing efforts required to address existing limitations and fully harness their potential in practical deployments.

Title: Seeing the Whole in the Parts in Self-Supervised Representation Learning

• Authors: Arthur Aubret, Céline Teulière, Jochen Triesch
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02860
• Pdf URL: https://arxiv.org/pdf/2501.02860
• Copy Paste: [[2501.02860]] Seeing the Whole in the Parts in Self-Supervised Representation Learning(https://arxiv.org/abs/2501.02860)
• Keywords: attack, robust
• Abstract: Recent successes in self-supervised learning (SSL) model spatial co-occurrences of visual features either by masking portions of an image or by aggressively cropping it. Here, we propose a new way to model spatial co-occurrences by aligning local representations (before pooling) with a global image representation. We present CO-SSL, a family of instance discrimination methods and show that it outperforms previous methods on several datasets, including ImageNet-1K where it achieves 71.5% of Top-1 accuracy with 100 pre-training epochs. CO-SSL is also more robust to noise corruption, internal corruption, small adversarial attacks, and large training crop sizes. Our analysis further indicates that CO-SSL learns highly redundant local representations, which offers an explanation for its robustness. Overall, our work suggests that aligning local and global representations may be a powerful principle of unsupervised category learning.

Title: IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment

• Authors: Yiming Zhang, Zheng Chang, Wentao Cai, MengXing Ren, Kang Yuan, Yining Sun, Zenghui Ding
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02869
• Pdf URL: https://arxiv.org/pdf/2501.02869
• Copy Paste: [[2501.02869]] IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment(https://arxiv.org/abs/2501.02869)
• Keywords: large language model
• Abstract: Recent researches of large language models(LLM), which is pre-trained on massive general-purpose corpora, have achieved breakthroughs in responding human queries. However, these methods face challenges including limited data insufficiency to support extensive pre-training and can not align responses with users' instructions. To address these issues, we introduce a medical instruction dataset, CMedINS, containing six medical instructions derived from actual medical tasks, which effectively fine-tunes LLM in conjunction with other data. Subsequently, We launch our medical model, IIMedGPT, employing an efficient preference alignment method, Direct preference Optimization(DPO). The results show that our final model outperforms existing medical models in medical this http URL, Code and model checkpoints will be released upon acceptance.

Title: Conditional Mutual Information Based Diffusion Posterior Sampling for Solving Inverse Problems

• Authors: Shayan Mohajer Hamidi, En-Hui Yang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.02880
• Pdf URL: https://arxiv.org/pdf/2501.02880
• Copy Paste: [[2501.02880]] Conditional Mutual Information Based Diffusion Posterior Sampling for Solving Inverse Problems(https://arxiv.org/abs/2501.02880)
• Keywords: diffusion
• Abstract: Inverse problems are prevalent across various disciplines in science and engineering. In the field of computer vision, tasks such as inpainting, deblurring, and super-resolution are commonly formulated as inverse problems. Recently, diffusion models (DMs) have emerged as a promising approach for addressing noisy linear inverse problems, offering effective solutions without requiring additional task-specific training. Specifically, with the prior provided by DMs, one can sample from the posterior by finding the likelihood. Since the likelihood is intractable, it is often approximated in the literature. However, this approximation compromises the quality of the generated images. To overcome this limitation and improve the effectiveness of DMs in solving inverse problems, we propose an information-theoretic approach. Specifically, we maximize the conditional mutual information $\mathrm{I}(\boldsymbol{x}_0; \boldsymbol{y} | \boldsymbol{x}_t)$, where $\boldsymbol{x}_0$ represents the reconstructed signal, $\boldsymbol{y}$ is the measurement, and $\boldsymbol{x}_t$ is the intermediate signal at stage $t$. This ensures that the intermediate signals $\boldsymbol{x}_t$ are generated in a way that the final reconstructed signal $\boldsymbol{x}_0$ retains as much information as possible about the measurement $\boldsymbol{y}$. We demonstrate that this method can be seamlessly integrated with recent approaches and, once incorporated, enhances their performance both qualitatively and quantitatively.

Title: PARF-Net: integrating pixel-wise adaptive receptive fields into hybrid Transformer-CNN network for medical image segmentation

• Authors: Xu Ma, Mengsheng Chen, Junhui Zhang, Lijuan Song, Fang Du, Zhenhua Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02882
• Pdf URL: https://arxiv.org/pdf/2501.02882
• Copy Paste: [[2501.02882]] PARF-Net: integrating pixel-wise adaptive receptive fields into hybrid Transformer-CNN network for medical image segmentation(https://arxiv.org/abs/2501.02882)
• Keywords: extraction, transformer, segmentation
• Abstract: Convolutional neural networks (CNNs) excel in local feature extraction while Transformers are superior in processing global semantic information. By leveraging the strengths of both, hybrid Transformer-CNN networks have become the major architectures in medical image segmentation tasks. However, existing hybrid methods still suffer deficient learning of local semantic features due to the fixed receptive fields of convolutions, and also fall short in effectively integrating local and long-range dependencies. To address these issues, we develop a new method PARF-Net to integrate convolutions of Pixel-wise Adaptive Receptive Fields (Conv-PARF) into hybrid Network for medical image segmentation. The Conv-PARF is introduced to cope with inter-pixel semantic differences and dynamically adjust convolutional receptive fields for each pixel, thus providing distinguishable features to disentangle the lesions with varying shapes and scales from the background. The features derived from the Conv-PARF layers are further processed using hybrid Transformer-CNN blocks under a lightweight manner, to effectively capture local and long-range dependencies, thus boosting the segmentation performance. By assessing PARF-Net on four widely used medical image datasets including MoNuSeg, GlaS, DSB2018 and multi-organ Synapse, we showcase the advantages of our method over the state-of-the-arts. For instance, PARF-Net achieves 84.27% mean Dice on the Synapse dataset, surpassing existing methods by a large margin.

Title: MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMs

• Authors: Hui Sun, Shiyin Lu, Huanyu Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Ming Li
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02885
• Pdf URL: https://arxiv.org/pdf/2501.02885
• Copy Paste: [[2501.02885]] MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMs(https://arxiv.org/abs/2501.02885)
• Keywords: robust, large language model
• Abstract: Video large language models (Video-LLMs) have made significant progress in understanding videos. However, processing multiple frames leads to lengthy visual token sequences, presenting challenges such as the limited context length cannot accommodate the entire video, and the inclusion of irrelevant frames hinders visual perception. Hence, effective frame selection is crucial. This paper emphasizes that frame selection should follow three key principles: query relevance, list-wise diversity, and sequentiality. Existing methods, such as uniform frame sampling and query-frame matching, do not capture all of these principles. Thus, we propose Markov decision determinantal point process with dynamic programming (MDP3) for frame selection, a training-free and model-agnostic method that can be seamlessly integrated into existing Video-LLMs. Our method first estimates frame similarities conditioned on the query using a conditional Gaussian kernel within the reproducing kernel Hilbert space~(RKHS). We then apply the determinantal point process~(DPP) to the similarity matrix to capture both query relevance and list-wise diversity. To incorporate sequentiality, we segment the video and apply DPP within each segment, conditioned on the preceding segment selection, modeled as a Markov decision process~(MDP) for allocating selection sizes across segments. Theoretically, MDP3 provides a \((1 - 1/e)\)-approximate solution to the NP-hard list-wise frame selection problem with pseudo-polynomial time complexity, demonstrating its efficiency. Empirically, MDP3 significantly outperforms existing methods, verifying its effectiveness and robustness.

Title: FoundPAD: Foundation Models Reloaded for Face Presentation Attack Detection

• Authors: Guray Ozgur, Eduarda Caldeira, Tahar Chettaoui, Fadi Boutros, Raghavendra Ramachandra, Naser Damer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02892
• Pdf URL: https://arxiv.org/pdf/2501.02892
• Copy Paste: [[2501.02892]] FoundPAD: Foundation Models Reloaded for Face Presentation Attack Detection(https://arxiv.org/abs/2501.02892)
• Keywords: attack
• Abstract: Although face recognition systems have seen a massive performance enhancement in recent years, they are still targeted by threats such as presentation attacks, leading to the need for generalizable presentation attack detection (PAD) algorithms. Current PAD solutions suffer from two main problems: low generalization to unknown cenarios and large training data requirements. Foundation models (FM) are pre-trained on extensive datasets, achieving remarkable results when generalizing to unseen domains and allowing for efficient task-specific adaption even when little training data are available. In this work, we recognize the potential of FMs to address common PAD problems and tackle the PAD task with an adapted FM for the first time. The FM under consideration is adapted with LoRA weights while simultaneously training a classification header. The resultant architecture, FoundPAD, is highly generalizable to unseen domains, achieving competitive results in several settings under different data availability scenarios and even when using synthetic training data. To encourage reproducibility and facilitate further research in PAD, we publicly release the implementation of FoundPAD at this https URL .

Title: Skillful High-Resolution Ensemble Precipitation Forecasting with an Integrated Deep Learning Framework

• Authors: Shuangshuang He, Hongli Liang, Yuanting Zhang, Xingyuan Yuan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02905
• Pdf URL: https://arxiv.org/pdf/2501.02905
• Copy Paste: [[2501.02905]] Skillful High-Resolution Ensemble Precipitation Forecasting with an Integrated Deep Learning Framework(https://arxiv.org/abs/2501.02905)
• Keywords: diffusion, transformer
• Abstract: High-resolution precipitation forecasts are crucial for providing accurate weather prediction and supporting effective responses to extreme weather events. Traditional numerical models struggle with stochastic subgrid-scale processes, while recent deep learning models often produce blurry results. To address these challenges, we propose a physics-inspired deep learning framework for high-resolution (0.05\textdegree{} $\times$ 0.05\textdegree{}) ensemble precipitation forecasting. Trained on ERA5 and CMPA high-resolution precipitation datasets, the framework integrates deterministic and probabilistic components. The deterministic model, based on a 3D SwinTransformer, captures average precipitation at mesoscale resolution and incorporates strategies to enhance performance, particularly for moderate to heavy rainfall. The probabilistic model employs conditional diffusion in latent space to account for uncertainties in residual precipitation at convective scales. During inference, ensemble members are generated by repeatedly sampling latent variables, enabling the model to represent precipitation uncertainty. Our model significantly enhances spatial resolution and forecast accuracy. Rank histogram shows that the ensemble system is reliable and unbiased. In a case study of heavy precipitation in southern China, the model outputs align more closely with observed precipitation distributions than ERA5, demonstrating superior capability in capturing extreme precipitation events. Additionally, 5-day real-time forecasts show good performance in terms of CSI scores.

Title: Comprehensive Pathological Image Segmentation via Teacher Aggregation for Tumor Microenvironment Analysis

• Authors: Daisuke Komura, Maki Takao, Mieko Ochi, Takumi Onoyama, Hiroto Katoh, Hiroyuki Abe, Hiroyuki Sano, Teppei Konishi, Toshio Kumasaka, Tomoyuki Yokose, Yohei Miyagi, Tetsuo Ushiku, Shumpei Ishikawa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02909
• Pdf URL: https://arxiv.org/pdf/2501.02909
• Copy Paste: [[2501.02909]] Comprehensive Pathological Image Segmentation via Teacher Aggregation for Tumor Microenvironment Analysis(https://arxiv.org/abs/2501.02909)
• Keywords: segmentation
• Abstract: The tumor microenvironment (TME) plays a crucial role in cancer progression and treatment response, yet current methods for its comprehensive analysis in H&E-stained tissue slides face significant limitations in the diversity of tissue cell types and accuracy. Here, we present PAGET (Pathological image segmentation via AGgrEgated Teachers), a new knowledge distillation approach that integrates multiple segmentation models while considering the hierarchical nature of cell types in the TME. By leveraging a unique dataset created through immunohistochemical restaining techniques and existing segmentation models, PAGET enables simultaneous identification and classification of 14 key TME components. We demonstrate PAGET's ability to perform rapid, comprehensive TME segmentation across various tissue types and medical institutions, advancing the quantitative analysis of tumor microenvironments. This method represents a significant step forward in enhancing our understanding of cancer biology and supporting precise clinical decision-making from large-scale histopathology images.

Title: Pointmap-Conditioned Diffusion for Consistent Novel View Synthesis

• Authors: Thang-Anh-Quan Nguyen, Nathan Piasco, Luis Roldão, Moussab Bennehar, Dzmitry Tsishkou, Laurent Caraffa, Jean-Philippe Tarel, Roland Brémond
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02913
• Pdf URL: https://arxiv.org/pdf/2501.02913
• Copy Paste: [[2501.02913]] Pointmap-Conditioned Diffusion for Consistent Novel View Synthesis(https://arxiv.org/abs/2501.02913)
• Keywords: diffusion, generative
• Abstract: In this paper, we present PointmapDiffusion, a novel framework for single-image novel view synthesis (NVS) that utilizes pre-trained 2D diffusion models. Our method is the first to leverage pointmaps (i.e. rasterized 3D scene coordinates) as a conditioning signal, capturing geometric prior from the reference images to guide the diffusion process. By embedding reference attention blocks and a ControlNet for pointmap features, our model balances between generative capability and geometric consistency, enabling accurate view synthesis across varying viewpoints. Extensive experiments on diverse real-world datasets demonstrate that PointmapDiffusion achieves high-quality, multi-view consistent results with significantly fewer trainable parameters compared to other baselines for single-image NVS tasks.

Title: Label-free Concept Based Multiple Instance Learning for Gigapixel Histopathology

• Authors: Susu Sun, Leslie Tessier, Frédérique Meeuwsen, Clément Grisi, Dominique van Midden, Geert Litjens, Christian F. Baumgartner
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02922
• Pdf URL: https://arxiv.org/pdf/2501.02922
• Copy Paste: [[2501.02922]] Label-free Concept Based Multiple Instance Learning for Gigapixel Histopathology(https://arxiv.org/abs/2501.02922)
• Keywords: interpretability
• Abstract: Multiple Instance Learning (MIL) methods allow for gigapixel Whole-Slide Image (WSI) analysis with only slide-level annotations. Interpretability is crucial for safely deploying such algorithms in high-stakes medical domains. Traditional MIL methods offer explanations by highlighting salient regions. However, such spatial heatmaps provide limited insights for end users. To address this, we propose a novel inherently interpretable WSI-classification approach that uses human-understandable pathology concepts to generate explanations. Our proposed Concept MIL model leverages recent advances in vision-language models to directly predict pathology concepts based on image features. The model's predictions are obtained through a linear combination of the concepts identified on the top-K patches of a WSI, enabling inherent explanations by tracing each concept's influence on the prediction. In contrast to traditional concept-based interpretable models, our approach eliminates the need for costly human annotations by leveraging the vision-language model. We validate our method on two widely used pathology datasets: Camelyon16 and PANDA. On both datasets, Concept MIL achieves AUC and accuracy scores over 0.9, putting it on par with state-of-the-art models. We further find that 87.1\% (Camelyon16) and 85.3\% (PANDA) of the top 20 patches fall within the tumor region. A user study shows that the concepts identified by our model align with the concepts used by pathologists, making it a promising strategy for human-interpretable WSI classification.

Title: Self-Attention as a Parametric Endofunctor: A Categorical Framework for Transformer Architectures

• Authors: Charles O'Neill
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02931
• Pdf URL: https://arxiv.org/pdf/2501.02931
• Copy Paste: [[2501.02931]] Self-Attention as a Parametric Endofunctor: A Categorical Framework for Transformer Architectures(https://arxiv.org/abs/2501.02931)
• Keywords: interpretability, transformer
• Abstract: Self-attention mechanisms have revolutionised deep learning architectures, but their mathematical foundations remain incompletely understood. We establish that these mechanisms can be formalised through categorical algebra, presenting a framework that focuses on the linear components of self-attention. We prove that the query, key, and value maps in self-attention naturally form a parametric endofunctor in the 2-category $\mathbf{Para}(\mathbf{Vect})$ of parametric morphisms. We show that stacking multiple self-attention layers corresponds to constructing the free monad on this endofunctor. For positional encodings, we demonstrate that strictly additive position embeddings constitute monoid actions on the embedding space, while standard sinusoidal encodings, though not additive, possess a universal property among faithful position-preserving functors. We establish that the linear portions of self-attention exhibit natural equivariance properties with respect to permutations of input tokens. Finally, we prove that the ``circuits'' identified in mechanistic interpretability correspond precisely to compositions of parametric morphisms in our framework. This categorical perspective unifies geometric, algebraic, and interpretability-based approaches to transformer analysis, while making explicit the mathematical structures underlying attention mechanisms. Our treatment focuses exclusively on linear maps, setting aside nonlinearities like softmax and layer normalisation, which require more sophisticated categorical structures. Our results extend recent work on categorical foundations for deep learning while providing insights into the algebraic structure of attention mechanisms.

Title: Echomix: a Strong Anonymity System with Messaging

• Authors: Ewa J Infeld, David Stainton, Leif Ryge, Threebit Hacker
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.02933
• Pdf URL: https://arxiv.org/pdf/2501.02933
• Copy Paste: [[2501.02933]] Echomix: a Strong Anonymity System with Messaging(https://arxiv.org/abs/2501.02933)
• Keywords: privacy, protect, attack, robust
• Abstract: Echomix is a practical mix network framework and a suite of associated protocols providing strong metadata privacy against realistic modern adversaries. It is distinguished from other anonymity systems by a resistance to traffic analysis by global adversaries, compromised contacts and network infrastructure, quantum decryption algorithms, and statistical and confirmation attacks typical for multi-client messaging setting. It is implemented as Katzenpost, a robust software project, and used in multiple deployed systems, and features relatively low latency and bandwidth overhead. The contributions of this paper are: (1) Improvements on leading mix network designs, supported by rigorous analysis. These include solutions to crucial vulnerabilities to traffic analysis, malicious servers and active attacks. (2) A cryptographic group messaging protocol with strong metadata protection guarantees and reliability. (3) Hybrid post-quantum nested packet encryption.

Title: 4D-CS: Exploiting Cluster Prior for 4D Spatio-Temporal LiDAR Semantic Segmentation

• Authors: Jiexi Zhong, Zhiheng Li, Yubo Cui, Zheng Fang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02937
• Pdf URL: https://arxiv.org/pdf/2501.02937
• Copy Paste: [[2501.02937]] 4D-CS: Exploiting Cluster Prior for 4D Spatio-Temporal LiDAR Semantic Segmentation(https://arxiv.org/abs/2501.02937)
• Keywords: segmentation
• Abstract: Semantic segmentation of LiDAR points has significant value for autonomous driving and mobile robot systems. Most approaches explore spatio-temporal information of multi-scan to identify the semantic classes and motion states for each point. However, these methods often overlook the segmentation consistency in space and time, which may result in point clouds within the same object being predicted as different categories. To handle this issue, our core idea is to generate cluster labels across multiple frames that can reflect the complete spatial structure and temporal information of objects. These labels serve as explicit guidance for our dual-branch network, 4D-CS, which integrates point-based and cluster-based branches to enable more consistent segmentation. Specifically, in the point-based branch, we leverage historical knowledge to enrich the current feature through temporal fusion on multiple views. In the cluster-based branch, we propose a new strategy to produce cluster labels of foreground objects and apply them to gather point-wise information to derive cluster features. We then merge neighboring clusters across multiple scans to restore missing features due to occlusion. Finally, in the point-cluster fusion stage, we adaptively fuse the information from the two branches to optimize segmentation results. Extensive experiments confirm the effectiveness of the proposed method, and we achieve state-of-the-art results on the multi-scan semantic and moving object segmentation on SemanticKITTI and nuScenes datasets. The code will be available at this https URL.

Title: MSA-CNN: A Lightweight Multi-Scale CNN with Attention for Sleep Stage Classification

• Authors: Stephan Goerttler, Yucheng Wang, Emadeldeen Eldele, Min Wu, Fei He
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2501.02949
• Pdf URL: https://arxiv.org/pdf/2501.02949
• Copy Paste: [[2501.02949]] MSA-CNN: A Lightweight Multi-Scale CNN with Attention for Sleep Stage Classification(https://arxiv.org/abs/2501.02949)
• Keywords: extraction
• Abstract: Recent advancements in machine learning-based signal analysis, coupled with open data initiatives, have fuelled efforts in automatic sleep stage classification. Despite the proliferation of classification models, few have prioritised reducing model complexity, which is a crucial factor for practical applications. In this work, we introduce Multi-Scale and Attention Convolutional Neural Network (MSA-CNN), a lightweight architecture featuring as few as ~10,000 parameters. MSA-CNN leverages a novel multi-scale module employing complementary pooling to eliminate redundant filter parameters and dense convolutions. Model complexity is further reduced by separating temporal and spatial feature extraction and using cost-effective global spatial convolutions. This separation of tasks not only reduces model complexity but also mirrors the approach used by human experts in sleep stage scoring. We evaluated both small and large configurations of MSA-CNN against nine state-of-the-art baseline models across three public datasets, treating univariate and multivariate models separately. Our evaluation, based on repeated cross-validation and re-evaluation of all baseline models, demonstrated that the large MSA-CNN outperformed all baseline models on all three datasets in terms of accuracy and Cohen's kappa, despite its significantly reduced parameter count. Lastly, we explored various model variants and conducted an in-depth analysis of the key modules and techniques, providing deeper insights into the underlying mechanisms. The code for our models, baselines, and evaluation procedures is available at this https URL.

Title: SceneVTG++: Controllable Multilingual Visual Text Generation in the Wild

• Authors: Jiawei Liu, Yuanzhi Zhu, Feiyu Gao, Zhibo Yang, Peng Wang, Junyang Lin, Xinggang Wang, Wenyu Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02962
• Pdf URL: https://arxiv.org/pdf/2501.02962
• Copy Paste: [[2501.02962]] SceneVTG++: Controllable Multilingual Visual Text Generation in the Wild(https://arxiv.org/abs/2501.02962)
• Keywords: diffusion, large language model
• Abstract: Generating visual text in natural scene images is a challenging task with many unsolved problems. Different from generating text on artificially designed images (such as posters, covers, cartoons, etc.), the text in natural scene images needs to meet the following four key criteria: (1) Fidelity: the generated text should appear as realistic as a photograph and be completely accurate, with no errors in any of the strokes. (2) Reasonability: the text should be generated on reasonable carrier areas (such as boards, signs, walls, etc.), and the generated text content should also be relevant to the scene. (3) Utility: the generated text can facilitate to the training of natural scene OCR (Optical Character Recognition) tasks. (4) Controllability: The attribute of the text (such as font and color) should be controllable as this http URL this paper, we propose a two stage method, SceneVTG++, which simultaneously satisfies the four aspects mentioned above. SceneVTG++ consists of a Text Layout and Content Generator (TLCG) and a Controllable Local Text Diffusion (CLTD). The former utilizes the world knowledge of multi modal large language models to find reasonable text areas and recommend text content according to the nature scene background images, while the latter generates controllable multilingual text based on the diffusion model. Through extensive experiments, we respectively verified the effectiveness of TLCG and CLTD, and demonstrated the state-of-the-art text generation performance of SceneVTG++. In addition, the generated images have superior utility in OCR tasks like text detection and text recognition. Codes and datasets will be available.

Title: Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild

• Authors: Wanpeng Hu, Haodi Liu, Lin Chen, Feng Zhou, Changming Xiao, Qi Yang, Changshui Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02964
• Pdf URL: https://arxiv.org/pdf/2501.02964
• Copy Paste: [[2501.02964]] Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild(https://arxiv.org/abs/2501.02964)
• Keywords: large language model
• Abstract: Complex visual reasoning remains a key challenge today. Typically, the challenge is tackled using methodologies such as Chain of Thought (COT) and visual instruction tuning. However, how to organically combine these two methodologies for greater success remains unexplored. Also, issues like hallucinations and high training cost still need to be addressed. In this work, we devise an innovative multi-round training and reasoning framework suitable for lightweight Multimodal Large Language Models (MLLMs). Our self-questioning approach heuristically guides MLLMs to focus on visual clues relevant to the target problem, reducing hallucinations and enhancing the model's ability to describe fine-grained image details. This ultimately enables the model to perform well in complex visual reasoning and question-answering tasks. We have named this framework Socratic Questioning(SQ). To facilitate future research, we create a multimodal mini-dataset named CapQA, which includes 1k images of fine-grained activities, for visual instruction tuning and evaluation, our proposed SQ method leads to a 31.2% improvement in the hallucination score. Our extensive experiments on various benchmarks demonstrate SQ's remarkable capabilities in heuristic self-questioning, zero-shot visual reasoning and hallucination mitigation. Our model and code will be publicly available.

Title: Leader Rotation Is Not Enough: Scrutinizing Leadership Democracy of Chained BFT Consensus

• Authors: Yining Tang, Runchao Han, Jianyu Niu, Chen Feng, Yinqian Zhang
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.02970
• Pdf URL: https://arxiv.org/pdf/2501.02970
• Copy Paste: [[2501.02970]] Leader Rotation Is Not Enough: Scrutinizing Leadership Democracy of Chained BFT Consensus(https://arxiv.org/abs/2501.02970)
• Keywords: attack
• Abstract: With the growing popularity of blockchains, modern chained BFT protocols combining chaining and leader rotation to obtain better efficiency and leadership democracy have received increasing interest. Although the efficiency provisions of chained BFT protocols have been thoroughly analyzed, the leadership democracy has received little attention in prior work. In this paper, we scrutinize the leadership democracy of four representative chained BFT protocols, especially under attack. To this end, we propose a unified framework with two evaluation metrics, i.e., chain quality and censorship resilience, and quantitatively analyze chosen protocols through the Markov Decision Process (MDP). With this framework, we further examine the impact of two key components, i.e., voting pattern and leader rotation on leadership democracy. Our results indicate that leader rotation is not enough to provide the leadership democracy guarantee; an adversary could utilize the design, e.g., voting pattern, to deteriorate the leadership democracy significantly. Based on the analysis results, we propose customized countermeasures for three evaluated protocols to improve their leadership democracy with only slight protocol overhead and no change of consensus rules. We also discuss future directions toward building more democratic chained BFT protocols.

Title: Proof-of-Data: A Consensus Protocol for Collaborative Intelligence

• Authors: Huiwen Liu, Feida Zhu, Ling Cheng
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02971
• Pdf URL: https://arxiv.org/pdf/2501.02971
• Copy Paste: [[2501.02971]] Proof-of-Data: A Consensus Protocol for Collaborative Intelligence(https://arxiv.org/abs/2501.02971)
• Keywords: privacy, attack, federate, fair
• Abstract: Existing research on federated learning has been focused on the setting where learning is coordinated by a centralized entity. Yet the greatest potential of future collaborative intelligence would be unleashed in a more open and democratized setting with no central entity in a dominant role, referred to as "decentralized federated learning". New challenges arise accordingly in achieving both correct model training and fair reward allocation with collective effort among all participating nodes, especially with the threat of the Byzantine node jeopardising both tasks. In this paper, we propose a blockchain-based decentralized Byzantine fault-tolerant federated learning framework based on a novel Proof-of-Data (PoD) consensus protocol to resolve both the "trust" and "incentive" components. By decoupling model training and contribution accounting, PoD is able to enjoy not only the benefit of learning efficiency and system liveliness from asynchronous societal-scale PoW-style learning but also the finality of consensus and reward allocation from epoch-based BFT-style voting. To mitigate false reward claims by data forgery from Byzantine attacks, a privacy-aware data verification and contribution-based reward allocation mechanism is designed to complete the framework. Our evaluation results show that PoD demonstrates performance in model training close to that of the centralized counterpart while achieving trust in consensus and fairness for reward allocation with a fault tolerance ratio of 1/3.

Title: HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos

• Authors: Jinglei Zhang, Jiankang Deng, Chao Ma, Rolandos Alexandros Potamias
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02973
• Pdf URL: https://arxiv.org/pdf/2501.02973
• Copy Paste: [[2501.02973]] HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos(https://arxiv.org/abs/2501.02973)
• Keywords: robust
• Abstract: Despite the advent in 3D hand pose estimation, current methods predominantly focus on single-image 3D hand reconstruction in the camera frame, overlooking the world-space motion of the hands. Such limitation prohibits their direct use in egocentric video settings, where hands and camera are continuously in motion. In this work, we propose HaWoR, a high-fidelity method for hand motion reconstruction in world coordinates from egocentric videos. We propose to decouple the task by reconstructing the hand motion in the camera space and estimating the camera trajectory in the world coordinate system. To achieve precise camera trajectory estimation, we propose an adaptive egocentric SLAM framework that addresses the shortcomings of traditional SLAM methods, providing robust performance under challenging camera dynamics. To ensure robust hand motion trajectories, even when the hands move out of view frustum, we devise a novel motion infiller network that effectively completes the missing frames of the sequence. Through extensive quantitative and qualitative evaluations, we demonstrate that HaWoR achieves state-of-the-art performance on both hand motion reconstruction and world-frame camera trajectory estimation under different egocentric benchmark datasets. Code and models are available on this https URL .

Title: STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution

• Authors: Rui Xie, Yinhong Liu, Penghao Zhou, Chen Zhao, Jun Zhou, Kai Zhang, Zhenyu Zhang, Jian Yang, Zhenheng Yang, Ying Tai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02976
• Pdf URL: https://arxiv.org/pdf/2501.02976
• Copy Paste: [[2501.02976]] STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution(https://arxiv.org/abs/2501.02976)
• Keywords: robust, diffusion, generative
• Abstract: Image diffusion models have been adapted for real-world video super-resolution to tackle over-smoothing issues in GAN-based methods. However, these models struggle to maintain temporal consistency, as they are trained on static images, limiting their ability to capture temporal dynamics effectively. Integrating text-to-video (T2V) models into video super-resolution for improved temporal modeling is straightforward. However, two key challenges remain: artifacts introduced by complex degradations in real-world scenarios, and compromised fidelity due to the strong generative capacity of powerful T2V models (\textit{e.g.}, CogVideoX-5B). To enhance the spatio-temporal quality of restored videos, we introduce\textbf{~\name} (\textbf{S}patial-\textbf{T}emporal \textbf{A}ugmentation with T2V models for \textbf{R}eal-world video super-resolution), a novel approach that leverages T2V models for real-world video super-resolution, achieving realistic spatial details and robust temporal consistency. Specifically, we introduce a Local Information Enhancement Module (LIEM) before the global attention block to enrich local details and mitigate degradation artifacts. Moreover, we propose a Dynamic Frequency (DF) Loss to reinforce fidelity, guiding the model to focus on different frequency components across diffusion steps. Extensive experiments demonstrate\textbf{~\name}~outperforms state-of-the-art methods on both synthetic and real-world datasets.

Title: Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation

• Authors: Zhi Qu, Yiran Wang, Jiannan Mao, Chenchen Ding, Hideki Tanaka, Masao Utiyama, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02979
• Pdf URL: https://arxiv.org/pdf/2501.02979
• Copy Paste: [[2501.02979]] Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation(https://arxiv.org/abs/2501.02979)
• Keywords: large language model
• Abstract: The multilingual neural machine translation (MNMT) enables arbitrary translations across multiple languages by training a model with limited parameters using parallel data only. However, the performance of such MNMT models still lags behind that of large language models (LLMs), limiting their practicality. In this work, we address this limitation by introducing registering to achieve the new state-of-the-art of decoder-only MNMT models. Specifically, we insert a set of artificial tokens specifying the target language, called registers, into the input sequence between the source and target tokens. By modifying the attention mask, the target token generation only pays attention to the activation of registers, representing the source tokens in the target language space. Experiments on EC-40, a large-scale benchmark, show that our method outperforms related methods driven by optimizing multilingual representations. We further scale up and collect 9.3 billion sentence pairs across 24 languages from public datasets to pre-train two models, namely MITRE (multilingual translation with registers). One of them, MITRE-913M, outperforms NLLB-3.3B, achieves comparable performance with commercial LLMs, and shows strong adaptability in fine-tuning. Finally, we open-source our models to facilitate further research and development in MNMT: this https URL.

Title: CONTINUUM: Detecting APT Attacks through Spatial-Temporal Graph Neural Networks

• Authors: Atmane Ayoub Mansour Bahara, Kamel Soaïd Ferrahia, Mohamed-Lamine Messai, Hamida Seba, Karima Amrouche
• Subjects: cs.CR, cs.AI, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.02981
• Pdf URL: https://arxiv.org/pdf/2501.02981
• Copy Paste: [[2501.02981]] CONTINUUM: Detecting APT Attacks through Spatial-Temporal Graph Neural Networks(https://arxiv.org/abs/2501.02981)
• Keywords: security, privacy, defense, attack, steal, federate
• Abstract: Advanced Persistent Threats (APTs) represent a significant challenge in cybersecurity due to their sophisticated and stealthy nature. Traditional Intrusion Detection Systems (IDS) often fall short in detecting these multi-stage attacks. Recently, Graph Neural Networks (GNNs) have been employed to enhance IDS capabilities by analyzing the complex relationships within networked data. However, existing GNN-based solutions are hampered by high false positive rates and substantial resource consumption. In this paper, we present a novel IDS designed to detect APTs using a Spatio-Temporal Graph Neural Network Autoencoder. Our approach leverages spatial information to understand the interactions between entities within a graph and temporal information to capture the evolution of the graph over time. This dual perspective is crucial for identifying the sequential stages of APTs. Furthermore, to address privacy and scalability concerns, we deploy our architecture in a federated learning environment. This setup ensures that local data remains on-premise while encrypted model-weights are shared and aggregated using homomorphic encryption, maintaining data privacy and security. Our evaluation shows that this system effectively detects APTs with lower false positive rates and optimized resource usage compared to existing methods, highlighting the potential of spatio-temporal analysis and federated learning in enhancing cybersecurity defenses.

Title: SurgRIPE challenge: Benchmark of Surgical Robot Instrument Pose Estimation

• Authors: Haozheng Xu, Alistair Weld, Chi Xu, Alfie Roddan, Joao Cartucho, Mert Asim Karaoglu, Alexander Ladikos, Yangke Li, Yiping Li, Daiyun Shen, Shoujie Yang, Geonhee Lee, Seyeon Park, Jongho Shin, Young-Gon Kim, Lucy Fothergill, Dominic Jones, Pietro Valdastri, Duygu Sarikaya, Stamatia Giannarou
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.02990
• Pdf URL: https://arxiv.org/pdf/2501.02990
• Copy Paste: [[2501.02990]] SurgRIPE challenge: Benchmark of Surgical Robot Instrument Pose Estimation(https://arxiv.org/abs/2501.02990)
• Keywords: robust
• Abstract: Accurate instrument pose estimation is a crucial step towards the future of robotic surgery, enabling applications such as autonomous surgical task execution. Vision-based methods for surgical instrument pose estimation provide a practical approach to tool tracking, but they often require markers to be attached to the instruments. Recently, more research has focused on the development of marker-less methods based on deep learning. However, acquiring realistic surgical data, with ground truth instrument poses, required for deep learning training, is challenging. To address the issues in surgical instrument pose estimation, we introduce the Surgical Robot Instrument Pose Estimation (SurgRIPE) challenge, hosted at the 26th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) in 2023. The objectives of this challenge are: (1) to provide the surgical vision community with realistic surgical video data paired with ground truth instrument poses, and (2) to establish a benchmark for evaluating markerless pose estimation methods. The challenge led to the development of several novel algorithms that showcased improved accuracy and robustness over existing methods. The performance evaluation study on the SurgRIPE dataset highlights the potential of these advanced algorithms to be integrated into robotic surgery systems, paving the way for more precise and autonomous surgical procedures. The SurgRIPE challenge has successfully established a new benchmark for the field, encouraging further research and development in surgical robot instrument pose estimation.

Title: TransPixar: Advancing Text-to-Video Generation with Transparency

• Authors: Luozhou Wang, Yijun Li, Zhifei Chen, Jui-Hsien Wang, Zhifei Zhang, He Zhang, Zhe Lin, Yingcong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03006
• Pdf URL: https://arxiv.org/pdf/2501.03006
• Copy Paste: [[2501.03006]] TransPixar: Advancing Text-to-Video Generation with Transparency(https://arxiv.org/abs/2501.03006)
• Keywords: diffusion, transformer, generative
• Abstract: Text-to-video generative models have made significant strides, enabling diverse applications in entertainment, advertising, and education. However, generating RGBA video, which includes alpha channels for transparency, remains a challenge due to limited datasets and the difficulty of adapting existing models. Alpha channels are crucial for visual effects (VFX), allowing transparent elements like smoke and reflections to blend seamlessly into scenes. We introduce TransPixar, a method to extend pretrained video models for RGBA generation while retaining the original RGB capabilities. TransPixar leverages a diffusion transformer (DiT) architecture, incorporating alpha-specific tokens and using LoRA-based fine-tuning to jointly generate RGB and alpha channels with high consistency. By optimizing attention mechanisms, TransPixar preserves the strengths of the original RGB model and achieves strong alignment between RGB and alpha channels despite limited training data. Our approach effectively generates diverse and consistent RGBA videos, advancing the possibilities for VFX and interactive content creation.

Title: Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning

• Authors: Zhen Li, Yupeng Su, Runming Yang, Zhongwei Xie, Ngai Wong, Hongxia Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03035
• Pdf URL: https://arxiv.org/pdf/2501.03035
• Copy Paste: [[2501.03035]] Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning(https://arxiv.org/abs/2501.03035)
• Keywords: large language model
• Abstract: Large language models have achieved significant advancements in complex mathematical reasoning benchmarks, such as MATH. However, their substantial computational requirements present challenges for practical deployment. Model quantization has emerged as an effective strategy to reduce memory usage and computational costs by employing lower precision and bit-width representations. In this study, we systematically evaluate the impact of quantization on mathematical reasoning tasks. We introduce a multidimensional evaluation framework that qualitatively assesses specific capability dimensions and conduct quantitative analyses on the step-by-step outputs of various quantization methods. Our results demonstrate that quantization differentially affects numerical computation and reasoning planning abilities, identifying key areas where quantized models experience performance degradation.

Title: ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events

• Authors: Duygu Sezen Islakoglu, Jan-Christoph Kalo
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03040
• Pdf URL: https://arxiv.org/pdf/2501.03040
• Copy Paste: [[2501.03040]] ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events(https://arxiv.org/abs/2501.03040)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have achieved remarkable success in various NLP tasks, yet they still face significant challenges in reasoning and arithmetic. Temporal reasoning, a critical component of natural language understanding, has raised increasing research attention. However, comprehensive testing of Allen's interval relations (e.g., before, after, during) -- a fundamental framework for temporal relationships -- remains underexplored. To fill this gap, we present ChronoSense, a new benchmark for evaluating LLMs' temporal understanding. It includes 16 tasks, focusing on identifying the Allen relation between two temporal events and temporal arithmetic, using both abstract events and real-world data from Wikidata. We assess the performance of seven recent LLMs using this benchmark and the results indicate that models handle Allen relations, even symmetrical ones, quite differently. Moreover, the findings suggest that the models may rely on memorization to answer time-related questions. Overall, the models' low performance highlights the need for improved temporal understanding in LLMs and ChronoSense offers a robust framework for future research in this area. Our dataset and the source code are available at this https URL.

Title: Survival Analysis Revisited: Understanding and Unifying Poisson, Exponential, and Cox Models in Fall Risk Analysis

• Authors: Tianhua Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03058
• Pdf URL: https://arxiv.org/pdf/2501.03058
• Copy Paste: [[2501.03058]] Survival Analysis Revisited: Understanding and Unifying Poisson, Exponential, and Cox Models in Fall Risk Analysis(https://arxiv.org/abs/2501.03058)
• Keywords: robust, interpretability, explainability
• Abstract: This paper explores foundational and applied aspects of survival analysis, using fall risk assessment as a case study. It revisits key time-related probability distributions and statistical methods, including logistic regression, Poisson regression, Exponential regression, and the Cox Proportional Hazards model, offering a unified perspective on their relationships within the survival analysis framework. A contribution of this work is the step-by-step derivation and clarification of the relationships among these models, particularly demonstrating that Poisson regression in the survival context is a specific case of the Cox model. These insights address gaps in understanding and reinforce the simplicity and interpretability of survival models. The paper also emphasizes the practical utility of survival analysis by connecting theoretical insights with real-world applications. In the context of fall detection, it demonstrates how these models can simultaneously predict fall risk, analyze contributing factors, and estimate time-to-event outcomes within a single streamlined framework. In contrast, advanced deep learning methods often require complex post-hoc interpretation and separate training for different tasks particularly when working with structured numerical data. This highlights the enduring relevance of classical statistical frameworks and makes survival models especially valuable in healthcare settings, where explainability and robustness are critical. By unifying foundational concepts and offering a cohesive perspective on time-to-event analysis, this work serves as an accessible resource for understanding survival models and applying them effectively to diverse analytical challenges.

Title: Design and implementation of tools to build an ontology of Security Requirements for Internet of Medical Things

• Authors: Daniel Naro, Jaime Delgado, Silvia Llorente, Amanda Palomo
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.03067
• Pdf URL: https://arxiv.org/pdf/2501.03067
• Copy Paste: [[2501.03067]] Design and implementation of tools to build an ontology of Security Requirements for Internet of Medical Things(https://arxiv.org/abs/2501.03067)
• Keywords: security
• Abstract: When developing devices, architectures and services for the Internet of Medical Things (IoMT) world, manufacturers or integrators must be aware of the security requirements expressed by both laws and specifications. To provide tools guiding through these requirements and to assure a third party of the correct compliance, an ontology charting the relevant laws and specifications (for the European context) is very useful. We here address the development of this ontology. Due to the very high number and size of the considered specification documents, we have put in place a methodology and tools to simplify the transition from natural text to an ontology. The first step is a manual highlighting of relevant concepts in the corpus, then a manual translation to XML/XSD is operated. We have developed a tool allowing us to convert this semi-structured data into an ontology. Because the different specifications use similar but different wording, our approach favors the creation of similar instances in the ontology. To improve the ontology simplification through instance merging, we consider the use of LLMs. The responses of the LLMs are compared against our manually defined correct responses. The quality of the responses of the automated system does not prove to be good enough to be trusted blindly, and should only be used as a starting point for a manual correction.

Title: AIF-SFDA: Autonomous Information Filter-driven Source-Free Domain Adaptation for Medical Image Segmentation

• Authors: Haojin Li, Heng Li, Jianyu Chen, Rihan Zhong, Ke Niu, Huazhu Fu, Jiang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03074
• Pdf URL: https://arxiv.org/pdf/2501.03074
• Copy Paste: [[2501.03074]] AIF-SFDA: Autonomous Information Filter-driven Source-Free Domain Adaptation for Medical Image Segmentation(https://arxiv.org/abs/2501.03074)
• Keywords: privacy, segmentation
• Abstract: Decoupling domain-variant information (DVI) from domain-invariant information (DII) serves as a prominent strategy for mitigating domain shifts in the practical implementation of deep learning algorithms. However, in medical settings, concerns surrounding data collection and privacy often restrict access to both training and test data, hindering the empirical decoupling of information by existing methods. To tackle this issue, we propose an Autonomous Information Filter-driven Source-free Domain Adaptation (AIF-SFDA) algorithm, which leverages a frequency-based learnable information filter to autonomously decouple DVI and DII. Information Bottleneck (IB) and Self-supervision (SS) are incorporated to optimize the learnable frequency filter. The IB governs the information flow within the filter to diminish redundant DVI, while SS preserves DII in alignment with the specific task and image modality. Thus, the autonomous information filter can overcome domain shifts relying solely on target data. A series of experiments covering various medical image modalities and segmentation tasks were conducted to demonstrate the benefits of AIF-SFDA through comparisons with leading algorithms and ablation studies. The code is available at this https URL.

Title: LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases

• Authors: Dylan Bouchard, Mohit Singh Chauhan, David Skarbrevik, Viren Bajaj, Zeya Ahmad
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03112
• Pdf URL: https://arxiv.org/pdf/2501.03112
• Copy Paste: [[2501.03112]] LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases(https://arxiv.org/abs/2501.03112)
• Keywords: protect, fair, large language model
• Abstract: Large Language Models (LLMs) have been observed to exhibit bias in numerous ways, potentially creating or worsening outcomes for specific groups identified by protected attributes such as sex, race, sexual orientation, or age. To help address this gap, we introduce LangFair, an open-source Python package that aims to equip LLM practitioners with the tools to evaluate bias and fairness risks relevant to their specific use cases. The package offers functionality to easily generate evaluation datasets, comprised of LLM responses to use-case-specific prompts, and subsequently calculate applicable metrics for the practitioner's use case. To guide in metric selection, LangFair offers an actionable decision framework.

Title: TEE-based Key-Value Stores: a Survey

• Authors: Aghiles Ait Messaoud, Sonia Ben Mokhtar, Anthony Simonet-Boulogne
• Subjects: cs.CR, cs.DB
• Abstract URL: https://arxiv.org/abs/2501.03118
• Pdf URL: https://arxiv.org/pdf/2501.03118
• Copy Paste: [[2501.03118]] TEE-based Key-Value Stores: a Survey(https://arxiv.org/abs/2501.03118)
• Keywords: secure, security
• Abstract: Key-Value Stores (KVSs) are No-SQL databases that store data as key-value pairs and have gained popularity due to their simplicity, scalability, and fast retrieval capabilities. However, storing sensitive data in KVSs requires strong security properties to prevent data leakage and unauthorized tampering. While software (SW)-based encryption techniques are commonly used to maintain data confidentiality and integrity, they suffer from several drawbacks. They strongly assume trust in the hosting system stack and do not secure data during processing unless using performance-heavy techniques (e.g., homomorphic encryption). Alternatively, Trusted Execution Environments (TEEs) provide a solution that enforces the confidentiality and integrity of code and data at the CPU level, allowing users to build trusted applications in an untrusted environment. They also secure data in use by providing an encapsulated processing environment called enclave. Nevertheless, TEEs come with their own set of drawbacks, including performance issues due to memory size limitations and CPU context switching. This paper examines the state of the art in TEE-based confidential KVSs and highlights common design strategies used in KVSs to leverage TEE security features while overcoming their inherent limitations. This work aims to provide a comprehensive understanding of the use of TEEs in KVSs and to identify research directions for future work.

Title: From Models to Network Topologies: A Topology Inference Attack in Decentralized Federated Learning

• Authors: Chao Feng, Yuanzhe Gao, Alberto Huertas Celdran, Gerome Bovet, Burkhard Stiller
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03119
• Pdf URL: https://arxiv.org/pdf/2501.03119
• Copy Paste: [[2501.03119]] From Models to Network Topologies: A Topology Inference Attack in Decentralized Federated Learning(https://arxiv.org/abs/2501.03119)
• Keywords: security, privacy, attack, robust, federate
• Abstract: Federated Learning (FL) is widely recognized as a privacy-preserving machine learning paradigm due to its model-sharing mechanism that avoids direct data exchange. However, model training inevitably leaves exploitable traces that can be used to infer sensitive information. In Decentralized FL (DFL), the overlay topology significantly influences its models' convergence, robustness, and security. This study explores the feasibility of inferring the overlay topology of DFL systems based solely on model behavior, introducing a novel Topology Inference Attack. A taxonomy of topology inference attacks is proposed, categorizing them by the attacker's capabilities and knowledge. Practical attack strategies are developed for different scenarios, and quantitative experiments are conducted to identify key factors influencing the attack effectiveness. Experimental results demonstrate that analyzing only the public models of individual nodes can accurately infer the DFL topology, underscoring the risk of sensitive information leakage in DFL systems. This finding offers valuable insights for improving privacy preservation in decentralized learning environments.

Title: CAT: Content-Adaptive Image Tokenization

• Authors: Junhong Shen, Kushal Tirumala, Michihiro Yasunaga, Ishan Misra, Luke Zettlemoyer, Lili Yu, Chunting Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03120
• Pdf URL: https://arxiv.org/pdf/2501.03120
• Copy Paste: [[2501.03120]] CAT: Content-Adaptive Image Tokenization(https://arxiv.org/abs/2501.03120)
• Keywords: robust, diffusion, transformer, large language model
• Abstract: Most existing image tokenizers encode images into a fixed number of tokens or patches, overlooking the inherent variability in image complexity. To address this, we introduce Content-Adaptive Tokenizer (CAT), which dynamically adjusts representation capacity based on the image content and encodes simpler images into fewer tokens. We design a caption-based evaluation system that leverages large language models (LLMs) to predict content complexity and determine the optimal compression ratio for a given image, taking into account factors critical to human perception. Trained on images with diverse compression ratios, CAT demonstrates robust performance in image reconstruction. We also utilize its variable-length latent representations to train Diffusion Transformers (DiTs) for ImageNet generation. By optimizing token allocation, CAT improves the FID score over fixed-ratio baselines trained with the same flops and boosts the inference throughput by 18.5%.

Title: PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models

• Authors: Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, Yu Cheng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03124
• Pdf URL: https://arxiv.org/pdf/2501.03124
• Copy Paste: [[2501.03124]] PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models(https://arxiv.org/abs/2501.03124)
• Keywords: robust, large language model
• Abstract: Process-level Reward Models (PRMs) are crucial for complex reasoning and decision-making tasks, where each intermediate step plays an important role in the reasoning process. Since language models are prone to various types of errors during the reasoning process, PRMs are required to possess nuanced capabilities for detecting various implicit error types in real-world scenarios. However, current benchmarks primarily focus on step correctness, failing to evaluate PRMs' performance systematically. To address this gap, we introduce PRMBench, a process-level benchmark specifically designed to assess the fine-grained error detection capabilities of PRMs. PRMBench comprises 6,216 carefully designed problems and 83,456 step-level labels, evaluating models across multiple dimensions, including simplicity, soundness, and sensitivity. In our experiments on 15 models, spanning both open-source PRMs and closed-source large language models prompted as critic models, we uncover significant weaknesses in current PRMs. These findings underscore the challenges inherent in process-level evaluation and highlight key directions for future research. We hope PRMBench can be a robust bench for advancing research on PRM evaluation and development.

Title: VicSim: Enhancing Victim Simulation with Emotional and Linguistic Fidelity

• Authors: Yerong Li, Yiren Liu, Yun Huang
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.03139
• Pdf URL: https://arxiv.org/pdf/2501.03139
• Copy Paste: [[2501.03139]] VicSim: Enhancing Victim Simulation with Emotional and Linguistic Fidelity(https://arxiv.org/abs/2501.03139)
• Keywords: large language model
• Abstract: Scenario-based training has been widely adopted in many public service sectors. Recent advancements in Large Language Models (LLMs) have shown promise in simulating diverse personas to create these training scenarios. However, little is known about how LLMs can be developed to simulate victims for scenario-based training purposes. In this paper, we introduce VicSim (victim simulator), a novel model that addresses three key dimensions of user simulation: informational faithfulness, emotional dynamics, and language style (e.g., grammar usage). We pioneer the integration of scenario-based victim modeling with GAN-based training workflow and key-information-based prompting, aiming to enhance the realism of simulated victims. Our adversarial training approach teaches the discriminator to recognize grammar and emotional cues as reliable indicators of synthetic content. According to evaluations by human raters, the VicSim model outperforms GPT-4 in terms of human-likeness.

Title: Geometry Restoration and Dewarping of Camera-Captured Document Images

• Authors: Valery Istomin, Oleg Pereziabov, Ilya Afanasyev
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03145
• Pdf URL: https://arxiv.org/pdf/2501.03145
• Copy Paste: [[2501.03145]] Geometry Restoration and Dewarping of Camera-Captured Document Images(https://arxiv.org/abs/2501.03145)
• Keywords: segmentation
• Abstract: This research focuses on developing a method for restoring the topology of digital images of paper documents captured by a camera, using algorithms for detection, segmentation, geometry restoration, and dewarping. Our methodology employs deep learning (DL) for document outline detection, followed by computer vision (CV) to create a topological 2D grid using cubic polynomial interpolation and correct nonlinear distortions by remapping the image. Using classical CV methods makes the document topology restoration process more efficient and faster, as it requires significantly fewer computational resources and memory. We developed a new pipeline for automatic document dewarping and reconstruction, along with a framework and annotated dataset to demonstrate its efficiency. Our experiments confirm the promise of our methodology and its superiority over existing benchmarks (including mobile apps and popular DL solutions, such as RectiNet, DocGeoNet, and DocTr++) both visually and in terms of document readability via Optical Character Recognition (OCR) and geometry restoration metrics. This paves the way for creating high-quality digital copies of paper documents and enhancing the efficiency of OCR systems. Project page: this https URL

Title: Segment Anything Model for Zero-shot Single Particle Tracking in Liquid Phase Transmission Electron Microscopy

• Authors: Risha Goel, Zain Shabeeb, Isabel Panicker, Vida Jamali
• Subjects: cs.CV, physics.data-an
• Abstract URL: https://arxiv.org/abs/2501.03153
• Pdf URL: https://arxiv.org/pdf/2501.03153
• Copy Paste: [[2501.03153]] Segment Anything Model for Zero-shot Single Particle Tracking in Liquid Phase Transmission Electron Microscopy(https://arxiv.org/abs/2501.03153)
• Keywords: segmentation
• Abstract: Liquid phase transmission electron microscopy (LPTEM) offers an unparalleled combination of spatial and temporal resolution, making it a promising tool for single particle tracking at the nanoscale. However, the absence of a standardized framework for identifying and tracking nanoparticles in noisy LPTEM videos has impeded progress in the field to develop this technique as a single particle tracking tool. To address this, we leveraged Segment Anything Model 2 (SAM 2), released by Meta, which is a foundation model developed for segmenting videos and images. Here, we demonstrate that SAM 2 can successfully segment LPTEM videos in a zero-shot manner and without requiring fine-tuning. Building on this capability, we introduce SAM4EM, a comprehensive framework that integrates promptable video segmentation with particle tracking and statistical analysis, providing an end-to-end LPTEM analysis framework for single particle tracking. SAM4EM achieves nearly 50-fold higher accuracy in segmenting and analyzing LPTEM videos compared to state-of-the-art methods, paving the way for broader applications of LPTEM in nanoscale imaging.

Title: Deep-Relative-Trust-Based Diffusion for Decentralized Deep Learning

• Authors: Muyun Li, Aaron Fainman, Stefan Vlaski
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2501.03162
• Pdf URL: https://arxiv.org/pdf/2501.03162
• Copy Paste: [[2501.03162]] Deep-Relative-Trust-Based Diffusion for Decentralized Deep Learning(https://arxiv.org/abs/2501.03162)
• Keywords: diffusion
• Abstract: Decentralized learning strategies allow a collection of agents to learn efficiently from local data sets without the need for central aggregation or orchestration. Current decentralized learning paradigms typically rely on an averaging mechanism to encourage agreement in the parameter space. We argue that in the context of deep neural networks, which are often over-parameterized, encouraging consensus of the neural network outputs, as opposed to their parameters can be more appropriate. This motivates the development of a new decentralized learning algorithm, termed DRT diffusion, based on deep relative trust (DRT), a recently introduced similarity measure for neural networks. We provide convergence analysis for the proposed strategy, and numerically establish its benefit to generalization, especially with sparse topologies, in an image classification task.

Title: Semantic Captioning: Benchmark Dataset and Graph-Aware Few-Shot In-Context Learning for SQL2Text

• Authors: Ali Al-Lawati, Jason Lucas, Prasenjit Mitra
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03166
• Pdf URL: https://arxiv.org/pdf/2501.03166
• Copy Paste: [[2501.03166]] Semantic Captioning: Benchmark Dataset and Graph-Aware Few-Shot In-Context Learning for SQL2Text(https://arxiv.org/abs/2501.03166)
• Keywords: security, robust, large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable performance in various NLP tasks, including semantic parsing, which trans lates natural language into formal code representations. However, the reverse process, translating code into natural language, termed semantic captioning, has received less attention. This task is becoming increasingly important as LLMs are integrated into platforms for code generation, security analysis, and educational purposes. In this paper, we focus on the captioning of SQL query (SQL2Text) to address the critical need for understanding and explaining SQL queries in an era where LLM-generated code poses potential security risks. We repurpose Text2SQL datasets for SQL2Text by introducing an iterative ICL prompt using GPT-4o to generate multiple additional utterances, which enhances the robustness of the datasets for the reverse task. We conduct our experiments using in-context learning (ICL) based on different sample selection methods, emphasizing smaller, more computationally efficient LLMs. Our findings demonstrate that leveraging the inherent graph properties of SQL for ICL sample selection significantly outperforms random selection by up to 39% on BLEU score and provides better results than alternative methods. Dataset and codes are published: \url{this https URL}.

Title: GLiREL -- Generalist Model for Zero-Shot Relation Extraction

• Authors: Jack Boylan, Chris Hokamp, Demian Gholipour Ghalandari
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03172
• Pdf URL: https://arxiv.org/pdf/2501.03172
• Copy Paste: [[2501.03172]] GLiREL -- Generalist Model for Zero-Shot Relation Extraction(https://arxiv.org/abs/2501.03172)
• Keywords: extraction
• Abstract: We introduce GLiREL (Generalist Lightweight model for zero-shot Relation Extraction), an efficient architecture and training paradigm for zero-shot relation classification. Inspired by recent advancements in zero-shot named entity recognition, this work presents an approach to efficiently and accurately predict zero-shot relationship labels between multiple entities in a single forward pass. Experiments using the FewRel and WikiZSL benchmarks demonstrate that our approach achieves state-of-the-art results on the zero-shot relation classification task. In addition, we contribute a protocol for synthetically-generating datasets with diverse relation labels.

Title: MObI: Multimodal Object Inpainting Using Diffusion Models

• Authors: Alexandru Buburuzan, Anuj Sharma, John Redford, Puneet K. Dokania, Romain Mueller
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03173
• Pdf URL: https://arxiv.org/pdf/2501.03173
• Copy Paste: [[2501.03173]] MObI: Multimodal Object Inpainting Using Diffusion Models(https://arxiv.org/abs/2501.03173)
• Keywords: diffusion
• Abstract: Safety-critical applications, such as autonomous driving, require extensive multimodal data for rigorous testing. Methods based on synthetic data are gaining prominence due to the cost and complexity of gathering real-world data but require a high degree of realism and controllability in order to be useful. This paper introduces MObI, a novel framework for Multimodal Object Inpainting that leverages a diffusion model to create realistic and controllable object inpaintings across perceptual modalities, demonstrated for both camera and lidar simultaneously. Using a single reference RGB image, MObI enables objects to be seamlessly inserted into existing multimodal scenes at a 3D location specified by a bounding box, while maintaining semantic consistency and multimodal coherence. Unlike traditional inpainting methods that rely solely on edit masks, our 3D bounding box conditioning gives objects accurate spatial positioning and realistic scaling. As a result, our approach can be used to insert novel objects flexibly into multimodal scenes, providing significant advantages for testing perception models.

Title: Boosting Explainability through Selective Rationalization in Pre-trained Language Models

• Authors: Libing Yuan, Shuaibo Hu, Kui Yu, Le Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03182
• Pdf URL: https://arxiv.org/pdf/2501.03182
• Copy Paste: [[2501.03182]] Boosting Explainability through Selective Rationalization in Pre-trained Language Models(https://arxiv.org/abs/2501.03182)
• Keywords: explainability
• Abstract: The widespread application of pre-trained language models (PLMs) in natural language processing (NLP) has led to increasing concerns about their explainability. Selective rationalization is a self-explanatory framework that selects human-intelligible input subsets as rationales for predictions. Recent studies have shown that applying existing rationalization frameworks to PLMs will result in severe degeneration and failure problems, producing sub-optimal or meaningless rationales. Such failures severely damage trust in rationalization methods and constrain the application of rationalization techniques on PLMs. In this paper, we find that the homogeneity of tokens in the sentences produced by PLMs is the primary contributor to these problems. To address these challenges, we propose a method named Pre-trained Language Model's Rationalization (PLMR), which splits PLMs into a generator and a predictor to deal with NLP tasks while providing interpretable rationales. The generator in PLMR also alleviates homogeneity by pruning irrelevant tokens, while the predictor uses full-text information to standardize predictions. Experiments conducted on two widely used datasets across multiple PLMs demonstrate the effectiveness of the proposed method PLMR in addressing the challenge of applying selective rationalization to PLMs. Codes: this https URL.

Title: CLIX: Cross-Lingual Explanations of Idiomatic Expressions

• Authors: Aaron Gluck, Katharina von der Wense, Maria Pacheco
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03191
• Pdf URL: https://arxiv.org/pdf/2501.03191
• Copy Paste: [[2501.03191]] CLIX: Cross-Lingual Explanations of Idiomatic Expressions(https://arxiv.org/abs/2501.03191)
• Keywords: large language model
• Abstract: Automated definition generation systems have been proposed to support vocabulary expansion for language learners. The main barrier to the success of these systems is that learners often struggle to understand definitions due to the presence of potentially unfamiliar words and grammar, particularly when non-standard language is involved. To address these challenges, we propose CLIX, the task of Cross-Lingual explanations of Idiomatic eXpressions. We explore the capabilities of current NLP models for this task, and observe that while it remains challenging, large language models show promise. Finally, we perform a detailed error analysis to highlight the key challenges that need to be addressed before we can reliably incorporate these systems into educational tools.

Title: Detecting AI-Generated Text in Educational Content: Leveraging Machine Learning and Explainable AI for Academic Integrity

• Authors: Ayat A. Najjar, Huthaifa I. Ashqar, Omar A. Darwish, Eman Hammad
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.03203
• Pdf URL: https://arxiv.org/pdf/2501.03203
• Copy Paste: [[2501.03203]] Detecting AI-Generated Text in Educational Content: Leveraging Machine Learning and Explainable AI for Academic Integrity(https://arxiv.org/abs/2501.03203)
• Keywords: large language model
• Abstract: This study seeks to enhance academic integrity by providing tools to detect AI-generated content in student work using advanced technologies. The findings promote transparency and accountability, helping educators maintain ethical standards and supporting the responsible integration of AI in education. A key contribution of this work is the generation of the CyberHumanAI dataset, which has 1000 observations, 500 of which are written by humans and the other 500 produced by ChatGPT. We evaluate various machine learning (ML) and deep learning (DL) algorithms on the CyberHumanAI dataset comparing human-written and AI-generated content from Large Language Models (LLMs) (i.e., ChatGPT). Results demonstrate that traditional ML algorithms, specifically XGBoost and Random Forest, achieve high performance (83% and 81% accuracies respectively). Results also show that classifying shorter content seems to be more challenging than classifying longer content. Further, using Explainable Artificial Intelligence (XAI) we identify discriminative features influencing the ML model's predictions, where human-written content tends to use a practical language (e.g., use and allow). Meanwhile AI-generated text is characterized by more abstract and formal terms (e.g., realm and employ). Finally, a comparative analysis with GPTZero show that our narrowly focused, simple, and fine-tuned model can outperform generalized systems like GPTZero. The proposed model achieved approximately 77.5% accuracy compared to GPTZero's 48.5% accuracy when tasked to classify Pure AI, Pure Human, and mixed class. GPTZero showed a tendency to classify challenging and small-content cases as either mixed or unrecognized while our proposed model showed a more balanced performance across the three classes.

Title: Leveraging Explainable AI for LLM Text Attribution: Differentiating Human-Written and Multiple LLMs-Generated Text

• Authors: Ayat Najjar, Huthaifa I. Ashqar, Omar Darwish, Eman Hammad
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.03212
• Pdf URL: https://arxiv.org/pdf/2501.03212
• Copy Paste: [[2501.03212]] Leveraging Explainable AI for LLM Text Attribution: Differentiating Human-Written and Multiple LLMs-Generated Text(https://arxiv.org/abs/2501.03212)
• Keywords: robust, generative, large language model
• Abstract: The development of Generative AI Large Language Models (LLMs) raised the alarm regarding identifying content produced through generative AI or humans. In one case, issues arise when students heavily rely on such tools in a manner that can affect the development of their writing or coding skills. Other issues of plagiarism also apply. This study aims to support efforts to detect and identify textual content generated using LLM tools. We hypothesize that LLMs-generated text is detectable by machine learning (ML), and investigate ML models that can recognize and differentiate texts generated by multiple LLMs tools. We leverage several ML and Deep Learning (DL) algorithms such as Random Forest (RF), and Recurrent Neural Networks (RNN), and utilized Explainable Artificial Intelligence (XAI) to understand the important features in attribution. Our method is divided into 1) binary classification to differentiate between human-written and AI-text, and 2) multi classification, to differentiate between human-written text and the text generated by the five different LLM tools (ChatGPT, LLaMA, Google Bard, Claude, and Perplexity). Results show high accuracy in the multi and binary classification. Our model outperformed GPTZero with 98.5\% accuracy to 78.3\%. Notably, GPTZero was unable to recognize about 4.2\% of the observations, but our model was able to recognize the complete test dataset. XAI results showed that understanding feature importance across different classes enables detailed author/source profiles. Further, aiding in attribution and supporting plagiarism detection by highlighting unique stylistic and structural elements ensuring robust content originality verification.

Title: ProTracker: Probabilistic Integration for Robust and Accurate Point Tracking

• Authors: Tingyang Zhang, Chen Wang, Zhiyang Dou, Qingzhe Gao, Jiahui Lei, Baoquan Chen, Lingjie Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03220
• Pdf URL: https://arxiv.org/pdf/2501.03220
• Copy Paste: [[2501.03220]] ProTracker: Probabilistic Integration for Robust and Accurate Point Tracking(https://arxiv.org/abs/2501.03220)
• Keywords: robust
• Abstract: In this paper, we propose ProTracker, a novel framework for robust and accurate long-term dense tracking of arbitrary points in videos. The key idea of our method is incorporating probabilistic integration to refine multiple predictions from both optical flow and semantic features for robust short-term and long-term tracking. Specifically, we integrate optical flow estimations in a probabilistic manner, producing smooth and accurate trajectories by maximizing the likelihood of each prediction. To effectively re-localize challenging points that disappear and reappear due to occlusion, we further incorporate long-term feature correspondence into our flow predictions for continuous trajectory generation. Extensive experiments show that ProTracker achieves the state-of-the-art performance among unsupervised and self-supervised approaches, and even outperforms supervised methods on several benchmarks. Our code and model will be publicly available upon publication.

Title: RW-Net: Enhancing Few-Shot Point Cloud Classification with a Wavelet Transform Projection-based Network

• Authors: Haosheng Zhang, Hao Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03221
• Pdf URL: https://arxiv.org/pdf/2501.03221
• Copy Paste: [[2501.03221]] RW-Net: Enhancing Few-Shot Point Cloud Classification with a Wavelet Transform Projection-based Network(https://arxiv.org/abs/2501.03221)
• Keywords: robust
• Abstract: In the domain of 3D object classification, a fundamental challenge lies in addressing the scarcity of labeled data, which limits the applicability of traditional data-intensive learning paradigms. This challenge is particularly pronounced in few-shot learning scenarios, where the objective is to achieve robust generalization from minimal annotated samples. To overcome these limitations, it is crucial to identify and leverage the most salient and discriminative features of 3D objects, thereby enhancing learning efficiency and reducing dependency on large-scale labeled datasets. This work introduces RW-Net, a novel framework designed to address the challenges above by integrating Rate-Distortion Explanation (RDE) and wavelet transform into a state-of-the-art projection-based 3D object classification architecture. The proposed method capitalizes on RDE to extract critical features by identifying and preserving the most informative data components while reducing redundancy. This process ensures the retention of essential information for effective decision-making, optimizing the model's ability to learn from limited data. Complementing RDE, incorporating the wavelet transform further enhances the framework's capability to generalize in low-data regimes. By emphasizing low-frequency components of the input data, the wavelet transform captures fundamental geometric and structural attributes of 3D objects. These attributes are instrumental in mitigating overfitting and improving the robustness of the learned representations across diverse tasks and domains. To validate the effectiveness of our RW-Net, we conduct extensive experiments on three datasets: ModelNet40, ModelNet40-C, and ScanObjectNN for few-shot 3D object classification. The results demonstrate that our approach achieves state-of-the-art performance and exhibits superior generalization and robustness in few-shot learning scenarios.

Title: Characterizing the Accuracy-Communication-Privacy Trade-off in Distributed Stochastic Convex Optimization

• Authors: Sudeep Salgia, Nikola Pavlovic, Yuejie Chi, Qing Zhao
• Subjects: cs.LG, cs.IT, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.03222
• Pdf URL: https://arxiv.org/pdf/2501.03222
• Copy Paste: [[2501.03222]] Characterizing the Accuracy-Communication-Privacy Trade-off in Distributed Stochastic Convex Optimization(https://arxiv.org/abs/2501.03222)
• Keywords: privacy
• Abstract: We consider the problem of differentially private stochastic convex optimization (DP-SCO) in a distributed setting with $M$ clients, where each of them has a local dataset of $N$ i.i.d. data samples from an underlying data distribution. The objective is to design an algorithm to minimize a convex population loss using a collaborative effort across $M$ clients, while ensuring the privacy of the local datasets. In this work, we investigate the accuracy-communication-privacy trade-off for this problem. We establish matching converse and achievability results using a novel lower bound and a new algorithm for distributed DP-SCO based on Vaidya's plane cutting method. Thus, our results provide a complete characterization of the accuracy-communication-privacy trade-off for DP-SCO in the distributed setting.

Title: Rate-My-LoRA: Efficient and Adaptive Federated Model Tuning for Cardiac MRI Segmentation

• Authors: Xiaoxiao He, Haizhou Shi, Ligong Han, Chaowei Tan, Bo Liu, Zihao Xu, Meng Ye, Leon Axel, Kang Li, Dimitris Metaxas
• Subjects: cs.CV, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03223
• Pdf URL: https://arxiv.org/pdf/2501.03223
• Copy Paste: [[2501.03223]] Rate-My-LoRA: Efficient and Adaptive Federated Model Tuning for Cardiac MRI Segmentation(https://arxiv.org/abs/2501.03223)
• Keywords: privacy, federate, segmentation
• Abstract: Cardiovascular disease (CVD) and cardiac dyssynchrony are major public health problems in the United States. Precise cardiac image segmentation is crucial for extracting quantitative measures that help categorize cardiac dyssynchrony. However, achieving high accuracy often depends on centralizing large datasets from different hospitals, which can be challenging due to privacy concerns. To solve this problem, Federated Learning (FL) is proposed to enable decentralized model training on such data without exchanging sensitive information. However, bandwidth limitations and data heterogeneity remain as significant challenges in conventional FL algorithms. In this paper, we propose a novel efficient and adaptive federate learning method for cardiac segmentation that improves model performance while reducing the bandwidth requirement. Our method leverages the low-rank adaptation (LoRA) to regularize model weight update and reduce communication overhead. We also propose a \mymethod{} aggregation technique to address data heterogeneity among clients. This technique adaptively penalizes the aggregated weights from different clients by comparing the validation accuracy in each client, allowing better generalization performance and fast local adaptation. In-client and cross-client evaluations on public cardiac MR datasets demonstrate the superiority of our method over other LoRA-based federate learning approaches.

Title: BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoning

• Authors: Beichen Zhang, Yuhong Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Haodong Duan, Yuhang Cao, Dahua Lin, Jiaqi Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03226
• Pdf URL: https://arxiv.org/pdf/2501.03226
• Copy Paste: [[2501.03226]] BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoning(https://arxiv.org/abs/2501.03226)
• Keywords: robust, large language model
• Abstract: Cutting-edge large language models (LLMs) demonstrate promising performance in solving complex math problems with a divide-and-conquer pipeline and the assistance of in-context learning (ICL) examples. However, their potential for improvement is limited by two critical problems within their ICL examples: granularity-mismatch and the ensuing negative-effect noise problem. Specifically, the LLMs are capable of the dividing process yet mostly failed by inaccurate reasoning within a few conquer steps, while the ICL examples retrieved in question-grained sometimes lack relevant steps for a specific challenging reasoning step. Further, this disconnect may hinder the correct reasoning due to its irrelevance. To this end, we focus on improving the reasoning quality within each step and present BoostStep. BoostStep aligns the granularity between the retrieving and reasoning on step grained, and provides highly related ICL examples for each reasoning step with a novel `first-try' strategy. BoostStep provides more relevant examples than the coarse question-grained strategy, enhancing the model reasoning quality within each step steadily. BoostStep is a general and robust reasoning-enhancing method that not only improves standalone reasoning performance but also integrates seamlessly with Monte Carlo Tree Search methods (MCTS) to refine both candidate generation and decision-making. Quantitatively, it improves GPT-4o and Qwen2.5-Math-72B by 3.6\% and 2.0\% respectively on various mathematical benchmarks, and 7.5\% gain combined with MCTS.

Title: When Should Selfish Miners Double-Spend?

• Authors: Mustafa Doger, Sennur Ulukus
• Subjects: cs.CR, cs.DC, cs.DM, cs.IT, math.PR
• Abstract URL: https://arxiv.org/abs/2501.03227
• Pdf URL: https://arxiv.org/pdf/2501.03227
• Copy Paste: [[2501.03227]] When Should Selfish Miners Double-Spend?(https://arxiv.org/abs/2501.03227)
• Keywords: attack
• Abstract: Although, both double-spending and selfish-mining attacks have been extensively studied since the ``Bitcoin'' whitepaper of Nakamoto and the ``majority is not enough'' paper of Eyal and Sirer, there has been no rigorous stochastic analysis of an attack that combines the two, except for the complicated MDP models. In this paper, we first combine stubborn and selfish mining attacks, i.e., construct a strategy where the attacker acts stubborn until its private branch reaches a certain length and then switches to act selfish. We provide the optimal stubbornness for each parameter regime. Next, we provide the maximum stubbornness that is still more profitable than honest mining and argue a connection between the level of stubbornness and the $k$-confirmation rule. We show that, at each attack cycle, if the level of stubbornness is higher than $k$, there is a risk of double-spending which comes at no-cost to the adversary. The result can be seen as a guide for picking $k$ in the $k$-confirmation rule in a blockchain design. At each cycle, for a given stubbornness level, we rigorously formulate how great the risk of double-spending is. We provide the minimum double-spend value needed for an attack to be profitable in the regimes where the scheme is less profitable than honest mining. We further modify the attack in the stubborn regime in order to conceal the attack and increase the double-spending probability. Finally, we evaluate the results and provide the optimal and the maximum stubbornness levels for each parameter regime as well as the revenue. As a case study, with Bitcoin's $k=6$ block confirmation rule, we evaluate the revenue and double-spending risk of the attacks for each pool parameter.

Title: Gaussian Masked Autoencoders

• Authors: Jathushan Rajasegaran, Xinlei Chen, Rulilong Li, Christoph Feichtenhofer, Jitendra Malik, Shiry Ginosar
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03229
• Pdf URL: https://arxiv.org/pdf/2501.03229
• Copy Paste: [[2501.03229]] Gaussian Masked Autoencoders(https://arxiv.org/abs/2501.03229)
• Keywords: segmentation
• Abstract: This paper explores Masked Autoencoders (MAE) with Gaussian Splatting. While reconstructive self-supervised learning frameworks such as MAE learns good semantic abstractions, it is not trained for explicit spatial awareness. Our approach, named Gaussian Masked Autoencoder, or GMAE, aims to learn semantic abstractions and spatial understanding jointly. Like MAE, it reconstructs the image end-to-end in the pixel space, but beyond MAE, it also introduces an intermediate, 3D Gaussian-based representation and renders images via splatting. We show that GMAE can enable various zero-shot learning capabilities of spatial understanding (e.g., figure-ground segmentation, image layering, edge detection, etc.) while preserving the high-level semantics of self-supervised representation quality from MAE. To our knowledge, we are the first to employ Gaussian primitives in an image representation learning framework beyond optimization-based single-scene reconstructions. We believe GMAE will inspire further research in this direction and contribute to developing next-generation techniques for modeling high-fidelity visual data. More details at this https URL