2025-02-06

Title: MIND: Microstructure INverse Design with Generative Hybrid Neural Representation

• Authors: Tianyang Xue, Haochen Li, Longdu Liu, Paul Henderson, Pengbin Tang, Lin Lu, Jikai Liu, Haisen Zhao, Hao Peng, Bernd Bickel
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02607
• Pdf URL: https://arxiv.org/pdf/2502.02607
• Copy Paste: [[2502.02607]] MIND: Microstructure INverse Design with Generative Hybrid Neural Representation(https://arxiv.org/abs/2502.02607)
• Keywords: diffusion, generative
• Abstract: The inverse design of microstructures plays a pivotal role in optimizing metamaterials with specific, targeted physical properties. While traditional forward design methods are constrained by their inability to explore the vast combinatorial design space, inverse design offers a compelling alternative by directly generating structures that fulfill predefined performance criteria. However, achieving precise control over both geometry and material properties remains a significant challenge due to their intricate interdependence. Existing approaches, which typically rely on voxel or parametric representations, often limit design flexibility and structural diversity. In this work, we present a novel generative model that integrates latent diffusion with Holoplane, an advanced hybrid neural representation that simultaneously encodes both geometric and physical properties. This combination ensures superior alignment between geometry and properties. Our approach generalizes across multiple microstructure classes, enabling the generation of diverse, tileable microstructures with significantly improved property accuracy and enhanced control over geometric validity, surpassing the performance of existing methods. We introduce a multi-class dataset encompassing a variety of geometric morphologies, including truss, shell, tube, and plate structures, to train and validate our model. Experimental results demonstrate the model's ability to generate microstructures that meet target properties, maintain geometric validity, and integrate seamlessly into complex assemblies. Additionally, we explore the potential of our framework through the generation of new microstructures, cross-class interpolation, and the infilling of heterogeneous microstructures. The dataset and source code will be open-sourced upon publication.

Title: PolarQuant: Quantizing KV Caches with Polar Transformation

• Authors: Insu Han, Praneeth Kacham, Amin Karbasi, Vahab Mirrokni, Amir Zandieh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02617
• Pdf URL: https://arxiv.org/pdf/2502.02617
• Copy Paste: [[2502.02617]] PolarQuant: Quantizing KV Caches with Polar Transformation(https://arxiv.org/abs/2502.02617)
• Keywords: large language model
• Abstract: Large language models (LLMs) require significant memory to store Key-Value (KV) embeddings in their KV cache, especially when handling long-range contexts. Quantization of these KV embeddings is a common technique to reduce memory consumption. This work introduces PolarQuant, a novel quantization method employing random preconditioning and polar transformation. Our method transforms the KV embeddings into polar coordinates using an efficient recursive algorithm and then quantizes resulting angles. Our key insight is that, after random preconditioning, the angles in the polar representation exhibit a tightly bounded and highly concentrated distribution with an analytically computable form. This nice distribution eliminates the need for explicit normalization, a step required by traditional quantization methods which introduces significant memory overhead because quantization parameters (e.g., zero point and scale) must be stored in full precision per each data block. PolarQuant bypasses this normalization step, enabling substantial memory savings. The long-context evaluation demonstrates that PolarQuant compresses the KV cache by over x4.2 while achieving the best quality scores compared to the state-of-the-art methods.

Title: Deep Learning-Based Facial Expression Recognition for the Elderly: A Systematic Review

• Authors: F. Xavier Gaya-Morey, Jose M. Buades-Rubio, Philippe Palanque, Raquel Lacuesta, Cristina Manresa-Yee
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02618
• Pdf URL: https://arxiv.org/pdf/2502.02618
• Copy Paste: [[2502.02618]] Deep Learning-Based Facial Expression Recognition for the Elderly: A Systematic Review(https://arxiv.org/abs/2502.02618)
• Keywords: privacy
• Abstract: The rapid aging of the global population has highlighted the need for technologies to support elderly, particularly in healthcare and emotional well-being. Facial expression recognition (FER) systems offer a non-invasive means of monitoring emotional states, with applications in assisted living, mental health support, and personalized care. This study presents a systematic review of deep learning-based FER systems, focusing on their applications for the elderly population. Following a rigorous methodology, we analyzed 31 studies published over the last decade, addressing challenges such as the scarcity of elderly-specific datasets, class imbalances, and the impact of age-related facial expression differences. Our findings show that convolutional neural networks remain dominant in FER, and especially lightweight versions for resource-constrained environments. However, existing datasets often lack diversity in age representation, and real-world deployment remains limited. Additionally, privacy concerns and the need for explainable artificial intelligence emerged as key barriers to adoption. This review underscores the importance of developing age-inclusive datasets, integrating multimodal solutions, and adopting XAI techniques to enhance system usability, reliability, and trustworthiness. We conclude by offering recommendations for future research to bridge the gap between academic progress and real-world implementation in elderly care.

Title: Sample Complexity of Bias Detection with Subsampled Point-to-Subspace Distances

• Authors: German Martinez Matilla, Jakub Marecek
• Subjects: cs.LG, cs.AI, math.ST
• Abstract URL: https://arxiv.org/abs/2502.02623
• Pdf URL: https://arxiv.org/pdf/2502.02623
• Copy Paste: [[2502.02623]] Sample Complexity of Bias Detection with Subsampled Point-to-Subspace Distances(https://arxiv.org/abs/2502.02623)
• Keywords: protect
• Abstract: Sample complexity of bias estimation is a lower bound on the runtime of any bias detection method. Many regulatory frameworks require the bias to be tested for all subgroups, whose number grows exponentially with the number of protected attributes. Unless one wishes to run a bias detection with a doubly-exponential run-time, one should like to have polynomial complexity of bias detection for a single subgroup. At the same time, the reference data may be based on surveys, and thus come with non-trivial uncertainty. Here, we reformulate bias detection as a point-to-subspace problem on the space of measures and show that, for supremum norm, it can be subsampled efficiently. In particular, our probabilistically approximately correct (PAC) results are corroborated by tests on well-known instances.

Title: e-SimFT: Alignment of Generative Models with Simulation Feedback for Pareto-Front Design Exploration

• Authors: Hyunmin Cheong, Mohammadmehdi Ataei, Amir Hosein Khasahmadi, Pradeep Kumar Jayaraman
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02628
• Pdf URL: https://arxiv.org/pdf/2502.02628
• Copy Paste: [[2502.02628]] e-SimFT: Alignment of Generative Models with Simulation Feedback for Pareto-Front Design Exploration(https://arxiv.org/abs/2502.02628)
• Keywords: generative, large language model
• Abstract: Deep generative models have recently shown success in solving complex engineering design problems where models predict solutions that address the design requirements specified as input. However, there remains a challenge in aligning such models for effective design exploration. For many design problems, finding a solution that meets all the requirements is infeasible. In such a case, engineers prefer to obtain a set of Pareto optimal solutions with respect to those requirements, but uniform sampling of generative models may not yield a useful Pareto front. To address this gap, we introduce a new framework for Pareto-front design exploration with simulation fine-tuned generative models. First, the framework adopts preference alignment methods developed for Large Language Models (LLMs) and showcases the first application in fine-tuning a generative model for engineering design. The important distinction here is that we use a simulator instead of humans to provide accurate and scalable feedback. Next, we propose epsilon-sampling, inspired by the epsilon-constraint method used for Pareto-front generation with classical optimization algorithms, to construct a high-quality Pareto front with the fine-tuned models. Our framework, named e-SimFT, is shown to produce better-quality Pareto fronts than existing multi-objective alignment methods.

Title: A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)

• Authors: Yan Li, Tianyi Zhang, Zechuan Li, Soyeon Caren Han
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02659
• Pdf URL: https://arxiv.org/pdf/2502.02659
• Copy Paste: [[2502.02659]] A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)(https://arxiv.org/abs/2502.02659)
• Keywords: transformer, large language model
• Abstract: Transformer-based Large Language Models (LLMs) struggle to process inputs exceeding their training context window, with performance degrading due to positional out-of-distribution (O.O.D.) that disrupt attention computations. Existing solutions, fine-tuning and training-free methods, are limited by computational inefficiency, attention logit outliers or loss of local positional information. To address this, we propose Greedy Attention Logit Interpolation (GALI), a training-free length extrapolation method that maximizes the utilization of pretrained positional intervals while avoiding attention logit outliers through attention logit interpolation. The result demonstrates that GALI consistently outperforms state-of-the-art training-free methods. Our findings reveal that LLMs interpret positional intervals unevenly within their training context window, suggesting that extrapolating within a smaller positional interval range yields superior results-even for short-context tasks. GALI represents a significant step toward resolving the positional O.O.D. challenge, enabling more reliable long-text understanding in LLMs. Our implementation of GALI, along with the experiments from our paper, is open-sourced at this https URL.

Title: On Teacher Hacking in Language Model Distillation

• Authors: Daniil Tiapkin, Daniele Calandriello, Johan Ferret, Sarah Perrin, Nino Vieillard, Alexandre Ramé, Mathieu Blondel
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.02671
• Pdf URL: https://arxiv.org/pdf/2502.02671
• Copy Paste: [[2502.02671]] On Teacher Hacking in Language Model Distillation(https://arxiv.org/abs/2502.02671)
• Keywords: robust
• Abstract: Post-training of language models (LMs) increasingly relies on the following two stages: (i) knowledge distillation, where the LM is trained to imitate a larger teacher LM, and (ii) reinforcement learning from human feedback (RLHF), where the LM is aligned by optimizing a reward model. In the second RLHF stage, a well-known challenge is reward hacking, where the LM over-optimizes the reward model. Such phenomenon is in line with Goodhart's law and can lead to degraded performance on the true objective. In this paper, we investigate whether a similar phenomenon, that we call teacher hacking, can occur during knowledge distillation. This could arise because the teacher LM is itself an imperfect approximation of the true distribution. To study this, we propose a controlled experimental setup involving: (i) an oracle LM representing the ground-truth distribution, (ii) a teacher LM distilled from the oracle, and (iii) a student LM distilled from the teacher. Our experiments reveal the following insights. When using a fixed offline dataset for distillation, teacher hacking occurs; moreover, we can detect it by observing when the optimization process deviates from polynomial convergence laws. In contrast, employing online data generation techniques effectively mitigates teacher hacking. More precisely, we identify data diversity as the key factor in preventing hacking. Overall, our findings provide a deeper understanding of the benefits and limitations of distillation for building robust and efficient LMs.

Title: Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes

• Authors: Mayuka Jayawardhana (1), Renbo Tu (2), Samuel Dooley (3), Valeriia Cherepanova (4), Andrew Gordon Wilson (5), Frank Hutter (6), Colin White (7), Tom Goldstein (1), Micah Goldblum (8) ((1) University of Maryland, (2) University of Toronto, (3) Meta, (4) Amazon, (5) New York University, (6) University of Freiburg, (7) this http URL, (8) Columbia University)
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02672
• Pdf URL: https://arxiv.org/pdf/2502.02672
• Copy Paste: [[2502.02672]] Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes(https://arxiv.org/abs/2502.02672)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) perform remarkably well on tabular datasets in zero- and few-shot settings, since they can extract meaning from natural language column headers that describe features and labels. Similarly, TabPFN, a recent non-LLM transformer pretrained on numerous tables for in-context learning, has demonstrated excellent performance for dataset sizes up to a thousand samples. In contrast, gradient-boosted decision trees (GBDTs) are typically trained from scratch on each dataset without benefiting from pretraining data and must learn the relationships between columns from their entries alone since they lack natural language understanding. LLMs and TabPFN excel on small tabular datasets where a strong prior is essential, yet they are not competitive with GBDTs on medium or large datasets, since their context lengths are limited. In this paper, we propose a simple and lightweight approach for fusing large language models and TabPFN with gradient-boosted decision trees, which allows scalable GBDTs to benefit from the natural language capabilities and pretraining of transformers. We name our fusion methods LLM-Boost and PFN-Boost, respectively. While matching or surpassing the performance of the transformer at sufficiently small dataset sizes and GBDTs at sufficiently large sizes, LLM-Boost and PFN-Boost outperform both standalone components on a wide range of dataset sizes in between. We demonstrate state-of-the-art performance against numerous baselines and ensembling algorithms. We find that PFN-Boost achieves the best average performance among all methods we test for all but very small dataset sizes. We release our code at this http URL .

Title: MedRAX: Medical Reasoning Agent for Chest X-ray

• Authors: Adibvafa Fallahpour, Jun Ma, Alif Munim, Hongwei Lyu, Bo Wang
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2502.02673
• Pdf URL: https://arxiv.org/pdf/2502.02673
• Copy Paste: [[2502.02673]] MedRAX: Medical Reasoning Agent for Chest X-ray(https://arxiv.org/abs/2502.02673)
• Keywords: large language model
• Abstract: Chest X-rays (CXRs) play an integral role in driving critical decisions in disease management and patient care. While recent innovations have led to specialized models for various CXR interpretation tasks, these solutions often operate in isolation, limiting their practical utility in clinical practice. We present MedRAX, the first versatile AI agent that seamlessly integrates state-of-the-art CXR analysis tools and multimodal large language models into a unified framework. MedRAX dynamically leverages these models to address complex medical queries without requiring additional training. To rigorously evaluate its capabilities, we introduce ChestAgentBench, a comprehensive benchmark containing 2,500 complex medical queries across 7 diverse categories. Our experiments demonstrate that MedRAX achieves state-of-the-art performance compared to both open-source and proprietary models, representing a significant step toward the practical deployment of automated CXR interpretation systems. Data and code have been publicly available at this https URL

Title: Blind Visible Watermark Removal with Morphological Dilation

• Authors: Preston K. Robinette, Taylor T. Johnson
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02676
• Pdf URL: https://arxiv.org/pdf/2502.02676
• Copy Paste: [[2502.02676]] Blind Visible Watermark Removal with Morphological Dilation(https://arxiv.org/abs/2502.02676)
• Keywords: robust, watermark
• Abstract: Visible watermarks pose significant challenges for image restoration techniques, especially when the target background is unknown. Toward this end, we present MorphoMod, a novel method for automated visible watermark removal that operates in a blind setting -- without requiring target images. Unlike existing methods, MorphoMod effectively removes opaque and transparent watermarks while preserving semantic content, making it well-suited for real-world applications. Evaluations on benchmark datasets, including the Colored Large-scale Watermark Dataset (CLWD), LOGO-series, and the newly introduced Alpha1 datasets, demonstrate that MorphoMod achieves up to a 50.8% improvement in watermark removal effectiveness compared to state-of-the-art methods. Ablation studies highlight the impact of prompts used for inpainting, pre-removal filling strategies, and inpainting model performance on watermark removal. Additionally, a case study on steganographic disorientation reveals broader applications for watermark removal in disrupting high-level hidden messages. MorphoMod offers a robust, adaptable solution for watermark removal and opens avenues for further advancements in image restoration and adversarial manipulation.

Title: Controllable Video Generation with Provable Disentanglement

• Authors: Yifan Shen, Peiyuan Zhu, Zijian Li, Shaoan Xie, Zeyu Tang, Namrata Deka, Zongfang Liu, Guangyi Chen, Kun Zhang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02690
• Pdf URL: https://arxiv.org/pdf/2502.02690
• Copy Paste: [[2502.02690]] Controllable Video Generation with Provable Disentanglement(https://arxiv.org/abs/2502.02690)
• Keywords: generative
• Abstract: Controllable video generation remains a significant challenge, despite recent advances in generating high-quality and consistent videos. Most existing methods for controlling video generation treat the video as a whole, neglecting intricate fine-grained spatiotemporal relationships, which limits both control precision and efficiency. In this paper, we propose Controllable Video Generative Adversarial Networks (CoVoGAN) to disentangle the video concepts, thus facilitating efficient and independent control over individual concepts. Specifically, following the minimal change principle, we first disentangle static and dynamic latent variables. We then leverage the sufficient change property to achieve component-wise identifiability of dynamic latent variables, enabling independent control over motion and identity. To establish the theoretical foundation, we provide a rigorous analysis demonstrating the identifiability of our approach. Building on these theoretical insights, we design a Temporal Transition Module to disentangle latent dynamics. To enforce the minimal change principle and sufficient change property, we minimize the dimensionality of latent dynamic variables and impose temporal conditional independence. To validate our approach, we integrate this module as a plug-in for GANs. Extensive qualitative and quantitative experiments on various video generation benchmarks demonstrate that our method significantly improves generation quality and controllability across diverse real-world scenarios.

Title: Enforcing Demographic Coherence: A Harms Aware Framework for Reasoning about Private Data Release

• Authors: Mark Bun, Marco Carmosino, Palak Jain, Gabriel Kaptchuk, Satchit Sivakumar
• Subjects: cs.CR, cs.DB
• Abstract URL: https://arxiv.org/abs/2502.02709
• Pdf URL: https://arxiv.org/pdf/2502.02709
• Copy Paste: [[2502.02709]] Enforcing Demographic Coherence: A Harms Aware Framework for Reasoning about Private Data Release(https://arxiv.org/abs/2502.02709)
• Keywords: privacy, attack
• Abstract: The technical literature about data privacy largely consists of two complementary approaches: formal definitions of conditions sufficient for privacy preservation and attacks that demonstrate privacy breaches. Differential privacy is an accepted standard in the former sphere. However, differential privacy's powerful adversarial model and worst-case guarantees may make it too stringent in some situations, especially when achieving it comes at a significant cost to data utility. Meanwhile, privacy attacks aim to expose real and worrying privacy risks associated with existing data release processes but often face criticism for being unrealistic. Moreover, the literature on attacks generally does not identify what properties are necessary to defend against them. We address the gap between these approaches by introducing demographic coherence, a condition inspired by privacy attacks that we argue is necessary for data privacy. This condition captures privacy violations arising from inferences about individuals that are incoherent with respect to the demographic patterns in the data. Our framework focuses on confidence rated predictors, which can in turn be distilled from almost any data-informed process. Thus, we capture privacy threats that exist even when no attack is explicitly being carried out. Our framework not only provides a condition with respect to which data release algorithms can be analysed but suggests natural experimental evaluation methodologies that could be used to build practical intuition and make tangible assessment of risks. Finally, we argue that demographic coherence is weaker than differential privacy: we prove that every differentially private data release is also demographically coherent, and that there are demographically coherent algorithms which are not differentially private.

Title: A Unified Understanding and Evaluation of Steering Methods

• Authors: Shawn Im, Yixuan Li
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02716
• Pdf URL: https://arxiv.org/pdf/2502.02716
• Copy Paste: [[2502.02716]] A Unified Understanding and Evaluation of Steering Methods(https://arxiv.org/abs/2502.02716)
• Keywords: large language model
• Abstract: Steering methods provide a practical approach to controlling large language models by applying steering vectors to intermediate activations, guiding outputs toward desired behaviors while avoiding retraining. Despite their growing importance, the field lacks a unified understanding and consistent evaluation across tasks and datasets, hindering progress. This paper introduces a unified framework for analyzing and evaluating steering methods, formalizing their core principles and offering theoretical insights into their effectiveness. Through comprehensive empirical evaluations on multiple-choice and open-ended text generation tasks, we validate these insights, identifying key factors that influence performance and demonstrating the superiority of certain methods. Our work bridges theoretical and practical perspectives, offering actionable guidance for advancing the design, optimization, and deployment of steering methods in LLMs.

Title: Beyond Topological Self-Explainable GNNs: A Formal Explainability Perspective

• Authors: Steve Azzolin, Sagar Malhotra, Andrea Passerini, Stefano Teso
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02719
• Pdf URL: https://arxiv.org/pdf/2502.02719
• Copy Paste: [[2502.02719]] Beyond Topological Self-Explainable GNNs: A Formal Explainability Perspective(https://arxiv.org/abs/2502.02719)
• Keywords: explainability
• Abstract: Self-Explainable Graph Neural Networks (SE-GNNs) are popular explainable-by-design GNNs, but the properties and the limitations of their explanations are not well understood. Our first contribution fills this gap by formalizing the explanations extracted by SE-GNNs, referred to as Trivial Explanations (TEs), and comparing them to established notions of explanations, namely Prime Implicant (PI) and faithful explanations. Our analysis reveals that TEs match PI explanations for a restricted but significant family of tasks. In general, however, they can be less informative than PI explanations and are surprisingly misaligned with widely accepted notions of faithfulness. Although faithful and PI explanations are informative, they are intractable to find and we show that they can be prohibitively large. Motivated by this, we propose Dual-Channel GNNs that integrate a white-box rule extractor and a standard SE-GNN, adaptively combining both channels when the task benefits. Our experiments show that even a simple instantiation of Dual-Channel GNNs can recover succinct rules and perform on par or better than widely used SE-GNNs. Our code can be found in the supplementary material.

Title: Risk-Aware Sensitive Property-Driven Resource Management in Cloud Datacenters

• Authors: Muhamad Felemban, Abdulrahman Almutairi, Arif Ghafoor
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2502.02720
• Pdf URL: https://arxiv.org/pdf/2502.02720
• Copy Paste: [[2502.02720]] Risk-Aware Sensitive Property-Driven Resource Management in Cloud Datacenters(https://arxiv.org/abs/2502.02720)
• Keywords: security
• Abstract: Organizations are increasingly moving towards the cloud computing paradigm, in which an on-demand access to a pool of shared configurable resources is provided. However, security challenges, which are particularly exacerbated by the multitenancy and virtualization features of cloud computing, present a major obstacle. In particular, sharing of resources among potentially untrusted tenants in access controlled cloud datacenters can result in increased risk of data leakage. To address such risk, we propose an efficient risk-aware sensitive property-driven virtual resource assignment mechanism for cloud datacenters. We have used two information-theoretic measures, i.e., KL-divergence and mutual information, to represent sensitive properties in the dataset. Based on the vulnerabilities of cloud architecture and the sensitive property profile, we have formulated the problem as a cost-drive optimization problem. The problem is shown to be NP-complete. Accordingly, we have proposed two heuristics and presented simulation based performance results for cloud datacenters with multiple sensitivity.

Title: Cross-Lingual Transfer for Low-Resource Natural Language Processing

• Authors: Iker García-Ferrero
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02722
• Pdf URL: https://arxiv.org/pdf/2502.02722
• Copy Paste: [[2502.02722]] Cross-Lingual Transfer for Low-Resource Natural Language Processing(https://arxiv.org/abs/2502.02722)
• Keywords: extraction, large language model
• Abstract: Natural Language Processing (NLP) has seen remarkable advances in recent years, particularly with the emergence of Large Language Models that have achieved unprecedented performance across many tasks. However, these developments have mainly benefited a small number of high-resource languages such as English. The majority of languages still face significant challenges due to the scarcity of training data and computational resources. To address this issue, this thesis focuses on cross-lingual transfer learning, a research area aimed at leveraging data and models from high-resource languages to improve NLP performance for low-resource languages. Specifically, we focus on Sequence Labeling tasks such as Named Entity Recognition, Opinion Target Extraction, and Argument Mining. The research is structured around three main objectives: (1) advancing data-based cross-lingual transfer learning methods through improved translation and annotation projection techniques, (2) developing enhanced model-based transfer learning approaches utilizing state-of-the-art multilingual models, and (3) applying these methods to real-world problems while creating open-source resources that facilitate future research in low-resource NLP. More specifically, this thesis presents a new method to improve data-based transfer with T-Projection, a state-of-the-art annotation projection method that leverages text-to-text multilingual models and machine translation systems. T-Projection significantly outperforms previous annotation projection methods by a wide margin. For model-based transfer, we introduce a constrained decoding algorithm that enhances cross-lingual Sequence Labeling in zero-shot settings using text-to-text models. Finally, we develop Medical mT5, the first multilingual text-to-text medical model, demonstrating the practical impact of our research on real-world applications.

Title: Parameter Tracking in Federated Learning with Adaptive Optimization

• Authors: Evan Chen. Jianing Zhang, Shiqiang Wang, Chaoyue Liu, Christopher Brinton
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2502.02727
• Pdf URL: https://arxiv.org/pdf/2502.02727
• Copy Paste: [[2502.02727]] Parameter Tracking in Federated Learning with Adaptive Optimization(https://arxiv.org/abs/2502.02727)
• Keywords: federate
• Abstract: In Federated Learning (FL), model training performance is strongly impacted by data heterogeneity across clients. Gradient Tracking (GT) has recently emerged as a solution which mitigates this issue by introducing correction terms to local model updates. To date, GT has only been considered under Stochastic Gradient Descent (SGD)-based model training, while modern FL frameworks increasingly employ adaptive optimizers for improved convergence. In this work, we generalize the GT framework to a more flexible Parameter Tracking (PT) paradigm and propose two novel adaptive optimization algorithms, {\tt FAdamET} and {\tt FAdamGT}, that integrate PT into Adam-based FL. We provide a rigorous convergence analysis of these algorithms under non-convex settings. Our experimental results demonstrate that both proposed algorithms consistently outperform existing methods when evaluating total communication cost and total computation cost across varying levels of data heterogeneity, showing the effectiveness of correcting first-order information in federated adaptive optimization.

Title: Semantic Entanglement-Based Ransomware Detection via Probabilistic Latent Encryption Mapping

• Authors: Mohammad Eisa, Quentin Yardley, Rafael Witherspoon, Harriet Pendlebury, Clement Rutherford
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2502.02730
• Pdf URL: https://arxiv.org/pdf/2502.02730
• Copy Paste: [[2502.02730]] Semantic Entanglement-Based Ransomware Detection via Probabilistic Latent Encryption Mapping(https://arxiv.org/abs/2502.02730)
• Keywords: security, attack, robust
• Abstract: Encryption-based attacks have introduced significant challenges for detection mechanisms that rely on predefined signatures, heuristic indicators, or static rule-based classifications. Probabilistic Latent Encryption Mapping presents an alternative detection framework that models ransomware-induced encryption behaviors through statistical representations of entropy deviations and probabilistic dependencies in execution traces. Unlike conventional approaches that depend on explicit bytecode analysis or predefined cryptographic function call monitoring, probabilistic inference techniques classify encryption anomalies based on their underlying statistical characteristics, ensuring greater adaptability to polymorphic attack strategies. Evaluations demonstrate that entropy-driven classification reduces false positive rates while maintaining high detection accuracy across diverse ransomware families and encryption methodologies. Experimental results further highlight the framework's ability to differentiate between benign encryption workflows and adversarial cryptographic manipulations, ensuring that classification performance remains effective across cloud-based and localized execution environments. Benchmark comparisons illustrate that probabilistic modeling exhibits advantages over heuristic and machine learning-based detection approaches, particularly in handling previously unseen encryption techniques and adversarial obfuscation strategies. Computational efficiency analysis confirms that detection latency remains within operational feasibility constraints, reinforcing the viability of probabilistic encryption classification for real-time security infrastructures. The ability to systematically infer encryption-induced deviations without requiring static attack signatures strengthens detection robustness against adversarial evasion techniques.

Title: Peri-LN: Revisiting Layer Normalization in the Transformer Architecture

• Authors: Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, Kang Min Yoo
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02732
• Pdf URL: https://arxiv.org/pdf/2502.02732
• Copy Paste: [[2502.02732]] Peri-LN: Revisiting Layer Normalization in the Transformer Architecture(https://arxiv.org/abs/2502.02732)
• Keywords: transformer, large language model
• Abstract: Designing Transformer architectures with the optimal layer normalization (LN) strategy that ensures large-scale training stability and expedite convergence has remained elusive, even in this era of large language models (LLMs). To this end, we present a comprehensive analytical foundation for understanding how different LN strategies influence training dynamics in large-scale Transformer training. Until recently, Pre-LN and Post-LN have long dominated standard practices despite their limitations in large-scale training. However, several open-source large-scale models have recently begun silently adopting a third strategy without much explanation. This strategy places layer normalization (LN) peripherally around sublayers, a design we term Peri-LN. While Peri-LN has demonstrated promising empirical performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis shows that Peri-LN strikes an ideal balance in variance growth -- unlike Pre-LN and Post-LN, which are prone to vanishing gradients and ``massive activations.'' To validate our theoretical insight, we conduct large-scale experiments on Transformers up to 3.2B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and convergence stability. Our results suggest that Peri-LN warrants broader consideration for large-scale Transformer architectures, providing renewed insights into the optimal placement and application of LN.

Title: SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model

• Authors: Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, Thomas Wolf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02737
• Pdf URL: https://arxiv.org/pdf/2502.02737
• Copy Paste: [[2502.02737]] SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model(https://arxiv.org/abs/2502.02737)
• Keywords: large language model
• Abstract: While large language models have facilitated breakthroughs in many applications of artificial intelligence, their inherent largeness makes them computationally expensive and challenging to deploy in resource-constrained settings. In this paper, we document the development of SmolLM2, a state-of-the-art "small" (1.7 billion parameter) language model (LM). To attain strong performance, we overtrain SmolLM2 on ~11 trillion tokens of data using a multi-stage training process that mixes web text with specialized math, code, and instruction-following data. We additionally introduce new specialized datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing datasets to be problematically small or low-quality. To inform our design decisions, we perform both small-scale ablations as well as a manual refinement process that updates the dataset mixing rates at each stage based on the performance at the previous stage. Ultimately, we demonstrate that SmolLM2 outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To facilitate future research on LM development as well as applications of small LMs, we release both SmolLM2 as well as all of the datasets we prepared in the course of this project.

Title: RFMedSAM 2: Automatic Prompt Refinement for Enhanced Volumetric Medical Image Segmentation with SAM 2

• Authors: Bin Xie, Hao Tang, Yan Yan, Gady Agam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02741
• Pdf URL: https://arxiv.org/pdf/2502.02741
• Copy Paste: [[2502.02741]] RFMedSAM 2: Automatic Prompt Refinement for Enhanced Volumetric Medical Image Segmentation with SAM 2(https://arxiv.org/abs/2502.02741)
• Keywords: segmentation
• Abstract: Segment Anything Model 2 (SAM 2), a prompt-driven foundation model extending SAM to both image and video domains, has shown superior zero-shot performance compared to its predecessor. Building on SAM's success in medical image segmentation, SAM 2 presents significant potential for further advancement. However, similar to SAM, SAM 2 is limited by its output of binary masks, inability to infer semantic labels, and dependence on precise prompts for the target object area. Additionally, direct application of SAM and SAM 2 to medical image segmentation tasks yields suboptimal results. In this paper, we explore the upper performance limit of SAM 2 using custom fine-tuning adapters, achieving a Dice Similarity Coefficient (DSC) of 92.30% on the BTCV dataset, surpassing the state-of-the-art nnUNet by 12%. Following this, we address the prompt dependency by investigating various prompt generators. We introduce a UNet to autonomously generate predicted masks and bounding boxes, which serve as input to SAM 2. Subsequent dual-stage refinements by SAM 2 further enhance performance. Extensive experiments show that our method achieves state-of-the-art results on the AMOS2022 dataset, with a Dice improvement of 2.9% compared to nnUNet, and outperforms nnUNet by 6.4% on the BTCV dataset.

Title: LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing

• Authors: Yang Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02743
• Pdf URL: https://arxiv.org/pdf/2502.02743
• Copy Paste: [[2502.02743]] LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing(https://arxiv.org/abs/2502.02743)
• Keywords: large language model
• Abstract: The rapid advancement in large language models (LLMs) has brought forth a diverse range of models with varying capabilities that excel in different tasks and domains. However, selecting the optimal LLM for user queries often involves a challenging trade-off between accuracy and cost, a problem exacerbated by the diverse demands of individual queries. In this work, we present a novel framework that formulates the LLM selection process as a multi-armed bandit problem, enabling dynamic and intelligent routing of queries to the most appropriate model. Our approach incorporates a preference-conditioned dynamic routing mechanism, allowing users to specify their preferences at inference time, thereby offering a customizable balance between performance and cost. Additionally, our selection policy is designed to generalize to unseen LLMs, ensuring adaptability to new models as they emerge. Experimental results demonstrate that our method achieves significant improvements in both accuracy and cost-effectiveness across various LLM platforms, showcasing the potential of our framework to adaptively optimize LLM selection in real-world scenarios.

Title: ClarAVy: A Tool for Scalable and Accurate Malware Family Labeling

• Authors: Robert J. Joyce, Derek Everett, Maya Fuchs, Edward Raff, James Holt
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2502.02759
• Pdf URL: https://arxiv.org/pdf/2502.02759
• Copy Paste: [[2502.02759]] ClarAVy: A Tool for Scalable and Accurate Malware Family Labeling(https://arxiv.org/abs/2502.02759)
• Keywords: attack
• Abstract: Determining the family to which a malicious file belongs is an essential component of cyberattack investigation, attribution, and remediation. Performing this task manually is time consuming and requires expert knowledge. Automated tools using that label malware using antivirus detections lack accuracy and/or scalability, making them insufficient for real-world applications. Three pervasive shortcomings in these tools are responsible: (1) incorrect parsing of antivirus detections, (2) errors during family alias resolution, and (3) an inappropriate antivirus aggregation strategy. To address each of these, we created our own malware family labeling tool called ClarAVy. ClarAVy utilizes a Variational Bayesian approach to aggregate detections from a collection of antivirus products into accurate family labels. Our tool scales to enormous malware datasets, and we evaluated it by labeling $\approx$40 million malicious files. ClarAVy has 8 and 12 percentage points higher accuracy than the prior leading tool in labeling the MOTIF and MalPedia datasets, respectively.

Title: Federated Low-Rank Tensor Estimation for Multimodal Image Reconstruction

• Authors: Anh Van Nguyen, Diego Klabjan, Minseok Ryu, Kibaek Kim, Zichao Di
• Subjects: cs.LG, cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2502.02761
• Pdf URL: https://arxiv.org/pdf/2502.02761
• Copy Paste: [[2502.02761]] Federated Low-Rank Tensor Estimation for Multimodal Image Reconstruction(https://arxiv.org/abs/2502.02761)
• Keywords: federate
• Abstract: Low-rank tensor estimation offers a powerful approach to addressing high-dimensional data challenges and can substantially improve solutions to ill-posed inverse problems, such as image reconstruction under noisy or undersampled conditions. Meanwhile, tensor decomposition has gained prominence in federated learning (FL) due to its effectiveness in exploiting latent space structure and its capacity to enhance communication efficiency. In this paper, we present a federated image reconstruction method that applies Tucker decomposition, incorporating joint factorization and randomized sketching to manage large-scale, multimodal data. Our approach avoids reconstructing full-size tensors and supports heterogeneous ranks, allowing clients to select personalized decomposition ranks based on prior knowledge or communication capacity. Numerical results demonstrate that our method achieves superior reconstruction quality and communication compression compared to existing approaches, thereby highlighting its potential for multimodal inverse problems in the FL setting.

Title: Rethinking Vision Transformer for Object Centric Foundation Models

• Authors: Manuel Traub, Martin V. Butz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02763
• Pdf URL: https://arxiv.org/pdf/2502.02763
• Copy Paste: [[2502.02763]] Rethinking Vision Transformer for Object Centric Foundation Models(https://arxiv.org/abs/2502.02763)
• Keywords: transformer, segmentation
• Abstract: Recent state-of-the-art object segmentation mechanisms, such as the Segment Anything Model (SAM) and FastSAM, first encode the full image over several layers and then focus on generating the mask for one particular object or area. We present an off-grid Fovea-Like Input Patching (FLIP) approach, which selects image input and encodes it from the beginning in an object-focused manner. While doing so, it separates locational encoding from an object-centric perceptual code. FLIP is more data-efficient and yields improved segmentation performance when masking relatively small objects in high-resolution visual scenes. On standard benchmarks such as Hypersim, KITTI-360, and OpenImages, FLIP achieves Intersection over Union (IoU) scores that approach the performance of SAM with much less compute effort. It surpasses FastSAM in all IoU measurements. We also introduce an additional semi-natural but highly intuitive dataset where FLIP outperforms SAM and FastSAM overall and particularly on relatively small objects. Seeing that FLIP is an end-to-end object-centric segmentation approach, it has high potential particularly for applications that benefit from computationally efficient, spatially highly selective object tracking.

Title: Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning

• Authors: Chaofan Lin, Jiaming Tang, Shuo Yang, Hanshuo Wang, Tian Tang, Boyu Tian, Ion Stoica, Song Han, Mingyu Gao
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02770
• Pdf URL: https://arxiv.org/pdf/2502.02770
• Copy Paste: [[2502.02770]] Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning(https://arxiv.org/abs/2502.02770)
• Keywords: large language model
• Abstract: Leveraging attention sparsity to accelerate long-context large language models (LLMs) has been a hot research topic. However, current algorithms such as sparse attention or key-value (KV) cache compression tend to use a fixed budget, which presents a significant challenge during deployment because it fails to account for the dynamic nature of real-world scenarios, where the optimal balance between accuracy and efficiency can vary greatly. In this paper, we find that borrowing top-$p$ sampling (nucleus sampling) to sparse attention can surprisingly achieve adaptive budgeting. Based on this, we propose Twilight, a framework to bring adaptive sparsity to any existing sparse attention algorithm without sacrificing their accuracy. Empirical results show that Twilight can adaptively prune at most 98% of redundant tokens, leading to $15.4\times$ acceleration in self-attention operations and $3.9\times$ acceleration in end-to-end per token latency in long context LLM decoding.

Title: 3D Foundation AI Model for Generalizable Disease Detection in Head Computed Tomography

• Authors: Weicheng Zhu, Haoxu Huang, Huanze Tang, Rushabh Musthyala, Boyang Yu, Long Chen, Emilio Vega, Thomas O'Donnell, Seena Dehkharghani, Jennifer A. Frontera, Arjun V. Masurkar, Kara Melmed, Narges Razavian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02779
• Pdf URL: https://arxiv.org/pdf/2502.02779
• Copy Paste: [[2502.02779]] 3D Foundation AI Model for Generalizable Disease Detection in Head Computed Tomography(https://arxiv.org/abs/2502.02779)
• Keywords: robust
• Abstract: Head computed tomography (CT) imaging is a widely-used imaging modality with multitudes of medical indications, particularly in assessing pathology of the brain, skull, and cerebrovascular system. It is commonly the first-line imaging in neurologic emergencies given its rapidity of image acquisition, safety, cost, and ubiquity. Deep learning models may facilitate detection of a wide range of diseases. However, the scarcity of high-quality labels and annotations, particularly among less common conditions, significantly hinders the development of powerful models. To address this challenge, we introduce FM-CT: a Foundation Model for Head CT for generalizable disease detection, trained using self-supervised learning. Our approach pre-trains a deep learning model on a large, diverse dataset of 361,663 non-contrast 3D head CT scans without the need for manual annotations, enabling the model to learn robust, generalizable features. To investigate the potential of self-supervised learning in head CT, we employed both discrimination with self-distillation and masked image modeling, and we construct our model in 3D rather than at the slice level (2D) to exploit the structure of head CT scans more comprehensively and efficiently. The model's downstream classification performance is evaluated using internal and three external datasets, encompassing both in-distribution (ID) and out-of-distribution (OOD) data. Our results demonstrate that the self-supervised foundation model significantly improves performance on downstream diagnostic tasks compared to models trained from scratch and previous 3D CT foundation models on scarce annotated datasets. This work highlights the effectiveness of self-supervised learning in medical imaging and sets a new benchmark for head CT image analysis in 3D, enabling broader use of artificial intelligence for head CT-based diagnosis.

Title: OpenSTARLab: Open Approach for Spatio-Temporal Agent Data Analysis in Soccer

• Authors: Calvin Yeung, Kenjiro Ide, Taiga Someya, Keisuke Fujii
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02785
• Pdf URL: https://arxiv.org/pdf/2502.02785
• Copy Paste: [[2502.02785]] OpenSTARLab: Open Approach for Spatio-Temporal Agent Data Analysis in Soccer(https://arxiv.org/abs/2502.02785)
• Keywords: robust
• Abstract: Sports analytics has become both more professional and sophisticated, driven by the growing availability of detailed performance data. This progress enables applications such as match outcome prediction, player scouting, and tactical analysis. In soccer, the effective utilization of event and tracking data is fundamental for capturing and analyzing the dynamics of the game. However, there are two primary challenges: the limited availability of event data, primarily restricted to top-tier teams and leagues, and the scarcity and high cost of tracking data, which complicates its integration with event data for comprehensive analysis. Here we propose OpenSTARLab, an open-source framework designed to democratize spatio-temporal agent data analysis in sports by addressing these key challenges. OpenSTARLab includes the Pre-processing Package that standardizes event and tracking data through Unified and Integrated Event Data and State-Action-Reward formats, the Event Modeling Package that implements deep learning-based event prediction, alongside the RLearn Package for reinforcement learning tasks. These technical components facilitate the handling of diverse data sources and support advanced analytical tasks, thereby enhancing the overall functionality and usability of the framework. To assess OpenSTARLab's effectiveness, we conducted several experimental evaluations. These demonstrate the superior performance of the specific event prediction model in terms of action and time prediction accuracies and maintained its robust event simulation performance. Furthermore, reinforcement learning experiments reveal a trade-off between action accuracy and temporal difference loss and show comprehensive visualization. Overall, OpenSTARLab serves as a robust platform for researchers and practitioners, enhancing innovation and collaboration in the field of soccer data analytics.

Title: When Machine Learning Gets Personal: Understanding Fairness of Personalized Models

• Authors: Louisa Cornelis, Guillermo Bernárdez, Haewon Jeong, Nina Miolane
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02786
• Pdf URL: https://arxiv.org/pdf/2502.02786
• Copy Paste: [[2502.02786]] When Machine Learning Gets Personal: Understanding Fairness of Personalized Models(https://arxiv.org/abs/2502.02786)
• Keywords: fair, interpretability, explainability
• Abstract: Personalization in machine learning involves tailoring models to individual users by incorporating personal attributes such as demographic or medical data. While personalization can improve prediction accuracy, it may also amplify biases and reduce explainability. This work introduces a unified framework to evaluate the impact of personalization on both prediction accuracy and explanation quality across classification and regression tasks. We derive novel upper bounds for the number of personal attributes that can be used to reliably validate benefits of personalization. Our analysis uncovers key trade-offs. We show that regression models can potentially utilize more personal attributes than classification models. We also demonstrate that improvements in prediction accuracy due to personalization do not necessarily translate to enhanced explainability -- underpinning the importance to evaluate both metrics when personalizing machine learning models in critical settings such as healthcare. Validated with a real-world dataset, this framework offers practical guidance for balancing accuracy, fairness, and interpretability in personalized models.

Title: SimMark: A Robust Sentence-Level Similarity-Based Watermarking Algorithm for Large Language Models

• Authors: Amirhossein Dabiriaghdam, Lele Wang
• Subjects: cs.CL, cs.CR, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02787
• Pdf URL: https://arxiv.org/pdf/2502.02787
• Copy Paste: [[2502.02787]] SimMark: A Robust Sentence-Level Similarity-Based Watermarking Algorithm for Large Language Models(https://arxiv.org/abs/2502.02787)
• Keywords: attack, robust, watermark, large language model
• Abstract: The rapid proliferation of large language models (LLMs) has created an urgent need for reliable methods to detect whether a text is generated by such models. In this paper, we propose SimMark, a posthoc watermarking algorithm that makes LLMs' outputs traceable without requiring access to the model's internal logits, enabling compatibility with a wide range of LLMs, including API-only models. By leveraging the similarity of semantic sentence embeddings and rejection sampling to impose detectable statistical patterns imperceptible to humans, and employing a soft counting mechanism, SimMark achieves robustness against paraphrasing attacks. Experimental results demonstrate that SimMark sets a new benchmark for robust watermarking of LLM-generated content, surpassing prior sentence-level watermarking techniques in robustness, sampling efficiency, and applicability across diverse domains, all while preserving the text quality.

Title: Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation

• Authors: Jingyu Liu, Beidi Chen, Ce Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02789
• Pdf URL: https://arxiv.org/pdf/2502.02789
• Copy Paste: [[2502.02789]] Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation(https://arxiv.org/abs/2502.02789)
• Keywords: large language model
• Abstract: Improving time-to-first-token (TTFT) is an essentially important objective in modern large language model (LLM) inference engines. Because optimizing TTFT directly results in higher maximal QPS and meets the requirements of many critical applications. However, boosting TTFT is notoriously challenging since it is purely compute-bounded and the performance bottleneck shifts from the self-attention to the MLP part. We present SpecPrefill, a training free framework that accelerates the inference TTFT for both long and medium context queries based on the following insight: LLMs are generalized enough to still preserve the quality given only a carefully chosen subset of prompt tokens. At its core, SpecPrefill leverages a lightweight model to speculate locally important tokens based on the context. These tokens, along with the necessary positional information, are then sent to the main model for processing. We evaluate SpecPrefill with a diverse set of tasks, followed by a comprehensive benchmarking of performance improvement both in a real end-to-end setting and ablation studies. SpecPrefill manages to serve Llama-3.1-405B-Instruct-FP8 with up to $7\times$ maximal end-to-end QPS on real downstream tasks and $7.66\times$ TTFT improvement during benchmarking.

Title: Leveraging the true depth of LLMs

• Authors: Ramón Calvo González, Daniele Paliotta, Matteo Pagliardini, Martin Jaggi, François Fleuret
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02790
• Pdf URL: https://arxiv.org/pdf/2502.02790
• Copy Paste: [[2502.02790]] Leveraging the true depth of LLMs(https://arxiv.org/abs/2502.02790)
• Keywords: large language model
• Abstract: Large Language Models demonstrate remarkable capabilities at the cost of high compute requirements. While recent research has shown that intermediate layers can be removed or have their order shuffled without impacting performance significantly, these findings have not been employed to reduce the computational cost of inference. We investigate several potential ways to reduce the depth of pre-trained LLMs without significantly affecting performance. Leveraging our insights, we present a novel approach that exploits this decoupling between layers by grouping some of them into pairs that can be evaluated in parallel. This modification of the computational graph -- through better parallelism -- results in an average improvement of around 1.20x on the number of tokens generated per second, without re-training nor fine-tuning, while retaining 95%-99% of the original accuracy. Empirical evaluation demonstrates that this approach significantly improves serving efficiency while maintaining model performance, offering a practical improvement for large-scale LLM deployment.

Title: CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Exploration

• Authors: Yizhe Yang, Palakorn Achananuparp, Heyan Huang, Jing Jiang, Kit Phey Leng, Nicholas Gabriel Lim, Cameron Tan Shi Ern, Ee-peng Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02807
• Pdf URL: https://arxiv.org/pdf/2502.02807
• Copy Paste: [[2502.02807]] CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Exploration(https://arxiv.org/abs/2502.02807)
• Keywords: large language model
• Abstract: Conversational counselor agents have become essential tools for addressing the rising demand for scalable and accessible mental health support. This paper introduces CAMI, a novel automated counselor agent grounded in Motivational Interviewing (MI) -- a client-centered counseling approach designed to address ambivalence and facilitate behavior change. CAMI employs a novel STAR framework, consisting of client's state inference, motivation topic exploration, and response generation modules, leveraging large language models (LLMs). These components work together to evoke change talk, aligning with MI principles and improving counseling outcomes for clients from diverse backgrounds. We evaluate CAMI's performance through both automated and manual evaluations, utilizing simulated clients to assess MI skill competency, client's state inference accuracy, topic exploration proficiency, and overall counseling success. Results show that CAMI not only outperforms several state-of-the-art methods but also shows more realistic counselor-like behavior. Additionally, our ablation study underscores the critical roles of state inference and topic exploration in achieving this performance.

Title: Mol-LLM: Generalist Molecular LLM with Improved Graph Utilization

• Authors: Chanhui Lee, Yuheon Song, YongJun Jeong, Hanbum Ko, Rodrigo Hormazabal, Sehui Han, Kyunghoon Bae, Sungbin Lim, Sungwoong Kim
• Subjects: cs.LG, cs.AI, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2502.02810
• Pdf URL: https://arxiv.org/pdf/2502.02810
• Copy Paste: [[2502.02810]] Mol-LLM: Generalist Molecular LLM with Improved Graph Utilization(https://arxiv.org/abs/2502.02810)
• Keywords: large language model
• Abstract: Recent advances in Large Language Models (LLMs) have motivated the development of general LLMs for molecular tasks. While several studies have demonstrated that fine-tuned LLMs can achieve impressive benchmark performances, they are far from genuine generalist molecular LLMs due to a lack of fundamental understanding of molecular structure. Specifically, when given molecular task instructions, LLMs trained with naive next-token prediction training assign similar likelihood scores to both original and negatively corrupted molecules, revealing their lack of molecular structure understanding that is crucial for reliable and general molecular LLMs. To overcome this limitation and obtain a true generalist molecular LLM, we introduce a novel multi-modal training method based on a thorough multi-modal instruction tuning as well as a molecular structure preference optimization between chosen and rejected graphs. On various molecular benchmarks, the proposed generalist molecular LLM, called Mol-LLM, achieves state-of-the-art performances among generalist LLMs on most tasks, at the same time, surpassing or comparable to state-of-the-art specialist LLMs. Moreover, Mol-LLM also shows superior generalization performances in reaction prediction tasks, demonstrating the effect of the molecular structure understanding for generalization perspective.

Title: A Survey of Sample-Efficient Deep Learning for Change Detection in Remote Sensing: Tasks, Strategies, and Challenges

• Authors: Lei Ding, Danfeng Hong, Maofan Zhao, Hongruixuan Chen, Chenyu Li, Jie Deng, Naoto Yokoya, Lorenzo Bruzzone, Jocelyn Chanussot
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02835
• Pdf URL: https://arxiv.org/pdf/2502.02835
• Copy Paste: [[2502.02835]] A Survey of Sample-Efficient Deep Learning for Change Detection in Remote Sensing: Tasks, Strategies, and Challenges(https://arxiv.org/abs/2502.02835)
• Keywords: robust
• Abstract: In the last decade, the rapid development of deep learning (DL) has made it possible to perform automatic, accurate, and robust Change Detection (CD) on large volumes of Remote Sensing Images (RSIs). However, despite advances in CD methods, their practical application in real-world contexts remains limited due to the diverse input data and the applicational context. For example, the collected RSIs can be time-series observations, and more informative results are required to indicate the time of change or the specific change category. Moreover, training a Deep Neural Network (DNN) requires a massive amount of training samples, whereas in many cases these samples are difficult to collect. To address these challenges, various specific CD methods have been developed considering different application scenarios and training resources. Additionally, recent advancements in image generation, self-supervision, and visual foundation models (VFMs) have opened up new approaches to address the 'data-hungry' issue of DL-based CD. The development of these methods in broader application scenarios requires further investigation and discussion. Therefore, this article summarizes the literature methods for different CD tasks and the available strategies and techniques to train and deploy DL-based CD methods in sample-limited scenarios. We expect that this survey can provide new insights and inspiration for researchers in this field to develop more effective CD methods that can be applied in a wider range of contexts.

Title: Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning

• Authors: Sunwoo Lee, Jaebak Hwang, Yonghyeon Jo, Seungyul Han
• Subjects: cs.LG, cs.AI, cs.CR, cs.MA
• Abstract URL: https://arxiv.org/abs/2502.02844
• Pdf URL: https://arxiv.org/pdf/2502.02844
• Copy Paste: [[2502.02844]] Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning(https://arxiv.org/abs/2502.02844)
• Keywords: attack, robust
• Abstract: Traditional robust methods in multi-agent reinforcement learning (MARL) often struggle against coordinated adversarial attacks in cooperative scenarios. To address this limitation, we propose the Wolfpack Adversarial Attack framework, inspired by wolf hunting strategies, which targets an initial agent and its assisting agents to disrupt cooperation. Additionally, we introduce the Wolfpack-Adversarial Learning for MARL (WALL) framework, which trains robust MARL policies to defend against the proposed Wolfpack attack by fostering system-wide collaboration. Experimental results underscore the devastating impact of the Wolfpack attack and the significant robustness improvements achieved by WALL.

Title: 5G-AKA-HPQC: Hybrid Post-Quantum Cryptography Protocol for Quantum-Resilient 5G Primary Authentication with Forward Secrecy

• Authors: Yongho Ko, I Wayan Adi Juliawan Pawana, Ilsun You
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2502.02851
• Pdf URL: https://arxiv.org/pdf/2502.02851
• Copy Paste: [[2502.02851]] 5G-AKA-HPQC: Hybrid Post-Quantum Cryptography Protocol for Quantum-Resilient 5G Primary Authentication with Forward Secrecy(https://arxiv.org/abs/2502.02851)
• Keywords: secure, security, attack, robust
• Abstract: 5G enables digital innovation by integrating diverse services, making security especially primary authentication crucial. Two standardized protocols, 5G AKA and EAP AKA', handle authentication for 3GPP and non 3GPP devices. However, 5G AKA has vulnerabilities, including linkability attacks. Additionally, quantum computing poses threats, requiring quantum resistant cryptography. While post-quantum cryptography (PQC) is being standardized, its real world robustness remains unproven. Conventional cryptographic schemes offer reliability due to decades of practical use. To bridge this gap, IETF is standardizing hybrid PQC (HPQC), combining classical and quantum resistant methods. Ensuring forward secrecy and quantum resilience in 5G-AKA is critical. To address these issues, we propose 5G AKA HPQC, a protocol maintaining compatibility with existing standards while enhancing security by integrating keys derived from Elliptic Curve Integrated Encryption Scheme (ECIES) and PQC Key Encapsulation Mechanism (KEM). We validate its security using SVO Logic and ProVerif, confirming its robustness. Performance evaluations assess computational and communication overheads, demonstrating a balance between security and efficiency. This research provides key insights into quantum-safe authentication, contributing to future standardization of secure mobile authentication protocols.

Title: PH-VAE: A Polynomial Hierarchical Variational Autoencoder Towards Disentangled Representation Learning

• Authors: Xi Chen, Shaofan Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02856
• Pdf URL: https://arxiv.org/pdf/2502.02856
• Copy Paste: [[2502.02856]] PH-VAE: A Polynomial Hierarchical Variational Autoencoder Towards Disentangled Representation Learning(https://arxiv.org/abs/2502.02856)
• Keywords: interpretability, generative
• Abstract: The variational autoencoder (VAE) is a simple and efficient generative artificial intelligence method for modeling complex probability distributions of various types of data, such as images and texts. However, it suffers some main shortcomings, such as lack of interpretability in the latent variables, difficulties in tuning hyperparameters while training, producing blurry, unrealistic downstream outputs or loss of information due to how it calculates loss functions and recovers data distributions, overfitting, and origin gravity effect for small data sets, among other issues. These and other limitations have caused unsatisfactory generation effects for the data with complex distributions. In this work, we proposed and developed a polynomial hierarchical variational autoencoder (PH-VAE), in which we used a polynomial hierarchical date format to generate or to reconstruct the data distributions. In doing so, we also proposed a novel Polynomial Divergence in the loss function to replace or generalize the Kullback-Leibler (KL) divergence, which results in systematic and drastic improvements in both accuracy and reproducibility of the re-constructed distribution function as well as the quality of re-constructed data images while keeping the dataset size the same but capturing fine resolution of the data. Moreover, we showed that the proposed PH-VAE has some form of disentangled representation learning ability.

Title: Domain-Invariant Per-Frame Feature Extraction for Cross-Domain Imitation Learning with Visual Observations

• Authors: Minung Kim, Kawon Lee, Jungmo Kim, Sungho Choi, Seungyul Han
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02867
• Pdf URL: https://arxiv.org/pdf/2502.02867
• Copy Paste: [[2502.02867]] Domain-Invariant Per-Frame Feature Extraction for Cross-Domain Imitation Learning with Visual Observations(https://arxiv.org/abs/2502.02867)
• Keywords: extraction
• Abstract: Imitation learning (IL) enables agents to mimic expert behavior without reward signals but faces challenges in cross-domain scenarios with high-dimensional, noisy, and incomplete visual observations. To address this, we propose Domain-Invariant Per-Frame Feature Extraction for Imitation Learning (DIFF-IL), a novel IL method that extracts domain-invariant features from individual frames and adapts them into sequences to isolate and replicate expert behaviors. We also introduce a frame-wise time labeling technique to segment expert behaviors by timesteps and assign rewards aligned with temporal contexts, enhancing task performance. Experiments across diverse visual environments demonstrate the effectiveness of DIFF-IL in addressing complex visual tasks.

Title: Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning

• Authors: Yibo Yan, Shen Wang, Jiahao Huo, Jingheng Ye, Zhendong Chu, Xuming Hu, Philip S. Yu, Carla Gomes, Bart Selman, Qingsong Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02871
• Pdf URL: https://arxiv.org/pdf/2502.02871
• Copy Paste: [[2502.02871]] Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning(https://arxiv.org/abs/2502.02871)
• Keywords: large language model
• Abstract: Scientific reasoning, the process through which humans apply logic, evidence, and critical thinking to explore and interpret scientific phenomena, is essential in advancing knowledge reasoning across diverse fields. However, despite significant progress, current scientific reasoning models still struggle with generalization across domains and often fall short of multimodal perception. Multimodal Large Language Models (MLLMs), which integrate text, images, and other modalities, present an exciting opportunity to overcome these limitations and enhance scientific reasoning. Therefore, this position paper argues that MLLMs can significantly advance scientific reasoning across disciplines such as mathematics, physics, chemistry, and biology. First, we propose a four-stage research roadmap of scientific reasoning capabilities, and highlight the current state of MLLM applications in scientific reasoning, noting their ability to integrate and reason over diverse data types. Second, we summarize the key challenges that remain obstacles to achieving MLLM's full potential. To address these challenges, we propose actionable insights and suggestions for the future. Overall, our work offers a novel perspective on MLLM integration with scientific reasoning, providing the LLM community with a valuable vision for achieving Artificial General Intelligence (AGI).

Title: Expertized Caption Auto-Enhancement for Video-Text Retrieval

• Authors: Junxiang Chen, Baoyao yang, Wenbin Yao
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02885
• Pdf URL: https://arxiv.org/pdf/2502.02885
• Copy Paste: [[2502.02885]] Expertized Caption Auto-Enhancement for Video-Text Retrieval(https://arxiv.org/abs/2502.02885)
• Keywords: large language model
• Abstract: The burgeoning field of video-text retrieval has witnessed significant advancements with the advent of deep learning. However, the challenge of matching text and video persists due to inadequate textual descriptions of videos. The substantial information gap between the two modalities hinders a comprehensive understanding of videos, resulting in ambiguous retrieval results. While rewriting methods based on large language models have been proposed to broaden text expressions, carefully crafted prompts are essential to ensure the reasonableness and completeness of the rewritten texts. This paper proposes an automatic caption enhancement method that enhances expression quality and mitigates empiricism in augmented captions through self-learning. Additionally, an expertized caption selection mechanism is designed and introduced to customize augmented captions for each video, facilitating video-text matching. Our method is entirely data-driven, which not only dispenses with heavy data collection and computation workload but also improves self-adaptability by circumventing lexicon dependence and introducing personalized matching. The superiority of our method is validated by state-of-the-art results on various benchmarks, specifically achieving Top-1 recall accuracy of 68.5% on MSR-VTT, 68.1% on MSVD, and 62.0% on DiDeMo.

Title: Lowering the Barrier of Machine Learning: Achieving Zero Manual Labeling in Review Classification Using LLMs

• Authors: Yejian Zhang, Shingo Takada
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02893
• Pdf URL: https://arxiv.org/pdf/2502.02893
• Copy Paste: [[2502.02893]] Lowering the Barrier of Machine Learning: Achieving Zero Manual Labeling in Review Classification Using LLMs(https://arxiv.org/abs/2502.02893)
• Keywords: transformer, generative, large language model
• Abstract: With the internet's evolution, consumers increasingly rely on online reviews for service or product choices, necessitating that businesses analyze extensive customer feedback to enhance their offerings. While machine learning-based sentiment classification shows promise in this realm, its technical complexity often bars small businesses and individuals from leveraging such advancements, which may end up making the competitive gap between small and large businesses even bigger in terms of improving customer satisfaction. This paper introduces an approach that integrates large language models (LLMs), specifically Generative Pre-trained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT)-based models, making it accessible to a wider audience. Our experiments across various datasets confirm that our approach retains high classification accuracy without the need for manual labeling, expert knowledge in tuning and data annotation, or substantial computational power. By significantly lowering the barriers to applying sentiment classification techniques, our methodology enhances competitiveness and paves the way for making machine learning technology accessible to a broader audience.

Title: A Benchmark for the Detection of Metalinguistic Disagreements between LLMs and Knowledge Graphs

• Authors: Bradley P. Allen, Paul T. Groth
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02896
• Pdf URL: https://arxiv.org/pdf/2502.02896
• Copy Paste: [[2502.02896]] A Benchmark for the Detection of Metalinguistic Disagreements between LLMs and Knowledge Graphs(https://arxiv.org/abs/2502.02896)
• Keywords: extraction, large language model
• Abstract: Evaluating large language models (LLMs) for tasks like fact extraction in support of knowledge graph construction frequently involves computing accuracy metrics using a ground truth benchmark based on a knowledge graph (KG). These evaluations assume that errors represent factual disagreements. However, human discourse frequently features metalinguistic disagreement, where agents differ not on facts but on the meaning of the language used to express them. Given the complexity of natural language processing and generation using LLMs, we ask: do metalinguistic disagreements occur between LLMs and KGs? Based on an investigation using the T-REx knowledge alignment dataset, we hypothesize that metalinguistic disagreement does in fact occur between LLMs and KGs, with potential relevance for the practice of knowledge graph engineering. We propose a benchmark for evaluating the detection of factual and metalinguistic disagreements between LLMs and KGs. An initial proof of concept of such a benchmark is available on Github.

Title: PoleStack: Robust Pole Estimation of Irregular Objects from Silhouette Stacking

• Authors: Jacopo Villa, Jay W. McMahon, Issa A. D. Nesnas
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02907
• Pdf URL: https://arxiv.org/pdf/2502.02907
• Copy Paste: [[2502.02907]] PoleStack: Robust Pole Estimation of Irregular Objects from Silhouette Stacking(https://arxiv.org/abs/2502.02907)
• Keywords: robust
• Abstract: We present an algorithm to estimate the rotation pole of a principal-axis rotator using silhouette images collected from multiple camera poses. First, a set of images is stacked to form a single silhouette-stack image, where the object's rotation introduces reflective symmetry about the imaged pole direction. We estimate this projected-pole direction by identifying maximum symmetry in the silhouette stack. To handle unknown center-of-mass image location, we apply the Discrete Fourier Transform to produce the silhouette-stack amplitude spectrum, achieving translation invariance and increased robustness to noise. Second, the 3D pole orientation is estimated by combining two or more projected-pole measurements collected from different camera orientations. We demonstrate degree-level pole estimation accuracy using low-resolution imagery, showing robustness to severe surface shadowing and centroid-based image-registration errors. The proposed approach could be suitable for pole estimation during both the approach phase toward a target object and while hovering.

Title: SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in LLMs

• Authors: Dinithi Jayasuriya, Sina Tayebati, Davide Ettori, Ranganath Krishnan, Amit Ranjan Trivedi (Intel Labs, Oregon)
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02909
• Pdf URL: https://arxiv.org/pdf/2502.02909
• Copy Paste: [[2502.02909]] SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in LLMs(https://arxiv.org/abs/2502.02909)
• Keywords: robust, large language model
• Abstract: We propose SPARC, a lightweight continual learning framework for large language models (LLMs) that enables efficient task adaptation through prompt tuning in a lower-dimensional space. By leveraging principal component analysis (PCA), we identify a compact subspace of the training data. Optimizing prompts in this lower-dimensional space enhances training efficiency, as it focuses updates on the most relevant features while reducing computational overhead. Furthermore, since the model's internal structure remains unaltered, the extensive knowledge gained from pretraining is fully preserved, ensuring that previously learned information is not compromised during adaptation. Our method achieves high knowledge retention in both task-incremental and domain-incremental continual learning setups while fine-tuning only 0.04% of the model's parameters. Additionally, by integrating LoRA, we enhance adaptability to computational constraints, allowing for a tradeoff between accuracy and training cost. Experiments on the SuperGLUE benchmark demonstrate that our PCA-based prompt tuning combined with LoRA maintains full knowledge retention while improving accuracy, utilizing only 1% of the model's parameters. These results establish our approach as a scalable and resource-efficient solution for continual learning in LLMs.

Title: Privacy Token: Surprised to Find Out What You Accidentally Revealed

• Authors: Jiayang Meng, Tao Huang, Xin Shi, Qingyu Huang, Chen Hou, Hong Chen
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2502.02913
• Pdf URL: https://arxiv.org/pdf/2502.02913
• Copy Paste: [[2502.02913]] Privacy Token: Surprised to Find Out What You Accidentally Revealed(https://arxiv.org/abs/2502.02913)
• Keywords: privacy, attack, robust
• Abstract: The widespread deployment of deep learning models in privacy-sensitive domains has amplified concerns regarding privacy risks, particularly those stemming from gradient leakage during training. Current privacy assessments primarily rely on post-training attack simulations. However, these methods are inherently reactive, unable to encompass all potential attack scenarios, and often based on idealized adversarial assumptions. These limitations underscore the need for proactive approaches to privacy risk assessment during the training process. To address this gap, we propose the concept of privacy tokens, which are derived directly from private gradients during training. Privacy tokens encapsulate gradient features and, when combined with data features, offer valuable insights into the extent of private information leakage from training data, enabling real-time measurement of privacy risks without relying on adversarial attack simulations. Additionally, we employ Mutual Information (MI) as a robust metric to quantify the relationship between training data and gradients, providing precise and continuous assessments of privacy leakage throughout the training process. Extensive experiments validate our framework, demonstrating the effectiveness of privacy tokens and MI in identifying and quantifying privacy risks. This proactive approach marks a significant advancement in privacy monitoring, promoting the safer deployment of deep learning models in sensitive applications.

Title: Interactive Symbolic Regression through Offline Reinforcement Learning: A Co-Design Framework

• Authors: Yuan Tian, Wenqi Zhou, Michele Viscione, Hao Dong, David Kammer, Olga Fink
• Subjects: cs.LG, cs.AI, cs.SC
• Abstract URL: https://arxiv.org/abs/2502.02917
• Pdf URL: https://arxiv.org/pdf/2502.02917
• Copy Paste: [[2502.02917]] Interactive Symbolic Regression through Offline Reinforcement Learning: A Co-Design Framework(https://arxiv.org/abs/2502.02917)
• Keywords: transformer
• Abstract: Symbolic Regression (SR) holds great potential for uncovering underlying mathematical and physical relationships from observed data. However, the vast combinatorial space of possible expressions poses significant challenges for both online search methods and pre-trained transformer models. Additionally, current state-of-the-art approaches typically do not consider the integration of domain experts' prior knowledge and do not support iterative interactions with the model during the equation discovery process. To address these challenges, we propose the Symbolic Q-network (Sym-Q), an advanced interactive framework for large-scale symbolic regression. Unlike previous large-scale transformer-based SR approaches, Sym-Q leverages reinforcement learning without relying on a transformer-based decoder. This formulation allows the agent to learn through offline reinforcement learning using any type of tree encoder, enabling more efficient training and inference. Furthermore, we propose a co-design mechanism, where the reinforcement learning-based Sym-Q facilitates effective interaction with domain experts at any stage of the equation discovery process. Users can dynamically modify generated nodes of the expression, collaborating with the agent to tailor the mathematical expression to best fit the problem and align with the assumed physical laws, particularly when there is prior partial knowledge of the expected behavior. Our experiments demonstrate that the pre-trained Sym-Q surpasses existing SR algorithms on the challenging SSDNC benchmark. Moreover, we experimentally show on real-world cases that its performance can be further enhanced by the interactive co-design mechanism, with Sym-Q achieving greater performance gains than other state-of-the-art models. Our reproducible code is available at this https URL.

Title: Maximizing the Position Embedding for Vision Transformers with Global Average Pooling

• Authors: Wonjun Lee, Bumsub Ham, Suhyun Kim
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02919
• Pdf URL: https://arxiv.org/pdf/2502.02919
• Copy Paste: [[2502.02919]] Maximizing the Position Embedding for Vision Transformers with Global Average Pooling(https://arxiv.org/abs/2502.02919)
• Keywords: transformer
• Abstract: In vision transformers, position embedding (PE) plays a crucial role in capturing the order of tokens. However, in vision transformer structures, there is a limitation in the expressiveness of PE due to the structure where position embedding is simply added to the token embedding. A layer-wise method that delivers PE to each layer and applies independent Layer Normalizations for token embedding and PE has been adopted to overcome this limitation. In this paper, we identify the conflicting result that occurs in a layer-wise structure when using the global average pooling (GAP) method instead of the class token. To overcome this problem, we propose MPVG, which maximizes the effectiveness of PE in a layer-wise structure with GAP. Specifically, we identify that PE counterbalances token embedding values at each layer in a layer-wise structure. Furthermore, we recognize that the counterbalancing role of PE is insufficient in the layer-wise structure, and we address this by maximizing the effectiveness of PE through MPVG. Through experiments, we demonstrate that PE performs a counterbalancing role and that maintaining this counterbalancing directionality significantly impacts vision transformers. As a result, the experimental results show that MPVG outperforms existing methods across vision transformers on various tasks.

Title: Robust Reward Alignment in Hypothesis Space

• Authors: Zhixian Xie, Haode Zhang, Yizhe Feng, Wanxin Jin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02921
• Pdf URL: https://arxiv.org/pdf/2502.02921
• Copy Paste: [[2502.02921]] Robust Reward Alignment in Hypothesis Space(https://arxiv.org/abs/2502.02921)
• Keywords: robust
• Abstract: Reward design for reinforcement learning and optimal control agents is challenging. Preference-based alignment addresses this by enabling agents to learn rewards from ranked trajectory pairs provided by humans. However, existing methods often struggle from poor robustness to unknown false human preferences. In this work, we propose a robust and efficient reward alignment method based on a novel and geometrically interpretable perspective: hypothesis space batched cutting. Our method iteratively refines the reward hypothesis space through "cuts" based on batches of human preferences. Within each batch, human preferences, queried based on disagreement, are grouped using a voting function to determine the appropriate cut, ensuring a bounded human query complexity. To handle unknown erroneous preferences, we introduce a conservative cutting method within each batch, preventing erroneous human preferences from making overly aggressive cuts to the hypothesis space. This guarantees provable robustness against false preferences. We evaluate our method in a model predictive control setting across diverse tasks, including DM-Control, dexterous in-hand manipulation, and locomotion. The results demonstrate that our framework achieves comparable or superior performance to state-of-the-art methods in error-free settings while significantly outperforming existing method when handling high percentage of erroneous human preferences.

Title: Elucidating the Preconditioning in Consistency Distillation

• Authors: Kaiwen Zheng, Guande He, Jianfei Chen, Fan Bao, Jun Zhu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02922
• Pdf URL: https://arxiv.org/pdf/2502.02922
• Copy Paste: [[2502.02922]] Elucidating the Preconditioning in Consistency Distillation(https://arxiv.org/abs/2502.02922)
• Keywords: diffusion
• Abstract: Consistency distillation is a prevalent way for accelerating diffusion models adopted in consistency (trajectory) models, in which a student model is trained to traverse backward on the probability flow (PF) ordinary differential equation (ODE) trajectory determined by the teacher model. Preconditioning is a vital technique for stabilizing consistency distillation, by linear combining the input data and the network output with pre-defined coefficients as the consistency function. It imposes the boundary condition of consistency functions without restricting the form and expressiveness of the neural network. However, previous preconditionings are hand-crafted and may be suboptimal choices. In this work, we offer the first theoretical insights into the preconditioning in consistency distillation, by elucidating its design criteria and the connection to the teacher ODE trajectory. Based on these analyses, we further propose a principled way dubbed \textit{Analytic-Precond} to analytically optimize the preconditioning according to the consistency gap (defined as the gap between the teacher denoiser and the optimal student denoiser) on a generalized teacher ODE. We demonstrate that Analytic-Precond can facilitate the learning of trajectory jumpers, enhance the alignment of the student trajectory with the teacher's, and achieve $2\times$ to $3\times$ training acceleration of consistency trajectory models in multi-step generation across various datasets.

Title: Every Angle Is Worth A Second Glance: Mining Kinematic Skeletal Structures from Multi-view Joint Cloud

• Authors: Junkun Jiang, Jie Chen, Ho Yin Au, Mingyuan Chen, Wei Xue, Yike Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02936
• Pdf URL: https://arxiv.org/pdf/2502.02936
• Copy Paste: [[2502.02936]] Every Angle Is Worth A Second Glance: Mining Kinematic Skeletal Structures from Multi-view Joint Cloud(https://arxiv.org/abs/2502.02936)
• Keywords: transformer
• Abstract: Multi-person motion capture over sparse angular observations is a challenging problem under interference from both self- and mutual-occlusions. Existing works produce accurate 2D joint detection, however, when these are triangulated and lifted into 3D, available solutions all struggle in selecting the most accurate candidates and associating them to the correct joint type and target identity. As such, in order to fully utilize all accurate 2D joint location information, we propose to independently triangulate between all same-typed 2D joints from all camera views regardless of their target ID, forming the Joint Cloud. Joint Cloud consist of both valid joints lifted from the same joint type and target ID, as well as falsely constructed ones that are from different 2D sources. These redundant and inaccurate candidates are processed over the proposed Joint Cloud Selection and Aggregation Transformer (JCSAT) involving three cascaded encoders which deeply explore the trajectile, skeletal structural, and view-dependent correlations among all 3D point candidates in the cross-embedding space. An Optimal Token Attention Path (OTAP) module is proposed which subsequently selects and aggregates informative features from these redundant observations for the final prediction of human motion. To demonstrate the effectiveness of JCSAT, we build and publish a new multi-person motion capture dataset BUMocap-X with complex interactions and severe occlusions. Comprehensive experiments over the newly presented as well as benchmark datasets validate the effectiveness of the proposed framework, which outperforms all existing state-of-the-art methods, especially under challenging occlusion scenarios.

Title: Fast T2T: Optimization Consistency Speeds Up Diffusion-Based Training-to-Testing Solving for Combinatorial Optimization

• Authors: Yang Li, Jinpei Guo, Runzhong Wang, Hongyuan Zha, Junchi Yan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02941
• Pdf URL: https://arxiv.org/pdf/2502.02941
• Copy Paste: [[2502.02941]] Fast T2T: Optimization Consistency Speeds Up Diffusion-Based Training-to-Testing Solving for Combinatorial Optimization(https://arxiv.org/abs/2502.02941)
• Keywords: diffusion, generative
• Abstract: Diffusion models have recently advanced Combinatorial Optimization (CO) as a powerful backbone for neural solvers. However, their iterative sampling process requiring denoising across multiple noise levels incurs substantial overhead. We propose to learn direct mappings from different noise levels to the optimal solution for a given instance, facilitating high-quality generation with minimal shots. This is achieved through an optimization consistency training protocol, which, for a given instance, minimizes the difference among samples originating from varying generative trajectories and time steps relative to the optimal solution. The proposed model enables fast single-step solution generation while retaining the option of multi-step sampling to trade for sampling quality, which offers a more effective and efficient alternative backbone for neural solvers. In addition, within the training-to-testing (T2T) framework, to bridge the gap between training on historical instances and solving new instances, we introduce a novel consistency-based gradient search scheme during the test stage, enabling more effective exploration of the solution space learned during training. It is achieved by updating the latent solution probabilities under objective gradient guidance during the alternation of noise injection and denoising steps. We refer to this model as Fast T2T. Extensive experiments on two popular tasks, the Traveling Salesman Problem (TSP) and Maximal Independent Set (MIS), demonstrate the superiority of Fast T2T regarding both solution quality and efficiency, even outperforming LKH given limited time budgets. Notably, Fast T2T with merely one-step generation and one-step gradient search can mostly outperform the SOTA diffusion-based counterparts that require hundreds of steps, while achieving tens of times speedup.

Title: LLM-KT: Aligning Large Language Models with Knowledge Tracing using a Plug-and-Play Instruction

• Authors: Ziwei Wang, Jie Zhou, Qin Chen, Min Zhang, Bo Jiang, Aimin Zhou, Qinchun Bai, Liang He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02945
• Pdf URL: https://arxiv.org/pdf/2502.02945
• Copy Paste: [[2502.02945]] LLM-KT: Aligning Large Language Models with Knowledge Tracing using a Plug-and-Play Instruction(https://arxiv.org/abs/2502.02945)
• Keywords: large language model
• Abstract: The knowledge tracing (KT) problem is an extremely important topic in personalized education, which aims to predict whether students can correctly answer the next question based on their past question-answer records. Prior work on this task mainly focused on learning the sequence of behaviors based on the IDs or textual information. However, these studies usually fail to capture students' sufficient behavioral patterns without reasoning with rich world knowledge about questions. In this paper, we propose a large language models (LLMs)-based framework for KT, named \texttt{\textbf{LLM-KT}}, to integrate the strengths of LLMs and traditional sequence interaction models. For task-level alignment, we design Plug-and-Play instruction to align LLMs with KT, leveraging LLMs' rich knowledge and powerful reasoning capacity. For modality-level alignment, we design the plug-in context and sequence to integrate multiple modalities learned by traditional methods. To capture the long context of history records, we present a plug-in context to flexibly insert the compressed context embedding into LLMs using question-specific and concept-specific tokens. Furthermore, we introduce a plug-in sequence to enhance LLMs with sequence interaction behavior representation learned by traditional sequence models using a sequence adapter. Extensive experiments show that \texttt{\textbf{LLM-KT}} obtains state-of-the-art performance on four typical datasets by comparing it with approximately 20 strong baselines.

Title: Direct Distributional Optimization for Provable Alignment of Diffusion Models

• Authors: Ryotaro Kawata, Kazusato Oko, Atsushi Nitanda, Taiji Suzuki
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02954
• Pdf URL: https://arxiv.org/pdf/2502.02954
• Copy Paste: [[2502.02954]] Direct Distributional Optimization for Provable Alignment of Diffusion Models(https://arxiv.org/abs/2502.02954)
• Keywords: diffusion
• Abstract: We introduce a novel alignment method for diffusion models from distribution optimization perspectives while providing rigorous convergence guarantees. We first formulate the problem as a generic regularized loss minimization over probability distributions and directly optimize the distribution using the Dual Averaging method. Next, we enable sampling from the learned distribution by approximating its score function via Doob's $h$-transform technique. The proposed framework is supported by rigorous convergence guarantees and an end-to-end bound on the sampling error, which imply that when the original distribution's score is known accurately, the complexity of sampling from shifted distributions is independent of isoperimetric conditions. This framework is broadly applicable to general distribution optimization problems, including alignment tasks in Reinforcement Learning with Human Feedback (RLHF), Direct Preference Optimization (DPO), and Kahneman-Tversky Optimization (KTO). We empirically validate its performance on synthetic and image datasets using the DPO objective.

Title: Position: Editing Large Language Models Poses Serious Safety Risks

• Authors: Paul Youssef, Zhixue Zhao, Daniel Braun, Jörg Schlötterer, Christin Seifert
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02958
• Pdf URL: https://arxiv.org/pdf/2502.02958
• Copy Paste: [[2502.02958]] Position: Editing Large Language Models Poses Serious Safety Risks(https://arxiv.org/abs/2502.02958)
• Keywords: steal, large language model
• Abstract: Large Language Models (LLMs) contain large amounts of facts about the world. These facts can become outdated over time, which has led to the development of knowledge editing methods (KEs) that can change specific facts in LLMs with limited side effects. This position paper argues that editing LLMs poses serious safety risks that have been largely overlooked. First, we note the fact that KEs are widely available, computationally inexpensive, highly performant, and stealthy makes them an attractive tool for malicious actors. Second, we discuss malicious use cases of KEs, showing how KEs can be easily adapted for a variety of malicious purposes. Third, we highlight vulnerabilities in the AI ecosystem that allow unrestricted uploading and downloading of updated models without verification. Fourth, we argue that a lack of social and institutional awareness exacerbates this risk, and discuss the implications for different stakeholders. We call on the community to (i) research tamper-resistant models and countermeasures against malicious model editing, and (ii) actively engage in securing the AI ecosystem.

Title: Large Language Model Adversarial Landscape Through the Lens of Attack Objectives

• Authors: Nan Wang, Kane Walter, Yansong Gao, Alsharif Abuadbba
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2502.02960
• Pdf URL: https://arxiv.org/pdf/2502.02960
• Copy Paste: [[2502.02960]] Large Language Model Adversarial Landscape Through the Lens of Attack Objectives(https://arxiv.org/abs/2502.02960)
• Keywords: security, privacy, defense, attack, robust, large language model
• Abstract: Large Language Models (LLMs) represent a transformative leap in artificial intelligence, enabling the comprehension, generation, and nuanced interaction with human language on an unparalleled scale. However, LLMs are increasingly vulnerable to a range of adversarial attacks that threaten their privacy, reliability, security, and trustworthiness. These attacks can distort outputs, inject biases, leak sensitive information, or disrupt the normal functioning of LLMs, posing significant challenges across various applications. In this paper, we provide a novel comprehensive analysis of the adversarial landscape of LLMs, framed through the lens of attack objectives. By concentrating on the core goals of adversarial actors, we offer a fresh perspective that examines threats from the angles of privacy, integrity, availability, and misuse, moving beyond conventional taxonomies that focus solely on attack techniques. This objective-driven adversarial landscape not only highlights the strategic intent behind different adversarial approaches but also sheds light on the evolving nature of these threats and the effectiveness of current defenses. Our analysis aims to guide researchers and practitioners in better understanding, anticipating, and mitigating these attacks, ultimately contributing to the development of more resilient and robust LLM systems.

Title: Membership Inference Attack Should Move On to Distributional Statistics for Distilled Generative Models

• Authors: Muxing Li, Zesheng Ye, Yixuan Li, Andy Song, Guangquan Zhang, Feng Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02970
• Pdf URL: https://arxiv.org/pdf/2502.02970
• Copy Paste: [[2502.02970]] Membership Inference Attack Should Move On to Distributional Statistics for Distilled Generative Models(https://arxiv.org/abs/2502.02970)
• Keywords: privacy, attack, membership infer, generative
• Abstract: Membership inference attacks (MIAs) determine whether certain data instances were used to train a model by exploiting the differences in how the model responds to seen versus unseen instances. This capability makes MIAs important in assessing privacy leakage within modern generative AI systems. However, this paper reveals an oversight in existing MIAs against \emph{distilled generative models}: attackers can no longer detect a teacher model's training instances individually when targeting the distilled student model, as the student learns from the teacher-generated data rather than its original member data, preventing direct instance-level memorization. Nevertheless, we find that student-generated samples exhibit a significantly stronger distributional alignment with teacher's member data than non-member data. This leads us to posit that MIAs \emph{on distilled generative models should shift from instance-level to distribution-level statistics}. We thereby introduce a \emph{set-based} MIA framework that measures \emph{relative} distributional discrepancies between student-generated data\emph{sets} and potential member/non-member data\emph{sets}, Empirically, distributional statistics reliably distinguish a teacher's member data from non-member data through the distilled model. Finally, we discuss scenarios in which our setup faces limitations.

Title: Disentangling CLIP Features for Enhanced Localized Understanding

• Authors: Samyak Rawelekar, Yujun Cai, Yiwei Wang, Ming-Hsuan Yang, Narendra Ahuja
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02977
• Pdf URL: https://arxiv.org/pdf/2502.02977
• Copy Paste: [[2502.02977]] Disentangling CLIP Features for Enhanced Localized Understanding(https://arxiv.org/abs/2502.02977)
• Keywords: segmentation
• Abstract: Vision-language models (VLMs) demonstrate impressive capabilities in coarse-grained tasks like image classification and retrieval. However, they struggle with fine-grained tasks that require localized understanding. To investigate this weakness, we comprehensively analyze CLIP features and identify an important issue: semantic features are highly correlated. Specifically, the features of a class encode information about other classes, which we call mutual feature information (MFI). This mutual information becomes evident when we query a specific class and unrelated objects are activated along with the target class. To address this issue, we propose Unmix-CLIP, a novel framework designed to reduce MFI and improve feature disentanglement. We introduce MFI loss, which explicitly separates text features by projecting them into a space where inter-class similarity is minimized. To ensure a corresponding separation in image features, we use multi-label recognition (MLR) to align the image features with the separated text features. This ensures that both image and text features are disentangled and aligned across modalities, improving feature separation for downstream tasks. For the COCO- 14 dataset, Unmix-CLIP reduces feature similarity by 24.9%. We demonstrate its effectiveness through extensive evaluations of MLR and zeroshot semantic segmentation (ZS3). In MLR, our method performs competitively on the VOC2007 and surpasses SOTA approaches on the COCO-14 dataset, using fewer training parameters. Additionally, Unmix-CLIP consistently outperforms existing ZS3 methods on COCO and VOC

Title: Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons

• Authors: Renjun Hu, Yi Cheng, Libin Meng, Jiaxin Xia, Yi Zong, Xing Shi, Wei Lin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02988
• Pdf URL: https://arxiv.org/pdf/2502.02988
• Copy Paste: [[2502.02988]] Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons(https://arxiv.org/abs/2502.02988)
• Keywords: large language model
• Abstract: The rapid advancement of large language models (LLMs) has opened new possibilities for their adoption as evaluative judges. This paper introduces Themis, a fine-tuned LLM judge that delivers sophisticated context-aware evaluations. We provide a comprehensive overview of the development pipeline for Themis, highlighting its scenario-dependent evaluation prompts and two novel methods for controlled instruction generation. These designs enable Themis to effectively distill evaluative skills from teacher models, while retaining flexibility for continuous development. We introduce two human-labeled benchmarks for meta-evaluation, demonstrating that Themis can achieve high alignment with human preferences in an economical manner. Additionally, we explore insights into the LLM-as-a-judge paradigm, revealing nuances in performance and the varied effects of reference answers. Notably, we observe that pure knowledge distillation from strong LLMs, though common, does not guarantee performance improvement through scaling. We propose a mitigation strategy based on instruction-following difficulty. Furthermore, we provide practical guidelines covering data balancing, prompt customization, multi-objective training, and metric aggregation. We aim for our method and findings, along with the fine-tuning data, benchmarks, and model checkpoints, to support future research and development in this area.

Title: Lightweight Protocols for Distributed Private Quantile Estimation

• Authors: Anders Aamand, Fabrizio Boninsegna, Abigail Gentle, Jacob Imola, Rasmus Pagh
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2502.02990
• Pdf URL: https://arxiv.org/pdf/2502.02990
• Copy Paste: [[2502.02990]] Lightweight Protocols for Distributed Private Quantile Estimation(https://arxiv.org/abs/2502.02990)
• Keywords: privacy
• Abstract: Distributed data analysis is a large and growing field driven by a massive proliferation of user devices, and by privacy concerns surrounding the centralised storage of data. We consider two \emph{adaptive} algorithms for estimating one quantile (e.g.~the median) when each user holds a single data point lying in a domain $[B]$ that can be queried once through a private mechanism; one under local differential privacy (LDP) and another for shuffle differential privacy (shuffle-DP). In the adaptive setting we present an $\varepsilon$-LDP algorithm which can estimate any quantile within error $\alpha$ only requiring $O(\frac{\log B}{\varepsilon^2\alpha^2})$ users, and an $(\varepsilon,\delta)$-shuffle DP algorithm requiring only $\widetilde{O}((\frac{1}{\varepsilon^2}+\frac{1}{\alpha^2})\log B)$ users. Prior (nonadaptive) algorithms require more users by several logarithmic factors in $B$. We further provide a matching lower bound for adaptive protocols, showing that our LDP algorithm is optimal in the low-$\varepsilon$ regime. Additionally, we establish lower bounds against non-adaptive protocols which paired with our understanding of the adaptive case, proves a fundamental separation between these models.

Title: MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation

• Authors: Seonok Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03004
• Pdf URL: https://arxiv.org/pdf/2502.03004
• Copy Paste: [[2502.03004]] MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation(https://arxiv.org/abs/2502.03004)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities across natural language processing tasks. However, their application to specialized domains such as medicine and biology requires further optimization to ensure factual accuracy, reliability, and contextual depth. We introduce MedBioLM, a domain-adapted biomedical question-answering model designed to enhance both short-form and long-form queries. By integrating fine-tuning and retrieval-augmented generation (RAG), MedBioLM dynamically incorporates domain-specific knowledge, improving reasoning abilities and factual accuracy. To evaluate its effectiveness, we fine-tuned the model on diverse biomedical QA datasets, covering structured multiple-choice assessments and complex clinical reasoning tasks. Fine-tuning significantly improves accuracy on benchmark datasets, while RAG enhances factual consistency. These results highlight the potential of domain-optimized LLMs in advancing biomedical research, medical education, and clinical decision support.

Title: Driver Assistance System Based on Multimodal Data Hazard Detection

• Authors: Long Zhouxiang, Ovanes Petrosian
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03005
• Pdf URL: https://arxiv.org/pdf/2502.03005
• Copy Paste: [[2502.03005]] Driver Assistance System Based on Multimodal Data Hazard Detection(https://arxiv.org/abs/2502.03005)
• Keywords: extraction
• Abstract: Autonomous driving technology has advanced significantly, yet detecting driving anomalies remains a major challenge due to the long-tailed distribution of driving events. Existing methods primarily rely on single-modal road condition video data, which limits their ability to capture rare and unpredictable driving incidents. This paper proposes a multimodal driver assistance detection system that integrates road condition video, driver facial video, and audio data to enhance incident recognition accuracy. Our model employs an attention-based intermediate fusion strategy, enabling end-to-end learning without separate feature extraction. To support this approach, we develop a new three-modality dataset using a driving simulator. Experimental results demonstrate that our method effectively captures cross-modal correlations, reducing misjudgments and improving driving safety.

Title: Scaling Laws for Upcycling Mixture-of-Experts Language Models

• Authors: Seng Pei Liew, Takuya Kato, Sho Takase
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03009
• Pdf URL: https://arxiv.org/pdf/2502.03009
• Copy Paste: [[2502.03009]] Scaling Laws for Upcycling Mixture-of-Experts Language Models(https://arxiv.org/abs/2502.03009)
• Keywords: large language model
• Abstract: Pretraining large language models (LLMs) is resource-intensive, often requiring months of training time even with high-end GPU clusters. There are two approaches of mitigating such computational demands: reusing smaller models to train larger ones (upcycling), and training computationally efficient models like mixture-of-experts (MoE). In this paper, we study the upcycling of LLMs to MoE models, of which the scaling behavior remains underexplored. Through extensive experiments, we identify empirical scaling laws that describe how performance depends on dataset size and model configuration. Particularly, we show that, while scaling these factors improves performance, there is a novel interaction term between the dense and upcycled training dataset that limits the efficiency of upcycling at large computational budgets. Based on these findings, we provide guidance to scale upcycling, and establish conditions under which upcycling outperforms from-scratch trainings within budget constraints.

Title: xai_evals : A Framework for Evaluating Post-Hoc Local Explanation Methods

• Authors: Pratinav Seth, Yashwardhan Rathore, Neeraj Kumar Singh, Chintan Chitroda, Vinay Kumar Sankarapu
• Subjects: cs.LG, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2502.03014
• Pdf URL: https://arxiv.org/pdf/2502.03014
• Copy Paste: [[2502.03014]] xai_evals : A Framework for Evaluating Post-Hoc Local Explanation Methods(https://arxiv.org/abs/2502.03014)
• Keywords: robust, interpretability
• Abstract: The growing complexity of machine learning and deep learning models has led to an increased reliance on opaque "black box" systems, making it difficult to understand the rationale behind predictions. This lack of transparency is particularly challenging in high-stakes applications where interpretability is as important as accuracy. Post-hoc explanation methods are commonly used to interpret these models, but they are seldom rigorously evaluated, raising concerns about their reliability. The Python package xai_evals addresses this by providing a comprehensive framework for generating, benchmarking, and evaluating explanation methods across both tabular and image data modalities. It integrates popular techniques like SHAP, LIME, Grad-CAM, Integrated Gradients (IG), and Backtrace, while supporting evaluation metrics such as faithfulness, sensitivity, and robustness. xai_evals enhances the interpretability of machine learning models, fostering transparency and trust in AI systems. The library is open-sourced at this https URL .

Title: On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation

• Authors: Nghiem T. Diep, Huy Nguyen, Chau Nguyen, Minh Le, Duy M. H. Nguyen, Daniel Sonntag, Mathias Niepert, Nhat Ho
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03029
• Pdf URL: https://arxiv.org/pdf/2502.03029
• Copy Paste: [[2502.03029]] On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation(https://arxiv.org/abs/2502.03029)
• Keywords: robust
• Abstract: The LLaMA-Adapter has recently emerged as an efficient fine-tuning technique for LLaMA models, leveraging zero-initialized attention to stabilize training and enhance performance. However, despite its empirical success, the theoretical foundations of zero-initialized attention remain largely unexplored. In this paper, we provide a rigorous theoretical analysis, establishing a connection between zero-initialized attention and mixture-of-expert models. We prove that both linear and non-linear prompts, along with gating functions, can be optimally estimated, with non-linear prompts offering greater flexibility for future applications. Empirically, we validate our findings on the open LLM benchmarks, demonstrating that non-linear prompts outperform linear ones. Notably, even with limited training data, both prompt types consistently surpass vanilla attention, highlighting the robustness and adaptability of zero-initialized attention.

Title: Analyze Feature Flow to Enhance Interpretation and Steering in Language Models

• Authors: Daniil Laptev, Nikita Balagansky, Yaroslav Aksenov, Daniil Gavrilov
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03032
• Pdf URL: https://arxiv.org/pdf/2502.03032
• Copy Paste: [[2502.03032]] Analyze Feature Flow to Enhance Interpretation and Steering in Language Models(https://arxiv.org/abs/2502.03032)
• Keywords: interpretability, data-free, large language model
• Abstract: We introduce a new approach to systematically map features discovered by sparse autoencoder across consecutive layers of large language models, extending earlier work that examined inter-layer feature links. By using a data-free cosine similarity technique, we trace how specific features persist, transform, or first appear at each stage. This method yields granular flow graphs of feature evolution, enabling fine-grained interpretability and mechanistic insights into model computations. Crucially, we demonstrate how these cross-layer feature maps facilitate direct steering of model behavior by amplifying or suppressing chosen features, achieving targeted thematic control in text generation. Together, our findings highlight the utility of a causal, cross-layer interpretability framework that not only clarifies how features develop through forward passes but also provides new means for transparent manipulation of large language models.

Title: Aggregate to Adapt: Node-Centric Aggregation for Multi-Source-Free Graph Domain Adaptation

• Authors: Zhen Zhang, Bingsheng He
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03033
• Pdf URL: https://arxiv.org/pdf/2502.03033
• Copy Paste: [[2502.03033]] Aggregate to Adapt: Node-Centric Aggregation for Multi-Source-Free Graph Domain Adaptation(https://arxiv.org/abs/2502.03033)
• Keywords: privacy
• Abstract: Unsupervised graph domain adaptation (UGDA) focuses on transferring knowledge from labeled source graph to unlabeled target graph under domain discrepancies. Most existing UGDA methods are designed to adapt information from a single source domain, which cannot effectively exploit the complementary knowledge from multiple source domains. Furthermore, their assumptions that the labeled source graphs are accessible throughout the training procedure might not be practical due to privacy, regulation, and storage concerns. In this paper, we investigate multi-source-free unsupervised graph domain adaptation, i.e., adapting knowledge from multiple source domains to an unlabeled target domain without utilizing labeled source graphs but relying solely on source pre-trained models. Unlike previous multi-source domain adaptation approaches that aggregate predictions at model level, we introduce a novel model named GraphATA which conducts adaptation at node granularity. Specifically, we parameterize each node with its own graph convolutional matrix by automatically aggregating weight matrices from multiple source models according to its local context, thus realizing dynamic adaptation over graph structured data. We also demonstrate the capability of GraphATA to generalize to both model-centric and layer-centric methods. Comprehensive experiments on various public datasets show that our GraphATA can consistently surpass recent state-of-the-art baselines with different gains.

Title: Knowledge Distillation from Large Language Models for Household Energy Modeling

• Authors: Mohannad Takrouri, Nicolás M. Cuadrado, Martin Takáč
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03034
• Pdf URL: https://arxiv.org/pdf/2502.03034
• Copy Paste: [[2502.03034]] Knowledge Distillation from Large Language Models for Household Energy Modeling(https://arxiv.org/abs/2502.03034)
• Keywords: privacy, large language model
• Abstract: Machine learning (ML) is increasingly vital for smart-grid research, yet restricted access to realistic, diverse data - often due to privacy concerns - slows progress and fuels doubts within the energy sector about adopting ML-based strategies. We propose integrating Large Language Models (LLMs) in energy modeling to generate realistic, culturally sensitive, and behavior-specific data for household energy usage across diverse geographies. In this study, we employ and compare five different LLMs to systematically produce family structures, weather patterns, and daily consumption profiles for households in six distinct countries. A four-stage methodology synthesizes contextual daily data, including culturally nuanced activities, realistic weather ranges, HVAC operations, and distinct `energy signatures' that capture unique consumption footprints. Additionally, we explore an alternative strategy where external weather datasets can be directly integrated, bypassing intermediate weather modeling stages while ensuring physically consistent data inputs. The resulting dataset provides insights into how cultural, climatic, and behavioral factors converge to shape carbon emissions, offering a cost-effective avenue for scenario-based energy optimization. This approach underscores how prompt engineering, combined with knowledge distillation, can advance sustainable energy research and climate mitigation efforts. Source code is available at this https URL .

Title: RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts

• Authors: Tuan Truong, Chau Nguyen, Huy Nguyen, Minh Le, Trung Le, Nhat Ho
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03044
• Pdf URL: https://arxiv.org/pdf/2502.03044
• Copy Paste: [[2502.03044]] RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts(https://arxiv.org/abs/2502.03044)
• Keywords: robust
• Abstract: Low-rank adaptation (LoRA) has emerged as a powerful method for fine-tuning large-scale foundation models. Despite its popularity, the theoretical understanding of LoRA has remained limited. This paper presents a theoretical analysis of LoRA by examining its connection to the Mixture of Experts models. Under this framework, we show that simple reparameterizations of the LoRA matrices can notably accelerate the low-rank matrix estimation process. In particular, we prove that reparameterization can reduce the data needed to achieve a desired estimation error from an exponential to a polynomial scale. Motivated by this insight, we propose Reparameterized Low-rank Adaptation (RepLoRA), which incorporates lightweight MLPs to reparameterize the LoRA matrices. Extensive experiments across multiple domains demonstrate that RepLoRA consistently outperforms vanilla LoRA. Notably, with limited data, RepLoRA surpasses LoRA by a margin of up to 40.0% and achieves LoRA's performance with only 30.0% of the training data, highlighting both the theoretical and empirical robustness of our PEFT method.

Title: Understanding and Enhancing the Transferability of Jailbreaking Attacks

• Authors: Runqi Lin, Bo Han, Fengwang Li, Tongling Liu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2502.03052
• Pdf URL: https://arxiv.org/pdf/2502.03052
• Copy Paste: [[2502.03052]] Understanding and Enhancing the Transferability of Jailbreaking Attacks(https://arxiv.org/abs/2502.03052)
• Keywords: attack, large language model
• Abstract: Jailbreaking attacks can effectively manipulate open-source large language models (LLMs) to produce harmful responses. However, these attacks exhibit limited transferability, failing to disrupt proprietary LLMs consistently. To reliably identify vulnerabilities in proprietary LLMs, this work investigates the transferability of jailbreaking attacks by analysing their impact on the model's intent perception. By incorporating adversarial sequences, these attacks can redirect the source LLM's focus away from malicious-intent tokens in the original input, thereby obstructing the model's intent recognition and eliciting harmful responses. Nevertheless, these adversarial sequences fail to mislead the target LLM's intent perception, allowing the target LLM to refocus on malicious-intent tokens and abstain from responding. Our analysis further reveals the inherent distributional dependency within the generated adversarial sequences, whose effectiveness stems from overfitting the source LLM's parameters, resulting in limited transferability to target LLMs. To this end, we propose the Perceived-importance Flatten (PiF) method, which uniformly disperses the model's focus across neutral-intent tokens in the original input, thus obscuring malicious-intent tokens without relying on overfitted adversarial sequences. Extensive experiments demonstrate that PiF provides an effective and efficient red-teaming evaluation for proprietary LLMs.

Title: IAO Prompting: Making Knowledge Flow Explicit in LLMs through Structured Reasoning Templates

• Authors: Aissatou Diallo, Antonis Bikakis, Luke Dickens, Anthony Hunter, Rob Miller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03080
• Pdf URL: https://arxiv.org/pdf/2502.03080
• Copy Paste: [[2502.03080]] IAO Prompting: Making Knowledge Flow Explicit in LLMs through Structured Reasoning Templates(https://arxiv.org/abs/2502.03080)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) demonstrate impressive reasoning capabilities, understanding and validating their knowledge utilization remains challenging. Chain-of-thought (CoT) prompting partially addresses this by revealing intermediate reasoning steps, but the knowledge flow and application remain implicit. We introduce IAO (Input-Action-Output) prompting, a structured template-based method that explicitly models how LLMs access and apply their knowledge during complex reasoning tasks. IAO decomposes problems into sequential steps, each clearly identifying the input knowledge being used, the action being performed, and the resulting output. This structured decomposition enables us to trace knowledge flow, verify factual consistency, and identify potential knowledge gaps or misapplications. Through experiments across diverse reasoning tasks, we demonstrate that IAO not only improves zero-shot performance but also provides transparency in how LLMs leverage their stored knowledge. Human evaluation confirms that this structured approach enhances our ability to verify knowledge utilization and detect potential hallucinations or reasoning errors. Our findings provide insights into both knowledge representation within LLMs and methods for more reliable knowledge application.

Title: E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing

• Authors: Yuhao Zhou, Yuxin Tian, Mingjia Shi, Yuanxi Li, Yanan Sun, Qing Ye, Jiancheng Lv
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2502.03092
• Pdf URL: https://arxiv.org/pdf/2502.03092
• Copy Paste: [[2502.03092]] E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing(https://arxiv.org/abs/2502.03092)
• Keywords: federate
• Abstract: The exponential growth in model sizes has significantly increased the communication burden in Federated Learning (FL). Existing methods to alleviate this burden by transmitting compressed gradients often face high compression errors, which slow down the model's convergence. To simultaneously achieve high compression effectiveness and lower compression errors, we study the gradient compression problem from a novel perspective. Specifically, we propose a systematical algorithm termed Extended Single-Step Synthetic Features Compressing (E-3SFC), which consists of three sub-components, i.e., the Single-Step Synthetic Features Compressor (3SFC), a double-way compression algorithm, and a communication budget scheduler. First, we regard the process of gradient computation of a model as decompressing gradients from corresponding inputs, while the inverse process is considered as compressing the gradients. Based on this, we introduce a novel gradient compression method termed 3SFC, which utilizes the model itself as a decompressor, leveraging training priors such as model weights and objective functions. 3SFC compresses raw gradients into tiny synthetic features in a single-step simulation, incorporating error feedback to minimize overall compression errors. To further reduce communication overhead, 3SFC is extended to E-3SFC, allowing double-way compression and dynamic communication budget scheduling. Our theoretical analysis under both strongly convex and non-convex conditions demonstrates that 3SFC achieves linear and sub-linear convergence rates with aggregation noise. Extensive experiments across six datasets and six models reveal that 3SFC outperforms state-of-the-art methods by up to 13.4% while reducing communication costs by 111.6 times. These findings suggest that 3SFC can significantly enhance communication efficiency in FL without compromising model performance.

Title: Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms

• Authors: Xuerui Su, Yue Wang, Jinhua Zhu, Mingyang Yi, Feng Xu, Zhiming Ma, Yuting Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03095
• Pdf URL: https://arxiv.org/pdf/2502.03095
• Copy Paste: [[2502.03095]] Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms(https://arxiv.org/abs/2502.03095)
• Keywords: large language model
• Abstract: With the rapid development of Large Language Models (LLMs), numerous Reinforcement Learning from Human Feedback (RLHF) algorithms have been introduced to improve model safety and alignment with human preferences. These algorithms can be divided into two main frameworks based on whether they require an explicit reward (or value) function for training: actor-critic-based Proximal Policy Optimization (PPO) and alignment-based Direct Preference Optimization (DPO). The mismatch between DPO and PPO, such as DPO's use of a classification loss driven by human-preferred data, has raised confusion about whether DPO should be classified as a Reinforcement Learning (RL) algorithm. To address these ambiguities, we focus on three key aspects related to DPO, RL, and other RLHF algorithms: (1) the construction of the loss function; (2) the target distribution at which the algorithm converges; (3) the impact of key components within the loss function. Specifically, we first establish a unified framework named UDRRA connecting these algorithms based on the construction of their loss functions. Next, we uncover their target policy distributions within this framework. Finally, we investigate the critical components of DPO to understand their impact on the convergence rate. Our work provides a deeper understanding of the relationship between DPO, RL, and other RLHF algorithms, offering new insights for improving existing algorithms.

Title: Structured Token Retention and Computational Memory Paths in Large Language Models

• Authors: Jonathan Delena, Augustin Moreau, Dominic Ravensdale, Frederick Chatterton
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03102
• Pdf URL: https://arxiv.org/pdf/2502.03102
• Copy Paste: [[2502.03102]] Structured Token Retention and Computational Memory Paths in Large Language Models(https://arxiv.org/abs/2502.03102)
• Keywords: generative, large language model
• Abstract: Memory retention mechanisms play a central role in determining the efficiency of computational architectures designed for processing extended sequences. Conventional methods for token management often impose fixed retention thresholds or rely on uniform attention weight distributions, leading to inefficient memory utilization and premature information loss in extended sequence modeling. Structured Token Retention (STR) introduces a probabilistic selection framework that dynamically adjusts token persistence based on contextual significance, ensuring that computational resources are allocated to semantically relevant elements. Computational Memory Paths (CMP) extend this framework through hierarchical memory allocation, refining retention efficiency through structured reallocation of token embeddings. Comparative assessments against baseline models demonstrate that STR and CMP improve token survival rates across long input sequences while reducing cumulative error propagation across processing layers. Experimental results further indicate reductions in computational overhead, improving inference speed without degrading contextual coherence. Token distribution analyses reveal that structured memory allocation prevents excessive redundancy in attention weight calculations, optimizing information retrieval efficiency in large-scale generative architectures. The integration of STR and CMP into an open-source model illustrates the adaptability of structured memory retention methodologies, highlighting their applicability in generative text processing, long-context comprehension, and scalable sequence modeling.

Title: Edge Attention Module for Object Classification

• Authors: Santanu Roy, Ashvath Suresh, Archit Gupta
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03103
• Pdf URL: https://arxiv.org/pdf/2502.03103
• Copy Paste: [[2502.03103]] Edge Attention Module for Object Classification(https://arxiv.org/abs/2502.03103)
• Keywords: transformer
• Abstract: A novel ``edge attention-based Convolutional Neural Network (CNN)'' is proposed in this research for object classification task. With the advent of advanced computing technology, CNN models have achieved to remarkable success, particularly in computer vision applications. Nevertheless, the efficacy of the conventional CNN is often hindered due to class imbalance and inter-class similarity problems, which are particularly prominent in the computer vision field. In this research, we introduce for the first time an ``Edge Attention Module (EAM)'' consisting of a Max-Min pooling layer, followed by convolutional layers. This Max-Min pooling is entirely a novel pooling technique, specifically designed to capture only the edge information that is crucial for any object classification task. Therefore, by integrating this novel pooling technique into the attention module, the CNN network inherently prioritizes on essential edge features, thereby boosting the accuracy and F1-score of the model significantly. We have implemented our proposed EAM or 2EAMs on several standard pre-trained CNN models for Caltech-101, Caltech-256, CIFAR-100 and Tiny ImageNet-200 datasets. The extensive experiments reveal that our proposed framework (that is, EAM with CNN and 2EAMs with CNN), outperforms all pre-trained CNN models as well as recent trend models ``Pooling-based Vision Transformer (PiT)'', ``Convolutional Block Attention Module (CBAM)'', and ConvNext, by substantial margins. We have achieved the accuracy of 95.5% and 86% by the proposed framework on Caltech-101 and Caltech-256 datasets, respectively. So far, this is the best results on these datasets, to the best of our knowledge.

Title: Multi-objective methods in Federated Learning: A survey and taxonomy

• Authors: Maria Hartmann, Grégoire Danoy, Pascal Bouvry
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2502.03108
• Pdf URL: https://arxiv.org/pdf/2502.03108
• Copy Paste: [[2502.03108]] Multi-objective methods in Federated Learning: A survey and taxonomy(https://arxiv.org/abs/2502.03108)
• Keywords: federate, fair
• Abstract: The Federated Learning paradigm facilitates effective distributed machine learning in settings where training data is decentralized across multiple clients. As the popularity of the strategy grows, increasingly complex real-world problems emerge, many of which require balancing conflicting demands such as fairness, utility, and resource consumption. Recent works have begun to recognise the use of a multi-objective perspective in answer to this challenge. However, this novel approach of combining federated methods with multi-objective optimisation has never been discussed in the broader context of both fields. In this work, we offer a first clear and systematic overview of the different ways the two fields can be integrated. We propose a first taxonomy on the use of multi-objective methods in connection with Federated Learning, providing a targeted survey of the state-of-the-art and proposing unambiguous labels to categorise contributions. Given the developing nature of this field, our taxonomy is designed to provide a solid basis for further research, capturing existing works while anticipating future additions. Finally, we outline open challenges and possible directions for further research.

Title: Teaching Large Language Models Number-Focused Headline Generation With Key Element Rationales

• Authors: Zhen Qian, Xiuzhen Zhang, Xiaofei Xu, Feng Xia
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03129
• Pdf URL: https://arxiv.org/pdf/2502.03129
• Copy Paste: [[2502.03129]] Teaching Large Language Models Number-Focused Headline Generation With Key Element Rationales(https://arxiv.org/abs/2502.03129)
• Keywords: large language model
• Abstract: Number-focused headline generation is a summarization task requiring both high textual quality and precise numerical accuracy, which poses a unique challenge for Large Language Models (LLMs). Existing studies in the literature focus only on either textual quality or numerical reasoning and thus are inadequate to address this challenge. In this paper, we propose a novel chain-of-thought framework for using rationales comprising key elements of the Topic, Entities, and Numerical reasoning (TEN) in news articles to enhance the capability for LLMs to generate topic-aligned high-quality texts with precise numerical accuracy. Specifically, a teacher LLM is employed to generate TEN rationales as supervision data, which are then used to teach and fine-tune a student LLM. Our approach teaches the student LLM automatic generation of rationales with enhanced capability for numerical reasoning and topic-aligned numerical headline generation. Experiments show that our approach achieves superior performance in both textual quality and numerical accuracy.

Title: Gotham Dataset 2025: A Reproducible Large-Scale IoT Network Dataset for Intrusion Detection and Security Research

• Authors: Othmane Belarbi, Theodoros Spyridopoulos, Eirini Anthi, Omer Rana, Pietro Carnelli, Aftab Khan
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03134
• Pdf URL: https://arxiv.org/pdf/2502.03134
• Copy Paste: [[2502.03134]] Gotham Dataset 2025: A Reproducible Large-Scale IoT Network Dataset for Intrusion Detection and Security Research(https://arxiv.org/abs/2502.03134)
• Keywords: security, attack, extraction
• Abstract: In this paper, a dataset of IoT network traffic is presented. Our dataset was generated by utilising the Gotham testbed, an emulated large-scale Internet of Things (IoT) network designed to provide a realistic and heterogeneous environment for network security research. The testbed includes 78 emulated IoT devices operating on various protocols, including MQTT, CoAP, and RTSP. Network traffic was captured in Packet Capture (PCAP) format using tcpdump, and both benign and malicious traffic were recorded. Malicious traffic was generated through scripted attacks, covering a variety of attack types, such as Denial of Service (DoS), Telnet Brute Force, Network Scanning, CoAP Amplification, and various stages of Command and Control (C&C) communication. The data were subsequently processed in Python for feature extraction using the Tshark tool, and the resulting data was converted to Comma Separated Values (CSV) format and labelled. The data repository includes the raw network traffic in PCAP format and the processed labelled data in CSV format. Our dataset was collected in a distributed manner, where network traffic was captured separately for each IoT device at the interface between the IoT gateway and the device. Our dataset was collected in a distributed manner, where network traffic was separately captured for each IoT device at the interface between the IoT gateway and the device. With its diverse traffic patterns and attack scenarios, this dataset provides a valuable resource for developing Intrusion Detection Systems and security mechanisms tailored to complex, large-scale IoT environments. The dataset is publicly available at Zenodo.

Title: Machine Learning-Driven Student Performance Prediction for Enhancing Tiered Instruction

• Authors: Yawen Chen, Jiande Sun, Jinhui Wang, Liang Zhao, Xinmin Song, Linbo Zhai
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.03143
• Pdf URL: https://arxiv.org/pdf/2502.03143
• Copy Paste: [[2502.03143]] Machine Learning-Driven Student Performance Prediction for Enhancing Tiered Instruction(https://arxiv.org/abs/2502.03143)
• Keywords: extraction
• Abstract: Student performance prediction is one of the most important subjects in educational data mining. As a modern technology, machine learning offers powerful capabilities in feature extraction and data modeling, providing essential support for diverse application scenarios, as evidenced by recent studies confirming its effectiveness in educational data mining. However, despite extensive prediction experiments, machine learning methods have not been effectively integrated into practical teaching strategies, hindering their application in modern education. In addition, massive features as input variables for machine learning algorithms often leads to information redundancy, which can negatively impact prediction accuracy. Therefore, how to effectively use machine learning methods to predict student performance and integrate the prediction results with actual teaching scenarios is a worthy research subject. To this end, this study integrates the results of machine learning-based student performance prediction with tiered instruction, aiming to enhance student outcomes in target course, which is significant for the application of educational data mining in contemporary teaching scenarios. Specifically, we collect original educational data and perform feature selection to reduce information redundancy. Then, the performance of five representative machine learning methods is analyzed and discussed with Random Forest showing the best performance. Furthermore, based on the results of the classification of students, tiered instruction is applied accordingly, and different teaching objectives and contents are set for all levels of students. The comparison of teaching outcomes between the control and experimental classes, along with the analysis of questionnaire results, demonstrates the effectiveness of the proposed framework.

Title: Symmetry-Aware Bayesian Flow Networks for Crystal Generation

• Authors: Laura Ruple, Luca Torresi, Henrik Schopmans, Pascal Friederich
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2502.03146
• Pdf URL: https://arxiv.org/pdf/2502.03146
• Copy Paste: [[2502.03146]] Symmetry-Aware Bayesian Flow Networks for Crystal Generation(https://arxiv.org/abs/2502.03146)
• Keywords: generative
• Abstract: The discovery of new crystalline materials is essential to scientific and technological progress. However, traditional trial-and-error approaches are inefficient due to the vast search space. Recent advancements in machine learning have enabled generative models to predict new stable materials by incorporating structural symmetries and to condition the generation on desired properties. In this work, we introduce SymmBFN, a novel symmetry-aware Bayesian Flow Network (BFN) for crystalline material generation that accurately reproduces the distribution of space groups found in experimentally observed crystals. SymmBFN substantially improves efficiency, generating stable structures at least 50 times faster than the next-best method. Furthermore, we demonstrate its capability for property-conditioned generation, enabling the design of materials with tailored properties. Our findings establish BFNs as an effective tool for accelerating the discovery of crystalline materials.

Title: Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models

• Authors: Xumeng Wen, Shun Zheng, Zhen Xu, Yiming Sun, Jiang Bian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03147
• Pdf URL: https://arxiv.org/pdf/2502.03147
• Copy Paste: [[2502.03147]] Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models(https://arxiv.org/abs/2502.03147)
• Keywords: large language model
• Abstract: Recent studies have shown that large language models (LLMs), when customized with post-training on tabular data, can acquire general tabular in-context learning (TabICL) capabilities. These models are able to transfer effectively across diverse data schemas and different task domains. However, existing LLM-based TabICL approaches are constrained to few-shot scenarios due to the sequence length limitations of LLMs, as tabular instances represented in plain text consume substantial tokens. To address this limitation and enable scalable TabICL for any data size, we propose retrieval-augmented LLMs tailored to tabular data. Our approach incorporates a customized retrieval module, combined with retrieval-guided instruction-tuning for LLMs. This enables LLMs to effectively leverage larger datasets, achieving significantly improved performance across 69 widely recognized datasets and demonstrating promising scaling behavior. Extensive comparisons with state-of-the-art tabular models reveal that, while LLM-based TabICL still lags behind well-tuned numeric models in overall performance, it uncovers powerful algorithms under limited contexts, enhances ensemble diversity, and excels on specific datasets. These unique properties underscore the potential of language as a universal and accessible interface for scalable tabular data learning.

Title: Secure Resource Management in Cloud Computing: Challenges, Strategies and Meta-Analysis

• Authors: Deepika Saxena, Smruti Rekha Swain, Jatinder Kumar, Sakshi Patni, Kishu Gupta, Ashutosh Kumar Singh, Volker Lindenstruth
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2502.03149
• Pdf URL: https://arxiv.org/pdf/2502.03149
• Copy Paste: [[2502.03149]] Secure Resource Management in Cloud Computing: Challenges, Strategies and Meta-Analysis(https://arxiv.org/abs/2502.03149)
• Keywords: secure, security
• Abstract: Secure resource management (SRM) within a cloud computing environment is a critical yet infrequently studied research topic. This paper provides a comprehensive survey and comparative performance evaluation of potential cyber threat countermeasure strategies that address security challenges during cloud workload execution and resource management. Cybersecurity is explored specifically in the context of cloud resource management, with an emphasis on identifying the associated challenges. The cyber threat countermeasure methods are categorized into three classes: defensive strategies, mitigating strategies, and hybrid strategies. The existing countermeasure strategies belonging to each class are thoroughly discussed and compared. In addition to conceptual and theoretical analysis, the leading countermeasure strategies within these categories are implemented on a common platform and examined using two real-world virtual machine (VM) data traces. Based on this comprehensive study and performance evaluation, the paper discusses the trade-offs among these countermeasure strategies and their utility, providing imperative concluding remarks on the holistic study of cloud cyber threat countermeasures and secure resource management. Furthermore, the study suggests future methodologies that could effectively address the emerging challenges of secure cloud resource management.

Title: PICBench: Benchmarking LLMs for Photonic Integrated Circuits Design

• Authors: Yuchao Wu, Xiaofei Yu, Hao Chen, Yang Luo, Yeyu Tong, Yuzhe Ma
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03159
• Pdf URL: https://arxiv.org/pdf/2502.03159
• Copy Paste: [[2502.03159]] PICBench: Benchmarking LLMs for Photonic Integrated Circuits Design(https://arxiv.org/abs/2502.03159)
• Keywords: large language model
• Abstract: While large language models (LLMs) have shown remarkable potential in automating various tasks in digital chip design, the field of Photonic Integrated Circuits (PICs)-a promising solution to advanced chip designs-remains relatively unexplored in this context. The design of PICs is time-consuming and prone to errors due to the extensive and repetitive nature of code involved in photonic chip design. In this paper, we introduce PICBench, the first benchmarking and evaluation framework specifically designed to automate PIC design generation using LLMs, where the generated output takes the form of a netlist. Our benchmark consists of dozens of meticulously crafted PIC design problems, spanning from fundamental device designs to more complex circuit-level designs. It automatically evaluates both the syntax and functionality of generated PIC designs by comparing simulation outputs with expert-written solutions, leveraging an open-source simulator. We evaluate a range of existing LLMs, while also conducting comparative tests on various prompt engineering techniques to enhance LLM performance in automated PIC design. The results reveal the challenges and potential of LLMs in the PIC design domain, offering insights into the key areas that require further research and development to optimize automation in this field. Our benchmark and evaluation code is available at this https URL.

Title: LED there be DoS: Exploiting variable bitrate IP cameras for network DoS

• Authors: Emmanuel Goldberg, Oleg Brodt, Aviad Elyashar, Rami Puzis
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2502.03177
• Pdf URL: https://arxiv.org/pdf/2502.03177
• Copy Paste: [[2502.03177]] LED there be DoS: Exploiting variable bitrate IP cameras for network DoS(https://arxiv.org/abs/2502.03177)
• Keywords: attack
• Abstract: Variable-bitrate video streaming is ubiquitous in video surveillance and CCTV, enabling high-quality video streaming while conserving network bandwidth. However, as the name suggests, variable-bitrate IP cameras can generate sharp traffic spikes depending on the dynamics of the visual input. In this paper, we show that the effectiveness of video compression can be reduced by up to 6X using a simple laser LED pointing at a variable-bitrate IP camera, forcing the camera to generate excessive network traffic. Experiments with IP cameras connected to wired and wireless networks indicate that a laser attack on a single camera can cause significant packet loss in systems sharing the network with the camera and reduce the available bandwidth of a shared network link by 90%. This attack represents a new class of cyber-physical attacks that manipulate variable bitrate devices through changes in the physical environment without a digital presence on the device or the network. We also analyze the broader view of multidimensional cyberattacks that involve both the physical and digital realms and present a taxonomy that categorizes attacks based on their direction of influence (physical-to-digital or digital-to-physical) and their method of operation (environment-driven or device-driven), highlighting multiple areas for future research.

Title: MaxInfo: A Training-Free Key-Frame Selection Method Using Maximum Volume for Enhanced Video Understanding

• Authors: Pengyi Li, Irina Abdullaeva, Alexander Gambashidze, Andrey Kuznetsov, Ivan Oseledets
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03183
• Pdf URL: https://arxiv.org/pdf/2502.03183
• Copy Paste: [[2502.03183]] MaxInfo: A Training-Free Key-Frame Selection Method Using Maximum Volume for Enhanced Video Understanding(https://arxiv.org/abs/2502.03183)
• Keywords: large language model
• Abstract: Modern Video Large Language Models (VLLMs) often rely on uniform frame sampling for video understanding, but this approach frequently fails to capture critical information due to frame redundancy and variations in video content. We propose MaxInfo, a training-free method based on the maximum volume principle, which selects and retains the most representative frames from the input video. By maximizing the geometric volume formed by selected embeddings, MaxInfo ensures that the chosen frames cover the most informative regions of the embedding space, effectively reducing redundancy while preserving diversity. This method enhances the quality of input representations and improves long video comprehension performance across benchmarks. For instance, MaxInfo achieves a 3.28% improvement on LongVideoBench and a 6.4% improvement on EgoSchema for LLaVA-Video-7B. It also achieves a 3.47% improvement for LLaVA-Video-72B. The approach is simple to implement and works with existing VLLMs without the need for additional training, making it a practical and effective alternative to traditional uniform sampling methods.

Title: Improve Decoding Factuality by Token-wise Cross Layer Entropy of Large Language Models

• Authors: Jialiang Wu, Yi Shen, Sijia Liu, Yi Tang, Sen Song, Xiaoyi Wang, Longjun Cai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03199
• Pdf URL: https://arxiv.org/pdf/2502.03199
• Copy Paste: [[2502.03199]] Improve Decoding Factuality by Token-wise Cross Layer Entropy of Large Language Models(https://arxiv.org/abs/2502.03199)
• Keywords: robust, large language model
• Abstract: Despite their impressive capacities, Large language models (LLMs) often struggle with the hallucination issue of generating inaccurate or fabricated content even when they possess correct knowledge. In this paper, we extend the exploration of the correlation between hidden-state prediction changes and output factuality into a deeper, token-wise level. Based on the insights , we propose cross-layer Entropy eNhanced Decoding (END), a decoding method that mitigates hallucinations without requiring extra training. END leverages inner probability changes across layers to individually quantify the factual knowledge required for each candidate token, and adjusts the final predicting distribution to prioritize tokens with higher factuality. Experiments on both hallucination and QA benchmarks demonstrate that END significantly enhances the truthfulness and informativeness of generated content while maintaining robust QA accuracy. Moreover, our work provides a deeper perspective on understanding the correlations between inherent knowledge and output factuality.

Title: FSLH: Flexible Mechanized Speculative Load Hardening

• Authors: Roberto Blanco, Léon Ducruet, Sebastian Harwig, Catalin Hritcu
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2502.03203
• Pdf URL: https://arxiv.org/pdf/2502.03203
• Copy Paste: [[2502.03203]] FSLH: Flexible Mechanized Speculative Load Hardening(https://arxiv.org/abs/2502.03203)
• Keywords: security, protect, attack
• Abstract: The Spectre speculative side-channel attacks pose formidable threats for computer system security. Research has shown that cryptographic constant-time code can be efficiently protected against Spectre v1 using a selective variant of Speculative Load Hardening (SLH). SLH was, however, not strong enough for protecting non-cryptographic code, leading to the introduction of Ultimate SLH, which provides protection for arbitrary programs, but has too large overhead for general use, since it conservatively assumes that all data is secret. In this paper we introduce a flexible SLH notion that achieves the best of both worlds by formally generalizing both Selective and Ultimate SLH. We give a suitable security definition for such transformations protecting arbitrary programs: any transformed program running with speculation should not leak more than what the source program leaks sequentially. We formally prove using the Rocq prover that two flexible SLH variants enforce this relative security guarantee. As easy corollaries we also obtain that Ultimate SLH enforces our relative security notion, and also that the selective variants of value SLH and address SLH enforce speculative constant-time security.

Title: MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agent

• Authors: Xinyao Liao, Xianfang Zeng, Liao Wang, Gang Yu, Guosheng Lin, Chi Zhang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2502.03207
• Pdf URL: https://arxiv.org/pdf/2502.03207
• Copy Paste: [[2502.03207]] MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agent(https://arxiv.org/abs/2502.03207)
• Keywords: diffusion
• Abstract: We propose MotionAgent, enabling fine-grained motion control for text-guided image-to-video generation. The key technique is the motion field agent that converts motion information in text prompts into explicit motion fields, providing flexible and precise motion guidance. Specifically, the agent extracts the object movement and camera motion described in the text and converts them into object trajectories and camera extrinsics, respectively. An analytical optical flow composition module integrates these motion representations in 3D space and projects them into a unified optical flow. An optical flow adapter takes the flow to control the base image-to-video diffusion model for generating fine-grained controlled videos. The significant improvement in the Video-Text Camera Motion metrics on VBench indicates that our method achieves precise control over camera motion. We construct a subset of VBench to evaluate the alignment of motion information in the text and the generated video, outperforming other advanced models on motion generation accuracy.

Title: A Unified Framework for Semi-Supervised Image Segmentation and Registration

• Authors: Ruizhe Li, Grazziela Figueredo, Dorothee Auer, Rob Dineen, Paul Morgan, Xin Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03229
• Pdf URL: https://arxiv.org/pdf/2502.03229
• Copy Paste: [[2502.03229]] A Unified Framework for Semi-Supervised Image Segmentation and Registration(https://arxiv.org/abs/2502.03229)
• Keywords: segmentation
• Abstract: Semi-supervised learning, which leverages both annotated and unannotated data, is an efficient approach for medical image segmentation, where obtaining annotations for the whole dataset is time-consuming and costly. Traditional semi-supervised methods primarily focus on extracting features and learning data distributions from unannotated data to enhance model training. In this paper, we introduce a novel approach incorporating an image registration model to generate pseudo-labels for the unannotated data, producing more geometrically correct pseudo-labels to improve the model training. Our method was evaluated on a 2D brain data set, showing excellent performance even using only 1\% of the annotated data. The results show that our approach outperforms conventional semi-supervised segmentation methods (e.g. teacher-student model), particularly in a low percentage of annotation scenario. GitHub: this https URL.

Title: The Other Side of the Coin: Unveiling the Downsides of Model Aggregation in Federated Learning from a Layer-peeled Perspective

• Authors: Guogang Zhu, Xuefeng Liu, Jianwei Niu, Shaojie Tang, Xinghao Wu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03231
• Pdf URL: https://arxiv.org/pdf/2502.03231
• Copy Paste: [[2502.03231]] The Other Side of the Coin: Unveiling the Downsides of Model Aggregation in Federated Learning from a Layer-peeled Perspective(https://arxiv.org/abs/2502.03231)
• Keywords: federate
• Abstract: In federated learning (FL), model aggregation is a critical step by which multiple clients share their knowledge with one another. However, it is also widely recognized that the aggregated model, when sent back to each client, performs poorly on local data until after several rounds of local training. This temporary performance drop can potentially slow down the convergence of the FL model. Most research in FL regards this performance drop as an inherent cost of knowledge sharing among clients and does not give it special attention. While some studies directly focus on designing techniques to alleviate the issue, an in-depth investigation of the reasons behind this performance drop has yet to be this http URL address this gap, we conduct a layer-peeled analysis of model aggregation across various datasets and model architectures. Our findings reveal that the performance drop can be attributed to two major consequences of the aggregation process: (1) it disrupts feature variability suppression in deep neural networks (DNNs), and (2) it weakens the coupling between features and subsequent this http URL on these findings, we propose several simple yet effective strategies to mitigate the negative impacts of model aggregation while still enjoying the benefit it brings. To the best of our knowledge, our work is the first to conduct a layer-peeled analysis of model aggregation, potentially paving the way for the development of more effective FL algorithms.

Title: Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation

• Authors: Bo Lin, Shangwen Wang, Liqian Chen, Xiaoguang Mao
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2502.03233
• Pdf URL: https://arxiv.org/pdf/2502.03233
• Copy Paste: [[2502.03233]] Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation(https://arxiv.org/abs/2502.03233)
• Keywords: security, attack, large language model
• Abstract: The integration of Large Language Models (LLMs) into software development has revolutionized the field, particularly through the use of Retrieval-Augmented Code Generation (RACG) systems that enhance code generation with information from external knowledge bases. However, the security implications of RACG systems, particularly the risks posed by vulnerable code examples in the knowledge base, remain largely unexplored. This risk is particularly concerning given that public code repositories, which often serve as the sources for knowledge base collection in RACG systems, are usually accessible to anyone in the community. Malicious attackers can exploit this accessibility to inject vulnerable code into the knowledge base, making it toxic. Once these poisoned samples are retrieved and incorporated into the generated code, they can propagate security vulnerabilities into the final product. This paper presents the first comprehensive study on the security risks associated with RACG systems, focusing on how vulnerable code in the knowledge base compromises the security of generated code. We investigate the LLM-generated code security across different settings through extensive experiments using four major LLMs, two retrievers, and two poisoning scenarios. Our findings highlight the significant threat of knowledge base poisoning, where even a single poisoned code example can compromise up to 48% of generated code. Our findings provide crucial insights into vulnerability introduction in RACG systems and offer practical mitigation recommendations, thereby helping improve the security of LLM-generated code in future works.

Title: RiemannGFM: Learning a Graph Foundation Model from Riemannian Geometry

• Authors: Li Sun, Zhenhao Huang, Suyang Zhou, Qiqi Wan, Hao Peng, Philip Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03251
• Pdf URL: https://arxiv.org/pdf/2502.03251
• Copy Paste: [[2502.03251]] RiemannGFM: Learning a Graph Foundation Model from Riemannian Geometry(https://arxiv.org/abs/2502.03251)
• Keywords: large language model
• Abstract: The foundation model has heralded a new era in artificial intelligence, pretraining a single model to offer cross-domain transferability on different datasets. Graph neural networks excel at learning graph data, the omnipresent non-Euclidean structure, but often lack the generalization capacity. Hence, graph foundation model is drawing increasing attention, and recent efforts have been made to leverage Large Language Models. On the one hand, existing studies primarily focus on text-attributed graphs, while a wider range of real graphs do not contain fruitful textual attributes. On the other hand, the sequential graph description tailored for the Large Language Model neglects the structural complexity, which is a predominant characteristic of the graph. Such limitations motivate an important question: Can we go beyond Large Language Models, and pretrain a universal model to learn the structural knowledge for any graph? The answer in the language or vision domain is a shared vocabulary. We observe the fact that there also exist shared substructures underlying graph domain, and thereby open a new opportunity of graph foundation model with structural vocabulary. The key innovation is the discovery of a simple yet effective structural vocabulary of trees and cycles, and we explore its inherent connection to Riemannian geometry. Herein, we present a universal pretraining model, RiemannGFM. Concretely, we first construct a novel product bundle to incorporate the diverse geometries of the vocabulary. Then, on this constructed space, we stack Riemannian layers where the structural vocabulary, regardless of specific graph, is learned in Riemannian manifold offering cross-domain transferability. Extensive experiments show the effectiveness of RiemannGFM on a diversity of real graphs.

Title: Efficient extraction of medication information from clinical notes: an evaluation in two languages

• Authors: Thibaut Fabacher, Erik-André Sauleau, Emmanuelle Arcay, Bineta Faye, Maxime Alter, Archia Chahard, Nathan Miraillet, Adrien Coulet, Aurélie Névéol
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.03257
• Pdf URL: https://arxiv.org/pdf/2502.03257
• Copy Paste: [[2502.03257]] Efficient extraction of medication information from clinical notes: an evaluation in two languages(https://arxiv.org/abs/2502.03257)
• Keywords: extraction, transformer
• Abstract: Objective: To evaluate the accuracy, computational cost and portability of a new Natural Language Processing (NLP) method for extracting medication information from clinical narratives. Materials and Methods: We propose an original transformer-based architecture for the extraction of entities and their relations pertaining to patients' medication regimen. First, we used this approach to train and evaluate a model on French clinical notes, using a newly annotated corpus from Hôpitaux Universitaires de Strasbourg. Second, the portability of the approach was assessed by conducting an evaluation on clinical documents in English from the 2018 n2c2 shared task. Information extraction accuracy and computational cost were assessed by comparison with an available method using transformers. Results: The proposed architecture achieves on the task of relation extraction itself performance that are competitive with the state-of-the-art on both French and English (F-measures 0.82 and 0.96 vs 0.81 and 0.95), but reduce the computational cost by 10. End-to-end (Named Entity recognition and Relation Extraction) F1 performance is 0.69 and 0.82 for French and English corpus. Discussion: While an existing system developed for English notes was deployed in a French hospital setting with reasonable effort, we found that an alternative architecture offered end-to-end drug information extraction with comparable extraction performance and lower computational impact for both French and English clinical text processing, respectively. Conclusion: The proposed architecture can be used to extract medication information from clinical text with high performance and low computational cost and consequently suits with usually limited hospital IT resources

Title: General Time-series Model for Universal Knowledge Representation of Multivariate Time-Series data

• Authors: Cheng He, Xu Huang, Gangwei Jiang, Zhaoyi Li, Defu Lian, Hong Xie, Enhong Chen, Xijie Liang, Zengrong Zheng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03264
• Pdf URL: https://arxiv.org/pdf/2502.03264
• Copy Paste: [[2502.03264]] General Time-series Model for Universal Knowledge Representation of Multivariate Time-Series data(https://arxiv.org/abs/2502.03264)
• Keywords: generative
• Abstract: Universal knowledge representation is a central problem for multivariate time series(MTS) foundation models and yet remains open. This paper investigates this problem from the first principle and it makes four folds of contributions. First, a new empirical finding is revealed: time series with different time granularities (or corresponding frequency resolutions) exhibit distinct joint distributions in the frequency domain. This implies a crucial aspect of learning universal knowledge, one that has been overlooked by previous studies. Second, a novel Fourier knowledge attention mechanism is proposed to enable learning time granularity-aware representations from both the temporal and frequency domains. Third, an autoregressive blank infilling pre-training framework is incorporated to time series analysis for the first time, leading to a generative tasks agnostic pre-training strategy. To this end, we develop the General Time-series Model (GTM), a unified MTS foundation model that addresses the limitation of contemporary time series models, which often require token, pre-training, or model-level customizations for downstream tasks adaption. Fourth, extensive experiments show that GTM outperforms state-of-the-art (SOTA) methods across all generative tasks, including long-term forecasting, anomaly detection, and imputation.

Title: ZISVFM: Zero-Shot Object Instance Segmentation in Indoor Robotic Environments with Vision Foundation Models

• Authors: Ying Zhang, Maoliang Yin, Wenfu Bi, Haibao Yan, Shaohan Bian, Cui-Hua Zhang, Changchun Hua
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2502.03266
• Pdf URL: https://arxiv.org/pdf/2502.03266
• Copy Paste: [[2502.03266]] ZISVFM: Zero-Shot Object Instance Segmentation in Indoor Robotic Environments with Vision Foundation Models(https://arxiv.org/abs/2502.03266)
• Keywords: transformer, segmentation
• Abstract: Service robots operating in unstructured environments must effectively recognize and segment unknown objects to enhance their functionality. Traditional supervised learningbased segmentation techniques require extensive annotated datasets, which are impractical for the diversity of objects encountered in real-world scenarios. Unseen Object Instance Segmentation (UOIS) methods aim to address this by training models on synthetic data to generalize to novel objects, but they often suffer from the simulation-to-reality gap. This paper proposes a novel approach (ZISVFM) for solving UOIS by leveraging the powerful zero-shot capability of the segment anything model (SAM) and explicit visual representations from a selfsupervised vision transformer (ViT). The proposed framework operates in three stages: (1) generating object-agnostic mask proposals from colorized depth images using SAM, (2) refining these proposals using attention-based features from the selfsupervised ViT to filter non-object masks, and (3) applying K-Medoids clustering to generate point prompts that guide SAM towards precise object segmentation. Experimental validation on two benchmark datasets and a self-collected dataset demonstrates the superior performance of ZISVFM in complex environments, including hierarchical settings such as cabinets, drawers, and handheld objects. Our source code is available at this https URL.

Title: TYPEPULSE: Detecting Type Confusion Bugs in Rust Programs

• Authors: Hung-Mao Chen, Xu He, Shu Wang, Xiaokuan Zhang, Kun Sun
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2502.03271
• Pdf URL: https://arxiv.org/pdf/2502.03271
• Copy Paste: [[2502.03271]] TYPEPULSE: Detecting Type Confusion Bugs in Rust Programs(https://arxiv.org/abs/2502.03271)
• Keywords: security, robust
• Abstract: Rust supports type conversions and safe Rust guarantees the security of these conversions through robust static type checking and strict ownership guidelines. However, there are instances where programmers need to use unsafe Rust for certain type conversions, especially those involving pointers. Consequently, these conversions may cause severe memory corruption problems. Despite extensive research on type confusion bugs in C/C++, studies on type confusion bugs in Rust are still lacking. Also, due to Rust's new features in the type system, existing solutions in C/C++ cannot be directly applied to Rust. In this paper, we develop a static analysis tool called TYPEPULSE to detect three main categories of type confusion bugs in Rust including misalignment, inconsistent layout, and mismatched scope. TYPEPULSE first performs a type conversion analysis to collect and determine trait bounds for type pairs. Moreover, it performs a pointer alias analysis to resolve the alias relationship of pointers. Following the integration of information into the property graph, it constructs type patterns and detects each type of bug in various conversion scenarios. We run TYPEPULSE on the top 3,000 Rust packages and uncover 71 new type confusion bugs, exceeding the total number of type confusion bugs reported in RUSTSEC over the past five years. We have received 32 confirmations from developers, along with one CVE ID and six RUSTSEC IDs.

Title: Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning

• Authors: DiJia Su, Hanlin Zhu, Yingchen Xu, Jiantao Jiao, Yuandong Tian, Qinqing Zheng
• Subjects: cs.CL, cs.AI, cs.LG, cs.LO
• Abstract URL: https://arxiv.org/abs/2502.03275
• Pdf URL: https://arxiv.org/pdf/2502.03275
• Copy Paste: [[2502.03275]] Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning(https://arxiv.org/abs/2502.03275)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) excel at reasoning and planning when trained on chainof-thought (CoT) data, where the step-by-step thought process is explicitly outlined by text tokens. However, this results in lengthy inputs where many words support textual coherence rather than core reasoning information, and processing these inputs consumes substantial computation resources. In this work, we propose a hybrid representation of the reasoning process, where we partially abstract away the initial reasoning steps using latent discrete tokens generated by VQ-VAE, significantly reducing the length of reasoning traces. We explore the use of latent trace abstractions in two scenarios: 1) training the model from scratch for the Keys-Finding Maze problem, 2) fine-tuning LLMs on this hybrid data with an extended vocabulary including unseen latent tokens, for both logical and mathematical reasoning problems. To facilitate effective learning, we introduce a simple training procedure that randomly mixes latent and text tokens, which enables fast adaptation to new latent tokens. Our approach consistently outperforms the baselines methods in various benchmarks.

Title: MeDiSumQA: Patient-Oriented Question-Answer Generation from Discharge Letters

• Authors: Amin Dada, Osman Alperen Koras, Marie Bauer, Amanda Butler, Kaleb E. Smith, Jens Kleesiek, Julian Friedrich
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03298
• Pdf URL: https://arxiv.org/pdf/2502.03298
• Copy Paste: [[2502.03298]] MeDiSumQA: Patient-Oriented Question-Answer Generation from Discharge Letters(https://arxiv.org/abs/2502.03298)
• Keywords: large language model
• Abstract: While increasing patients' access to medical documents improves medical care, this benefit is limited by varying health literacy levels and complex medical terminology. Large language models (LLMs) offer solutions by simplifying medical information. However, evaluating LLMs for safe and patient-friendly text generation is difficult due to the lack of standardized evaluation resources. To fill this gap, we developed MeDiSumQA. MeDiSumQA is a dataset created from MIMIC-IV discharge summaries through an automated pipeline combining LLM-based question-answer generation with manual quality checks. We use this dataset to evaluate various LLMs on patient-oriented question-answering. Our findings reveal that general-purpose LLMs frequently surpass biomedical-adapted models, while automated metrics correlate with human judgment. By releasing MeDiSumQA on PhysioNet, we aim to advance the development of LLMs to enhance patient understanding and ultimately improve care outcomes.

Title: MAP Image Recovery with Guarantees using Locally Convex Multi-Scale Energy (LC-MUSE) Model

• Authors: Jyothi Rikhab Chand, Mathews Jacob
• Subjects: cs.LG, cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2502.03302
• Pdf URL: https://arxiv.org/pdf/2502.03302
• Copy Paste: [[2502.03302]] MAP Image Recovery with Guarantees using Locally Convex Multi-Scale Energy (LC-MUSE) Model(https://arxiv.org/abs/2502.03302)
• Keywords: robust
• Abstract: We propose a multi-scale deep energy model that is strongly convex in the local neighbourhood around the data manifold to represent its probability density, with application in inverse problems. In particular, we represent the negative log-prior as a multi-scale energy model parameterized by a Convolutional Neural Network (CNN). We restrict the gradient of the CNN to be locally monotone, which constrains the model as a Locally Convex Multi-Scale Energy (LC-MuSE). We use the learned energy model in image-based inverse problems, where the formulation offers several desirable properties: i) uniqueness of the solution, ii) convergence guarantees to a minimum of the inverse problem, and iii) robustness to input perturbations. In the context of parallel Magnetic Resonance (MR) image reconstruction, we show that the proposed method performs better than the state-of-the-art convex regularizers, while the performance is comparable to plug-and-play regularizers and end-to-end trained methods.

Title: Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning

• Authors: Qitao Tan, Jun Liu, Zheng Zhan, Caiwei Ding, Yanzhi Wang, Jin Lu, Geng Yuan
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03304
• Pdf URL: https://arxiv.org/pdf/2502.03304
• Copy Paste: [[2502.03304]] Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning(https://arxiv.org/abs/2502.03304)
• Keywords: large language model
• Abstract: Large language models (LLMs) excel across various tasks, but standard first-order (FO) fine-tuning demands considerable memory, significantly limiting real-world deployment. Recently, zeroth-order (ZO) optimization stood out as a promising memory-efficient training paradigm, avoiding backward passes and relying solely on forward passes for gradient estimation, making it attractive for resource-constrained scenarios. However, ZO method lags far behind FO method in both convergence speed and accuracy. To bridge the gap, we introduce a novel layer-wise divergence analysis that uncovers the distinct update pattern of FO and ZO optimization. Aiming to resemble the learning capacity of FO method from the findings, we propose \textbf{Di}vergence-driven \textbf{Z}eroth-\textbf{O}rder (\textbf{DiZO}) optimization. DiZO conducts divergence-driven layer adaptation by incorporating projections to ZO updates, generating diverse-magnitude updates precisely scaled to layer-wise individual optimization needs. Our results demonstrate that DiZO significantly reduces the needed iterations for convergence without sacrificing throughput, cutting training GPU hours by up to 48\% on various datasets. Moreover, DiZO consistently outperforms the representative ZO baselines in fine-tuning RoBERTa-large, OPT-series, and Llama-series on downstream tasks and, in some cases, even surpasses memory-intensive FO fine-tuning.

Title: Out-of-Distribution Detection using Synthetic Data Generation

• Authors: Momin Abbas, Muneeza Azmat, Raya Horesh, Mikhail Yurochkin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03323
• Pdf URL: https://arxiv.org/pdf/2502.03323
• Copy Paste: [[2502.03323]] Out-of-Distribution Detection using Synthetic Data Generation(https://arxiv.org/abs/2502.03323)
• Keywords: generative, large language model
• Abstract: Distinguishing in- and out-of-distribution (OOD) inputs is crucial for reliable deployment of classification systems. However, OOD data is typically unavailable or difficult to collect, posing a significant challenge for accurate OOD detection. In this work, we present a method that harnesses the generative capabilities of Large Language Models (LLMs) to create high-quality synthetic OOD proxies, eliminating the dependency on any external OOD data source. We study the efficacy of our method on classical text classification tasks such as toxicity detection and sentiment classification as well as classification tasks arising in LLM development and deployment, such as training a reward model for RLHF and detecting misaligned generations. Extensive experiments on nine InD-OOD dataset pairs and various model sizes show that our approach dramatically lowers false positive rates (achieving a perfect zero in some cases) while maintaining high accuracy on in-distribution tasks, outperforming baseline methods by a significant margin.

Title: ECM: A Unified Electronic Circuit Model for Explaining the Emergence of In-Context Learning and Chain-of-Thought in Large Language Model

• Authors: Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiaqi Wang, Mengkang Hu, Zhi Chen, Wanxiang Che, Ting Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03325
• Pdf URL: https://arxiv.org/pdf/2502.03325
• Copy Paste: [[2502.03325]] ECM: A Unified Electronic Circuit Model for Explaining the Emergence of In-Context Learning and Chain-of-Thought in Large Language Model(https://arxiv.org/abs/2502.03325)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have led to significant successes across various applications, where the most noticeable is to a series of emerging capabilities, particularly in the areas of In-Context Learning (ICL) and Chain-of-Thought (CoT). To better understand and control model performance, many studies have begun investigating the underlying causes of these phenomena and their impact on task outcomes. However, existing explanatory frameworks predominantly focus on isolating and explaining ICL and CoT independently, leading to an incomplete understanding of their combined influence on model performance. To address this gap, we propose the Electronic Circuit Model (ECM), which provides a foundation for developing scalable, learnable policies and improving the management of AI-generated content. Specifically, ECM conceptualizes model behavior as an electronic circuit: ICL is represented as semantic magnetic field to providing an additional voltage following Faraday's Law, while CoT is modeled as series resistors to constrain the model output performance following Ohm's Law. Experimental results demonstrate that the ECM effectively predicts and explains LLM performance across a variety of prompting strategies. Furthermore, we apply ECM to advanced reasoning strategy optimization on a series of tasks, such as the International Olympiad in Informatics (IOI) and the International Mathematical Olympiad (IMO), achieving competitive performance that surpasses nearly 80% of top human competitors.

Title: Interaction-Aware Gaussian Weighting for Clustered Federated Learning

• Authors: Alessandro Licciardi, Davide Leo, Eros Faní, Barbara Caputo, Marco Ciccone
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03340
• Pdf URL: https://arxiv.org/pdf/2502.03340
• Copy Paste: [[2502.03340]] Interaction-Aware Gaussian Weighting for Clustered Federated Learning(https://arxiv.org/abs/2502.03340)
• Keywords: privacy, robust, federate
• Abstract: Federated Learning (FL) emerged as a decentralized paradigm to train models while preserving privacy. However, conventional FL struggles with data heterogeneity and class imbalance, which degrade model performance. Clustered FL balances personalization and decentralized training by grouping clients with analogous data distributions, enabling improved accuracy while adhering to privacy constraints. This approach effectively mitigates the adverse impact of heterogeneity in FL. In this work, we propose a novel clustered FL method, FedGWC (Federated Gaussian Weighting Clustering), which groups clients based on their data distribution, allowing training of a more robust and personalized model on the identified clusters. FedGWC identifies homogeneous clusters by transforming individual empirical losses to model client interactions with a Gaussian reward mechanism. Additionally, we introduce the Wasserstein Adjusted Score, a new clustering metric for FL to evaluate cluster cohesion with respect to the individual class distribution. Our experiments on benchmark datasets show that FedGWC outperforms existing FL algorithms in cluster quality and classification accuracy, validating the efficacy of our approach.

Title: Robust Autonomy Emerges from Self-Play

• Authors: Marco Cusumano-Towner, David Hafner, Alex Hertzberg, Brody Huval, Aleksei Petrenko, Eugene Vinitsky, Erik Wijmans, Taylor Killian, Stuart Bowers, Ozan Sener, Philipp Krähenbühl, Vladlen Koltun
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2502.03349
• Pdf URL: https://arxiv.org/pdf/2502.03349
• Copy Paste: [[2502.03349]] Robust Autonomy Emerges from Self-Play(https://arxiv.org/abs/2502.03349)
• Keywords: robust
• Abstract: Self-play has powered breakthroughs in two-player and multi-player games. Here we show that self-play is a surprisingly effective strategy in another domain. We show that robust and naturalistic driving emerges entirely from self-play in simulation at unprecedented scale -- 1.6~billion~km of driving. This is enabled by Gigaflow, a batched simulator that can synthesize and train on 42 years of subjective driving experience per hour on a single 8-GPU node. The resulting policy achieves state-of-the-art performance on three independent autonomous driving benchmarks. The policy outperforms the prior state of the art when tested on recorded real-world scenarios, amidst human drivers, without ever seeing human data during training. The policy is realistic when assessed against human references and achieves unprecedented robustness, averaging 17.5 years of continuous driving between incidents in simulation.

Title: GHOST: Gaussian Hypothesis Open-Set Technique

• Authors: Ryan Rabinowitz, Steve Cruz, Manuel Günther, Terrance E. Boult
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03359
• Pdf URL: https://arxiv.org/pdf/2502.03359
• Copy Paste: [[2502.03359]] GHOST: Gaussian Hypothesis Open-Set Technique(https://arxiv.org/abs/2502.03359)
• Keywords: fair
• Abstract: Evaluations of large-scale recognition methods typically focus on overall performance. While this approach is common, it often fails to provide insights into performance across individual classes, which can lead to fairness issues and misrepresentation. Addressing these gaps is crucial for accurately assessing how well methods handle novel or unseen classes and ensuring a fair evaluation. To address fairness in Open-Set Recognition (OSR), we demonstrate that per-class performance can vary dramatically. We introduce Gaussian Hypothesis Open Set Technique (GHOST), a novel hyperparameter-free algorithm that models deep features using class-wise multivariate Gaussian distributions with diagonal covariance matrices. We apply Z-score normalization to logits to mitigate the impact of feature magnitudes that deviate from the model's expectations, thereby reducing the likelihood of the network assigning a high score to an unknown sample. We evaluate GHOST across multiple ImageNet-1K pre-trained deep networks and test it with four different unknown datasets. Using standard metrics such as AUOSCR, AUROC and FPR95, we achieve statistically significant improvements, advancing the state-of-the-art in large-scale OSR. Source code is provided online.

Title: Scaling laws in wearable human activity recognition

• Authors: Tom Hoddes, Alex Bijamov, Saket Joshi, Daniel Roggen, Ali Etemad, Robert Harle, David Racz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03364
• Pdf URL: https://arxiv.org/pdf/2502.03364
• Copy Paste: [[2502.03364]] Scaling laws in wearable human activity recognition(https://arxiv.org/abs/2502.03364)
• Keywords: transformer
• Abstract: Many deep architectures and self-supervised pre-training techniques have been proposed for human activity recognition (HAR) from wearable multimodal sensors. Scaling laws have the potential to help move towards more principled design by linking model capacity with pre-training data volume. Yet, scaling laws have not been established for HAR to the same extent as in language and vision. By conducting an exhaustive grid search on both amount of pre-training data and Transformer architectures, we establish the first known scaling laws for HAR. We show that pre-training loss scales with a power law relationship to amount of data and parameter count and that increasing the number of users in a dataset results in a steeper improvement in performance than increasing data per user, indicating that diversity of pre-training data is important, which contrasts to some previously reported findings in self-supervised HAR. We show that these scaling laws translate to downstream performance improvements on three HAR benchmark datasets of postures, modes of locomotion and activities of daily living: UCI HAR and WISDM Phone and WISDM Watch. Finally, we suggest some previously published works should be revisited in light of these scaling laws with more adequate model capacities.

Title: Deep Learning-Based Approach for Identification of Potato Leaf Diseases Using Wrapper Feature Selection and Feature Concatenation

• Authors: Muhammad Ahtsam Naeem, Muhammad Asim Saleem, Muhammad Imran Sharif, Shahzad Akber, Sajjad Saleem, Zahid Akhtar, Kamran Siddique
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03370
• Pdf URL: https://arxiv.org/pdf/2502.03370
• Copy Paste: [[2502.03370]] Deep Learning-Based Approach for Identification of Potato Leaf Diseases Using Wrapper Feature Selection and Feature Concatenation(https://arxiv.org/abs/2502.03370)
• Keywords: extraction
• Abstract: The potato is a widely grown crop in many regions of the world. In recent decades, potato farming has gained incredible traction in the world. Potatoes are susceptible to several illnesses that stunt their development. This plant seems to have significant leaf disease. Early Blight and Late Blight are two prevalent leaf diseases that affect potato plants. The early detection of these diseases would be beneficial for enhancing the yield of this crop. The ideal solution is to use image processing to identify and analyze these disorders. Here, we present an autonomous method based on image processing and machine learning to detect late blight disease affecting potato leaves. The proposed method comprises four different phases: (1) Histogram Equalization is used to improve the quality of the input image; (2) feature extraction is performed using a Deep CNN model, then these extracted features are concatenated; (3) feature selection is performed using wrapper-based feature selection; (4) classification is performed using an SVM classifier and its variants. This proposed method achieves the highest accuracy of 99% using SVM by selecting 550 features.

Title: Demystifying Long Chain-of-Thought Reasoning in LLMs

• Authors: Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, Xiang Yue
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03373
• Pdf URL: https://arxiv.org/pdf/2502.03373
• Copy Paste: [[2502.03373]] Demystifying Long Chain-of-Thought Reasoning in LLMs(https://arxiv.org/abs/2502.03373)
• Keywords: large language model
• Abstract: Scaling inference compute enhances reasoning in large language models (LLMs), with long chains-of-thought (CoTs) enabling strategies like backtracking and error correction. Reinforcement learning (RL) has emerged as a crucial method for developing these capabilities, yet the conditions under which long CoTs emerge remain unclear, and RL training requires careful design choices. In this study, we systematically investigate the mechanics of long CoT reasoning, identifying the key factors that enable models to generate long CoT trajectories. Through extensive supervised fine-tuning (SFT) and RL experiments, we present four main findings: (1) While SFT is not strictly necessary, it simplifies training and improves efficiency; (2) Reasoning capabilities tend to emerge with increased training compute, but their development is not guaranteed, making reward shaping crucial for stabilizing CoT length growth; (3) Scaling verifiable reward signals is critical for RL. We find that leveraging noisy, web-extracted solutions with filtering mechanisms shows strong potential, particularly for out-of-distribution (OOD) tasks such as STEM reasoning; and (4) Core abilities like error correction are inherently present in base models, but incentivizing these skills effectively for complex tasks via RL demands significant compute, and measuring their emergence requires a nuanced approach. These insights provide practical guidance for optimizing training strategies to enhance long CoT reasoning in LLMs. Our code is available at: this https URL.

Title: Transformers and Their Roles as Time Series Foundation Models

• Authors: Dennis Wu, Yihan He, Yuan Cao, Jianqing Fan, Han Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03383
• Pdf URL: https://arxiv.org/pdf/2502.03383
• Copy Paste: [[2502.03383]] Transformers and Their Roles as Time Series Foundation Models(https://arxiv.org/abs/2502.03383)
• Keywords: transformer
• Abstract: We give a comprehensive analysis of transformers as time series foundation models, focusing on their approximation and generalization capabilities. First, we demonstrate that there exist transformers that fit an autoregressive model on input univariate time series via gradient descent. We then analyze MOIRAI, a multivariate time series foundation model capable of handling an arbitrary number of covariates. We prove that it is capable of automatically fitting autoregressive models with an arbitrary number of covariates, offering insights into its design and empirical success. For generalization, we establish bounds for pretraining when the data satisfies Dobrushin's condition. Experiments support our theoretical findings, highlighting the efficacy of transformers as time series foundation models.

Title: A Structured Reasoning Framework for Unbalanced Data Classification Using Probabilistic Models

• Authors: Junliang Du, Shiyu Dou, Bohuan Yang, Jiacheng Hu, Tai An
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03386
• Pdf URL: https://arxiv.org/pdf/2502.03386
• Copy Paste: [[2502.03386]] A Structured Reasoning Framework for Unbalanced Data Classification Using Probabilistic Models(https://arxiv.org/abs/2502.03386)
• Keywords: robust
• Abstract: This paper studies a Markov network model for unbalanced data, aiming to solve the problems of classification bias and insufficient minority class recognition ability of traditional machine learning models in environments with uneven class distribution. By constructing joint probability distribution and conditional dependency, the model can achieve global modeling and reasoning optimization of sample categories. The study introduced marginal probability estimation and weighted loss optimization strategies, combined with regularization constraints and structured reasoning methods, effectively improving the generalization ability and robustness of the model. In the experimental stage, a real credit card fraud detection dataset was selected and compared with models such as logistic regression, support vector machine, random forest and XGBoost. The experimental results show that the Markov network performs well in indicators such as weighted accuracy, F1 score, and AUC-ROC, significantly outperforming traditional classification models, demonstrating its strong decision-making ability and applicability in unbalanced data scenarios. Future research can focus on efficient model training, structural optimization, and deep learning integration in large-scale unbalanced data environments and promote its wide application in practical applications such as financial risk control, medical diagnosis, and intelligent monitoring.

Title: LIMO: Less is More for Reasoning

• Authors: Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, Pengfei Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03387
• Pdf URL: https://arxiv.org/pdf/2502.03387
• Copy Paste: [[2502.03387]] LIMO: Less is More for Reasoning(https://arxiv.org/abs/2502.03387)
• Keywords: large language model
• Abstract: We present a fundamental discovery that challenges our understanding of how complex reasoning emerges in large language models. While conventional wisdom suggests that sophisticated reasoning tasks demand extensive training data (>100,000 examples), we demonstrate that complex mathematical reasoning abilities can be effectively elicited with surprisingly few examples. Through comprehensive experiments, our proposed model LIMO demonstrates unprecedented performance in mathematical reasoning. With merely 817 curated training samples, LIMO achieves 57.1% accuracy on AIME and 94.8% on MATH, improving from previous SFT-based models' 6.5% and 59.2% respectively, while only using 1% of the training data required by previous approaches. LIMO demonstrates exceptional out-of-distribution generalization, achieving 40.5% absolute improvement across 10 diverse benchmarks, outperforming models trained on 100x more data, challenging the notion that SFT leads to memorization rather than generalization. Based on these results, we propose the Less-Is-More Reasoning Hypothesis (LIMO Hypothesis): In foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning capabilities can emerge through minimal but precisely orchestrated demonstrations of cognitive processes. This hypothesis posits that the elicitation threshold for complex reasoning is determined by two key factors: (1) the completeness of the model's encoded knowledge foundation during pre-training, and (2) the effectiveness of post-training examples as "cognitive templates" that show the model how to utilize its knowledge base to solve complex reasoning tasks. To facilitate reproducibility and future research in data-efficient reasoning, we release LIMO as a comprehensive open-source suite at this https URL.

Title: Benchmarking Time Series Forecasting Models: From Statistical Techniques to Foundation Models in Real-World Applications

• Authors: Issar Arab, Rodrigo Benitez
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03395
• Pdf URL: https://arxiv.org/pdf/2502.03395
• Copy Paste: [[2502.03395]] Benchmarking Time Series Forecasting Models: From Statistical Techniques to Foundation Models in Real-World Applications(https://arxiv.org/abs/2502.03395)
• Keywords: robust
• Abstract: Time series forecasting is essential for operational intelligence in the hospitality industry, and particularly challenging in large-scale, distributed systems. This study evaluates the performance of statistical, machine learning (ML), deep learning, and foundation models in forecasting hourly sales over a 14-day horizon using real-world data from a network of thousands of restaurants across Germany. The forecasting solution includes features such as weather conditions, calendar events, and time-of-day patterns. Results demonstrate the strong performance of ML-based meta-models and highlight the emerging potential of foundation models like Chronos and TimesFM, which deliver competitive performance with minimal feature engineering, leveraging only the pre-trained model (zero-shot inference). Additionally, a hybrid PySpark-Pandas approach proves to be a robust solution for achieving horizontal scalability in large-scale deployments.

Title: SPRI: Aligning Large Language Models with Context-Situated Principles

• Authors: Hongli Zhan, Muneeza Azmat, Raya Horesh, Junyi Jessy Li, Mikhail Yurochkin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03397
• Pdf URL: https://arxiv.org/pdf/2502.03397
• Copy Paste: [[2502.03397]] SPRI: Aligning Large Language Models with Context-Situated Principles(https://arxiv.org/abs/2502.03397)
• Keywords: large language model
• Abstract: Aligning Large Language Models to integrate and reflect human values, especially for tasks that demand intricate human oversight, is arduous since it is resource-intensive and time-consuming to depend on human expertise for context-specific guidance. Prior work has utilized predefined sets of rules or principles to steer the behavior of models (Bai et al., 2022; Sun et al., 2023). However, these principles tend to be generic, making it challenging to adapt them to each individual input query or context. In this work, we present Situated-PRInciples (SPRI), a framework requiring minimal or no human effort that is designed to automatically generate guiding principles in real-time for each input query and utilize them to align each response. We evaluate SPRI on three tasks, and show that 1) SPRI can derive principles in a complex domain-specific task that leads to on-par performance as expert-crafted ones; 2) SPRI-generated principles lead to instance-specific rubrics that outperform prior LLM-as-a-judge frameworks; 3) using SPRI to generate synthetic SFT data leads to substantial improvement on truthfulness. We release our code and model generations at this https URL.

Title: The Adoption of Artificial Intelligence in Different Network Security Concepts

• Authors: Mamoon A. Al Jbaar, Adel Jalal Yousif, Qutaiba I. Ali
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2502.03398
• Pdf URL: https://arxiv.org/pdf/2502.03398
• Copy Paste: [[2502.03398]] The Adoption of Artificial Intelligence in Different Network Security Concepts(https://arxiv.org/abs/2502.03398)
• Keywords: secure, security, attack
• Abstract: The obstacles of each security system combined with the increase of cyber-attacks, negatively affect the effectiveness of network security management and rise the activities to be taken by the security staff and network administrators. So, there is a growing need for the automated auditing and intelligent reporting strategies for reliable network security with as less model complexity as possible. Newly, artificial intelligence has been effectively applied to various network security issues, and numerous studies have been conducted that utilize various artificial intelligence techniques for the purposes of encryption and secure communication, in addition to using artificial intelligence to perform a large number of data encryption operations in record time. The aim of the study is to present and discuss the most prominent methods of artificial intelligence recently used in the field of network security including user authentication, Key exchanging, encryption/decryption, data integrity and intrusion detection system.

Title: Lightweight Authenticated Task Offloading in 6G-Cloud Vehicular Twin Networks

• Authors: Sarah Al-Shareeda, Fusun Ozguner, Keith Redmill, Trung Q. Duong, Berk Canberk
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03403
• Pdf URL: https://arxiv.org/pdf/2502.03403
• Copy Paste: [[2502.03403]] Lightweight Authenticated Task Offloading in 6G-Cloud Vehicular Twin Networks(https://arxiv.org/abs/2502.03403)
• Keywords: secure
• Abstract: Task offloading management in 6G vehicular networks is crucial for maintaining network efficiency, particularly as vehicles generate substantial data. Integrating secure communication through authentication introduces additional computational and communication overhead, significantly impacting offloading efficiency and latency. This paper presents a unified framework incorporating lightweight Identity-Based Cryptographic (IBC) authentication into task offloading within cloud-based 6G Vehicular Twin Networks (VTNs). Utilizing Proximal Policy Optimization (PPO) in Deep Reinforcement Learning (DRL), our approach optimizes authenticated offloading decisions to minimize latency and enhance resource allocation. Performance evaluation under varying network sizes, task sizes, and data rates reveals that IBC authentication can reduce offloading efficiency by up to 50% due to the added overhead. Besides, increasing network size and task size can further reduce offloading efficiency by up to 91.7%. As a countermeasure, increasing the transmission data rate can improve the offloading performance by as much as 63%, even in the presence of authentication overhead. The code for the simulations and experiments detailed in this paper is available on GitHub for further reference and reproducibility [1].

Title: Detecting Strategic Deception Using Linear Probes

• Authors: Nicholas Goldowsky-Dill, Bilal Chughtai, Stefan Heimersheim, Marius Hobbhahn
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03407
• Pdf URL: https://arxiv.org/pdf/2502.03407
• Copy Paste: [[2502.03407]] Detecting Strategic Deception Using Linear Probes(https://arxiv.org/abs/2502.03407)
• Keywords: robust
• Abstract: AI models might use deceptive strategies as part of scheming or misaligned behaviour. Monitoring outputs alone is insufficient, since the AI might produce seemingly benign outputs while their internal reasoning is misaligned. We thus evaluate if linear probes can robustly detect deception by monitoring model activations. We test two probe-training datasets, one with contrasting instructions to be honest or deceptive (following Zou et al., 2023) and one of responses to simple roleplaying scenarios. We test whether these probes generalize to realistic settings where Llama-3.3-70B-Instruct behaves deceptively, such as concealing insider trading (Scheurer et al., 2023) and purposely underperforming on safety evaluations (Benton et al., 2024). We find that our probe distinguishes honest and deceptive responses with AUROCs between 0.96 and 0.999 on our evaluation datasets. If we set the decision threshold to have a 1% false positive rate on chat data not related to deception, our probe catches 95-99% of the deceptive responses. Overall we think white-box probes are promising for future monitoring systems, but current performance is insufficient as a robust defence against deception. Our probes' outputs can be viewed at this http URL and our code at this http URL.

Title: From Features to Transformers: Redefining Ranking for Scalable Impact

• Authors: Fedor Borisyuk, Lars Hertel, Ganesh Parameswaran, Gaurav Srivastava, Sudarshan Srinivasa Ramanujam, Borja Ocejo, Peng Du, Andrei Akterskii, Neil Daftary, Shao Tang, Daqi Sun, Qiang Charles Xiao, Deepesh Nathani, Mohit Kothari, Yun Dai, Aman Gupta
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03417
• Pdf URL: https://arxiv.org/pdf/2502.03417
• Copy Paste: [[2502.03417]] From Features to Transformers: Redefining Ranking for Scalable Impact(https://arxiv.org/abs/2502.03417)
• Keywords: transformer
• Abstract: We present LiGR, a large-scale ranking framework developed at LinkedIn that brings state-of-the-art transformer-based modeling architectures into production. We introduce a modified transformer architecture that incorporates learned normalization and simultaneous set-wise attention to user history and ranked items. This architecture enables several breakthrough achievements, including: (1) the deprecation of most manually designed feature engineering, outperforming the prior state-of-the-art system using only few features (compared to hundreds in the baseline), (2) validation of the scaling law for ranking systems, showing improved performance with larger models, more training data, and longer context sequences, and (3) simultaneous joint scoring of items in a set-wise manner, leading to automated improvements in diversity. To enable efficient serving of large ranking models, we describe techniques to scale inference effectively using single-pass processing of user history and set-wise attention. We also summarize key insights from various ablation studies and A/B tests, highlighting the most impactful technical approaches.

Title: Think or Step-by-Step? UnZIPping the Black Box in Zero-Shot Prompts

• Authors: Nikta Gohari Sadr, Sangmitra Madhusudan, Ali Emami
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03418
• Pdf URL: https://arxiv.org/pdf/2502.03418
• Copy Paste: [[2502.03418]] Think or Step-by-Step? UnZIPping the Black Box in Zero-Shot Prompts(https://arxiv.org/abs/2502.03418)
• Keywords: interpretability, large language model
• Abstract: Zero-shot prompting techniques have significantly improved the performance of Large Language Models (LLMs). However, we lack a clear understanding of why zero-shot prompts are so effective. For example, in the prompt "Let's think step-by-step," is "think" or "step-by-step" more crucial to its success? Existing interpretability methods, such as gradient-based and attention-based approaches, are computationally intensive and restricted to open-source models. We introduce the ZIP score (Zero-shot Importance of Perturbation score), a versatile metric applicable to both open and closed-source models, based on systematic input word perturbations. Our experiments across four recent LLMs, seven widely-used prompts, and several tasks, reveal interesting patterns in word importance. For instance, while both 'step-by-step' and 'think' show high ZIP scores, which one is more influential depends on the model and task. We validate our method using controlled experiments and compare our results with human judgments, finding that proprietary models align more closely with human intuition regarding word significance. These findings enhance our understanding of LLM behavior and contribute to developing more effective zero-shot prompts and improved model analysis.

Title: Can Text-to-Image Generative Models Accurately Depict Age? A Comparative Study on Synthetic Portrait Generation and Age Estimation

• Authors: Alexey A. Novikov, Miroslav Vranka, François David, Artem Voronin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03420
• Pdf URL: https://arxiv.org/pdf/2502.03420
• Copy Paste: [[2502.03420]] Can Text-to-Image Generative Models Accurately Depict Age? A Comparative Study on Synthetic Portrait Generation and Age Estimation(https://arxiv.org/abs/2502.03420)
• Keywords: robust, generative
• Abstract: Text-to-image generative models have shown remarkable progress in producing diverse and photorealistic outputs. In this paper, we present a comprehensive analysis of their effectiveness in creating synthetic portraits that accurately represent various demographic attributes, with a special focus on age, nationality, and gender. Our evaluation employs prompts specifying detailed profiles (e.g., Photorealistic selfie photo of a 32-year-old Canadian male), covering a broad spectrum of 212 nationalities, 30 distinct ages from 10 to 78, and balanced gender representation. We compare the generated images against ground truth age estimates from two established age estimation models to assess how faithfully age is depicted. Our findings reveal that although text-to-image models can consistently generate faces reflecting different identities, the accuracy with which they capture specific ages and do so across diverse demographic backgrounds remains highly variable. These results suggest that current synthetic data may be insufficiently reliable for high-stakes age-related tasks requiring robust precision, unless practitioners are prepared to invest in significant filtering and curation. Nevertheless, they may still be useful in less sensitive or exploratory applications, where absolute age precision is not critical.

Title: Concept Based Explanations and Class Contrasting

• Authors: Rudolf Herdt, Daniel Otero Baguer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03422
• Pdf URL: https://arxiv.org/pdf/2502.03422
• Copy Paste: [[2502.03422]] Concept Based Explanations and Class Contrasting(https://arxiv.org/abs/2502.03422)
• Keywords: segmentation
• Abstract: Explaining deep neural networks is challenging, due to their large size and non-linearity. In this paper, we introduce a concept-based explanation method, in order to explain the prediction for an individual class, as well as contrasting any two classes, i.e. explain why the model predicts one class over the other. We test it on several openly available classification models trained on ImageNet1K, as well as on a segmentation model trained to detect tumor in stained tissue samples. We perform both qualitative and quantitative tests. For example, for a ResNet50 model from pytorch model zoo, we can use the explanation for why the model predicts a class 'A' to automatically select six dataset crops where the model does not predict class 'A'. The model then predicts class 'A' again for the newly combined image in 71\% of the cases (works for 710 out of the 1000 classes). The code including an .ipynb example is available on git: this https URL.

Title: TruePose: Human-Parsing-guided Attention Diffusion for Full-ID Preserving Pose Transfer

• Authors: Zhihong Xu, Dongxia Wang, Peng Du, Yang Cao, Qing Guo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03426
• Pdf URL: https://arxiv.org/pdf/2502.03426
• Copy Paste: [[2502.03426]] TruePose: Human-Parsing-guided Attention Diffusion for Full-ID Preserving Pose Transfer(https://arxiv.org/abs/2502.03426)
• Keywords: protect, extraction, diffusion
• Abstract: Pose-Guided Person Image Synthesis (PGPIS) generates images that maintain a subject's identity from a source image while adopting a specified target pose (e.g., skeleton). While diffusion-based PGPIS methods effectively preserve facial features during pose transformation, they often struggle to accurately maintain clothing details from the source image throughout the diffusion process. This limitation becomes particularly problematic when there is a substantial difference between the source and target poses, significantly impacting PGPIS applications in the fashion industry where clothing style preservation is crucial for copyright protection. Our analysis reveals that this limitation primarily stems from the conditional diffusion model's attention modules failing to adequately capture and preserve clothing patterns. To address this limitation, we propose human-parsing-guided attention diffusion, a novel approach that effectively preserves both facial and clothing appearance while generating high-quality results. We propose a human-parsing-aware Siamese network that consists of three key components: dual identical UNets (TargetNet for diffusion denoising and SourceNet for source image embedding extraction), a human-parsing-guided fusion attention (HPFA), and a CLIP-guided attention alignment (CAA). The HPFA and CAA modules can embed the face and clothes patterns into the target image generation adaptively and effectively. Extensive experiments on both the in-shop clothes retrieval benchmark and the latest in-the-wild human editing dataset demonstrate our method's significant advantages over 13 baseline approaches for preserving both facial and clothes appearance in the source image.

Title: A Hybrid Blockchain-IPFS Solution for Secure and Scalable Data Collection and Storage for Smart Water Meters

• Authors: Thandile Nododile, Clement Nyirenda
• Subjects: cs.CR, cs.ET
• Abstract URL: https://arxiv.org/abs/2502.03427
• Pdf URL: https://arxiv.org/pdf/2502.03427
• Copy Paste: [[2502.03427]] A Hybrid Blockchain-IPFS Solution for Secure and Scalable Data Collection and Storage for Smart Water Meters(https://arxiv.org/abs/2502.03427)
• Keywords: secure
• Abstract: Scalable and secure data management is important in Internet of Things (IoT) applications such as smart water meters, where traditional blockchain storage can be restrictive due to high data volumes. This paper investigates a hybrid blockchain and InterPlanetary File System (IPFS) approach designed to optimise storage efficiency, enhance throughput, and reduce block time by offloading large data off-chain to IPFS while preserving on-chain integrity. A substrate-based private blockchain was developed to store smart water meter (SWM) data, and controlled experiments were conducted to evaluate blockchain performance with and without IPFS. Key metrics, including block size, block time, and transaction throughput, were analysed across varying data volumes and node counts. Results show that integrating IPFS significantly reduces on-chain storage demands, leading to smaller block sizes, increased throughput, and improved block times compared to blockchain-only storage. These findings highlight the potential of hybrid blockchain-IPFS models for efficiently and securely managing high-volume IoT data.

Title: On Fairness of Unified Multimodal Large Language Model for Image Generation

• Authors: Ming Liu, Hao Chen, Jindong Wang, Liwen Wang, Bhiksha Raj Ramakrishnan, Wensheng Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03429
• Pdf URL: https://arxiv.org/pdf/2502.03429
• Copy Paste: [[2502.03429]] On Fairness of Unified Multimodal Large Language Model for Image Generation(https://arxiv.org/abs/2502.03429)
• Keywords: fair, diffusion, large language model
• Abstract: Unified multimodal large language models (U-MLLMs) have demonstrated impressive performance in visual understanding and generation in an end-to-end pipeline. Compared with generation-only models (e.g., Stable Diffusion), U-MLLMs may raise new questions about bias in their outputs, which can be affected by their unified capabilities. This gap is particularly concerning given the under-explored risk of propagating harmful stereotypes. In this paper, we benchmark the latest U-MLLMs and find that most exhibit significant demographic biases, such as gender and race bias. To better understand and mitigate this issue, we propose a locate-then-fix strategy, where we audit and show how the individual model component is affected by bias. Our analysis shows that bias originates primarily from the language model. More interestingly, we observe a "partial alignment" phenomenon in U-MLLMs, where understanding bias appears minimal, but generation bias remains substantial. Thus, we propose a novel balanced preference model to balance the demographic distribution with synthetic data. Experiments demonstrate that our approach reduces demographic bias while preserving semantic fidelity. We hope our findings underscore the need for more holistic interpretation and debiasing strategies of U-MLLMs in the future.

Title: A Temporal Convolutional Network-Based Approach and a Benchmark Dataset for Colonoscopy Video Temporal Segmentation

• Authors: Carlo Biffi, Giorgio Roffo, Pietro Salvagnini, Andrea Cherubini
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2502.03430
• Pdf URL: https://arxiv.org/pdf/2502.03430
• Copy Paste: [[2502.03430]] A Temporal Convolutional Network-Based Approach and a Benchmark Dataset for Colonoscopy Video Temporal Segmentation(https://arxiv.org/abs/2502.03430)
• Keywords: segmentation
• Abstract: Following recent advancements in computer-aided detection and diagnosis systems for colonoscopy, the automated reporting of colonoscopy procedures is set to further revolutionize clinical practice. A crucial yet underexplored aspect in the development of these systems is the creation of computer vision models capable of autonomously segmenting full-procedure colonoscopy videos into anatomical sections and procedural phases. In this work, we aim to create the first open-access dataset for this task and propose a state-of-the-art approach, benchmarked against competitive models. We annotated the publicly available REAL-Colon dataset, consisting of 2.7 million frames from 60 complete colonoscopy videos, with frame-level labels for anatomical locations and colonoscopy phases across nine categories. We then present ColonTCN, a learning-based architecture that employs custom temporal convolutional blocks designed to efficiently capture long temporal dependencies for the temporal segmentation of colonoscopy videos. We also propose a dual k-fold cross-validation evaluation protocol for this benchmark, which includes model assessment on unseen, multi-center this http URL achieves state-of-the-art performance in classification accuracy while maintaining a low parameter count when evaluated using the two proposed k-fold cross-validation settings, outperforming competitive models. We report ablation studies to provide insights into the challenges of this task and highlight the benefits of the custom temporal convolutional blocks, which enhance learning and improve model efficiency. We believe that the proposed open-access benchmark and the ColonTCN approach represent a significant advancement in the temporal segmentation of colonoscopy procedures, fostering further open-access research to address this clinical need.

Title: Masked Autoencoders Are Effective Tokenizers for Diffusion Models

• Authors: Hao Chen, Yujin Han, Fangyi Chen, Xiang Li, Yidong Wang, Jindong Wang, Ze Wang, Zicheng Liu, Difan Zou, Bhiksha Raj
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03444
• Pdf URL: https://arxiv.org/pdf/2502.03444
• Copy Paste: [[2502.03444]] Masked Autoencoders Are Effective Tokenizers for Diffusion Models(https://arxiv.org/abs/2502.03444)
• Keywords: diffusion
• Abstract: Recent advances in latent diffusion models have demonstrated their effectiveness for high-resolution image synthesis. However, the properties of the latent space from tokenizer for better learning and generation of diffusion models remain under-explored. Theoretically and empirically, we find that improved generation quality is closely tied to the latent distributions with better structure, such as the ones with fewer Gaussian Mixture modes and more discriminative features. Motivated by these insights, we propose MAETok, an autoencoder (AE) leveraging mask modeling to learn semantically rich latent space while maintaining reconstruction fidelity. Extensive experiments validate our analysis, demonstrating that the variational form of autoencoders is not necessary, and a discriminative latent space from AE alone enables state-of-the-art performance on ImageNet generation using only 128 tokens. MAETok achieves significant practical improvements, enabling a gFID of 1.69 with 76x faster training and 31x higher inference throughput for 512x512 generation. Our findings show that the structure of the latent space, rather than variational constraints, is crucial for effective diffusion models. Code and trained models are released.

Title: Dress-1-to-3: Single Image to Simulation-Ready 3D Outfit with Diffusion Prior and Differentiable Physics

• Authors: Xuan Li, Chang Yu, Wenxin Du, Ying Jiang, Tianyi Xie, Yunuo Chen, Yin Yang, Chenfanfu Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03449
• Pdf URL: https://arxiv.org/pdf/2502.03449
• Copy Paste: [[2502.03449]] Dress-1-to-3: Single Image to Simulation-Ready 3D Outfit with Diffusion Prior and Differentiable Physics(https://arxiv.org/abs/2502.03449)
• Keywords: diffusion
• Abstract: Recent advances in large models have significantly advanced image-to-3D reconstruction. However, the generated models are often fused into a single piece, limiting their applicability in downstream tasks. This paper focuses on 3D garment generation, a key area for applications like virtual try-on with dynamic garment animations, which require garments to be separable and simulation-ready. We introduce Dress-1-to-3, a novel pipeline that reconstructs physics-plausible, simulation-ready separated garments with sewing patterns and humans from an in-the-wild image. Starting with the image, our approach combines a pre-trained image-to-sewing pattern generation model for creating coarse sewing patterns with a pre-trained multi-view diffusion model to produce multi-view images. The sewing pattern is further refined using a differentiable garment simulator based on the generated multi-view images. Versatile experiments demonstrate that our optimization approach substantially enhances the geometric alignment of the reconstructed 3D garments and humans with the input image. Furthermore, by integrating a texture generation module and a human motion generation module, we produce customized physics-plausible and realistic dynamic garment demonstrations. Project page: this https URL

Title: A Schema-Guided Reason-while-Retrieve framework for Reasoning on Scene Graphs with Large-Language-Models (LLMs)

• Authors: Yiye Chen, Harpreet Sawhney, Nicholas Gydé, Yanan Jian, Jack Saunders, Patricio Vela, Ben Lundell
• Subjects: cs.LG, cs.AI, cs.MA, cs.RO
• Abstract URL: https://arxiv.org/abs/2502.03450
• Pdf URL: https://arxiv.org/pdf/2502.03450
• Copy Paste: [[2502.03450]] A Schema-Guided Reason-while-Retrieve framework for Reasoning on Scene Graphs with Large-Language-Models (LLMs)(https://arxiv.org/abs/2502.03450)
• Keywords: large language model
• Abstract: Scene graphs have emerged as a structured and serializable environment representation for grounded spatial reasoning with Large Language Models (LLMs). In this work, we propose SG-RwR, a Schema-Guided Retrieve-while-Reason framework for reasoning and planning with scene graphs. Our approach employs two cooperative, code-writing LLM agents: a (1) Reasoner for task planning and information queries generation, and a (2) Retriever for extracting corresponding graph information following the queries. Two agents collaborate iteratively, enabling sequential reasoning and adaptive attention to graph information. Unlike prior works, both agents are prompted only with the scene graph schema rather than the full graph data, which reduces the hallucination by limiting input tokens, and drives the Reasoner to generate reasoning trace this http URL the trace, the Retriever programmatically query the scene graph data based on the schema understanding, allowing dynamic and global attention on the graph that enhances alignment between reasoning and retrieval. Through experiments in multiple simulation environments, we show that our framework surpasses existing LLM-based approaches in numerical Q\&A and planning tasks, and can benefit from task-level few-shot examples, even in the absence of agent-level demonstrations. Project code will be released.

Title: SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living

• Authors: Arkaprava Sinha, Dominick Reilly, Francois Bremond, Pu Wang, Srijan Das
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03459
• Pdf URL: https://arxiv.org/pdf/2502.03459
• Copy Paste: [[2502.03459]] SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living(https://arxiv.org/abs/2502.03459)
• Keywords: robust
• Abstract: The introduction of vision-language models like CLIP has enabled the development of foundational video models capable of generalizing to unseen videos and human actions. However, these models are typically trained on web videos, which often fail to capture the challenges present in Activities of Daily Living (ADL) videos. Existing works address ADL-specific challenges, such as similar appearances, subtle motion patterns, and multiple viewpoints, by combining 3D skeletons and RGB videos. However, these approaches are not integrated with language, limiting their ability to generalize to unseen action classes. In this paper, we introduce SKI models, which integrate 3D skeletons into the vision-language embedding space. SKI models leverage a skeleton-language model, SkeletonCLIP, to infuse skeleton information into Vision Language Models (VLMs) and Large Vision Language Models (LVLMs) through collaborative training. Notably, SKI models do not require skeleton data during inference, enhancing their robustness for real-world applications. The effectiveness of SKI models is validated on three popular ADL datasets for zero-shot action recognition and video caption generation tasks.

Title: Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training

• Authors: Boyao Wang, Rui Pan, Shizhe Diao, Xingyuan Pan, Jipeng Zhang, Renjie Pi, Tong Zhang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03460
• Pdf URL: https://arxiv.org/pdf/2502.03460
• Copy Paste: [[2502.03460]] Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training(https://arxiv.org/abs/2502.03460)
• Keywords: large language model
• Abstract: Small language models (SLMs) have attracted considerable attention from both academia and industry due to their broad range of applications in edge devices. To obtain SLMs with strong performance, conventional approaches either pre-train the models from scratch, which incurs substantial computational costs, or compress/prune existing large language models (LLMs), which results in performance drops and falls short in comparison to pre-training. In this paper, we investigate the family of acceleration methods that involve both structured pruning and model training. We found 1) layer-wise adaptive pruning (Adapt-Pruner) is extremely effective in LLMs and yields significant improvements over existing pruning techniques, 2) adaptive pruning equipped with further training leads to models comparable to those pre-training from scratch, 3) incremental pruning brings non-trivial performance gain by interleaving pruning with training and only removing a small portion of neurons ($\sim$5%) at a time. Experimental results on LLaMA-3.1-8B demonstrate that Adapt-Pruner outperforms conventional pruning methods, such as LLM-Pruner, FLAP, and SliceGPT, by an average of 1%-7% in accuracy on commonsense benchmarks. Additionally, Adapt-Pruner restores the performance of MobileLLM-125M to 600M on the MMLU benchmark with 200$\times$ fewer tokens via pruning from its larger counterparts, and discovers a new 1B model that surpasses LLaMA-3.2-1B in multiple benchmarks.

Title: Do Large Language Model Benchmarks Test Reliability?

• Authors: Joshua Vendrow, Edward Vendrow, Sara Beery, Aleksander Madry
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03461
• Pdf URL: https://arxiv.org/pdf/2502.03461
• Copy Paste: [[2502.03461]] Do Large Language Model Benchmarks Test Reliability?(https://arxiv.org/abs/2502.03461)
• Keywords: large language model
• Abstract: When deploying large language models (LLMs), it is important to ensure that these models are not only capable, but also reliable. Many benchmarks have been created to track LLMs' growing capabilities, however there has been no similar focus on measuring their reliability. To understand the potential ramifications of this gap, we investigate how well current benchmarks quantify model reliability. We find that pervasive label errors can compromise these evaluations, obscuring lingering model failures and hiding unreliable behavior. Motivated by this gap in the evaluation of reliability, we then propose the concept of so-called platinum benchmarks, i.e., benchmarks carefully curated to minimize label errors and ambiguity. As a first attempt at constructing such benchmarks, we revise examples from fifteen existing popular benchmarks. We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks such as elementary-level math word problems. Analyzing these failures further reveals previously unidentified patterns of problems on which frontier models consistently struggle. We provide code at this https URL