2025-02-21

Title: Which Attention Heads Matter for In-Context Learning?

• Authors: Kayo Yin, Jacob Steinhardt
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.14010
• Pdf URL: https://arxiv.org/pdf/2502.14010
• Copy Paste: [[2502.14010]] Which Attention Heads Matter for In-Context Learning?(https://arxiv.org/abs/2502.14010)
• Keywords: in-context
• Abstract: Large language models (LLMs) exhibit impressive in-context learning (ICL) capability, enabling them to perform new tasks using only a few demonstrations in the prompt. Two different mechanisms have been proposed to explain ICL: induction heads that find and copy relevant tokens, and function vector (FV) heads whose activations compute a latent encoding of the ICL task. To better understand which of the two distinct mechanisms drives ICL, we study and compare induction heads and FV heads in 12 language models. Through detailed ablations, we discover that few-shot ICL performance depends primarily on FV heads, especially in larger models. In addition, we uncover that FV and induction heads are connected: many FV heads start as induction heads during training before transitioning to the FV mechanism. This leads us to speculate that induction facilitates learning the more complex FV mechanism that ultimately drives ICL.

Title: Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data

• Authors: Yucheng Shi, Quanzheng Li, Jin Sun, Xiang Li, Ninghao Liu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.14044
• Pdf URL: https://arxiv.org/pdf/2502.14044
• Copy Paste: [[2502.14044]] Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data(https://arxiv.org/abs/2502.14044)
• Keywords: foundation model
• Abstract: Large multimodal models (LMMs) have shown impressive capabilities in a wide range of visual tasks. However, they often struggle with fine-grained visual reasoning, failing to identify domain-specific objectives and provide justifiable explanations for their predictions. To address this, we propose a novel visual rejection sampling framework to improve the cognition and explainability of LMMs using self-synthesized data. Specifically, visual fine-tuning requires images, queries, and target answers. Our approach begins by synthesizing interpretable answers that include human-verifiable visual features. These features are based on expert-defined concepts, carefully selected based on their alignment with the image content. After each round of fine-tuning, we apply a reward model-free filtering mechanism to select the highest-quality interpretable answers for the next round of tuning. This iterative process of data synthesis and fine-tuning progressively improves the model's ability to generate accurate and reasonable explanations. Experimental results demonstrate the effectiveness of our method in improving both the accuracy and explainability of specialized visual classification tasks.

Title: Triad: Vision Foundation Model for 3D Magnetic Resonance Imaging

• Authors: Shansong Wang, Mojtaba Safari, Qiang Li, Chih-Wei Chang, Richard LJ Qiu, Justin Roper, David S. Yu, Xiaofeng Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.14064
• Pdf URL: https://arxiv.org/pdf/2502.14064
• Copy Paste: [[2502.14064]] Triad: Vision Foundation Model for 3D Magnetic Resonance Imaging(https://arxiv.org/abs/2502.14064)
• Keywords: foundation model
• Abstract: Vision foundation models (VFMs) are pre-trained on extensive image datasets to learn general representations for diverse types of data. These models can subsequently be fine-tuned for specific downstream tasks, significantly boosting performance across a broad range of applications. However, existing vision foundation models that claim to be applicable to various radiology tasks are mostly pre-trained on 3D computed tomography (CT), which benefits from the availability of extensive 3D CT databases. Significant differences between CT and magnetic resonance imaging (MRI) in imaging principles, signal characteristics, and data distribution may hinder their practical performance and versatility in MRI-specific applications. Here, we propose Triad, a vision foundation model for 3D MRI. Triad adopts a widely used autoencoder architecture to learn robust representations from 131,170 3D MRI volumes and uses organ-independent imaging descriptions to constrain the semantic distribution of the visual modality. The above pre-training dataset is called Triad-131K, which is currently the largest 3D MRI pre-training dataset. We evaluate Triad across three tasks, namely, organ/tumor segmentation, organ/cancer classification, and medical image registration, in two data modalities (within-domain and out-of-domain) settings using 25 downstream datasets. By initializing models with Triad's pre-trained weights, nnUNet-Triad improves segmentation performance by 6.88% compared to nnUNet-Scratch across 17 datasets. Swin-B-Triad achieves a 3.97% improvement over Swin-B-Scratch in classification tasks across five datasets. SwinUNETR-Triad improves by 4.00% compared to SwinUNETR-Scratch in registration tasks across two datasets. Our study demonstrates that pre-training can maximize performance when the data modalities and organs of upstream and downstream tasks are consistent.

Title: A Racing Dataset and Baseline Model for Track Detection in Autonomous Racing

• Authors: Shreya Ghosh, Yi-Huan Chen, Ching-Hsiang Huang, Abu Shafin Mohammad Mahdee Jameel, Chien Chou Ho, Aly El Gamal, Samuel Labi
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2502.14068
• Pdf URL: https://arxiv.org/pdf/2502.14068
• Copy Paste: [[2502.14068]] A Racing Dataset and Baseline Model for Track Detection in Autonomous Racing(https://arxiv.org/abs/2502.14068)
• Keywords: generative
• Abstract: A significant challenge in racing-related research is the lack of publicly available datasets containing raw images with corresponding annotations for the downstream task. In this paper, we introduce RoRaTrack, a novel dataset that contains annotated multi-camera image data from racing scenarios for track detection. The data is collected on a Dallara AV-21 at a racing circuit in Indiana, in collaboration with the Indy Autonomous Challenge (IAC). RoRaTrack addresses common problems such as blurriness due to high speed, color inversion from the camera, and absence of lane markings on the track. Consequently, we propose RaceGAN, a baseline model based on a Generative Adversarial Network (GAN) that effectively addresses these challenges. The proposed model demonstrates superior performance compared to current state-of-the-art machine learning models in track detection. The dataset and code for this work are available at this http URL.

Title: DiffExp: Efficient Exploration in Reward Fine-tuning for Text-to-Image Diffusion Models

• Authors: Daewon Chae, June Suk Choi, Jinkyu Kim, Kimin Lee
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.14070
• Pdf URL: https://arxiv.org/pdf/2502.14070
• Copy Paste: [[2502.14070]] DiffExp: Efficient Exploration in Reward Fine-tuning for Text-to-Image Diffusion Models(https://arxiv.org/abs/2502.14070)
• Keywords: diffusion
• Abstract: Fine-tuning text-to-image diffusion models to maximize rewards has proven effective for enhancing model performance. However, reward fine-tuning methods often suffer from slow convergence due to online sample generation. Therefore, obtaining diverse samples with strong reward signals is crucial for improving sample efficiency and overall performance. In this work, we introduce DiffExp, a simple yet effective exploration strategy for reward fine-tuning of text-to-image models. Our approach employs two key strategies: (a) dynamically adjusting the scale of classifier-free guidance to enhance sample diversity, and (b) randomly weighting phrases of the text prompt to exploit high-quality reward signals. We demonstrate that these strategies significantly enhance exploration during online sample generation, improving the sample efficiency of recent reward fine-tuning methods, such as DDPO and AlignProp.

Title: Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression

• Authors: Xuheng Li, Quanquan Gu
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.14123
• Pdf URL: https://arxiv.org/pdf/2502.14123
• Copy Paste: [[2502.14123]] Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression(https://arxiv.org/abs/2502.14123)
• Keywords: diffusion, generative
• Abstract: Exponential moving average (EMA) has recently gained significant popularity in training modern deep learning models, especially diffusion-based generative models. However, there have been few theoretical results explaining the effectiveness of EMA. In this paper, to better understand EMA, we establish the risk bound of online SGD with EMA for high-dimensional linear regression, one of the simplest overparameterized learning tasks that shares similarities with neural networks. Our results indicate that (i) the variance error of SGD with EMA is always smaller than that of SGD without averaging, and (ii) unlike SGD with iterate averaging from the beginning, the bias error of SGD with EMA decays exponentially in every eigen-subspace of the data covariance matrix. Additionally, we develop proof techniques applicable to the analysis of a broad class of averaging schemes.

Title: Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above

• Authors: Nishant Balepur, Rachel Rudinger, Jordan Lee Boyd-Graber
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.14127
• Pdf URL: https://arxiv.org/pdf/2502.14127
• Copy Paste: [[2502.14127]] Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above(https://arxiv.org/abs/2502.14127)
• Keywords: generative
• Abstract: Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing, but we argue for its reform. We first reveal flaws in MCQA's format, as it struggles to: 1) test generation/subjectivity; 2) match LLM use cases; and 3) fully test knowledge. We instead advocate for generative formats based on human testing-where LLMs construct and explain answers-better capturing user needs and knowledge while remaining easy to score. We then show even when MCQA is a useful format, its datasets suffer from: leakage; unanswerability; shortcuts; and saturation. In each issue, we give fixes from education, like rubrics to guide MCQ writing; scoring methods to bridle guessing; and Item Response Theory to build harder MCQs. Lastly, we discuss LLM errors in MCQA-robustness, biases, and unfaithful explanations-showing how our prior solutions better measure or address these issues. While we do not need to desert MCQA, we encourage more efforts in refining the task based on educational testing, advancing evaluations.

Title: ModSkill: Physical Character Skill Modularization

• Authors: Yiming Huang, Zhiyang Dou, Lingjie Liu
• Subjects: cs.CV, cs.GR, cs.RO
• Abstract URL: https://arxiv.org/abs/2502.14140
• Pdf URL: https://arxiv.org/pdf/2502.14140
• Copy Paste: [[2502.14140]] ModSkill: Physical Character Skill Modularization(https://arxiv.org/abs/2502.14140)
• Keywords: generative
• Abstract: Human motion is highly diverse and dynamic, posing challenges for imitation learning algorithms that aim to generalize motor skills for controlling simulated characters. Previous methods typically rely on a universal full-body controller for tracking reference motion (tracking-based model) or a unified full-body skill embedding space (skill embedding). However, these approaches often struggle to generalize and scale to larger motion datasets. In this work, we introduce a novel skill learning framework, ModSkill, that decouples complex full-body skills into compositional, modular skills for independent body parts. Our framework features a skill modularization attention layer that processes policy observations into modular skill embeddings that guide low-level controllers for each body part. We also propose an Active Skill Learning approach with Generative Adaptive Sampling, using large motion generation models to adaptively enhance policy learning in challenging tracking scenarios. Our results show that this modularized skill learning framework, enhanced by generative sampling, outperforms existing methods in precise full-body motion tracking and enables reusable skill embeddings for diverse goal-driven tasks.

Title: Bayesian SegNet for Semantic Segmentation with Improved Interpretation of Microstructural Evolution During Irradiation of Materials

• Authors: Marjolein Oostrom, Alex Hagen, Nicole LaHaye, Karl Pazdernik
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.14184
• Pdf URL: https://arxiv.org/pdf/2502.14184
• Copy Paste: [[2502.14184]] Bayesian SegNet for Semantic Segmentation with Improved Interpretation of Microstructural Evolution During Irradiation of Materials(https://arxiv.org/abs/2502.14184)
• Keywords: diffusion
• Abstract: Understanding the relationship between the evolution of microstructures of irradiated LiAlO2 pellets and tritium diffusion, retention and release could improve predictions of tritium-producing burnable absorber rod performance. Given expert-labeled segmented images of irradiated and unirradiated pellets, we trained Deep Convolutional Neural Networks to segment images into defect, grain, and boundary classes. Qualitative microstructural information was calculated from these segmented images to facilitate the comparison of unirradiated and irradiated pellets. We tested modifications to improve the sensitivity of the model, including incorporating meta-data into the model and utilizing uncertainty quantification. The predicted segmentation was similar to the expert-labeled segmentation for most methods of microstructural qualification, including pixel proportion, defect area, and defect density. Overall, the high performance metrics for the best models for both irradiated and unirradiated images shows that utilizing neural network models is a viable alternative to expert-labeled images.

Title: Adaptive Sparsified Graph Learning Framework for Vessel Behavior Anomalies

• Authors: Jeehong Kim, Minchan Kim, Jaeseong Ju, Youngseok Hwang, Wonhee Lee, Hyunwoo Park
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.14197
• Pdf URL: https://arxiv.org/pdf/2502.14197
• Copy Paste: [[2502.14197]] Adaptive Sparsified Graph Learning Framework for Vessel Behavior Anomalies(https://arxiv.org/abs/2502.14197)
• Keywords: anomaly
• Abstract: Graph neural networks have emerged as a powerful tool for learning spatiotemporal interactions. However, conventional approaches often rely on predefined graphs, which may obscure the precise relationships being modeled. Additionally, existing methods typically define nodes based on fixed spatial locations, a strategy that is ill-suited for dynamic environments like maritime environments. Our method introduces an innovative graph representation where timestamps are modeled as distinct nodes, allowing temporal dependencies to be explicitly captured through graph edges. This setup is extended to construct a multi-ship graph that effectively captures spatial interactions while preserving graph sparsity. The graph is processed using Graph Convolutional Network layers to capture spatiotemporal patterns, with a forecasting layer for feature prediction and a Variational Graph Autoencoder for reconstruction, enabling robust anomaly detection.

Title: Accurate Forgetting for Heterogeneous Federated Continual Learning

• Authors: Abudukelimu Wuerkaixi, Sen Cui, Jingfeng Zhang, Kunda Yan, Bo Han, Gang Niu, Lei Fang, Changshui Zhang, Masashi Sugiyama
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.14205
• Pdf URL: https://arxiv.org/pdf/2502.14205
• Copy Paste: [[2502.14205]] Accurate Forgetting for Heterogeneous Federated Continual Learning(https://arxiv.org/abs/2502.14205)
• Keywords: generative
• Abstract: Recent years have witnessed a burgeoning interest in federated learning (FL). However, the contexts in which clients engage in sequential learning remain under-explored. Bridging FL and continual learning (CL) gives rise to a challenging practical problem: federated continual learning (FCL). Existing research in FCL primarily focuses on mitigating the catastrophic forgetting issue of continual learning while collaborating with other clients. We argue that the forgetting phenomena are not invariably detrimental. In this paper, we consider a more practical and challenging FCL setting characterized by potentially unrelated or even antagonistic data/tasks across different clients. In the FL scenario, statistical heterogeneity and data noise among clients may exhibit spurious correlations which result in biased feature learning. While existing CL strategies focus on a complete utilization of previous knowledge, we found that forgetting biased information is beneficial in our study. Therefore, we propose a new concept accurate forgetting (AF) and develop a novel generative-replay method~\method~which selectively utilizes previous knowledge in federated networks. We employ a probabilistic framework based on a normalizing flow model to quantify the credibility of previous knowledge. Comprehensive experiments affirm the superiority of our method over baselines.

Title: Designing Parameter and Compute Efficient Diffusion Transformers using Distillation

• Authors: Vignesh Sundaresha
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2502.14226
• Pdf URL: https://arxiv.org/pdf/2502.14226
• Copy Paste: [[2502.14226]] Designing Parameter and Compute Efficient Diffusion Transformers using Distillation(https://arxiv.org/abs/2502.14226)
• Keywords: diffusion
• Abstract: Diffusion Transformers (DiTs) with billions of model parameters form the backbone of popular image and video generation models like DALL.E, Stable-Diffusion and SORA. Though these models are necessary in many low-latency applications like Augmented/Virtual Reality, they cannot be deployed on resource-constrained Edge devices (like Apple Vision Pro or Meta Ray-Ban glasses) due to their huge computational complexity. To overcome this, we turn to knowledge distillation and perform a thorough design-space exploration to achieve the best DiT for a given parameter size. In particular, we provide principles for how to choose design knobs such as depth, width, attention heads and distillation setup for a DiT. During the process, a three-way trade-off emerges between model performance, size and speed that is crucial for Edge implementation of diffusion. We also propose two distillation approaches - Teaching Assistant (TA) method and Multi-In-One (MI1) method - to perform feature distillation in the DiT context. Unlike existing solutions, we demonstrate and benchmark the efficacy of our approaches on practical Edge devices such as NVIDIA Jetson Orin Nano.

Title: LLM-EvRep: Learning an LLM-Compatible Event Representation Using a Self-Supervised Framework

• Authors: Zongyou Yu, Qiang Qu, Qian Zhang, Nan Zhang, Xiaoming Chen
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2502.14273
• Pdf URL: https://arxiv.org/pdf/2502.14273
• Copy Paste: [[2502.14273]] LLM-EvRep: Learning an LLM-Compatible Event Representation Using a Self-Supervised Framework(https://arxiv.org/abs/2502.14273)
• Keywords: self-supervised
• Abstract: Recent advancements in event-based recognition have demonstrated significant promise, yet most existing approaches rely on extensive training, limiting their adaptability for efficient processing of event-driven visual content. Meanwhile, large language models (LLMs) have exhibited remarkable zero-shot capabilities across diverse domains, but their application to event-based visual recognition remains largely unexplored. To bridge this gap, we propose \textbf{LLM-EvGen}, an event representation generator that produces LLM-compatible event representations \textbf{LLM-EvRep}, thereby enhancing the performance of LLMs on event recognition tasks. The generator is trained using a self-supervised framework, aligning the generated representations with semantic consistency and structural fidelity. Comprehensive experiments were conducted on three datasets: N-ImageNet, N-Caltech101, and N-MNIST. The results demonstrate that our method, \textbf{LLM-EvRep}, outperforms the event-to-video method, E2VID, by 15.93\%, 0.82\%, and 50.21\%, respectively, in recognition tasks when evaluated using GPT-4o.

Title: Correcting Noisy Multilabel Predictions: Modeling Label Noise through Latent Space Shifts

• Authors: Weipeng Huang, Qin Li, Yang Xiao, Cheng Qiao, Tie Cai, Junwei Liao, Neil J. Hurley, Guangyuan Piao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.14281
• Pdf URL: https://arxiv.org/pdf/2502.14281
• Copy Paste: [[2502.14281]] Correcting Noisy Multilabel Predictions: Modeling Label Noise through Latent Space Shifts(https://arxiv.org/abs/2502.14281)
• Keywords: generative
• Abstract: Noise in data appears to be inevitable in most real-world machine learning applications and would cause severe overfitting problems. Not only can data features contain noise, but labels are also prone to be noisy due to human input. In this paper, rather than noisy label learning in multiclass classifications, we instead focus on the less explored area of noisy label learning for multilabel classifications. Specifically, we investigate the post-correction of predictions generated from classifiers learned with noisy labels. The reasons are two-fold. Firstly, this approach can directly work with the trained models to save computational resources. Secondly, it could be applied on top of other noisy label correction techniques to achieve further improvements. To handle this problem, we appeal to deep generative approaches that are possible for uncertainty estimation. Our model posits that label noise arises from a stochastic shift in the latent variable, providing a more robust and beneficial means for noisy learning. We develop both unsupervised and semi-supervised learning methods for our model. The extensive empirical study presents solid evidence to that our approach is able to consistently improve the independent models and performs better than a number of existing methods across various noisy label settings. Moreover, a comprehensive empirical analysis of the proposed method is carried out to validate its robustness, including sensitivity analysis and an ablation study, among other elements.

Title: Graph Anomaly Detection via Adaptive Test-time Representation Learning across Out-of-Distribution Domains

• Authors: Delaram Pirhayati, Arlei Silva
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2502.14293
• Pdf URL: https://arxiv.org/pdf/2502.14293
• Copy Paste: [[2502.14293]] Graph Anomaly Detection via Adaptive Test-time Representation Learning across Out-of-Distribution Domains(https://arxiv.org/abs/2502.14293)
• Keywords: self-supervised, anomaly
• Abstract: Graph Anomaly Detection (GAD) has demonstrated great effectiveness in identifying unusual patterns within graph-structured data. However, while labeled anomalies are often scarce in emerging applications, existing supervised GAD approaches are either ineffective or not applicable when moved across graph domains due to distribution shifts and heterogeneous feature spaces. To address these challenges, we present AdaGraph-T3, a novel test-time training framework for cross-domain GAD. AdaGraph-T3 combines supervised and self-supervised learning during training while adapting to a new domain during test time using only self-supervised learning by leveraging a homophily-based affinity score that captures domain-invariant properties of anomalies. Our framework introduces four key innovations to cross-domain GAD: an effective self-supervision scheme, an attention-based mechanism that dynamically learns edge importance weights during message passing, domain-specific encoders for handling heterogeneous features, and class-aware regularization to address imbalance. Experiments across multiple cross-domain settings demonstrate that AdaGraph-T3 significantly outperforms existing approaches, achieving average improvements of over 6.6% in AUROC and 7.9% in AUPRC compared to the best competing model.

Title: Textured 3D Regenerative Morphing with 3D Diffusion Prior

• Authors: Songlin Yang, Yushi Lan, Honghua Chen, Xingang Pan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.14316
• Pdf URL: https://arxiv.org/pdf/2502.14316
• Copy Paste: [[2502.14316]] Textured 3D Regenerative Morphing with 3D Diffusion Prior(https://arxiv.org/abs/2502.14316)
• Keywords: diffusion, generative
• Abstract: Textured 3D morphing creates smooth and plausible interpolation sequences between two 3D objects, focusing on transitions in both shape and texture. This is important for creative applications like visual effects in filmmaking. Previous methods rely on establishing point-to-point correspondences and determining smooth deformation trajectories, which inherently restrict them to shape-only morphing on untextured, topologically aligned datasets. This restriction leads to labor-intensive preprocessing and poor generalization. To overcome these challenges, we propose a method for 3D regenerative morphing using a 3D diffusion prior. Unlike previous methods that depend on explicit correspondences and deformations, our method eliminates the additional need for obtaining correspondence and uses the 3D diffusion prior to generate morphing. Specifically, we introduce a 3D diffusion model and interpolate the source and target information at three levels: initial noise, model parameters, and condition features. We then explore an Attention Fusion strategy to generate more smooth morphing sequences. To further improve the plausibility of semantic interpolation and the generated 3D surfaces, we propose two strategies: (a) Token Reordering, where we match approximate tokens based on semantic analysis to guide implicit correspondences in the denoising process of the diffusion model, and (b) Low-Frequency Enhancement, where we enhance low-frequency signals in the tokens to improve the quality of generated surfaces. Experimental results show that our method achieves superior smoothness and plausibility in 3D morphing across diverse cross-category object pairs, offering a novel regenerative method for 3D morphing with textured representations.

Title: SegAnyPET: Universal Promptable Segmentation from Positron Emission Tomography Images

• Authors: Yichi Zhang, Le Xue, Wenbo Zhang, Lanlan Li, Yuchen Liu, Chen Jiang, Yuan Cheng, Yuan Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.14351
• Pdf URL: https://arxiv.org/pdf/2502.14351
• Copy Paste: [[2502.14351]] SegAnyPET: Universal Promptable Segmentation from Positron Emission Tomography Images(https://arxiv.org/abs/2502.14351)
• Keywords: foundation model
• Abstract: Positron Emission Tomography (PET) imaging plays a crucial role in modern medical diagnostics by revealing the metabolic processes within a patient's body, which is essential for quantification of therapy response and monitoring treatment progress. However, the segmentation of PET images presents unique challenges due to their lower contrast and less distinct boundaries compared to other structural medical modalities. Recent developments in segmentation foundation models have shown superior versatility across diverse natural image segmentation tasks. Despite the efforts of medical adaptations, these works primarily focus on structural medical images with detailed physiological structural information and exhibit poor generalization ability when adapted to molecular PET imaging. In this paper, we collect and construct PETS-5k, the largest PET segmentation dataset to date, comprising 5,731 three-dimensional whole-body PET images and encompassing over 1.3M 2D images. Based on the established dataset, we develop SegAnyPET, a modality-specific 3D foundation model for universal promptable segmentation from PET images. To issue the challenge of discrepant annotation quality of PET images, we adopt a cross prompting confident learning (CPCL) strategy with an uncertainty-guided self-rectification process to robustly learn segmentation from high-quality labeled data and low-quality noisy labeled data. Experimental results demonstrate that SegAnyPET can correctly segment seen and unseen targets using only one or a few prompt points, outperforming state-of-the-art foundation models and task-specific fully supervised models with higher accuracy and strong generalization ability for universal segmentation. As the first foundation model for PET images, we believe that SegAnyPET will advance the applications to various downstream tasks for molecular imaging.

Title: Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment

• Authors: Moxin Li, Yuantao Zhang, Wenjie Wang, Wentao Shi, Zhuo Liu, Fuli Feng, Tat-Seng Chua
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.14354
• Pdf URL: https://arxiv.org/pdf/2502.14354
• Copy Paste: [[2502.14354]] Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment(https://arxiv.org/abs/2502.14354)
• Keywords: self-supervised
• Abstract: Multi-Objective Alignment (MOA) aims to align LLMs' responses with multiple human preference objectives, with Direct Preference Optimization (DPO) emerging as a prominent approach. However, we find that DPO-based MOA approaches suffer from widespread preference conflicts in the data, where different objectives favor different responses. This results in conflicting optimization directions, hindering the optimization on the Pareto Front. To address this, we propose to construct Pareto-optimal responses to resolve preference conflicts. To efficiently obtain and utilize such responses, we propose a self-improving DPO framework that enables LLMs to self-generate and select Pareto-optimal responses for self-supervised preference alignment. Extensive experiments on two datasets demonstrate the superior Pareto Front achieved by our framework compared to various baselines. Code is available at \url{this https URL}.

Title: Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning

• Authors: Huimin Xu, Xin Mao, Feng-Lin Li, Xiaobao Wu, Wang Chen, Wei Zhang, Anh Tuan Luu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.14356
• Pdf URL: https://arxiv.org/pdf/2502.14356
• Copy Paste: [[2502.14356]] Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning(https://arxiv.org/abs/2502.14356)
• Keywords: self-supervised
• Abstract: Direct Preference Optimization (DPO) often struggles with long-chain mathematical reasoning. Existing approaches, such as Step-DPO, typically improve this by focusing on the first erroneous step in the reasoning chain. However, they overlook all other steps and rely heavily on humans or GPT-4 to identify erroneous steps. To address these issues, we propose Full-Step-DPO, a novel DPO framework tailored for mathematical reasoning. Instead of optimizing only the first erroneous step, it leverages step-wise rewards from the entire reasoning chain. This is achieved by training a self-supervised process reward model, which automatically scores each step, providing rewards while avoiding reliance on external signals. Furthermore, we introduce a novel step-wise DPO loss, which dynamically updates gradients based on these step-wise rewards. This endows stronger reasoning capabilities to language models. Extensive evaluations on both in-domain and out-of-domain mathematical reasoning benchmarks across various base language models, demonstrate that Full-Step-DPO achieves superior performance compared to state-of-the-art baselines.

Title: PPO-MI: Efficient Black-Box Model Inversion via Proximal Policy Optimization

• Authors: Xinpeng Shou
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.14370
• Pdf URL: https://arxiv.org/pdf/2502.14370
• Copy Paste: [[2502.14370]] PPO-MI: Efficient Black-Box Model Inversion via Proximal Policy Optimization(https://arxiv.org/abs/2502.14370)
• Keywords: generative
• Abstract: Model inversion attacks pose a significant privacy risk by attempting to reconstruct private training data from trained models. Most of the existing methods either depend on gradient estimation or require white-box access to model parameters, which limits their applicability in practical scenarios. In this paper, we propose PPO-MI, a novel reinforcement learning-based framework for black-box model inversion attacks. Our approach formulates the inversion task as a Markov Decision Process, where an agent navigates the latent space of a generative model to reconstruct private training samples using only model predictions. By employing Proximal Policy Optimization (PPO) with a momentum-based state transition mechanism, along with a reward function balancing prediction accuracy and exploration, PPO-MI ensures efficient latent space exploration and high query efficiency. We conduct extensive experiments illustrates that PPO-MI outperforms the existing methods while require less attack knowledge, and it is robust across various model architectures and datasets. These results underline its effectiveness and generalizability in practical black-box scenarios, raising important considerations for the privacy vulnerabilities of deployed machine learning models.

Title: RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers

• Authors: Ke Cao, Jing Wang, Ao Ma, Jiasong Feng, Zhanjie Zhang, Xuanhua He, Shanyuan Liu, Bo Cheng, Dawei Leng, Yuhui Yin, Jie Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.14377
• Pdf URL: https://arxiv.org/pdf/2502.14377
• Copy Paste: [[2502.14377]] RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers(https://arxiv.org/abs/2502.14377)
• Keywords: diffusion
• Abstract: The Diffusion Transformer plays a pivotal role in advancing text-to-image and text-to-video generation, owing primarily to its inherent scalability. However, existing controlled diffusion transformer methods incur significant parameter and computational overheads and suffer from inefficient resource allocation due to their failure to account for the varying relevance of control information across different transformer layers. To address this, we propose the Relevance-Guided Efficient Controllable Generation framework, RelaCtrl, enabling efficient and resource-optimized integration of control signals into the Diffusion Transformer. First, we evaluate the relevance of each layer in the Diffusion Transformer to the control information by assessing the "ControlNet Relevance Score"-i.e., the impact of skipping each control layer on both the quality of generation and the control effectiveness during inference. Based on the strength of the relevance, we then tailor the positioning, parameter scale, and modeling capacity of the control layers to reduce unnecessary parameters and redundant computations. Additionally, to further improve efficiency, we replace the self-attention and FFN in the commonly used copy block with the carefully designed Two-Dimensional Shuffle Mixer (TDSM), enabling efficient implementation of both the token mixer and channel mixer. Both qualitative and quantitative experimental results demonstrate that our approach achieves superior performance with only 15% of the parameters and computational complexity compared to PixArt-delta. More examples are available at this https URL.

Title: Affinity and Diversity: A Unified Metric for Demonstration Selection via Internal Representations

• Authors: Mariko Kato, Hakaze Cho, Yoshihiro Sakai, Naoya Inoue
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.14380
• Pdf URL: https://arxiv.org/pdf/2502.14380
• Copy Paste: [[2502.14380]] Affinity and Diversity: A Unified Metric for Demonstration Selection via Internal Representations(https://arxiv.org/abs/2502.14380)
• Keywords: in-context
• Abstract: The performance of In-Context Learning (ICL) is highly sensitive to the selected demonstrations. Existing approaches to demonstration selection optimize different objectives, yielding inconsistent results. To address this, we propose a unified metric--affinity and diversity--that leverages ICL model's internal representations. Our experiments show that both affinity and diversity strongly correlate with test accuracies, indicating their effectiveness for demonstration selection. Moreover, we show that our proposed metrics align well with various previous works to unify the inconsistency.

Title: dtaianomaly: A Python library for time series anomaly detection

• Authors: Louis Carpentier, Nick Seeuws, Wannes Meert, Mathias Verbeke
• Subjects: cs.LG, cs.DB
• Abstract URL: https://arxiv.org/abs/2502.14381
• Pdf URL: https://arxiv.org/pdf/2502.14381
• Copy Paste: [[2502.14381]] dtaianomaly: A Python library for time series anomaly detection(https://arxiv.org/abs/2502.14381)
• Keywords: anomaly
• Abstract: dtaianomaly is an open-source Python library for time series anomaly detection, designed to bridge the gap between academic research and real-world applications. Our goal is to (1) accelerate the development of novel state-of-the-art anomaly detection techniques through simple extensibility; (2) offer functionality for large-scale experimental validation; and thereby (3) bring cutting-edge research to business and industry through a standardized API, similar to scikit-learn to lower the entry barrier for both new and experienced users. Besides these key features, dtaianomaly offers (1) a broad range of built-in anomaly detectors, (2) support for time series preprocessing, (3) tools for visual analysis, (4) confidence prediction of anomaly scores, (5) runtime and memory profiling, (6) comprehensive documentation, and (7) cross-platform unit testing. The source code of dtaianomaly, documentation, code examples and installation guides are publicly available at this https URL.

Title: Enhancing Portuguese Variety Identification with Cross-Domain Approaches

• Authors: Hugo Sousa, Rúben Almeida, Purificação Silvano, Inês Cantante, Ricardo Campos, Alípio Jorge
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.14394
• Pdf URL: https://arxiv.org/pdf/2502.14394
• Copy Paste: [[2502.14394]] Enhancing Portuguese Variety Identification with Cross-Domain Approaches(https://arxiv.org/abs/2502.14394)
• Keywords: generative
• Abstract: Recent advances in natural language processing have raised expectations for generative models to produce coherent text across diverse language varieties. In the particular case of the Portuguese language, the predominance of Brazilian Portuguese corpora online introduces linguistic biases in these models, limiting their applicability outside of Brazil. To address this gap and promote the creation of European Portuguese resources, we developed a cross-domain language variety identifier (LVI) to discriminate between European and Brazilian Portuguese. Motivated by the findings of our literature review, we compiled the PtBrVarId corpus, a cross-domain LVI dataset, and study the effectiveness of transformer-based LVI classifiers for cross-domain scenarios. Although this research focuses on two Portuguese varieties, our contribution can be extended to other varieties and languages. We open source the code, corpus, and models to foster further research in this task.

Title: Evaluating Precise Geolocation Inference Capabilities of Vision Language Models

• Authors: Neel Jay, Hieu Minh Nguyen, Trung Dung Hoang, Jacob Haimes
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.14412
• Pdf URL: https://arxiv.org/pdf/2502.14412
• Copy Paste: [[2502.14412]] Evaluating Precise Geolocation Inference Capabilities of Vision Language Models(https://arxiv.org/abs/2502.14412)
• Keywords: foundation model
• Abstract: The prevalence of Vision-Language Models (VLMs) raises important questions about privacy in an era where visual information is increasingly available. While foundation VLMs demonstrate broad knowledge and learned capabilities, we specifically investigate their ability to infer geographic location from previously unseen image data. This paper introduces a benchmark dataset collected from Google Street View that represents its global distribution of coverage. Foundation models are evaluated on single-image geolocation inference, with many achieving median distance errors of <300 km. We further evaluate VLM "agents" with access to supplemental tools, observing up to a 30.6% decrease in distance error. Our findings establish that modern foundation VLMs can act as powerful image geolocation tools, without being specifically trained for this task. When coupled with increasing accessibility of these models, our findings have greater implications for online privacy. We discuss these risks, as well as future work in this area.

Title: Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models

• Authors: Artem Vazhentsev, Lyudmila Rvanova, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.14427
• Pdf URL: https://arxiv.org/pdf/2502.14427
• Copy Paste: [[2502.14427]] Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models(https://arxiv.org/abs/2502.14427)
• Keywords: generative
• Abstract: Uncertainty quantification (UQ) is a prominent approach for eliciting truthful answers from large language models (LLMs). To date, information-based and consistency-based UQ have been the dominant UQ methods for text generation via LLMs. Density-based methods, despite being very effective for UQ in text classification with encoder-based models, have not been very successful with generative LLMs. In this work, we adapt Mahalanobis Distance (MD) - a well-established UQ technique in classification tasks - for text generation and introduce a new supervised UQ method. Our method extracts token embeddings from multiple layers of LLMs, computes MD scores for each token, and uses linear regression trained on these features to provide robust uncertainty scores. Through extensive experiments on eleven datasets, we demonstrate that our approach substantially improves over existing UQ methods, providing accurate and computationally efficient uncertainty scores for both sequence-level selective generation and claim-level fact-checking tasks. Our method also exhibits strong generalization to out-of-domain data, making it suitable for a wide range of LLM-based applications.

Title: How Jailbreak Defenses Work and Ensemble? A Mechanistic Investigation

• Authors: Zhuohang Long, Siyuan Wang, Shujun Liu, Yuhang Lai, Xuanjing Huang, Zhongyu Wei
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.14486
• Pdf URL: https://arxiv.org/pdf/2502.14486
• Copy Paste: [[2502.14486]] How Jailbreak Defenses Work and Ensemble? A Mechanistic Investigation(https://arxiv.org/abs/2502.14486)
• Keywords: generative
• Abstract: Jailbreak attacks, where harmful prompts bypass generative models' built-in safety, raise serious concerns about model vulnerability. While many defense methods have been proposed, the trade-offs between safety and helpfulness, and their application to Large Vision-Language Models (LVLMs), are not well understood. This paper systematically examines jailbreak defenses by reframing the standard generation task as a binary classification problem to assess model refusal tendencies for both harmful and benign queries. We identify two key defense mechanisms: safety shift, which increases refusal rates across all queries, and harmfulness discrimination, which improves the model's ability to distinguish between harmful and benign inputs. Using these mechanisms, we develop two ensemble defense strategies-inter-mechanism ensembles and intra-mechanism ensembles-to balance safety and helpfulness. Experiments on the MM-SafetyBench and MOSSBench datasets with LLaVA-1.5 models show that these strategies effectively improve model safety or optimize the trade-off between safety and helpfulness.

Title: CrossFuse: Learning Infrared and Visible Image Fusion by Cross-Sensor Top-K Vision Alignment and Beyond

• Authors: Yukai Shi, Cidan Shi, Zhipeng Weng, Yin Tian, Xiaoyu Xian, Liang Lin
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.14493
• Pdf URL: https://arxiv.org/pdf/2502.14493
• Copy Paste: [[2502.14493]] CrossFuse: Learning Infrared and Visible Image Fusion by Cross-Sensor Top-K Vision Alignment and Beyond(https://arxiv.org/abs/2502.14493)
• Keywords: self-supervised
• Abstract: Infrared and visible image fusion (IVIF) is increasingly applied in critical fields such as video surveillance and autonomous driving systems. Significant progress has been made in deep learning-based fusion methods. However, these models frequently encounter out-of-distribution (OOD) scenes in real-world applications, which severely impact their performance and reliability. Therefore, addressing the challenge of OOD data is crucial for the safe deployment of these models in open-world environments. Unlike existing research, our focus is on the challenges posed by OOD data in real-world applications and on enhancing the robustness and generalization of models. In this paper, we propose an infrared-visible fusion framework based on Multi-View Augmentation. For external data augmentation, Top-k Selective Vision Alignment is employed to mitigate distribution shifts between datasets by performing RGB-wise transformations on visible images. This strategy effectively introduces augmented samples, enhancing the adaptability of the model to complex real-world scenarios. Additionally, for internal data augmentation, self-supervised learning is established using Weak-Aggressive Augmentation. This enables the model to learn more robust and general feature representations during the fusion process, thereby improving robustness and generalization. Extensive experiments demonstrate that the proposed method exhibits superior performance and robustness across various conditions and environments. Our approach significantly enhances the reliability and stability of IVIF tasks in practical applications.

Title: Generative adversarial networks vs large language models: a comparative study on synthetic tabular data generation

• Authors: Austin A. Barr, Robert Rozman, Eddie Guo
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.14523
• Pdf URL: https://arxiv.org/pdf/2502.14523
• Copy Paste: [[2502.14523]] Generative adversarial networks vs large language models: a comparative study on synthetic tabular data generation(https://arxiv.org/abs/2502.14523)
• Keywords: generative
• Abstract: We propose a new framework for zero-shot generation of synthetic tabular data. Using the large language model (LLM) GPT-4o and plain-language prompting, we demonstrate the ability to generate high-fidelity tabular data without task-specific fine-tuning or access to real-world data (RWD) for pre-training. To benchmark GPT-4o, we compared the fidelity and privacy of LLM-generated synthetic data against data generated with the conditional tabular generative adversarial network (CTGAN), across three open-access datasets: Iris, Fish Measurements, and Real Estate Valuation. Despite the zero-shot approach, GPT-4o outperformed CTGAN in preserving means, 95% confidence intervals, bivariate correlations, and data privacy of RWD, even at amplified sample sizes. Notably, correlations between parameters were consistently preserved with appropriate direction and strength. However, refinement is necessary to better retain distributional characteristics. These findings highlight the potential of LLMs in tabular data synthesis, offering an accessible alternative to generative adversarial networks and variational autoencoders.

Title: Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks

• Authors: Maya Bechler-Speicher, Ben Finkelshtein, Fabrizio Frasca, Luis Müller, Jan Tönshoff, Antoine Siraudin, Viktor Zaverkin, Michael M. Bronstein, Mathias Niepert, Bryan Perozzi, Mikhail Galkin, Christopher Morris
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2502.14546
• Pdf URL: https://arxiv.org/pdf/2502.14546
• Copy Paste: [[2502.14546]] Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks(https://arxiv.org/abs/2502.14546)
• Keywords: foundation model
• Abstract: While machine learning on graphs has demonstrated promise in drug design and molecular property prediction, significant benchmarking challenges hinder its further progress and relevance. Current benchmarking practices often lack focus on transformative, real-world applications, favoring narrow domains like two-dimensional molecular graphs over broader, impactful areas such as combinatorial optimization, relational databases, or chip design. Additionally, many benchmark datasets poorly represent the underlying data, leading to inadequate abstractions and misaligned use cases. Fragmented evaluations and an excessive focus on accuracy further exacerbate these issues, incentivizing overfitting rather than fostering generalizable insights. These limitations have prevented the development of truly useful graph foundation models. This position paper calls for a paradigm shift toward more meaningful benchmarks, rigorous evaluation protocols, and stronger collaboration with domain experts to drive impactful and reliable advances in graph learning research, unlocking the potential of graph learning.

Title: Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling

• Authors: Eric Egli, Matteo Manica, Jannis Born
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.14553
• Pdf URL: https://arxiv.org/pdf/2502.14553
• Copy Paste: [[2502.14553]] Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling(https://arxiv.org/abs/2502.14553)
• Keywords: foundation model
• Abstract: Bytes form the basis of the digital world and thus are a promising building block for multimodal foundation models. Recently, Byte Language Models (BLMs) have emerged to overcome tokenization, yet the excessive length of bytestreams requires new architectural paradigms. Therefore, we present the Multiscale Byte Language Model (MBLM), a model-agnostic hierarchical decoder stack that allows training with context windows of $5$M bytes on single GPU in full model precision. We thoroughly examine MBLM's performance with Transformer and Mamba blocks on both unimodal and multimodal tasks. Our experiments demonstrate that hybrid architectures are efficient in handling extremely long byte sequences during training while achieving near-linear generational efficiency. To the best of our knowledge, we present the first evaluation of BLMs on visual Q\&A tasks and find that, despite serializing images and the absence of an encoder, a MBLM with pure next token prediction can match custom CNN-LSTM architectures with designated classification heads. We show that MBLMs exhibit strong adaptability in integrating diverse data representations, including pixel and image filestream bytes, underlining their potential toward omnimodal foundation models. Source code is publicly available at: this https URL

Title: Can LLMs Predict Citation Intent? An Experimental Analysis of In-context Learning and Fine-tuning on Open LLMs

• Authors: Paris Koloveas, Serafeim Chatzopoulos, Thanasis Vergoulis, Christos Tryfonopoulos
• Subjects: cs.CL, cs.DL
• Abstract URL: https://arxiv.org/abs/2502.14561
• Pdf URL: https://arxiv.org/pdf/2502.14561
• Copy Paste: [[2502.14561]] Can LLMs Predict Citation Intent? An Experimental Analysis of In-context Learning and Fine-tuning on Open LLMs(https://arxiv.org/abs/2502.14561)
• Keywords: in-context
• Abstract: This work investigates the ability of open Large Language Models (LLMs) to predict citation intent through in-context learning and fine-tuning. Unlike traditional approaches that rely on pre-trained models like SciBERT, which require extensive domain-specific pretraining and specialized architectures, we demonstrate that general-purpose LLMs can be adapted to this task with minimal task-specific data. We evaluate twelve model variations across five prominent open LLM families using zero, one, few, and many-shot prompting to assess performance across scenarios. Our experimental study identifies the top-performing model through extensive experimentation of in-context learning-related parameters, which we fine-tune to further enhance task performance. The results highlight the strengths and limitations of LLMs in recognizing citation intents, providing valuable insights for model selection and prompt engineering. Additionally, we make our end-to-end evaluation framework and models openly available for future use.

Title: Self-supervised Monocular Depth Estimation Robust to Reflective Surface Leveraged by Triplet Mining

• Authors: Wonhyeok Choi, Kyumin Hwang, Wei Peng, Minwoo Choi, Sunghoon Im
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.14573
• Pdf URL: https://arxiv.org/pdf/2502.14573
• Copy Paste: [[2502.14573]] Self-supervised Monocular Depth Estimation Robust to Reflective Surface Leveraged by Triplet Mining(https://arxiv.org/abs/2502.14573)
• Keywords: self-supervised
• Abstract: Self-supervised monocular depth estimation (SSMDE) aims to predict the dense depth map of a monocular image, by learning depth from RGB image sequences, eliminating the need for ground-truth depth labels. Although this approach simplifies data acquisition compared to supervised methods, it struggles with reflective surfaces, as they violate the assumptions of Lambertian reflectance, leading to inaccurate training on such surfaces. To tackle this problem, we propose a novel training strategy for an SSMDE by leveraging triplet mining to pinpoint reflective regions at the pixel level, guided by the camera geometry between different viewpoints. The proposed reflection-aware triplet mining loss specifically penalizes the inappropriate photometric error minimization on the localized reflective regions while preserving depth accuracy in non-reflective areas. We also incorporate a reflection-aware knowledge distillation method that enables a student model to selectively learn the pixel-level knowledge from reflective and non-reflective regions. This results in robust depth estimation across areas. Evaluation results on multiple datasets demonstrate that our method effectively enhances depth quality on reflective surfaces and outperforms state-of-the-art SSMDE baselines.

Title: A Theory for Conditional Generative Modeling on Multiple Data Sources

• Authors: Rongzhen Wang, Yan Zhang, Chenyu Zheng, Chongxuan Li, Guoqiang Wu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.14583
• Pdf URL: https://arxiv.org/pdf/2502.14583
• Copy Paste: [[2502.14583]] A Theory for Conditional Generative Modeling on Multiple Data Sources(https://arxiv.org/abs/2502.14583)
• Keywords: generative
• Abstract: The success of large generative models has driven a paradigm shift, leveraging massive multi-source data to enhance model capabilities. However, the interaction among these sources remains theoretically underexplored. This paper takes the first step toward a rigorous analysis of multi-source training in conditional generative modeling, where each condition represents a distinct data source. Specifically, we establish a general distribution estimation error bound in average total variation distance for conditional maximum likelihood estimation based on the bracketing number. Our result shows that when source distributions share certain similarities and the model is expressive enough, multi-source training guarantees a sharper bound than single-source training. We further instantiate the general theory on conditional Gaussian estimation and deep generative models including autoregressive and flexible energy-based models, by characterizing their bracketing numbers. The results highlight that the number of sources and similarity among source distributions improve the advantage of multi-source training. Simulations and real-world experiments validate our theory. Code is available at: \url{this https URL}.

Title: PEARL: Towards Permutation-Resilient LLMs

• Authors: Liang Chen, Li Shen, Yang Deng, Xiaoyan Zhao, Bin Liang, Kam-Fai Wong
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.14628
• Pdf URL: https://arxiv.org/pdf/2502.14628
• Copy Paste: [[2502.14628]] PEARL: Towards Permutation-Resilient LLMs(https://arxiv.org/abs/2502.14628)
• Keywords: in-context
• Abstract: The in-context learning (ICL) capability of large language models (LLMs) enables them to perform challenging tasks using provided demonstrations. However, ICL is highly sensitive to the ordering of demonstrations, leading to instability in predictions. This paper shows that this vulnerability can be exploited to design a natural attack - difficult for model providers to detect - that achieves nearly 80% success rate on LLaMA-3 by simply permuting the demonstrations. Existing mitigation methods primarily rely on post-processing and fail to enhance the model's inherent robustness to input permutations, raising concerns about safety and reliability of LLMs. To address this issue, we propose Permutation-resilient learning (PEARL), a novel framework based on distributionally robust optimization (DRO), which optimizes model performance against the worst-case input permutation. Specifically, PEARL consists of a permutation-proposal network (P-Net) and the LLM. The P-Net generates the most challenging permutations by treating it as an optimal transport problem, which is solved using an entropy-constrained Sinkhorn algorithm. Through minimax optimization, the P-Net and the LLM iteratively optimize against each other, progressively improving the LLM's robustness. Experiments on synthetic pre-training and real-world instruction tuning tasks demonstrate that PEARL effectively mitigates permutation attacks and enhances performance. Notably, despite being trained on fewer shots and shorter contexts, PEARL achieves performance gains of up to 40% when scaled to many-shot and long-context scenarios, highlighting its efficiency and generalization capabilities.

Title: ReQFlow: Rectified Quaternion Flow for Efficient and High-Quality Protein Backbone Generation

• Authors: Angxiao Yue, Zichong Wang, Hongteng Xu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.14637
• Pdf URL: https://arxiv.org/pdf/2502.14637
• Copy Paste: [[2502.14637]] ReQFlow: Rectified Quaternion Flow for Efficient and High-Quality Protein Backbone Generation(https://arxiv.org/abs/2502.14637)
• Keywords: diffusion, generative
• Abstract: Protein backbone generation plays a central role in de novo protein design and is significant for many biological and medical applications. Although diffusion and flow-based generative models provide potential solutions to this challenging task, they often generate proteins with undesired designability and suffer computational inefficiency. In this study, we propose a novel rectified quaternion flow (ReQFlow) matching method for fast and high-quality protein backbone generation. In particular, our method generates a local translation and a 3D rotation from random noise for each residue in a protein chain, which represents each 3D rotation as a unit quaternion and constructs its flow by spherical linear interpolation (SLERP) in an exponential format. We train the model by quaternion flow (QFlow) matching with guaranteed numerical stability and rectify the QFlow model to accelerate its inference and improve the designability of generated protein backbones, leading to the proposed ReQFlow model. Experiments show that ReQFlow achieves state-of-the-art performance in protein backbone generation while requiring much fewer sampling steps and significantly less inference time (e.g., being 37x faster than RFDiffusion and 62x faster than Genie2 when generating a backbone of length 300), demonstrating its effectiveness and efficiency. The code is available at this https URL.

Title: LIFT: Improving Long Context Understanding of Large Language Models through Long Input Fine-Tuning

• Authors: Yansheng Mao, Yufei Xu, Jiaqi Li, Fanxu Meng, Haotong Yang, Zilong Zheng, Xiyuan Wang, Muhan Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.14644
• Pdf URL: https://arxiv.org/pdf/2502.14644
• Copy Paste: [[2502.14644]] LIFT: Improving Long Context Understanding of Large Language Models through Long Input Fine-Tuning(https://arxiv.org/abs/2502.14644)
• Keywords: in-context
• Abstract: Long context understanding remains challenging for large language models due to their limited context windows. This paper presents Long Input Fine-Tuning (LIFT), a novel framework for long-context modeling that can improve the long-context performance of arbitrary (short-context) LLMs by dynamically adapting model parameters based on the long input. Importantly, LIFT, rather than endlessly extending the context window size to accommodate increasingly longer inputs in context, chooses to store and absorb the long input in parameter. By fine-tuning the long input into model parameters, LIFT allows short-context LLMs to answer questions even when the required information is not provided in the context during inference. Furthermore, to enhance LIFT performance while maintaining the original in-context learning (ICL) capabilities, we introduce Gated Memory, a specialized attention adapter that automatically balances long input memorization and ICL. We provide a comprehensive analysis of the strengths and limitations of LIFT on long context understanding, offering valuable directions for future research.

Title: MAGO-SP: Detection and Correction of Water-Fat Swaps in Magnitude-Only VIBE MRI

• Authors: Robert Graf, Hendrik Möller, Sophie Starck, Matan Atad, Philipp Braun, Jonathan Stelter, Annette Peters, Lilian Krist, Stefan N. Willich, Henry Völzke, Robin Bülow, Klaus Berger, Tobias Pischon, Thoralf Niendorf, Johannes Paetzold, Dimitrios Karampinos, Daniel Rueckert, Jan Kirschke
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.14659
• Pdf URL: https://arxiv.org/pdf/2502.14659
• Copy Paste: [[2502.14659]] MAGO-SP: Detection and Correction of Water-Fat Swaps in Magnitude-Only VIBE MRI(https://arxiv.org/abs/2502.14659)
• Keywords: diffusion
• Abstract: Volume Interpolated Breath-Hold Examination (VIBE) MRI generates images suitable for water and fat signal composition estimation. While the two-point VIBE provides water-fat-separated images, the six-point VIBE allows estimation of the effective transversal relaxation rate R2* and the proton density fat fraction (PDFF), which are imaging markers for health and disease. Ambiguity during signal reconstruction can lead to water-fat swaps. This shortcoming challenges the application of VIBE-MRI for automated PDFF analyses of large-scale clinical data and of population studies. This study develops an automated pipeline to detect and correct water-fat swaps in non-contrast-enhanced VIBE images. Our three-step pipeline begins with training a segmentation network to classify volumes as "fat-like" or "water-like," using synthetic water-fat swaps generated by merging fat and water volumes with Perlin noise. Next, a denoising diffusion image-to-image network predicts water volumes as signal priors for correction. Finally, we integrate this prior into a physics-constrained model to recover accurate water and fat signals. Our approach achieves a < 1% error rate in water-fat swap detection for a 6-point VIBE. Notably, swaps disproportionately affect individuals in the Underweight and Class 3 Obesity BMI categories. Our correction algorithm ensures accurate solution selection in chemical phase MRIs, enabling reliable PDFF estimation. This forms a solid technical foundation for automated large-scale population imaging analysis.

Title: Data-Constrained Synthesis of Training Data for De-Identification

• Authors: Thomas Vakili, Aron Henriksson, Hercules Dalianis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.14677
• Pdf URL: https://arxiv.org/pdf/2502.14677
• Copy Paste: [[2502.14677]] Data-Constrained Synthesis of Training Data for De-Identification(https://arxiv.org/abs/2502.14677)
• Keywords: generative
• Abstract: Many sensitive domains -- such as the clinical domain -- lack widely available datasets due to privacy risks. The increasing generative capabilities of large language models (LLMs) have made synthetic datasets a viable path forward. In this study, we domain-adapt LLMs to the clinical domain and generate synthetic clinical texts that are machine-annotated with tags for personally identifiable information using capable encoder-based NER models. The synthetic corpora are then used to train synthetic NER models. The results show that training NER models using synthetic corpora incurs only a small drop in predictive performance. The limits of this process are investigated in a systematic ablation study -- using both Swedish and Spanish data. Our analysis shows that smaller datasets can be sufficient for domain-adapting LLMs for data synthesis. Instead, the effectiveness of this process is almost entirely contingent on the performance of the machine-annotating NER models trained using the original data.

Title: Bridging the Gap: Transforming Natural Language Questions into SQL Queries via Abstract Query Pattern and Contextual Schema Markup

• Authors: Yonghui Kong, Hongbing Hu, Dan Zhang, Siyuan Chai, Fan Zhang, Wei Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.14682
• Pdf URL: https://arxiv.org/pdf/2502.14682
• Copy Paste: [[2502.14682]] Bridging the Gap: Transforming Natural Language Questions into SQL Queries via Abstract Query Pattern and Contextual Schema Markup(https://arxiv.org/abs/2502.14682)
• Keywords: in-context
• Abstract: Large language models have demonstrated excellent performance in many tasks, including Text-to-SQL, due to their powerful in-context learning capabilities. They are becoming the mainstream approach for Text-to-SQL. However, these methods still have a significant gap compared to human performance, especially on complex questions. As the complexity of questions increases, the gap between questions and SQLs increases. We identify two important gaps: the structural mapping gap and the lexical mapping gap. To tackle these two gaps, we propose PAS-SQL, an efficient SQL generation pipeline based on LLMs, which alleviates gaps through Abstract Query Pattern (AQP) and Contextual Schema Markup (CSM). AQP aims to obtain the structural pattern of the question by removing database-related information, which enables us to find structurally similar demonstrations. CSM aims to associate database-related text span in the question with specific tables or columns in the database, which alleviates the lexical mapping gap. Experimental results on the Spider and BIRD datasets demonstrate the effectiveness of our proposed method. Specifically, PAS-SQL + GPT-4o sets a new state-of-the-art on the Spider benchmark with an execution accuracy of 87.9\%, and achieves leading results on the BIRD dataset with an execution accuracy of 64.67\%.

Title: Not All Data are Good Labels: On the Self-supervised Labeling for Time Series Forecasting

• Authors: Yuxuan Yang, Dalin Zhang, Yuxuan Liang, Hua Lu, Huan Li, Gang Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.14704
• Pdf URL: https://arxiv.org/pdf/2502.14704
• Copy Paste: [[2502.14704]] Not All Data are Good Labels: On the Self-supervised Labeling for Time Series Forecasting(https://arxiv.org/abs/2502.14704)
• Keywords: self-supervised
• Abstract: Time Series Forecasting (TSF) is a crucial task in various domains, yet existing TSF models rely heavily on high-quality data and insufficiently exploit all available data. This paper explores a novel self-supervised approach to re-label time series datasets by inherently constructing candidate datasets. During the optimization of a simple reconstruction network, intermediates are used as pseudo labels in a self-supervised paradigm, improving generalization for any predictor. We introduce the Self-Correction with Adaptive Mask (SCAM), which discards overfitted components and selectively replaces them with pseudo labels generated from reconstructions. Additionally, we incorporate Spectral Norm Regularization (SNR) to further suppress overfitting from a loss landscape perspective. Our experiments on eleven real-world datasets demonstrate that SCAM consistently improves the performance of various backbone models. This work offers a new perspective on constructing datasets and enhancing the generalization of TSF models through self-supervised learning.

Title: DC-ControlNet: Decoupling Inter- and Intra-Element Conditions in Image Generation with Diffusion Models

• Authors: Hongji Yang, Wencheng Han, Yucheng Zhou, Jianbing Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.14779
• Pdf URL: https://arxiv.org/pdf/2502.14779
• Copy Paste: [[2502.14779]] DC-ControlNet: Decoupling Inter- and Intra-Element Conditions in Image Generation with Diffusion Models(https://arxiv.org/abs/2502.14779)
• Keywords: diffusion, generative
• Abstract: In this paper, we introduce DC (Decouple)-ControlNet, a highly flexible and precisely controllable framework for multi-condition image generation. The core idea behind DC-ControlNet is to decouple control conditions, transforming global control into a hierarchical system that integrates distinct elements, contents, and layouts. This enables users to mix these individual conditions with greater flexibility, leading to more efficient and accurate image generation control. Previous ControlNet-based models rely solely on global conditions, which affect the entire image and lack the ability of element- or region-specific control. This limitation reduces flexibility and can cause condition misunderstandings in multi-conditional image generation. To address these challenges, we propose both intra-element and Inter-element Controllers in DC-ControlNet. The Intra-Element Controller handles different types of control signals within individual elements, accurately describing the content and layout characteristics of the object. For interactions between elements, we introduce the Inter-Element Controller, which accurately handles multi-element interactions and occlusion based on user-defined relationships. Extensive evaluations show that DC-ControlNet significantly outperforms existing ControlNet models and Layout-to-Image generative models in terms of control flexibility and precision in multi-condition control.

Title: SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

• Authors: Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, Xiaohua Zhai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.14786
• Pdf URL: https://arxiv.org/pdf/2502.14786
• Copy Paste: [[2502.14786]] SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features(https://arxiv.org/abs/2502.14786)
• Keywords: self-supervised
• Abstract: We introduce SigLIP 2, a family of new multilingual vision-language encoders that build on the success of the original SigLIP. In this second iteration, we extend the original image-text training objective with several prior, independently developed techniques into a unified recipe -- this includes captioning-based pretraining, self-supervised losses (self-distillation, masked prediction) and online data curation. With these changes, SigLIP 2 models outperform their SigLIP counterparts at all model scales in core capabilities, including zero-shot classification, image-text retrieval, and transfer performance when extracting visual representations for Vision-Language Models (VLMs). Furthermore, the new training recipe leads to significant improvements on localization and dense prediction tasks. We also train variants which support multiple resolutions and preserve the input's native aspect ratio. Finally, we train on a more diverse data-mixture that includes de-biasing techniques, leading to much better multilingual understanding and improved fairness. To allow users to trade off inference cost with performance, we release model checkpoints at four sizes: ViT-B (86M), L (303M), So400m (400M), and g (1B).

Title: Rapid Word Learning Through Meta In-Context Learning

• Authors: Wentao Wang, Guangyuan Jiang, Tal Linzen, Brenden M. Lake
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.14791
• Pdf URL: https://arxiv.org/pdf/2502.14791
• Copy Paste: [[2502.14791]] Rapid Word Learning Through Meta In-Context Learning(https://arxiv.org/abs/2502.14791)
• Keywords: generative, in-context
• Abstract: Humans can quickly learn a new word from a few illustrative examples, and then systematically and flexibly use it in novel contexts. Yet the abilities of current language models for few-shot word learning, and methods for improving these abilities, are underexplored. In this study, we introduce a novel method, Meta-training for IN-context learNing Of Words (Minnow). This method trains language models to generate new examples of a word's usage given a few in-context examples, using a special placeholder token to represent the new word. This training is repeated on many new words to develop a general word-learning ability. We find that training models from scratch with Minnow on human-scale child-directed language enables strong few-shot word learning, comparable to a large language model (LLM) pre-trained on orders of magnitude more data. Furthermore, through discriminative and generative evaluations, we demonstrate that finetuning pre-trained LLMs with Minnow improves their ability to discriminate between new words, identify syntactic categories of new words, and generate reasonable new usages and definitions for new words, based on one or a few in-context examples. These findings highlight the data efficiency of Minnow and its potential to improve language model performance in word learning tasks.

Title: RendBEV: Semantic Novel View Synthesis for Self-Supervised Bird's Eye View Segmentation

• Authors: Henrique Piñeiro Monteagudo, Leonardo Taccari, Aurel Pjetri, Francesco Sambo, Samuele Salti
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.14792
• Pdf URL: https://arxiv.org/pdf/2502.14792
• Copy Paste: [[2502.14792]] RendBEV: Semantic Novel View Synthesis for Self-Supervised Bird's Eye View Segmentation(https://arxiv.org/abs/2502.14792)
• Keywords: self-supervised
• Abstract: Bird's Eye View (BEV) semantic maps have recently garnered a lot of attention as a useful representation of the environment to tackle assisted and autonomous driving tasks. However, most of the existing work focuses on the fully supervised setting, training networks on large annotated datasets. In this work, we present RendBEV, a new method for the self-supervised training of BEV semantic segmentation networks, leveraging differentiable volumetric rendering to receive supervision from semantic perspective views computed by a 2D semantic segmentation model. Our method enables zero-shot BEV semantic segmentation, and already delivers competitive results in this challenging setting. When used as pretraining to then fine-tune on labeled BEV ground-truth, our method significantly boosts performance in low-annotation regimes, and sets a new state of the art when fine-tuning on all available labels.

Title: A Survey on Text-Driven 360-Degree Panorama Generation

• Authors: Hai Wang, Xiaoyu Xiang, Weihao Xia, Jing-Hao Xue
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.14799
• Pdf URL: https://arxiv.org/pdf/2502.14799
• Copy Paste: [[2502.14799]] A Survey on Text-Driven 360-Degree Panorama Generation(https://arxiv.org/abs/2502.14799)
• Keywords: diffusion
• Abstract: The advent of text-driven 360-degree panorama generation, enabling the synthesis of 360-degree panoramic images directly from textual descriptions, marks a transformative advancement in immersive visual content creation. This innovation significantly simplifies the traditionally complex process of producing such content. Recent progress in text-to-image diffusion models has accelerated the rapid development in this emerging field. This survey presents a comprehensive review of text-driven 360-degree panorama generation, offering an in-depth analysis of state-of-the-art algorithms and their expanding applications in 360-degree 3D scene generation. Furthermore, we critically examine current limitations and propose promising directions for future research. A curated project page with relevant resources and research papers is available at this https URL.

Title: AVD2: Accident Video Diffusion for Accident Video Description

• Authors: Cheng Li, Keyuan Zhou, Tong Liu, Yu Wang, Mingqiao Zhuang, Huan-ang Gao, Bu Jin, Hao Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.14801
• Pdf URL: https://arxiv.org/pdf/2502.14801
• Copy Paste: [[2502.14801]] AVD2: Accident Video Diffusion for Accident Video Description(https://arxiv.org/abs/2502.14801)
• Keywords: diffusion
• Abstract: Traffic accidents present complex challenges for autonomous driving, often featuring unpredictable scenarios that hinder accurate system interpretation and this http URL, prevailing methodologies fall short in elucidating the causes of accidents and proposing preventive measures due to the paucity of training data specific to accident this http URL this work, we introduce AVD2 (Accident Video Diffusion for Accident Video Description), a novel framework that enhances accident scene understanding by generating accident videos that aligned with detailed natural language descriptions and reasoning, resulting in the contributed EMM-AU (Enhanced Multi-Modal Accident Video Understanding) dataset. Empirical results reveal that the integration of the EMM-AU dataset establishes state-of-the-art performance across both automated metrics and human evaluations, markedly advancing the domains of accident analysis and prevention. Project resources are available at this https URL

Title: Improving the Diffusability of Autoencoders

• Authors: Ivan Skorokhodov, Sharath Girish, Benran Hu, Willi Menapace, Yanyu Li, Rameen Abdal, Sergey Tulyakov, Aliaksandr Siarohin
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.14831
• Pdf URL: https://arxiv.org/pdf/2502.14831
• Copy Paste: [[2502.14831]] Improving the Diffusability of Autoencoders(https://arxiv.org/abs/2502.14831)
• Keywords: diffusion
• Abstract: Latent diffusion models have emerged as the leading approach for generating high-quality images and videos, utilizing compressed latent representations to reduce the computational burden of the diffusion process. While recent advancements have primarily focused on scaling diffusion backbones and improving autoencoder reconstruction quality, the interaction between these components has received comparatively less attention. In this work, we perform a spectral analysis of modern autoencoders and identify inordinate high-frequency components in their latent spaces, which are especially pronounced in the autoencoders with a large bottleneck channel size. We hypothesize that this high-frequency component interferes with the coarse-to-fine nature of the diffusion synthesis process and hinders the generation quality. To mitigate the issue, we propose scale equivariance: a simple regularization strategy that aligns latent and RGB spaces across frequencies by enforcing scale equivariance in the decoder. It requires minimal code changes and only up to 20K autoencoder fine-tuning steps, yet significantly improves generation quality, reducing FID by 19% for image generation on ImageNet-1K 256x256 and FVD by at least 44% for video generation on Kinetics-700 17x256x256.

Title: Probabilistic Robustness in Deep Learning: A Concise yet Comprehensive Guide

• Authors: Xingyu Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.14833
• Pdf URL: https://arxiv.org/pdf/2502.14833
• Copy Paste: [[2502.14833]] Probabilistic Robustness in Deep Learning: A Concise yet Comprehensive Guide(https://arxiv.org/abs/2502.14833)
• Keywords: generative
• Abstract: Deep learning (DL) has demonstrated significant potential across various safety-critical applications, yet ensuring its robustness remains a key challenge. While adversarial robustness has been extensively studied in worst-case scenarios, probabilistic robustness (PR) offers a more practical perspective by quantifying the likelihood of failures under stochastic perturbations. This paper provides a concise yet comprehensive overview of PR, covering its formal definitions, evaluation and enhancement methods. We introduce a reformulated ''min-max'' optimisation framework for adversarial training specifically designed to improve PR. Furthermore, we explore the integration of PR verification evidence into system-level safety assurance, addressing challenges in translating DL model-level robustness to system-level claims. Finally, we highlight open research questions, including benchmarking PR evaluation methods, extending PR to generative AI tasks, and developing rigorous methodologies and case studies for system-level integration.