2025-02-07

Title: Teaching Language Models to Critique via Reinforcement Learning

• Authors: Zhihui Xie, Jie chen, Liyu Chen, Weichao Mao, Jingjing Xu, Lingpeng Kong
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03492
• Pdf URL: https://arxiv.org/pdf/2502.03492
• Copy Paste: [[2502.03492]] Teaching Language Models to Critique via Reinforcement Learning(https://arxiv.org/abs/2502.03492)
• Keywords: generative
• Abstract: Teaching large language models (LLMs) to critique and refine their outputs is crucial for building systems that can iteratively improve, yet it is fundamentally limited by the ability to provide accurate judgments and actionable suggestions. In this work, we study LLM critics for code generation and propose $\texttt{CTRL}$, a framework for $\texttt{C}$ritic $\texttt{T}$raining via $\texttt{R}$einforcement $\texttt{L}$earning, which trains a critic model to generate feedback that maximizes correction performance for a fixed generator model without human supervision. Our results demonstrate that critics trained with $\texttt{CTRL}$ significantly enhance pass rates and mitigate compounding errors across both base and stronger generator models. Furthermore, we show that these critic models act as accurate generative reward models and enable test-time scaling through iterative critique-revision, achieving up to 106.1% relative improvements across challenging code generation benchmarks.

Title: Path Planning for Masked Diffusion Model Sampling

• Authors: Fred Zhangzhi Peng, Zachary Bezemek, Sawan Patel, Sherwood Yao, Jarrid Rector-Brooks, Alexander Tong, Pranam Chatterjee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03540
• Pdf URL: https://arxiv.org/pdf/2502.03540
• Copy Paste: [[2502.03540]] Path Planning for Masked Diffusion Model Sampling(https://arxiv.org/abs/2502.03540)
• Keywords: diffusion, generative, in-context
• Abstract: In this paper, we investigate how the order in which tokens are unmasked during masked diffusion models (MDMs) inference affects generative quality. We derive an expanded evidence lower bound (ELBO) that introduces a planner, responsible for selecting which tokens to unmask at each step. Our analysis suggests that alternative unmasking strategies can improve generative performance. Based on these insights, we propose Path Planning (P2), a sampling framework that leverages pre-trained BERT or the denoiser itself to guide unmasking decisions. P2 generalizes all known MDM sampling strategies and enables significant improvements across diverse domains including language generation (in-context learning, code generation, story infilling, mathematical reasoning, reverse curse correction) and biological sequence generation (protein and RNA sequences).

Title: DynVFX: Augmenting Real Videos with Dynamic Content

• Authors: Danah Yatim, Rafail Fridman, Omer Bar-Tal, Tali Dekel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03621
• Pdf URL: https://arxiv.org/pdf/2502.03621
• Copy Paste: [[2502.03621]] DynVFX: Augmenting Real Videos with Dynamic Content(https://arxiv.org/abs/2502.03621)
• Keywords: diffusion
• Abstract: We present a method for augmenting real-world videos with newly generated dynamic content. Given an input video and a simple user-provided text instruction describing the desired content, our method synthesizes dynamic objects or complex scene effects that naturally interact with the existing scene over time. The position, appearance, and motion of the new content are seamlessly integrated into the original footage while accounting for camera motion, occlusions, and interactions with other dynamic objects in the scene, resulting in a cohesive and realistic output video. We achieve this via a zero-shot, training-free framework that harnesses a pre-trained text-to-video diffusion transformer to synthesize the new content and a pre-trained Vision Language Model to envision the augmented scene in detail. Specifically, we introduce a novel inference-based method that manipulates features within the attention mechanism, enabling accurate localization and seamless integration of the new content while preserving the integrity of the original scene. Our method is fully automated, requiring only a simple user instruction. We demonstrate its effectiveness on a wide range of edits applied to real-world videos, encompassing diverse objects and scenarios involving both camera and object motion.

Title: Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach

• Authors: Yunuo Chen, Junli Cao, Anil Kag, Vidit Goel, Sergei Korolev, Chenfanfu Jiang, Sergey Tulyakov, Jian Ren
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03639
• Pdf URL: https://arxiv.org/pdf/2502.03639
• Copy Paste: [[2502.03639]] Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach(https://arxiv.org/abs/2502.03639)
• Keywords: diffusion
• Abstract: We present a novel video generation framework that integrates 3-dimensional geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D point trajectories and align them in pixel space. The resulting 3D-aware video dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling it to track 2D objects with 3D Cartesian coordinates. Building on this, we regularize the shape and motion of objects in the video to eliminate undesired artifacts, \eg, nonphysical deformation. Consequently, we enhance the quality of generated RGB videos and alleviate common issues like object morphing, which are prevalent in current video models due to a lack of shape awareness. With our 3D augmentation and regularization, our model is capable of handling contact-rich scenarios such as task-oriented videos. These videos involve complex interactions of solids, where 3D information is essential for perceiving deformation and contact. Furthermore, our model improves the overall quality of video generation by promoting the 3D consistency of moving objects and reducing abrupt changes in shape and motion.

Title: Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics

• Authors: Indrashis Das, Mahmoud Safari, Steven Adriaensen, Frank Hutter
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03654
• Pdf URL: https://arxiv.org/pdf/2502.03654
• Copy Paste: [[2502.03654]] Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics(https://arxiv.org/abs/2502.03654)
• Keywords: diffusion
• Abstract: Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuron problem, which has been mitigated by variants such as LeakyReLU, PReLU, and ELU that better handle negative neuron outputs. Recently, self-gated activations like GELU and Swish have emerged as state-of-the-art alternatives, leveraging their smoothness to ensure stable gradient flow and prevent neuron inactivity. In this work, we introduce the Gompertz Linear Unit (GoLU), a novel self-gated activation function defined as $\mathrm{GoLU}(x) = x \, \mathrm{Gompertz}(x)$, where $\mathrm{Gompertz}(x) = e^{-e^{-x}}$. The GoLU activation leverages the asymmetry in the Gompertz function to reduce variance in the latent space more effectively compared to GELU and Swish, while preserving robust gradient flow. Extensive experiments across diverse tasks, including Image Classification, Language Modeling, Semantic Segmentation, Object Detection, Instance Segmentation, and Diffusion, highlight GoLU's superior performance relative to state-of-the-art activation functions, establishing GoLU as a robust alternative to existing activation functions.

Title: Privacy-Preserving Generative Models: A Comprehensive Survey

• Authors: Debalina Padariya, Isabel Wagner, Aboozar Taherkhani, Eerke Boiten
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2502.03668
• Pdf URL: https://arxiv.org/pdf/2502.03668
• Copy Paste: [[2502.03668]] Privacy-Preserving Generative Models: A Comprehensive Survey(https://arxiv.org/abs/2502.03668)
• Keywords: generative
• Abstract: Despite the generative model's groundbreaking success, the need to study its implications for privacy and utility becomes more urgent. Although many studies have demonstrated the privacy threats brought by GANs, no existing survey has systematically categorized the privacy and utility perspectives of GANs and VAEs. In this article, we comprehensively study privacy-preserving generative models, articulating the novel taxonomies for both privacy and utility metrics by analyzing 100 research publications. Finally, we discuss the current challenges and future research directions that help new researchers gain insight into the underlying concepts.

Title: Unrealized Expectations: Comparing AI Methods vs Classical Algorithms for Maximum Independent Set

• Authors: Yikai Wu, Haoyu Zhao, Sanjeev Arora
• Subjects: cs.LG, cs.AI, cs.DM, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.03669
• Pdf URL: https://arxiv.org/pdf/2502.03669
• Copy Paste: [[2502.03669]] Unrealized Expectations: Comparing AI Methods vs Classical Algorithms for Maximum Independent Set(https://arxiv.org/abs/2502.03669)
• Keywords: generative
• Abstract: AI methods, such as generative models and reinforcement learning, have recently been applied to combinatorial optimization (CO) problems, especially NP-hard ones. This paper compares such GPU-based methods with classical CPU-based methods on Maximum Independent Set (MIS). Experiments on standard graph families show that AI-based algorithms fail to outperform and, in many cases, to match the solution quality of the state-of-art classical solver KaMIS running on a single CPU. Some GPU-based methods even perform similarly to the simplest heuristic, degree-based greedy. Even with post-processing techniques like local search, AI-based methods still perform worse than CPU-based solvers. We develop a new mode of analysis to reveal that non-backtracking AI methods, e.g. LTFT (which is based on GFlowNets), end up reasoning similarly to the simplest degree-based greedy approach, and thus worse than KaMIS. We also find that CPU-based algorithms, notably KaMIS, have strong performance on sparse random graphs, which appears to refute a well-known conjectured upper bound for efficient algorithms from Coja-Oghlan & Efthymiou (2015).

Title: Advancing Reasoning in Large Language Models: Promising Methods and Approaches

• Authors: Avinash Patil
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03671
• Pdf URL: https://arxiv.org/pdf/2502.03671
• Copy Paste: [[2502.03671]] Advancing Reasoning in Large Language Models: Promising Methods and Approaches(https://arxiv.org/abs/2502.03671)
• Keywords: self-supervised
• Abstract: Large Language Models (LLMs) have succeeded remarkably in various natural language processing (NLP) tasks, yet their reasoning capabilities remain a fundamental challenge. While LLMs exhibit impressive fluency and factual recall, their ability to perform complex reasoning-spanning logical deduction, mathematical problem-solving, commonsense inference, and multi-step reasoning-often falls short of human expectations. This survey provides a comprehensive review of emerging techniques enhancing reasoning in LLMs. We categorize existing methods into key approaches, including prompting strategies (e.g., Chain-of-Thought reasoning, Self-Consistency, and Tree-of-Thought reasoning), architectural innovations (e.g., retrieval-augmented models, modular reasoning networks, and neuro-symbolic integration), and learning paradigms (e.g., fine-tuning with reasoning-specific datasets, reinforcement learning, and self-supervised reasoning objectives). Additionally, we explore evaluation frameworks used to assess reasoning in LLMs and highlight open challenges, such as hallucinations, robustness, and reasoning generalization across diverse tasks. By synthesizing recent advancements, this survey aims to provide insights into promising directions for future research and practical applications of reasoning-augmented LLMs.

Title: Variational Control for Guidance in Diffusion Models

• Authors: Kushagra Pandey, Farrin Marouf Sofian, Felix Draxler, Theofanis Karaletsos, Stephan Mandt
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.03686
• Pdf URL: https://arxiv.org/pdf/2502.03686
• Copy Paste: [[2502.03686]] Variational Control for Guidance in Diffusion Models(https://arxiv.org/abs/2502.03686)
• Keywords: diffusion
• Abstract: Diffusion models exhibit excellent sample quality, but existing guidance methods often require additional model training or are limited to specific tasks. We revisit guidance in diffusion models from the perspective of variational inference and control, introducing Diffusion Trajectory Matching (DTM) that enables guiding pretrained diffusion trajectories to satisfy a terminal cost. DTM unifies a broad class of guidance methods and enables novel instantiations. We introduce a new method within this framework that achieves state-of-the-art results on several linear and (blind) non-linear inverse problems without requiring additional model training or modifications. For instance, in ImageNet non-linear deblurring, our model achieves an FID score of 34.31, significantly improving over the best pretrained-method baseline (FID 78.07). We will make the code available in a future update.

Title: Conditional Diffusion Models are Medical Image Classifiers that Provide Explainability and Uncertainty for Free

• Authors: Gian Mario Favero, Parham Saremi, Emily Kaczmarek, Brennan Nichyporuk, Tal Arbel
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03687
• Pdf URL: https://arxiv.org/pdf/2502.03687
• Copy Paste: [[2502.03687]] Conditional Diffusion Models are Medical Image Classifiers that Provide Explainability and Uncertainty for Free(https://arxiv.org/abs/2502.03687)
• Keywords: diffusion, generative
• Abstract: Discriminative classifiers have become a foundational tool in deep learning for medical imaging, excelling at learning separable features of complex data distributions. However, these models often need careful design, augmentation, and training techniques to ensure safe and reliable deployment. Recently, diffusion models have become synonymous with generative modeling in 2D. These models showcase robustness across a range of tasks including natural image classification, where classification is performed by comparing reconstruction errors across images generated for each possible conditioning input. This work presents the first exploration of the potential of class conditional diffusion models for 2D medical image classification. First, we develop a novel majority voting scheme shown to improve the performance of medical diffusion classifiers. Next, extensive experiments on the CheXpert and ISIC Melanoma skin cancer datasets demonstrate that foundation and trained-from-scratch diffusion models achieve competitive performance against SOTA discriminative classifiers without the need for explicit supervision. In addition, we show that diffusion classifiers are intrinsically explainable, and can be used to quantify the uncertainty of their predictions, increasing their trustworthiness and reliability in safety-critical, clinical contexts. Further information is available on our project page: this https URL

Title: How vulnerable is my policy? Adversarial attacks on modern behavior cloning policies

• Authors: Basavasagar Patil, Akansha Kalra, Guanhong Tao, Daniel S. Brown
• Subjects: cs.LG, cs.CR, cs.RO
• Abstract URL: https://arxiv.org/abs/2502.03698
• Pdf URL: https://arxiv.org/pdf/2502.03698
• Copy Paste: [[2502.03698]] How vulnerable is my policy? Adversarial attacks on modern behavior cloning policies(https://arxiv.org/abs/2502.03698)
• Keywords: diffusion
• Abstract: Learning from Demonstration (LfD) algorithms have shown promising results in robotic manipulation tasks, but their vulnerability to adversarial attacks remains underexplored. This paper presents a comprehensive study of adversarial attacks on both classic and recently proposed algorithms, including Behavior Cloning (BC), LSTM-GMM, Implicit Behavior Cloning (IBC), Diffusion Policy (DP), and VQ-Behavior Transformer (VQ-BET). We study the vulnerability of these methods to untargeted, targeted and universal adversarial perturbations. While explicit policies, such as BC, LSTM-GMM and VQ-BET can be attacked in the same manner as standard computer vision models, we find that attacks for implicit and denoising policy models are nuanced and require developing novel attack methods. Our experiments on several simulated robotic manipulation tasks reveal that most of the current methods are highly vulnerable to adversarial perturbations. We also show that these attacks are transferable across algorithms, architectures, and tasks, raising concerning security vulnerabilities with potentially a white-box threat model. In addition, we test the efficacy of a randomized smoothing, a widely used adversarial defense technique, and highlight its limitation in defending against attacks on complex and multi-modal action distribution common in complex control tasks. In summary, our findings highlight the vulnerabilities of modern BC algorithms, paving way for future work in addressing such limitations.

Title: Detecting Backdoor Attacks via Similarity in Semantic Communication Systems

• Authors: Ziyang Wei, Yili Jiang, Jiaqi Huang, Fangtian Zhong, Sohan Gyawali
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03721
• Pdf URL: https://arxiv.org/pdf/2502.03721
• Copy Paste: [[2502.03721]] Detecting Backdoor Attacks via Similarity in Semantic Communication Systems(https://arxiv.org/abs/2502.03721)
• Keywords: generative
• Abstract: Semantic communication systems, which leverage Generative AI (GAI) to transmit semantic meaning rather than raw data, are poised to revolutionize modern communications. However, they are vulnerable to backdoor attacks, a type of poisoning manipulation that embeds malicious triggers into training datasets. As a result, Backdoor attacks mislead the inference for poisoned samples while clean samples remain unaffected. The existing defenses may alter the model structure (such as neuron pruning that potentially degrades inference performance on clean inputs, or impose strict requirements on data formats (such as ``Semantic Shield" that requires image-text pairs). To address these limitations, this work proposes a defense mechanism that leverages semantic similarity to detect backdoor attacks without modifying the model structure or imposing data format constraints. By analyzing deviations in semantic feature space and establishing a threshold-based detection framework, the proposed approach effectively identifies poisoned samples. The experimental results demonstrate high detection accuracy and recall across varying poisoning ratios, underlining the significant effectiveness of our proposed solution.

Title: DICE: Distilling Classifier-Free Guidance into Text Embeddings

• Authors: Zhenyu Zhou, Defang Chen, Can Wang, Chun Chen, Siwei Lyu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03726
• Pdf URL: https://arxiv.org/pdf/2502.03726
• Copy Paste: [[2502.03726]] DICE: Distilling Classifier-Free Guidance into Text Embeddings(https://arxiv.org/abs/2502.03726)
• Keywords: diffusion, generative
• Abstract: Text-to-image diffusion models are capable of generating high-quality images, but these images often fail to align closely with the given text prompts. Classifier-free guidance (CFG) is a popular and effective technique for improving text-image alignment in the generative process. However, using CFG introduces significant computational overhead and deviates from the established theoretical foundations of diffusion models. In this paper, we present DIstilling CFG by enhancing text Embeddings (DICE), a novel approach that removes the reliance on CFG in the generative process while maintaining the benefits it provides. DICE distills a CFG-based text-to-image diffusion model into a CFG-free version by refining text embeddings to replicate CFG-based directions. In this way, we avoid the computational and theoretical drawbacks of CFG, enabling high-quality, well-aligned image generation at a fast sampling speed. Extensive experiments on multiple Stable Diffusion v1.5 variants, SDXL and PixArt-$\alpha$ demonstrate the effectiveness of our method. Furthermore, DICE supports negative prompts for image editing to improve image quality further. Code will be available soon.

Title: It's All in The [MASK]: Simple Instruction-Tuning Enables BERT-like Masked Language Models As Generative Classifiers

• Authors: Benjamin Clavié, Nathan Cooper, Benjamin Warner
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03793
• Pdf URL: https://arxiv.org/pdf/2502.03793
• Copy Paste: [[2502.03793]] It's All in The [MASK]: Simple Instruction-Tuning Enables BERT-like Masked Language Models As Generative Classifiers(https://arxiv.org/abs/2502.03793)
• Keywords: generative
• Abstract: While encoder-only models such as BERT and ModernBERT are ubiquitous in real-world NLP applications, their conventional reliance on task-specific classification heads can limit their applicability compared to decoder-based large language models (LLMs). In this work, we introduce ModernBERT-Large-Instruct, a 0.4B-parameter encoder model that leverages its masked language modelling (MLM) head for generative classification. Our approach employs an intentionally simple training loop and inference mechanism that requires no heavy pre-processing, heavily engineered prompting, or architectural modifications. ModernBERT-Large-Instruct exhibits strong zero-shot performance on both classification and knowledge-based tasks, outperforming similarly sized LLMs on MMLU and achieving 93% of Llama3-1B's MMLU performance with 60% less parameters. We also demonstrate that, when fine-tuned, the generative approach using the MLM head matches or even surpasses traditional classification-head methods across diverse NLU this http URL capability emerges specifically in models trained on contemporary, diverse data mixes, with models trained on lower volume, less-diverse data yielding considerably weaker performance. Although preliminary, these results demonstrate the potential of using the original generative masked language modelling head over traditional task-specific heads for downstream tasks. Our work suggests that further exploration into this area is warranted, highlighting many avenues for future improvements.

Title: Distribution learning via neural differential equations: minimal energy regularization and approximation theory

• Authors: Youssef Marzouk, Zhi Ren, Jakob Zech
• Subjects: cs.LG, math.CA, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.03795
• Pdf URL: https://arxiv.org/pdf/2502.03795
• Copy Paste: [[2502.03795]] Distribution learning via neural differential equations: minimal energy regularization and approximation theory(https://arxiv.org/abs/2502.03795)
• Keywords: generative
• Abstract: Neural ordinary differential equations (ODEs) provide expressive representations of invertible transport maps that can be used to approximate complex probability distributions, e.g., for generative modeling, density estimation, and Bayesian inference. We show that for a large class of transport maps $T$, there exists a time-dependent ODE velocity field realizing a straight-line interpolation $(1-t)x + tT(x)$, $t \in [0,1]$, of the displacement induced by the map. Moreover, we show that such velocity fields are minimizers of a training objective containing a specific minimum-energy regularization. We then derive explicit upper bounds for the $C^k$ norm of the velocity field that are polynomial in the $C^k$ norm of the corresponding transport map $T$; in the case of triangular (Knothe--Rosenblatt) maps, we also show that these bounds are polynomial in the $C^k$ norms of the associated source and target densities. Combining these results with stability arguments for distribution approximation via ODEs, we show that Wasserstein or Kullback--Leibler approximation of the target distribution to any desired accuracy $\epsilon > 0$ can be achieved by a deep neural network representation of the velocity field whose size is bounded explicitly in terms of $\epsilon$, the dimension, and the smoothness of the source and target densities. The same neural network ansatz yields guarantees on the value of the regularized training objective.

Title: DeblurDiff: Real-World Image Deblurring with Generative Diffusion Models

• Authors: Lingshun Kong, Jiawei Zhang, Dongqing Zou, Jimmy Ren, Xiaohe Wu, Jiangxin Dong, Jinshan Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03810
• Pdf URL: https://arxiv.org/pdf/2502.03810
• Copy Paste: [[2502.03810]] DeblurDiff: Real-World Image Deblurring with Generative Diffusion Models(https://arxiv.org/abs/2502.03810)
• Keywords: diffusion, generative
• Abstract: Diffusion models have achieved significant progress in image generation. The pre-trained Stable Diffusion (SD) models are helpful for image deblurring by providing clear image priors. However, directly using a blurry image or pre-deblurred one as a conditional control for SD will either hinder accurate structure extraction or make the results overly dependent on the deblurring network. In this work, we propose a Latent Kernel Prediction Network (LKPN) to achieve robust real-world image deblurring. Specifically, we co-train the LKPN in latent space with conditional diffusion. The LKPN learns a spatially variant kernel to guide the restoration of sharp images in the latent space. By applying element-wise adaptive convolution (EAC), the learned kernel is utilized to adaptively process the input feature, effectively preserving the structural information of the input. This process thereby more effectively guides the generative process of Stable Diffusion (SD), enhancing both the deblurring efficacy and the quality of detail reconstruction. Moreover, the results at each diffusion step are utilized to iteratively estimate the kernels in LKPN to better restore the sharp latent by EAC. This iterative refinement enhances the accuracy and robustness of the deblurring process. Extensive experimental results demonstrate that the proposed method outperforms state-of-the-art image deblurring methods on both benchmark and real-world images.

Title: Adapting Human Mesh Recovery with Vision-Language Feedback

• Authors: Chongyang Xu, Buzhen Huang, Chengfang Zhang, Ziliang Feng, Yangang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03836
• Pdf URL: https://arxiv.org/pdf/2502.03836
• Copy Paste: [[2502.03836]] Adapting Human Mesh Recovery with Vision-Language Feedback(https://arxiv.org/abs/2502.03836)
• Keywords: diffusion
• Abstract: Human mesh recovery can be approached using either regression-based or optimization-based methods. Regression models achieve high pose accuracy but struggle with model-to-image alignment due to the lack of explicit 2D-3D correspondences. In contrast, optimization-based methods align 3D models to 2D observations but are prone to local minima and depth ambiguity. In this work, we leverage large vision-language models (VLMs) to generate interactive body part descriptions, which serve as implicit constraints to enhance 3D perception and limit the optimization space. Specifically, we formulate monocular human mesh recovery as a distribution adaptation task by integrating both 2D observations and language descriptions. To bridge the gap between text and 3D pose signals, we first train a text encoder and a pose VQ-VAE, aligning texts to body poses in a shared latent space using contrastive learning. Subsequently, we employ a diffusion-based framework to refine the initial parameters guided by gradients derived from both 2D observations and text descriptions. Finally, the model can produce poses with accurate 3D perception and image consistency. Experimental results on multiple benchmarks validate its effectiveness. The code will be made publicly available.

Title: BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation

• Authors: Bo Pang, Hanze Dong, Jiacheng Xu, Silvio Savarese, Yingbo Zhou, Caiming Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03860
• Pdf URL: https://arxiv.org/pdf/2502.03860
• Copy Paste: [[2502.03860]] BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation(https://arxiv.org/abs/2502.03860)
• Keywords: in-context
• Abstract: Large language models (LLMs), such as o1 from OpenAI, have demonstrated remarkable reasoning capabilities. o1 generates a long chain-of-thought (LongCoT) before answering a question. LongCoT allows LLMs to analyze problems, devise plans, reflect, and backtrack effectively. These actions empower LLM to solve complex problems. After the release of o1, many teams have attempted to replicate its LongCoT and reasoning capabilities. In terms of methods, they primarily rely on knowledge distillation with data from existing models with LongCoT capacities (e.g., OpenAI-o1, Qwen-QwQ, DeepSeek-R1-Preview), leaving significant uncertainties on systematically developing such reasoning abilities. In terms of data domains, these works focus narrowly on math while a few others include coding, limiting their generalizability. This paper introduces a novel approach to enable LLM's LongCoT capacity without distillation from o1-like models or expensive human annotations, where we bootstrap LongCoT (BOLT) from a standard instruct model. BOLT involves three stages: 1) LongCoT data bootstrapping with in-context learning on a standard instruct model; 2) LongCoT supervised finetuning; 3) online training to further refine LongCoT capacities. In BOLT, only a few in-context examples need to be constructed during the bootstrapping stage; in our experiments, we created 10 examples, demonstrating the feasibility of this approach. We use Llama-3.1-70B-Instruct to bootstrap LongCoT and apply our method to various model scales (7B, 8B, 70B). We achieve impressive performance on a variety of benchmarks, Arena-Hard, MT-Bench, WildBench, ZebraLogic, MATH500, which evaluate diverse task-solving and reasoning capabilities.

Title: Position: Untrained Machine Learning for Anomaly Detection

• Authors: Juan Du, Dongheng Chen, Hao Yan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03876
• Pdf URL: https://arxiv.org/pdf/2502.03876
• Copy Paste: [[2502.03876]] Position: Untrained Machine Learning for Anomaly Detection(https://arxiv.org/abs/2502.03876)
• Keywords: anomaly
• Abstract: Anomaly detection based on 3D point cloud data is an important research problem and receives more and more attention recently. Untrained anomaly detection based on only one sample is an emerging research problem motivated by real manufacturing industries such as personalized manufacturing that only one sample can be collected without any additional labels. How to accurately identify anomalies based on one 3D point cloud sample is a critical challenge in both industrial applications and the field of machine learning. This paper aims to provide a formal definition of untrained anomaly detection problem based on 3D point cloud data, discuss the differences between untrained anomaly detection and current unsupervised anomaly detection methods. Unlike unsupervised learning, untrained methods do not rely on any data, including unlabeled data. Instead, they leverage prior knowledge about the manufacturing surfaces and anomalies. Examples are used to illustrate these prior knowledge and untrained machine learning model. Afterwards, literature review on untrained anomaly detection based on 3D point cloud data is also provided, and the potential of untrained deep neural networks for anomaly detection is also discussed as outlooks.

Title: Hierarchical Entropic Diffusion for Ransomware Detection: A Probabilistic Approach to Behavioral Anomaly Isolation

• Authors: Vasili Iskorohodov, Maximilian Ravensdale, Matthias von Holstein, Hugo Petrovic, Adrian Yardley
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2502.03882
• Pdf URL: https://arxiv.org/pdf/2502.03882
• Copy Paste: [[2502.03882]] Hierarchical Entropic Diffusion for Ransomware Detection: A Probabilistic Approach to Behavioral Anomaly Isolation(https://arxiv.org/abs/2502.03882)
• Keywords: diffusion, anomaly
• Abstract: The increasing complexity of cryptographic extortion techniques has necessitated the development of adaptive detection frameworks capable of identifying adversarial encryption behaviors without reliance on predefined signatures. Hierarchical Entropic Diffusion (HED) introduces a structured entropy-based anomaly classification mechanism that systematically tracks fluctuations in entropy evolution to differentiate between benign cryptographic processes and unauthorized encryption attempts. The integration of hierarchical clustering, entropy profiling, and probabilistic diffusion modeling refines detection granularity, ensuring that encryption anomalies are identified despite obfuscation strategies or incremental execution methodologies. Experimental evaluations demonstrated that HED maintained high classification accuracy across diverse ransomware families, outperforming traditional heuristic-based and signature-driven approaches while reducing false positive occurrences. Comparative analysis highlighted that entropy-driven anomaly segmentation improved detection efficiency under variable system workload conditions, ensuring real-time classification feasibility. The computational overhead associated with entropy anomaly detection remained within operational constraints, reinforcing the suitability of entropy-driven classification for large-scale deployment. The ability to identify adversarial entropy manipulations before encryption completion contributes to broader cybersecurity defenses, offering a structured methodology for isolating unauthorized cryptographic activities within heterogeneous computing environments. The results further emphasized that entropy evolution modeling facilitates predictive anomaly detection, enhancing resilience against encryption evasion techniques designed to circumvent traditional detection mechanisms.

Title: LeAP: Consistent multi-domain 3D labeling using Foundation Models

• Authors: Simon Gebraad, Andras Palffy, Holger Caesar
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2502.03901
• Pdf URL: https://arxiv.org/pdf/2502.03901
• Copy Paste: [[2502.03901]] LeAP: Consistent multi-domain 3D labeling using Foundation Models(https://arxiv.org/abs/2502.03901)
• Keywords: foundation model
• Abstract: Availability of datasets is a strong driver for research on 3D semantic understanding, and whilst obtaining unlabeled 3D point cloud data is straightforward, manually annotating this data with semantic labels is time-consuming and costly. Recently, Vision Foundation Models (VFMs) enable open-set semantic segmentation on camera images, potentially aiding automatic labeling. However,VFMs for 3D data have been limited to adaptations of 2D models, which can introduce inconsistencies to 3D labels. This work introduces Label Any Pointcloud (LeAP), leveraging 2D VFMs to automatically label 3D data with any set of classes in any kind of application whilst ensuring label consistency. Using a Bayesian update, point labels are combined into voxels to improve spatio-temporal consistency. A novel 3D Consistency Network (3D-CN) exploits 3D information to further improve label quality. Through various experiments, we show that our method can generate high-quality 3D semantic labels across diverse fields without any manual labeling. Further, models adapted to new domains using our labels show up to a 34.2 mIoU increase in semantic segmentation tasks.

Title: No Free Lunch in Annotation either: An objective evaluation of foundation models for streamlining annotation in animal tracking

• Authors: Emil Mededovic, Valdy Laurentius, Yuli Wu, Marcin Kopaczka, Zhu Chen, Mareike Schulz, René Tolba, Johannes Stegmaier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03907
• Pdf URL: https://arxiv.org/pdf/2502.03907
• Copy Paste: [[2502.03907]] No Free Lunch in Annotation either: An objective evaluation of foundation models for streamlining annotation in animal tracking(https://arxiv.org/abs/2502.03907)
• Keywords: foundation model
• Abstract: We analyze the capabilities of foundation models addressing the tedious task of generating annotations for animal tracking. Annotating a large amount of data is vital and can be a make-or-break factor for the robustness of a tracking model. Robustness is particularly crucial in animal tracking, as accurate tracking over long time horizons is essential for capturing the behavior of animals. However, generating additional annotations using foundation models can be counterproductive, as the quality of the annotations is just as important. Poorly annotated data can introduce noise and inaccuracies, ultimately compromising the performance and accuracy of the trained model. Over-reliance on automated annotations without ensuring precision can lead to diminished results, making careful oversight and quality control essential in the annotation process. Ultimately, we demonstrate that a thoughtful combination of automated annotations and manually annotated data is a valuable strategy, yielding an IDF1 score of 80.8 against blind usage of SAM2 video with an IDF1 score of 65.6.

Title: HEP-JEPA: A foundation model for collider physics using joint embedding predictive architecture

• Authors: Jai Bardhan, Radhikesh Agrawal, Abhiram Tilak, Cyrin Neeraj, Subhadip Mitra
• Subjects: cs.LG, hep-ex, hep-ph
• Abstract URL: https://arxiv.org/abs/2502.03933
• Pdf URL: https://arxiv.org/pdf/2502.03933
• Copy Paste: [[2502.03933]] HEP-JEPA: A foundation model for collider physics using joint embedding predictive architecture(https://arxiv.org/abs/2502.03933)
• Keywords: self-supervised, foundation model
• Abstract: We present a transformer architecture-based foundation model for tasks at high-energy particle colliders such as the Large Hadron Collider. We train the model to classify jets using a self-supervised strategy inspired by the Joint Embedding Predictive Architecture. We use the JetClass dataset containing 100M jets of various known particles to pre-train the model with a data-centric approach -- the model uses a fraction of the jet constituents as the context to predict the embeddings of the unseen target constituents. Our pre-trained model fares well with other datasets for standard classification benchmark tasks. We test our model on two additional downstream tasks: top tagging and differentiating light-quark jets from gluon jets. We also evaluate our model with task-specific metrics and baselines and compare it with state-of-the-art models in high-energy physics. Project site: this https URL

Title: Unravelling Causal Genetic Biomarkers of Alzheimer's Disease via Neuron to Gene-token Backtracking in Neural Architecture: A Groundbreaking Reverse-Gene-Finder Approach

• Authors: Victor OK Li, Yang Han, Jacqueline CK Lam
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03938
• Pdf URL: https://arxiv.org/pdf/2502.03938
• Copy Paste: [[2502.03938]] Unravelling Causal Genetic Biomarkers of Alzheimer's Disease via Neuron to Gene-token Backtracking in Neural Architecture: A Groundbreaking Reverse-Gene-Finder Approach(https://arxiv.org/abs/2502.03938)
• Keywords: foundation model
• Abstract: Alzheimer's Disease (AD) affects over 55 million people globally, yet the key genetic contributors remain poorly understood. Leveraging recent advancements in genomic foundation models, we present the innovative Reverse-Gene-Finder technology, a ground-breaking neuron-to-gene-token backtracking approach in a neural network architecture to elucidate the novel causal genetic biomarkers driving AD onset. Reverse-Gene-Finder comprises three key innovations. Firstly, we exploit the observation that genes with the highest probability of causing AD, defined as the most causal genes (MCGs), must have the highest probability of activating those neurons with the highest probability of causing AD, defined as the most causal neurons (MCNs). Secondly, we utilize a gene token representation at the input layer to allow each gene (known or novel to AD) to be represented as a discrete and unique entity in the input space. Lastly, in contrast to the existing neural network architectures, which track neuron activations from the input layer to the output layer in a feed-forward manner, we develop an innovative backtracking method to track backwards from the MCNs to the input layer, identifying the Most Causal Tokens (MCTs) and the corresponding MCGs. Reverse-Gene-Finder is highly interpretable, generalizable, and adaptable, providing a promising avenue for application in other disease scenarios.

Title: LR0.FM: Low-Resolution Zero-shot Classification Benchmark For Foundation Models

• Authors: Priyank Pathak, Shyam Marjit, Shruti Vyas, Yogesh S Rawat
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03950
• Pdf URL: https://arxiv.org/pdf/2502.03950
• Copy Paste: [[2502.03950]] LR0.FM: Low-Resolution Zero-shot Classification Benchmark For Foundation Models(https://arxiv.org/abs/2502.03950)
• Keywords: foundation model
• Abstract: Visual-language foundation Models (FMs) exhibit remarkable zero-shot generalization across diverse tasks, largely attributed to extensive pre-training on large-scale datasets. However, their robustness on low-resolution/pixelated (LR) images, a common challenge in real-world scenarios, remains underexplored. We introduce this http URL, a comprehensive benchmark evaluating the impact of low resolution on the zero-shot classification performance of 10 FM(s) across 66 backbones and 15 datasets. We propose a novel metric, Weighted Aggregated Robustness, to address the limitations of existing metrics and better evaluate model performance across resolutions and datasets. Our key findings show that: (i) model size positively correlates with robustness to resolution degradation, (ii) pre-training dataset quality is more important than its size, and (iii) fine-tuned and higher-resolution models are less robust against LR. Our analysis further reveals that the model makes semantically reasonable predictions at LR, and the lack of fine-grained details in input adversely impacts the model's initial layers more than the deeper layers. We use these insights and introduce a simple strategy, LR-TK0, to enhance the robustness of models without compromising their pre-trained weights. We demonstrate the effectiveness of LR-TK0 for robustness against low-resolution across several datasets and its generalization capability across backbones and other approaches. Code is available at this this https URL

Title: MultiFloodSynth: Multi-Annotated Flood Synthetic Dataset Generation

• Authors: YoonJe Kang, Yonghoon Jung, Wonseop Shin, Bumsoo Kim, Sanghyun Seo
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03966
• Pdf URL: https://arxiv.org/pdf/2502.03966
• Copy Paste: [[2502.03966]] MultiFloodSynth: Multi-Annotated Flood Synthetic Dataset Generation(https://arxiv.org/abs/2502.03966)
• Keywords: generative
• Abstract: In this paper, we present synthetic data generation framework for flood hazard detection system. For high fidelity and quality, we characterize several real-world properties into virtual world and simulate the flood situation by controlling them. For the sake of efficiency, recent generative models in image-to-3D and urban city synthesis are leveraged to easily composite flood environments so that we avoid data bias due to the hand-crafted manner. Based on our framework, we build the flood synthetic dataset with 5 levels, dubbed MultiFloodSynth which contains rich annotation types like normal map, segmentation, 3D bounding box for a variety of downstream task. In experiments, our dataset demonstrate the enhanced performance of flood hazard detection with on-par realism compared with real dataset.

Title: Tight Bounds on Jensen's Gap: Novel Approach with Applications in Generative Modeling

• Authors: Marcin Mazur, Piotr Kościelniak, Łukasz Struski
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03988
• Pdf URL: https://arxiv.org/pdf/2502.03988
• Copy Paste: [[2502.03988]] Tight Bounds on Jensen's Gap: Novel Approach with Applications in Generative Modeling(https://arxiv.org/abs/2502.03988)
• Keywords: generative
• Abstract: Among various mathematical tools of particular interest are those that provide a common basis for researchers in different scientific fields. One of them is Jensen's inequality, which states that the expectation of a convex function is greater than or equal to the function evaluated at the expectation. The resulting difference, known as Jensen's gap, became the subject of investigation by both the statistical and machine learning communities. Among many related topics, finding lower and upper bounds on Jensen's gap (under different assumptions on the underlying function and distribution) has recently become a problem of particular interest. In our paper, we take another step in this direction by providing a novel general and mathematically rigorous technique, motivated by the recent results of Struski et al. (2023). In addition, by studying in detail the case of the logarithmic function and the log-normal distribution, we explore a method for tightly estimating the log-likelihood of generative models trained on real-world datasets. Furthermore, we present both analytical and experimental arguments in support of the superiority of our approach in comparison to existing state-of-the-art solutions, contingent upon fulfillment of the criteria set forth by theoretical studies and corresponding experiments on synthetic data.

Title: Exploring Imbalanced Annotations for Effective In-Context Learning

• Authors: Hongfu Gao, Feipeng Zhang, Hao Zeng, Deyu Meng, Bingyi Jing, Hongxin Wei
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.04037
• Pdf URL: https://arxiv.org/pdf/2502.04037
• Copy Paste: [[2502.04037]] Exploring Imbalanced Annotations for Effective In-Context Learning(https://arxiv.org/abs/2502.04037)
• Keywords: in-context
• Abstract: Large language models (LLMs) have shown impressive performance on downstream tasks through in-context learning (ICL), which heavily relies on the demonstrations selected from annotated datasets. Existing selection methods may hinge on the distribution of annotated datasets, which can often be long-tailed in real-world scenarios. In this work, we show that imbalanced class distributions in annotated datasets significantly degrade the performance of ICL across various tasks and selection methods. Moreover, traditional rebalance methods fail to ameliorate the issue of class imbalance in ICL. Our method is motivated by decomposing the distributional differences between annotated and test datasets into two-component weights: class-wise weights and conditional bias. The key idea behind our method is to estimate the conditional bias by minimizing the empirical error on a balanced validation dataset and to employ the two-component weights to modify the original scoring functions during selection. Our approach can prevent selecting too many demonstrations from a single class while preserving the effectiveness of the original selection methods. Extensive experiments demonstrate the effectiveness of our method, improving the average accuracy by up to 5.46 on common benchmarks with imbalanced datasets.

Title: PartEdit: Fine-Grained Image Editing using Pre-Trained Diffusion Models

• Authors: Aleksandar Cvejic (KAUST), Abdelrahman Eldesokey (KAUST), Peter Wonka (KAUST)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04050
• Pdf URL: https://arxiv.org/pdf/2502.04050
• Copy Paste: [[2502.04050]] PartEdit: Fine-Grained Image Editing using Pre-Trained Diffusion Models(https://arxiv.org/abs/2502.04050)
• Keywords: diffusion
• Abstract: We present the first text-based image editing approach for object parts based on pre-trained diffusion models. Diffusion-based image editing approaches capitalized on the deep understanding of diffusion models of image semantics to perform a variety of edits. However, existing diffusion models lack sufficient understanding of many object parts, hindering fine-grained edits requested by users. To address this, we propose to expand the knowledge of pre-trained diffusion models to allow them to understand various object parts, enabling them to perform fine-grained edits. We achieve this by learning special textual tokens that correspond to different object parts through an efficient token optimization process. These tokens are optimized to produce reliable localization masks at each inference step to localize the editing region. Leveraging these masks, we design feature-blending and adaptive thresholding strategies to execute the edits seamlessly. To evaluate our approach, we establish a benchmark and an evaluation protocol for part editing. Experiments show that our approach outperforms existing editing methods on all metrics and is preferred by users 77-90% of the time in conducted user studies.

Title: TQ-DiT: Efficient Time-Aware Quantization for Diffusion Transformers

• Authors: Younghye Hwang, Hyojin Lee, Joonhyuk Kang
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2502.04056
• Pdf URL: https://arxiv.org/pdf/2502.04056
• Copy Paste: [[2502.04056]] TQ-DiT: Efficient Time-Aware Quantization for Diffusion Transformers(https://arxiv.org/abs/2502.04056)
• Keywords: diffusion, generative
• Abstract: Diffusion transformers (DiTs) combine transformer architectures with diffusion models. However, their computational complexity imposes significant limitations on real-time applications and sustainability of AI systems. In this study, we aim to enhance the computational efficiency through model quantization, which represents the weights and activation values with lower precision. Multi-region quantization (MRQ) is introduced to address the asymmetric distribution of network values in DiT blocks by allocating two scaling parameters to sub-regions. Additionally, time-grouping quantization (TGQ) is proposed to reduce quantization error caused by temporal variation in activations. The experimental results show that the proposed algorithm achieves performance comparable to the original full-precision model with only a 0.29 increase in FID at W8A8. Furthermore, it outperforms other baselines at W6A6, thereby confirming its suitability for low-bit quantization. These results highlight the potential of our method to enable efficient real-time generative models.

Title: Generative Adversarial Networks Bridging Art and Machine Intelligence

• Authors: Junhao Song, Yichao Zhang, Ziqian Bi, Tianyang Wang, Keyu Chen, Ming Li, Qian Niu, Junyu Liu, Benji Peng, Sen Zhang, Ming Liu, Jiawei Xu, Xuanhe Pan, Jinlang Wang, Pohsun Feng, Yizhu Wen, Lawrence K.Q. Yan, Hong-Ming Tseng, Xinyuan Song, Jintao Ren, Silin Chen, Yunze Wang, Weiche Hsieh, Bowen Jing, Junjie Yang, Jun Zhou, Zheyu Yao, Chia Xin Liang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04116
• Pdf URL: https://arxiv.org/pdf/2502.04116
• Copy Paste: [[2502.04116]] Generative Adversarial Networks Bridging Art and Machine Intelligence(https://arxiv.org/abs/2502.04116)
• Keywords: diffusion, generative
• Abstract: This book begins with a detailed introduction to the fundamental principles and historical development of GANs, contrasting them with traditional generative models and elucidating the core adversarial mechanisms through illustrative Python examples. The text systematically addresses the mathematical and theoretical underpinnings including probability theory, statistics, and game theory providing a solid framework for understanding the objectives, loss functions, and optimisation challenges inherent to GAN training. Subsequent chapters review classic variants such as Conditional GANs, DCGANs, InfoGAN, and LAPGAN before progressing to advanced training methodologies like Wasserstein GANs, GANs with gradient penalty, least squares GANs, and spectral normalisation techniques. The book further examines architectural enhancements and task-specific adaptations in generators and discriminators, showcasing practical implementations in high resolution image generation, artistic style transfer, video synthesis, text to image generation and other multimedia applications. The concluding sections offer insights into emerging research trends, including self-attention mechanisms, transformer-based generative models, and a comparative analysis with diffusion models, thus charting promising directions for future developments in both academic and applied settings.

Title: MRAMG-Bench: A BeyondText Benchmark for Multimodal Retrieval-Augmented Multimodal Generation

• Authors: Qinhan Yu, Zhiyou Xiao, Binghui Li, Zhengren Wang, Chong Chen, Wentao Zhang
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.04176
• Pdf URL: https://arxiv.org/pdf/2502.04176
• Copy Paste: [[2502.04176]] MRAMG-Bench: A BeyondText Benchmark for Multimodal Retrieval-Augmented Multimodal Generation(https://arxiv.org/abs/2502.04176)
• Keywords: generative
• Abstract: Recent advancements in Retrieval-Augmented Generation (RAG) have shown remarkable performance in enhancing response accuracy and relevance by integrating external knowledge into generative models. However, existing RAG methods primarily focus on providing text-only answers, even in multimodal retrieval-augmented generation scenarios. In this work, we introduce the Multimodal Retrieval-Augmented Multimodal Generation (MRAMG) task, which aims to generate answers that combine both text and images, fully leveraging the multimodal data within a corpus. Despite the importance of this task, there is a notable absence of a comprehensive benchmark to effectively evaluate MRAMG performance. To bridge this gap, we introduce the MRAMG-Bench, a carefully curated, human-annotated dataset comprising 4,346 documents, 14,190 images, and 4,800 QA pairs, sourced from three categories: Web Data, Academic Papers, and Lifestyle. The dataset incorporates diverse difficulty levels and complex multi-image scenarios, providing a robust foundation for evaluating multimodal generation tasks. To facilitate rigorous evaluation, our MRAMG-Bench incorporates a comprehensive suite of both statistical and LLM-based metrics, enabling a thorough analysis of the performance of popular generative models in the MRAMG task. Besides, we propose an efficient multimodal answer generation framework that leverages both LLMs and MLLMs to generate multimodal responses. Our datasets are available at: this https URL.

Title: PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models?

• Authors: Mennatullah Siam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04192
• Pdf URL: https://arxiv.org/pdf/2502.04192
• Copy Paste: [[2502.04192]] PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models?(https://arxiv.org/abs/2502.04192)
• Keywords: foundation model
• Abstract: Multiple works have emerged to push the boundaries on multi-modal large language models (MLLMs) towards pixel-level understanding. Such approaches have shown strong performance on benchmarks for referring expression segmentation and grounded conversation generation. The current trend in pixel-level MLLMs is to train with pixel-level grounding supervision on large-scale labelled data. However, we show that such MLLMs when evaluated on recent challenging vision centric benchmarks, exhibit a weak ability in visual question answering. Surprisingly, some of these methods even downgrade the grounding ability of MLLMs that were never trained with such supervision. In this work, we propose two novel challenging benchmarks and show that MLLMs without pixel-level grounding supervision can outperform the state of the art in such tasks when evaluating both the pixel-level grounding and visual question answering. We propose simple baselines to extract the grounding information that can be plugged into any MLLM, which we call as PixFoundation. More importantly, we study the research question of ``When does grounding emerge in MLLMs that are not trained with pixel-level grounding supervision?'' We show that grounding can coincide with object parts or location/appearance information. Code repository is at this https URL.

Title: "Short-length" Adversarial Training Helps LLMs Defend "Long-length" Jailbreak Attacks: Theoretical and Empirical Evidence

• Authors: Shaopeng Fu, Liang Ding, Di Wang
• Subjects: cs.LG, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.04204
• Pdf URL: https://arxiv.org/pdf/2502.04204
• Copy Paste: [[2502.04204]] "Short-length" Adversarial Training Helps LLMs Defend "Long-length" Jailbreak Attacks: Theoretical and Empirical Evidence(https://arxiv.org/abs/2502.04204)
• Keywords: in-context
• Abstract: Jailbreak attacks against large language models (LLMs) aim to induce harmful behaviors in LLMs through carefully crafted adversarial prompts. To mitigate attacks, one way is to perform adversarial training (AT)-based alignment, i.e., training LLMs on some of the most adversarial prompts to help them learn how to behave safely under attacks. During AT, the length of adversarial prompts plays a critical role in the robustness of aligned LLMs. This paper focuses on adversarial suffix jailbreak attacks and unveils that to defend against a jailbreak attack with an adversarial suffix of length $\Theta(M)$, it is enough to align LLMs on prompts with adversarial suffixes of length $\Theta(\sqrt{M})$. Theoretically, we analyze the adversarial in-context learning of linear transformers on linear regression tasks and prove a robust generalization bound for trained transformers. The bound depends on the term $\Theta(\sqrt{M_{\text{test}}}/M_{\text{train}})$, where $M_{\text{train}}$ and $M_{\text{test}}$ are the number of adversarially perturbed in-context samples during training and testing. Empirically, we conduct AT on popular open-source LLMs and evaluate their robustness against jailbreak attacks of different adversarial suffix lengths. Results confirm a positive correlation between the attack success rate and the ratio of the square root of the adversarial suffix during jailbreaking to the length during AT. Our findings show that it is practical to defend "long-length" jailbreak attacks via efficient "short-length" AT. The code is available at this https URL.

Title: Realistic Image-to-Image Machine Unlearning via Decoupling and Knowledge Retention

• Authors: Ayush K. Varshney, Vicenç Torra
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.04260
• Pdf URL: https://arxiv.org/pdf/2502.04260
• Copy Paste: [[2502.04260]] Realistic Image-to-Image Machine Unlearning via Decoupling and Knowledge Retention(https://arxiv.org/abs/2502.04260)
• Keywords: generative
• Abstract: Machine Unlearning allows participants to remove their data from a trained machine learning model in order to preserve their privacy, and security. However, the machine unlearning literature for generative models is rather limited. The literature for image-to-image generative model (I2I model) considers minimizing the distance between Gaussian noise and the output of I2I model for forget samples as machine unlearning. However, we argue that the machine learning model performs fairly well on unseen data i.e., a retrained model will be able to catch generic patterns in the data and hence will not generate an output which is equivalent to Gaussian noise. In this paper, we consider that the model after unlearning should treat forget samples as out-of-distribution (OOD) data, i.e., the unlearned model should no longer recognize or encode the specific patterns found in the forget samples. To achieve this, we propose a framework which decouples the model parameters with gradient ascent, ensuring that forget samples are OOD for unlearned model with theoretical guarantee. We also provide $(\epsilon, \delta)$-unlearning guarantee for model updates with gradient ascent. The unlearned model is further fine-tuned on the remaining samples to maintain its performance. We also propose an attack model to ensure that the unlearned model has effectively removed the influence of forget samples. Extensive empirical evaluation on two large-scale datasets, ImageNet-1K and Places365 highlights the superiority of our approach. To show comparable performance with retrained model, we also show the comparison of a simple AutoEncoder on various baselines on CIFAR-10 dataset.

Title: Efficient Randomized Experiments Using Foundation Models

• Authors: Piersilvio De Bartolomeis, Javier Abad, Guanbo Wang, Konstantin Donhauser, Raymond M. Duch, Fanny Yang, Issa J. Dahabreh
• Subjects: cs.LG, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.04262
• Pdf URL: https://arxiv.org/pdf/2502.04262
• Copy Paste: [[2502.04262]] Efficient Randomized Experiments Using Foundation Models(https://arxiv.org/abs/2502.04262)
• Keywords: foundation model
• Abstract: Randomized experiments are the preferred approach for evaluating the effects of interventions, but they are costly and often yield estimates with substantial uncertainty. On the other hand, in silico experiments leveraging foundation models offer a cost-effective alternative that can potentially attain higher statistical precision. However, the benefits of in silico experiments come with a significant risk: statistical inferences are not valid if the models fail to accurately predict experimental responses to interventions. In this paper, we propose a novel approach that integrates the predictions from multiple foundation models with experimental data while preserving valid statistical inference. Our estimator is consistent and asymptotically normal, with asymptotic variance no larger than the standard estimator based on experimental data alone. Importantly, these statistical properties hold even when model predictions are arbitrarily biased. Empirical results across several randomized experiments show that our estimator offers substantial precision gains, equivalent to a reduction of up to 20% in the sample size needed to match the same precision as the standard estimator based on experimental data alone.

Title: Statistical guarantees for continuous-time policy evaluation: blessing of ellipticity and new tradeoffs

• Authors: Wenlong Mou
• Subjects: cs.LG, math.OC, math.PR, math.ST
• Abstract URL: https://arxiv.org/abs/2502.04297
• Pdf URL: https://arxiv.org/pdf/2502.04297
• Copy Paste: [[2502.04297]] Statistical guarantees for continuous-time policy evaluation: blessing of ellipticity and new tradeoffs(https://arxiv.org/abs/2502.04297)
• Keywords: diffusion
• Abstract: We study the estimation of the value function for continuous-time Markov diffusion processes using a single, discretely observed ergodic trajectory. Our work provides non-asymptotic statistical guarantees for the least-squares temporal-difference (LSTD) method, with performance measured in the first-order Sobolev norm. Specifically, the estimator attains an $O(1 / \sqrt{T})$ convergence rate when using a trajectory of length $T$; notably, this rate is achieved as long as $T$ scales nearly linearly with both the mixing time of the diffusion and the number of basis functions employed. A key insight of our approach is that the ellipticity inherent in the diffusion process ensures robust performance even as the effective horizon diverges to infinity. Moreover, we demonstrate that the Markovian component of the statistical error can be controlled by the approximation error, while the martingale component grows at a slower rate relative to the number of basis functions. By carefully balancing these two sources of error, our analysis reveals novel trade-offs between approximation and statistical errors.

Title: MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation

• Authors: Jinbo Xing, Long Mai, Cusuh Ham, Jiahui Huang, Aniruddha Mahapatra, Chi-Wing Fu, Tien-Tsin Wong, Feng Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04299
• Pdf URL: https://arxiv.org/pdf/2502.04299
• Copy Paste: [[2502.04299]] MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation(https://arxiv.org/abs/2502.04299)
• Keywords: diffusion
• Abstract: This paper presents a method that allows users to design cinematic video shots in the context of image-to-video generation. Shot design, a critical aspect of filmmaking, involves meticulously planning both camera movements and object motions in a scene. However, enabling intuitive shot design in modern image-to-video generation systems presents two main challenges: first, effectively capturing user intentions on the motion design, where both camera movements and scene-space object motions must be specified jointly; and second, representing motion information that can be effectively utilized by a video diffusion model to synthesize the image animations. To address these challenges, we introduce MotionCanvas, a method that integrates user-driven controls into image-to-video (I2V) generation models, allowing users to control both object and camera motions in a scene-aware manner. By connecting insights from classical computer graphics and contemporary video generation techniques, we demonstrate the ability to achieve 3D-aware motion control in I2V synthesis without requiring costly 3D-related training data. MotionCanvas enables users to intuitively depict scene-space motion intentions, and translates them into spatiotemporal motion-conditioning signals for video diffusion models. We demonstrate the effectiveness of our method on a wide range of real-world image content and shot-design scenarios, highlighting its potential to enhance the creative workflows in digital content creation and adapt to various image and video editing applications.

Title: HOG-Diff: Higher-Order Guided Diffusion for Graph Generation

• Authors: Yiming Huang, Tolga Birdal
• Subjects: cs.LG, cs.AI, cs.SI, physics.soc-ph
• Abstract URL: https://arxiv.org/abs/2502.04308
• Pdf URL: https://arxiv.org/pdf/2502.04308
• Copy Paste: [[2502.04308]] HOG-Diff: Higher-Order Guided Diffusion for Graph Generation(https://arxiv.org/abs/2502.04308)
• Keywords: diffusion
• Abstract: Graph generation is a critical yet challenging task as empirical analyses require a deep understanding of complex, non-Euclidean structures. Although diffusion models have recently made significant achievements in graph generation, these models typically adapt from the frameworks designed for image generation, making them ill-suited for capturing the topological properties of graphs. In this work, we propose a novel Higher-order Guided Diffusion (HOG-Diff) model that follows a coarse-to-fine generation curriculum and is guided by higher-order information, enabling the progressive generation of plausible graphs with inherent topological structures. We further prove that our model exhibits a stronger theoretical guarantee than classical diffusion frameworks. Extensive experiments on both molecular and generic graph generation tasks demonstrate that our method consistently outperforms or remains competitive with state-of-the-art baselines. Our code is available at this https URL.

Title: Finding Pegasus: Enhancing Unsupervised Anomaly Detection in High-Dimensional Data using a Manifold-Based Approach

• Authors: R. P. Nathan, Nikolaos Nikolaou, Ofer Lahav
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.04310
• Pdf URL: https://arxiv.org/pdf/2502.04310
• Copy Paste: [[2502.04310]] Finding Pegasus: Enhancing Unsupervised Anomaly Detection in High-Dimensional Data using a Manifold-Based Approach(https://arxiv.org/abs/2502.04310)
• Keywords: anomaly
• Abstract: Unsupervised machine learning methods are well suited to searching for anomalies at scale but can struggle with the high-dimensional representation of many modern datasets, hence dimensionality reduction (DR) is often performed first. In this paper we analyse unsupervised anomaly detection (AD) from the perspective of the manifold created in DR. We present an idealised illustration, "Finding Pegasus", and a novel formal framework with which we categorise AD methods and their results into "on manifold" and "off manifold". We define these terms and show how they differ. We then use this insight to develop an approach of combining AD methods which significantly boosts AD recall without sacrificing precision in situations employing high DR. When tested on MNIST data, our approach of combining AD methods improves recall by as much as 16 percent compared with simply combining with the best standalone AD method (Isolation Forest), a result which shows great promise for its application to real-world data.

Title: sshELF: Single-Shot Hierarchical Extrapolation of Latent Features for 3D Reconstruction from Sparse-Views

• Authors: Eyvaz Najafli, Marius Kästingschäfer, Sebastian Bernhard, Thomas Brox, Andreas Geiger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04318
• Pdf URL: https://arxiv.org/pdf/2502.04318
• Copy Paste: [[2502.04318]] sshELF: Single-Shot Hierarchical Extrapolation of Latent Features for 3D Reconstruction from Sparse-Views(https://arxiv.org/abs/2502.04318)
• Keywords: foundation model
• Abstract: Reconstructing unbounded outdoor scenes from sparse outward-facing views poses significant challenges due to minimal view overlap. Previous methods often lack cross-scene understanding and their primitive-centric formulations overload local features to compensate for missing global context, resulting in blurriness in unseen parts of the scene. We propose sshELF, a fast, single-shot pipeline for sparse-view 3D scene reconstruction via hierarchal extrapolation of latent features. Our key insights is that disentangling information extrapolation from primitive decoding allows efficient transfer of structural patterns across training scenes. Our method: (1) learns cross-scene priors to generate intermediate virtual views to extrapolate to unobserved regions, (2) offers a two-stage network design separating virtual view generation from 3D primitive decoding for efficient training and modular model design, and (3) integrates a pre-trained foundation model for joint inference of latent features and texture, improving scene understanding and generalization. sshELF can reconstruct 360 degree scenes from six sparse input views and achieves competitive results on synthetic and real-world datasets. We find that sshELF faithfully reconstructs occluded regions, supports real-time rendering, and provides rich latent features for downstream applications. The code will be released.

Title: ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features

• Authors: Alec Helbling, Tuna Han Salih Meral, Ben Hoover, Pinar Yanardag, Duen Horng Chau
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.04320
• Pdf URL: https://arxiv.org/pdf/2502.04320
• Copy Paste: [[2502.04320]] ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features(https://arxiv.org/abs/2502.04320)
• Keywords: diffusion, foundation model
• Abstract: Do the rich representations of multi-modal diffusion transformers (DiTs) exhibit unique properties that enhance their interpretability? We introduce ConceptAttention, a novel method that leverages the expressive power of DiT attention layers to generate high-quality saliency maps that precisely locate textual concepts within images. Without requiring additional training, ConceptAttention repurposes the parameters of DiT attention layers to produce highly contextualized concept embeddings, contributing the major discovery that performing linear projections in the output space of DiT attention layers yields significantly sharper saliency maps compared to commonly used cross-attention mechanisms. Remarkably, ConceptAttention even achieves state-of-the-art performance on zero-shot image segmentation benchmarks, outperforming 11 other zero-shot interpretability methods on the ImageNet-Segmentation dataset and on a single-class subset of PascalVOC. Our work contributes the first evidence that the representations of multi-modal DiT models like Flux are highly transferable to vision tasks like segmentation, even outperforming multi-modal foundation models like CLIP.