2025-02-03

Title: DebiasPI: Inference-time Debiasing by Prompt Iteration of a Text-to-Image Generative Model

• Authors: Sarah Bonna, Yu-Cheng Huang, Ekaterina Novozhilova, Sejin Paik, Zhengyang Shan, Michelle Yilin Feng, Ge Gao, Yonish Tayal, Rushil Kulkarni, Jialin Yu, Nupur Divekar, Deepti Ghadiyaram, Derry Wijaya, Margrit Betke
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.18642
• Pdf URL: https://arxiv.org/pdf/2501.18642
• Copy Paste: [[2501.18642]] DebiasPI: Inference-time Debiasing by Prompt Iteration of a Text-to-Image Generative Model(https://arxiv.org/abs/2501.18642)
• Keywords: generative
• Abstract: Ethical intervention prompting has emerged as a tool to counter demographic biases of text-to-image generative AI models. Existing solutions either require to retrain the model or struggle to generate images that reflect desired distributions on gender and race. We propose an inference-time process called DebiasPI for Debiasing-by-Prompt-Iteration that provides prompt intervention by enabling the user to control the distributions of individuals' demographic attributes in image generation. DebiasPI keeps track of which attributes have been generated either by probing the internal state of the model or by using external attribute classifiers. Its control loop guides the text-to-image model to select not yet sufficiently represented attributes, With DebiasPI, we were able to create images with equal representations of race and gender that visualize challenging concepts of news headlines. We also experimented with the attributes age, body type, profession, and skin tone, and measured how attributes change when our intervention prompt targets the distribution of an unrelated attribute type. We found, for example, if the text-to-image model is asked to balance racial representation, gender representation improves but the skin tone becomes less diverse. Attempts to cover a wide range of skin colors with various intervention prompts showed that the model struggles to generate the palest skin tones. We conducted various ablation studies, in which we removed DebiasPI's attribute control, that reveal the model's propensity to generate young, male characters. It sometimes visualized career success by generating two-panel images with a pre-success dark-skinned person becoming light-skinned with success, or switching gender from pre-success female to post-success male, thus further motivating ethical intervention prompting with DebiasPI.

Title: Prompt-oriented Output of Culture-Specific Items in Translated African Poetry by Large Language Model: An Initial Multi-layered Tabular Review

• Authors: Adeyola Opaluwah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18644
• Pdf URL: https://arxiv.org/pdf/2501.18644
• Copy Paste: [[2501.18644]] Prompt-oriented Output of Culture-Specific Items in Translated African Poetry by Large Language Model: An Initial Multi-layered Tabular Review(https://arxiv.org/abs/2501.18644)
• Keywords: generative
• Abstract: This paper examines the output of cultural items generated by Chat Generative PreTrained Transformer Pro in response to three structured prompts to translate three anthologies of African poetry. The first prompt was broad, the second focused on poetic structure, and the third prompt emphasized cultural specificity. To support this analysis, four comparative tables were created. The first table presents the results of the cultural items produced after the three prompts, the second categorizes these outputs based on Aixela framework of Proper nouns and Common expressions, the third table summarizes the cultural items generated by human translators, a custom translation engine, and a Large Language Model. The final table outlines the strategies employed by Chat Generative PreTrained Transformer Pro following the culture specific prompt. Compared to the outputs of cultural items from reference human translation and the custom translation engine in prior studies the findings indicate that the culture oriented prompts used with Chat Generative PreTrained Transformer Pro did not yield significant enhancements of cultural items during the translation of African poetry from English to French. Among the fifty four cultural items, the human translation produced thirty three cultural items in repetition, the custom translation engine generated Thirty eight cultural items in repetition while Chat Generative PreTrained Transformer Pro produced forty one cultural items in repetition. The untranslated cultural items revealed inconsistencies in Large language models approach to translating cultural items in African poetry from English to French.

Title: Unpaired Translation of Point Clouds for Modeling Detector Response

• Authors: Mingyang Li, Michelle Kuchera, Raghuram Ramanujan, Adam Anthony, Curtis Hunt, Yassid Ayyad
• Subjects: cs.CV, cs.LG, nucl-ex
• Abstract URL: https://arxiv.org/abs/2501.18674
• Pdf URL: https://arxiv.org/pdf/2501.18674
• Copy Paste: [[2501.18674]] Unpaired Translation of Point Clouds for Modeling Detector Response(https://arxiv.org/abs/2501.18674)
• Keywords: diffusion
• Abstract: Modeling detector response is a key challenge in time projection chambers. We cast this problem as an unpaired point cloud translation task, between data collected from simulations and from experimental runs. Effective translation can assist with both noise rejection and the construction of high-fidelity simulators. Building on recent work in diffusion probabilistic models, we present a novel framework for performing this mapping. We demonstrate the success of our approach in both synthetic domains and in data sourced from the Active-Target Time Projection Chamber.

Title: Regularized second-order optimization of tensor-network Born machines

• Authors: Matan Ben-Dov, Jing Chen
• Subjects: cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2501.18691
• Pdf URL: https://arxiv.org/pdf/2501.18691
• Copy Paste: [[2501.18691]] Regularized second-order optimization of tensor-network Born machines(https://arxiv.org/abs/2501.18691)
• Keywords: generative
• Abstract: Tensor-network Born machines (TNBMs) are quantum-inspired generative models for learning data distributions. Using tensor-network contraction and optimization techniques, the model learns an efficient representation of the target distribution, capable of capturing complex correlations with a compact parameterization. Despite their promise, the optimization of TNBMs presents several challenges. A key bottleneck of TNBMs is the logarithmic nature of the loss function that is commonly used for this problem. The single-tensor logarithmic optimization problem cannot be solved analytically, necessitating an iterative approach that slows down convergence and increases the risk of getting trapped in one of many non-optimal local minima. In this paper, we present an improved second-order optimization technique for TNBM training, which significantly enhances convergence rates and the quality of the optimized model. Our method employs a modified Newton's method on the manifold of normalized states, incorporating regularization of the loss landscape to mitigate local minima issues. We demonstrate the effectiveness of our approach by training a one-dimensional matrix product state (MPS) on both discrete and continuous datasets, showcasing its advantages in terms of stability, efficiency, and generalization.

Title: Strong and Controllable 3D Motion Generation

• Authors: Canxuan Gang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18726
• Pdf URL: https://arxiv.org/pdf/2501.18726
• Copy Paste: [[2501.18726]] Strong and Controllable 3D Motion Generation(https://arxiv.org/abs/2501.18726)
• Keywords: diffusion, generative
• Abstract: Human motion generation is a significant pursuit in generative computer vision with widespread applications in film-making, video games, AR/VR, and human-robot interaction. Current methods mainly utilize either diffusion-based generative models or autoregressive models for text-to-motion generation. However, they face two significant challenges: (1) The generation process is time-consuming, posing a major obstacle for real-time applications such as gaming, robot manipulation, and other online settings. (2) These methods typically learn a relative motion representation guided by text, making it difficult to generate motion sequences with precise joint-level control. These challenges significantly hinder progress and limit the real-world application of human motion generation techniques. To address this gap, we propose a simple yet effective architecture consisting of two key components. Firstly, we aim to improve hardware efficiency and computational complexity in transformer-based diffusion models for human motion generation. By customizing flash linear attention, we can optimize these models specifically for generating human motion efficiently. Furthermore, we will customize the consistency model in the motion latent space to further accelerate motion generation. Secondly, we introduce Motion ControlNet, which enables more precise joint-level control of human motion compared to previous text-to-motion generation methods. These contributions represent a significant advancement for text-to-motion generation, bringing it closer to real-world applications.

Title: Motion Diffusion Autoencoders: Enabling Attribute Manipulation in Human Motion Demonstrated on Karate Techniques

• Authors: Anthony Mendil, Felix Putze
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18729
• Pdf URL: https://arxiv.org/pdf/2501.18729
• Copy Paste: [[2501.18729]] Motion Diffusion Autoencoders: Enabling Attribute Manipulation in Human Motion Demonstrated on Karate Techniques(https://arxiv.org/abs/2501.18729)
• Keywords: diffusion
• Abstract: Attribute manipulation deals with the problem of changing individual attributes of a data point or a time series, while leaving all other aspects unaffected. This work focuses on the domain of human motion, more precisely karate movement patterns. To the best of our knowledge, it presents the first success at manipulating attributes of human motion data. One of the key requirements for achieving attribute manipulation on human motion is a suitable pose representation. Therefore, we design a novel rotation-based pose representation that enables the disentanglement of the human skeleton and the motion trajectory, while still allowing an accurate reconstruction of the original anatomy. The core idea of the manipulation approach is to use a transformer encoder for discovering high-level semantics, and a diffusion probabilistic model for modeling the remaining stochastic variations. We show that the embedding space obtained from the transformer encoder is semantically meaningful and linear. This enables the manipulation of high-level attributes, by discovering their linear direction of change in the semantic embedding space and moving the embedding along said direction. The code and data are available at this https URL.

Title: Synthetic Data Generation for Augmenting Small Samples

• Authors: Dan Liu, Samer El Kababji, Nicholas Mitsakakis, Lisa Pilgram, Thomas Walters, Mark Clemons, Greg Pond, Alaa El-Hussuna, Khaled El Emam
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18741
• Pdf URL: https://arxiv.org/pdf/2501.18741
• Copy Paste: [[2501.18741]] Synthetic Data Generation for Augmenting Small Samples(https://arxiv.org/abs/2501.18741)
• Keywords: generative
• Abstract: Small datasets are common in health research. However, the generalization performance of machine learning models is suboptimal when the training datasets are small. To address this, data augmentation is one solution. Augmentation increases sample size and is seen as a form of regularization that increases the diversity of small datasets, leading them to perform better on unseen data. We found that augmentation improves prognostic performance for datasets that: have fewer observations, with smaller baseline AUC, have higher cardinality categorical variables, and have more balanced outcome variables. No specific generative model consistently outperformed the others. We developed a decision support model that can be used to inform analysts if augmentation would be useful. For seven small application datasets, augmenting the existing data results in an increase in AUC between 4.31% (AUC from 0.71 to 0.75) and 43.23% (AUC from 0.51 to 0.73), with an average 15.55% relative improvement, demonstrating the nontrivial impact of augmentation on small datasets (p=0.0078). Augmentation AUC was higher than resampling only AUC (p=0.016). The diversity of augmented datasets was higher than the diversity of resampled datasets (p=0.046).

Title: Probabilistic Joint Recovery Method for CO$_2$ Plume Monitoring

• Authors: Zijun Deng, Rafael Orozco, Abhinav Prakash Gahlot, Felix J. Herrmann
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2501.18761
• Pdf URL: https://arxiv.org/pdf/2501.18761
• Copy Paste: [[2501.18761]] Probabilistic Joint Recovery Method for CO$_2$ Plume Monitoring(https://arxiv.org/abs/2501.18761)
• Keywords: generative
• Abstract: Reducing CO$_2$ emissions is crucial to mitigating climate change. Carbon Capture and Storage (CCS) is one of the few technologies capable of achieving net-negative CO$_2$ emissions. However, predicting fluid flow patterns in CCS remains challenging due to uncertainties in CO$_2$ plume dynamics and reservoir properties. Building on existing seismic imaging methods like the Joint Recovery Method (JRM), which lacks uncertainty quantification, we propose the Probabilistic Joint Recovery Method (pJRM). By estimating posterior distributions across surveys using a shared generative model, pJRM provides uncertainty information to improve risk assessment in CCS projects.

Title: Navigating the Fragrance space Via Graph Generative Models And Predicting Odors

• Authors: Mrityunjay Sharma, Sarabeshwar Balaji, Pinaki Saha, Ritesh Kumar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18777
• Pdf URL: https://arxiv.org/pdf/2501.18777
• Copy Paste: [[2501.18777]] Navigating the Fragrance space Via Graph Generative Models And Predicting Odors(https://arxiv.org/abs/2501.18777)
• Keywords: generative
• Abstract: We explore a suite of generative modelling techniques to efficiently navigate and explore the complex landscapes of odor and the broader chemical space. Unlike traditional approaches, we not only generate molecules but also predict the odor likeliness with ROC AUC score of 0.97 and assign probable odor labels. We correlate odor likeliness with physicochemical features of molecules using machine learning techniques and leverage SHAP (SHapley Additive exPlanations) to demonstrate the interpretability of the function. The whole process involves four key stages: molecule generation, stringent sanitization checks for molecular validity, fragrance likeliness screening and odor prediction of the generated molecules. By making our code and trained models publicly accessible, we aim to facilitate broader adoption of our research across applications in fragrance discovery and olfactory research.

Title: Compositional Generalization Requires More Than Disentangled Representations

• Authors: Qiyao Liang, Daoyuan Qian, Liu Ziyin, Ila Fiete
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18797
• Pdf URL: https://arxiv.org/pdf/2501.18797
• Copy Paste: [[2501.18797]] Compositional Generalization Requires More Than Disentangled Representations(https://arxiv.org/abs/2501.18797)
• Keywords: generative
• Abstract: Composition-the ability to generate myriad variations from finite means-is believed to underlie powerful generalization. However, compositional generalization remains a key challenge for deep learning. A widely held assumption is that learning disentangled (factorized) representations naturally supports this kind of extrapolation. Yet, empirical results are mixed, with many generative models failing to recognize and compose factors to generate out-of-distribution (OOD) samples. In this work, we investigate a controlled 2D Gaussian "bump" generation task, demonstrating that standard generative architectures fail in OOD regions when training with partial data, even when supplied with fully disentangled $(x, y)$ coordinates, re-entangling them through subsequent layers. By examining the model's learned kernels and manifold geometry, we show that this failure reflects a "memorization" strategy for generation through the superposition of training data rather than by combining the true factorized features. We show that models forced-through architectural modifications with regularization or curated training data-to create disentangled representations in the full-dimensional representational (pixel) space can be highly data-efficient and effective at learning to compose in OOD regions. These findings underscore that bottlenecks with factorized/disentangled representations in an abstract representation are insufficient: the model must actively maintain or induce factorization directly in the representational space in order to achieve robust compositional generalization.

Title: Every Image Listens, Every Image Dances: Music-Driven Image Animation

• Authors: Zhikang Dong, Weituo Hao, Ju-Chiang Wang, Peng Zhang, Pawel Polak
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18801
• Pdf URL: https://arxiv.org/pdf/2501.18801
• Copy Paste: [[2501.18801]] Every Image Listens, Every Image Dances: Music-Driven Image Animation(https://arxiv.org/abs/2501.18801)
• Keywords: diffusion
• Abstract: Image animation has become a promising area in multimodal research, with a focus on generating videos from reference images. While prior work has largely emphasized generic video generation guided by text, music-driven dance video generation remains underexplored. In this paper, we introduce MuseDance, an innovative end-to-end model that animates reference images using both music and text inputs. This dual input enables MuseDance to generate personalized videos that follow text descriptions and synchronize character movements with the music. Unlike existing approaches, MuseDance eliminates the need for complex motion guidance inputs, such as pose or depth sequences, making flexible and creative video generation accessible to users of all expertise levels. To advance research in this field, we present a new multimodal dataset comprising 2,904 dance videos with corresponding background music and text descriptions. Our approach leverages diffusion-based methods to achieve robust generalization, precise control, and temporal consistency, setting a new baseline for the music-driven image animation task.

Title: Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion

• Authors: Vitor Guizilini, Muhammad Zubair Irshad, Dian Chen, Greg Shakhnarovich, Rares Ambrus
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18804
• Pdf URL: https://arxiv.org/pdf/2501.18804
• Copy Paste: [[2501.18804]] Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion(https://arxiv.org/abs/2501.18804)
• Keywords: diffusion
• Abstract: Current methods for 3D scene reconstruction from sparse posed images employ intermediate 3D representations such as neural fields, voxel grids, or 3D Gaussians, to achieve multi-view consistent scene appearance and geometry. In this paper we introduce MVGD, a diffusion-based architecture capable of direct pixel-level generation of images and depth maps from novel viewpoints, given an arbitrary number of input views. Our method uses raymap conditioning to both augment visual features with spatial information from different viewpoints, as well as to guide the generation of images and depth maps from novel views. A key aspect of our approach is the multi-task generation of images and depth maps, using learnable task embeddings to guide the diffusion process towards specific modalities. We train this model on a collection of more than 60 million multi-view samples from publicly available datasets, and propose techniques to enable efficient and consistent learning in such diverse conditions. We also propose a novel strategy that enables the efficient training of larger models by incrementally fine-tuning smaller ones, with promising scaling behavior. Through extensive experiments, we report state-of-the-art results in multiple novel view synthesis benchmarks, as well as multi-view stereo and video depth estimation.

Title: An Optimal Cascade Feature-Level Spatiotemporal Fusion Strategy for Anomaly Detection in CAN Bus

• Authors: Mohammad Fatahi, Danial Sadrian Zadeh, Benyamin Ghojogh, Behzad Moshiri, Otman Basir
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.18821
• Pdf URL: https://arxiv.org/pdf/2501.18821
• Copy Paste: [[2501.18821]] An Optimal Cascade Feature-Level Spatiotemporal Fusion Strategy for Anomaly Detection in CAN Bus(https://arxiv.org/abs/2501.18821)
• Keywords: anomaly
• Abstract: Autonomous vehicles represent a revolutionary advancement driven by the integration of artificial intelligence within intelligent transportation systems. However, they remain vulnerable due to the absence of robust security mechanisms in the Controller Area Network (CAN) bus. In order to mitigate the security issue, many machine learning models and strategies have been proposed, which primarily focus on a subset of dominant patterns of anomalies and lack rigorous evaluation in terms of reliability and robustness. Therefore, to address the limitations of previous works and mitigate the security vulnerability in CAN bus, the current study develops a model based on the intrinsic nature of the problem to cover all dominant patterns of anomalies. To achieve this, a cascade feature-level fusion strategy optimized by a two-parameter genetic algorithm is proposed to combine temporal and spatial information. Subsequently, the model is evaluated using a paired t-test to ensure reliability and robustness. Finally, a comprehensive comparative analysis conducted on two widely used datasets advocates that the proposed model outperforms other models and achieves superior accuracy and F1-score, demonstrating the best performance among all models presented to date.

Title: REG: Rectified Gradient Guidance for Conditional Diffusion Models

• Authors: Zhengqi Gao, Kaiwen Zha, Tianyuan Zhang, Zihui Xue, Duane S. Boning
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18865
• Pdf URL: https://arxiv.org/pdf/2501.18865
• Copy Paste: [[2501.18865]] REG: Rectified Gradient Guidance for Conditional Diffusion Models(https://arxiv.org/abs/2501.18865)
• Keywords: diffusion
• Abstract: Guidance techniques are simple yet effective for improving conditional generation in diffusion models. Albeit their empirical success, the practical implementation of guidance diverges significantly from its theoretical motivation. In this paper, we reconcile this discrepancy by replacing the scaled marginal distribution target, which we prove theoretically invalid, with a valid scaled joint distribution objective. Additionally, we show that the established guidance implementations are approximations to the intractable optimal solution under no future foresight constraint. Building on these theoretical insights, we propose rectified gradient guidance (REG), a versatile enhancement designed to boost the performance of existing guidance methods. Experiments on 1D and 2D demonstrate that REG provides a better approximation to the optimal solution than prior guidance techniques, validating the proposed theoretical framework. Extensive experiments on class-conditional ImageNet and text-to-image generation tasks show that incorporating REG consistently improves FID and Inception/CLIP scores across various settings compared to its absence.

Title: Neural SDEs as a Unified Approach to Continuous-Domain Sequence Modeling

• Authors: Macheng Shen, Chen Cheng
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18871
• Pdf URL: https://arxiv.org/pdf/2501.18871
• Copy Paste: [[2501.18871]] Neural SDEs as a Unified Approach to Continuous-Domain Sequence Modeling(https://arxiv.org/abs/2501.18871)
• Keywords: diffusion, generative
• Abstract: Inspired by the ubiquitous use of differential equations to model continuous dynamics across diverse scientific and engineering domains, we propose a novel and intuitive approach to continuous sequence modeling. Our method interprets time-series data as \textit{discrete samples from an underlying continuous dynamical system}, and models its time evolution using Neural Stochastic Differential Equation (Neural SDE), where both the flow (drift) and diffusion terms are parameterized by neural networks. We derive a principled maximum likelihood objective and a \textit{simulation-free} scheme for efficient training of our Neural SDE model. We demonstrate the versatility of our approach through experiments on sequence modeling tasks across both embodied and generative AI. Notably, to the best of our knowledge, this is the first work to show that SDE-based continuous-time modeling also excels in such complex scenarios, and we hope that our work opens up new avenues for research of SDE models in high-dimensional and temporally intricate domains.

Title: Best Policy Learning from Trajectory Preference Feedback

• Authors: Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Zheng Wen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18873
• Pdf URL: https://arxiv.org/pdf/2501.18873
• Copy Paste: [[2501.18873]] Best Policy Learning from Trajectory Preference Feedback(https://arxiv.org/abs/2501.18873)
• Keywords: generative
• Abstract: We address the problem of best policy identification in preference-based reinforcement learning (PbRL), where learning occurs from noisy binary preferences over trajectory pairs rather than explicit numerical rewards. This approach is useful for post-training optimization of generative AI models during multi-turn user interactions, where preference feedback is more robust than handcrafted reward models. In this setting, learning is driven by both an offline preference dataset -- collected from a rater of unknown 'competence' -- and online data collected with pure exploration. Since offline datasets may exhibit out-of-distribution (OOD) biases, principled online data collection is necessary. To address this, we propose Posterior Sampling for Preference Learning ($\mathsf{PSPL}$), a novel algorithm inspired by Top-Two Thompson Sampling, that maintains independent posteriors over the true reward model and transition dynamics. We provide the first theoretical guarantees for PbRL in this setting, establishing an upper bound on the simple Bayesian regret of $\mathsf{PSPL}$. Since the exact algorithm can be computationally impractical, we also provide an approximate version that outperforms existing baselines.

Title: Self-Supervised Learning Using Nonlinear Dependence

• Authors: M.Hadi Sepanj, Benyamin Ghojogh, Paul Fieguth
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18875
• Pdf URL: https://arxiv.org/pdf/2501.18875
• Copy Paste: [[2501.18875]] Self-Supervised Learning Using Nonlinear Dependence(https://arxiv.org/abs/2501.18875)
• Keywords: self-supervised
• Abstract: Self-supervised learning has gained significant attention in contemporary applications, particularly due to the scarcity of labeled data. While existing SSL methodologies primarily address feature variance and linear correlations, they often neglect the intricate relations between samples and the nonlinear dependencies inherent in complex data. In this paper, we introduce Correlation-Dependence Self-Supervised Learning (CDSSL), a novel framework that unifies and extends existing SSL paradigms by integrating both linear correlations and nonlinear dependencies, encapsulating sample-wise and feature-wise interactions. Our approach incorporates the Hilbert-Schmidt Independence Criterion (HSIC) to robustly capture nonlinear dependencies within a Reproducing Kernel Hilbert Space, enriching representation learning. Experimental evaluations on diverse benchmarks demonstrate the efficacy of CDSSL in improving representation quality.

Title: Distorting Embedding Space for Safety: A Defense Mechanism for Adversarially Robust Diffusion Models

• Authors: Jaesin Ahn, Heechul Jung
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18877
• Pdf URL: https://arxiv.org/pdf/2501.18877
• Copy Paste: [[2501.18877]] Distorting Embedding Space for Safety: A Defense Mechanism for Adversarially Robust Diffusion Models(https://arxiv.org/abs/2501.18877)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models show remarkable generation performance following text prompts, but risk generating Not Safe For Work (NSFW) contents from unsafe prompts. Existing approaches, such as prompt filtering or concept unlearning, fail to defend against adversarial attacks while maintaining benign image quality. In this paper, we propose a novel approach called Distorting Embedding Space (DES), a text encoder-based defense mechanism that effectively tackles these issues through innovative embedding space control. DES transforms unsafe embeddings, extracted from a text encoder using unsafe prompts, toward carefully calculated safe embedding regions to prevent unsafe contents generation, while reproducing the original safe embeddings. DES also neutralizes the nudity embedding, extracted using prompt ``nudity", by aligning it with neutral embedding to enhance robustness against adversarial attacks. These methods ensure both robust defense and high-quality image generation. Additionally, DES can be adopted in a plug-and-play manner and requires zero inference overhead, facilitating its deployment. Extensive experiments on diverse attack types, including black-box and white-box scenarios, demonstrate DES's state-of-the-art performance in both defense capability and benign image generation quality. Our model is available at this https URL.

Title: Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posterior

• Authors: Tongda Xu, Xiyan Cai, Xinjie Zhang, Xingtong Ge, Dailan He, Ming Sun, Jingjing Liu, Ya-Qin Zhang, Jian Li, Yan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18913
• Pdf URL: https://arxiv.org/pdf/2501.18913
• Copy Paste: [[2501.18913]] Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posterior(https://arxiv.org/abs/2501.18913)
• Keywords: diffusion
• Abstract: Recent advancements in diffusion models have been leveraged to address inverse problems without additional training, and Diffusion Posterior Sampling (DPS) (Chung et al., 2022a) is among the most popular approaches. Previous analyses suggest that DPS accomplishes posterior sampling by approximating the conditional score. While in this paper, we demonstrate that the conditional score approximation employed by DPS is not as effective as previously assumed, but rather aligns more closely with the principle of maximizing a posterior (MAP). This assertion is substantiated through an examination of DPS on 512x512 ImageNet images, revealing that: 1) DPS's conditional score estimation significantly diverges from the score of a well-trained conditional diffusion model and is even inferior to the unconditional score; 2) The mean of DPS's conditional score estimation deviates significantly from zero, rendering it an invalid score estimation; 3) DPS generates high-quality samples with significantly lower diversity. In light of the above findings, we posit that DPS more closely resembles MAP than a conditional score estimator, and accordingly propose the following enhancements to DPS: 1) we explicitly maximize the posterior through multi-step gradient ascent and projection; 2) we utilize a light-weighted conditional score estimator trained with only 100 images and 8 GPU hours. Extensive experimental results indicate that these proposed improvements significantly enhance DPS's performance. The source code for these improvements is provided in this https URL.

Title: LLM Program Optimization via Retrieval Augmented Search

• Authors: Sagnik Anupam, Alexander Shypula, Osbert Bastani
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18916
• Pdf URL: https://arxiv.org/pdf/2501.18916
• Copy Paste: [[2501.18916]] LLM Program Optimization via Retrieval Augmented Search(https://arxiv.org/abs/2501.18916)
• Keywords: in-context
• Abstract: With the advent of large language models (LLMs), there has been a great deal of interest in applying them to solve difficult programming tasks. Recent work has demonstrated their potential at program optimization, a key challenge in programming languages research. We propose a blackbox adaptation method called Retrieval Augmented Search (RAS) that performs beam search over candidate optimizations; at each step, it retrieves in-context examples from a given training dataset of slow-fast program pairs to guide the LLM. Critically, we find that performing contextual retrieval based on an LLM-generated natural language description significantly outperforms retrieval based on the source code. In addition, we propose a method called AEGIS for improving interpretability by decomposing training examples into "atomic edits" that are significantly more incremental in nature. We show that RAS performs 1.8$\times$ better than prior state-of-the-art blackbox adaptation strategies, and that AEGIS performs 1.37$\times$ better while performing significantly smaller edits.

Title: Training-free Quantum-Inspired Image Edge Extraction Method

• Authors: Arti Jain, Pradeep Singh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18929
• Pdf URL: https://arxiv.org/pdf/2501.18929
• Copy Paste: [[2501.18929]] Training-free Quantum-Inspired Image Edge Extraction Method(https://arxiv.org/abs/2501.18929)
• Keywords: diffusion
• Abstract: Edge detection is a cornerstone of image processing, yet existing methods often face critical limitations. Traditional deep learning edge detection methods require extensive training datasets and fine-tuning, while classical techniques often fail in complex or noisy scenarios, limiting their real-world applicability. To address these limitations, we propose a training-free, quantum-inspired edge detection model. Our approach integrates classical Sobel edge detection, the Schrödinger wave equation refinement, and a hybrid framework combining Canny and Laplacian operators. By eliminating the need for training, the model is lightweight and adaptable to diverse applications. The Schrödinger wave equation refines gradient-based edge maps through iterative diffusion, significantly enhancing edge precision. The hybrid framework further strengthens the model by synergistically combining local and global features, ensuring robustness even under challenging conditions. Extensive evaluations on datasets like BIPED, Multicue, and NYUD demonstrate superior performance of the proposed model, achieving state-of-the-art metrics, including ODS, OIS, AP, and F-measure. Noise robustness experiments highlight its reliability, showcasing its practicality for real-world scenarios. Due to its versatile and adaptable nature, our model is well-suited for applications such as medical imaging, autonomous systems, and environmental monitoring, setting a new benchmark for edge detection.

Title: Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them

• Authors: Anh Bui, Trang Vu, Long Vuong, Trung Le, Paul Montague, Tamas Abraham, Junae Kim, Dinh Phung
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18950
• Pdf URL: https://arxiv.org/pdf/2501.18950
• Copy Paste: [[2501.18950]] Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them(https://arxiv.org/abs/2501.18950)
• Keywords: diffusion
• Abstract: Concept erasure has emerged as a promising technique for mitigating the risk of harmful content generation in diffusion models by selectively unlearning undesirable concepts. The common principle of previous works to remove a specific concept is to map it to a fixed generic concept, such as a neutral concept or just an empty text prompt. In this paper, we demonstrate that this fixed-target strategy is suboptimal, as it fails to account for the impact of erasing one concept on the others. To address this limitation, we model the concept space as a graph and empirically analyze the effects of erasing one concept on the remaining concepts. Our analysis uncovers intriguing geometric properties of the concept space, where the influence of erasing a concept is confined to a local region. Building on this insight, we propose the Adaptive Guided Erasure (AGE) method, which \emph{dynamically} selects optimal target concepts tailored to each undesirable concept, minimizing unintended side effects. Experimental results show that AGE significantly outperforms state-of-the-art erasure methods on preserving unrelated concepts while maintaining effective erasure performance. Our code is published at {this https URL}.

Title: Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Boostrapping

• Authors: Pu Yang, Yunzhen Feng, Ziyuan Chen, Yuhang Wu, Zhuoyuan Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18962
• Pdf URL: https://arxiv.org/pdf/2501.18962
• Copy Paste: [[2501.18962]] Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Boostrapping(https://arxiv.org/abs/2501.18962)
• Keywords: diffusion, foundation model
• Abstract: Modern foundation models often undergo iterative ``bootstrapping'' in their post-training phase: a model generates synthetic data, an external verifier filters out low-quality samples, and the high-quality subset is used for further fine-tuning. Over multiple iterations, the model's performance improves--raising a crucial question: how should the total budget on generation and training be allocated across iterations to maximize final performance? In this work, we develop a theoretical framework to analyze budget allocation strategies. Specifically, we show that constant policies fail to converge with high probability, while increasing policies--particularly exponential growth policies--exhibit significant theoretical advantages. Experiments on image denoising with diffusion probabilistic models and math reasoning with large language models show that both exponential and polynomial growth policies consistently outperform constant policies, with exponential policies often providing more stable performance.

Title: BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics

• Authors: Yuxuan Liu, Jingmin Sun, Hayden Schaeffer
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2501.18972
• Pdf URL: https://arxiv.org/pdf/2501.18972
• Copy Paste: [[2501.18972]] BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics(https://arxiv.org/abs/2501.18972)
• Keywords: foundation model
• Abstract: We introduce BCAT, a PDE foundation model designed for autoregressive prediction of solutions to two dimensional fluid dynamics problems. Our approach uses a block causal transformer architecture to model next frame predictions, leveraging previous frames as contextual priors rather than relying solely on sub-frames or pixel-based inputs commonly used in image generation methods. This block causal framework more effectively captures the spatial dependencies inherent in nonlinear spatiotemporal dynamics and physical phenomena. In an ablation study, next frame prediction demonstrated a 2.9x accuracy improvement over next token prediction. BCAT is trained on a diverse range of fluid dynamics datasets, including incompressible and compressible Navier-Stokes equations across various geometries and parameter regimes, as well as the shallow-water equations. The model's performance was evaluated on 6 distinct downstream prediction tasks and tested on about 8K trajectories to measure robustness on a variety of fluid dynamics simulations. BCAT achieved an average relative error of 1.92% across all evaluation tasks, outperforming prior approaches on standard benchmarks.

Title: OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation

• Authors: Yuchen Lin, Chenguo Lin, Jianjin Xu, Yadong Mu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18982
• Pdf URL: https://arxiv.org/pdf/2501.18982
• Copy Paste: [[2501.18982]] OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation(https://arxiv.org/abs/2501.18982)
• Keywords: diffusion
• Abstract: Recently, significant advancements have been made in the reconstruction and generation of 3D assets, including static cases and those with physical interactions. To recover the physical properties of 3D assets, existing methods typically assume that all materials belong to a specific predefined category (e.g., elasticity). However, such assumptions ignore the complex composition of multiple heterogeneous objects in real scenarios and tend to render less physically plausible animation given a wider range of objects. We propose OmniPhysGS for synthesizing a physics-based 3D dynamic scene composed of more general objects. A key design of OmniPhysGS is treating each 3D asset as a collection of constitutive 3D Gaussians. For each Gaussian, its physical material is represented by an ensemble of 12 physical domain-expert sub-models (rubber, metal, honey, water, etc.), which greatly enhances the flexibility of the proposed model. In the implementation, we define a scene by user-specified prompts and supervise the estimation of material weighting factors via a pretrained video diffusion model. Comprehensive experiments demonstrate that OmniPhysGS achieves more general and realistic physical dynamics across a broader spectrum of materials, including elastic, viscoelastic, plastic, and fluid substances, as well as interactions between different materials. Our method surpasses existing methods by approximately 3% to 16% in metrics of visual quality and text alignment.

Title: Visual Autoregressive Modeling for Image Super-Resolution

• Authors: Yunpeng Qu, Kun Yuan, Jinhua Hao, Kai Zhao, Qizhi Xie, Ming Sun, Chao Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18993
• Pdf URL: https://arxiv.org/pdf/2501.18993
• Copy Paste: [[2501.18993]] Visual Autoregressive Modeling for Image Super-Resolution(https://arxiv.org/abs/2501.18993)
• Keywords: diffusion, generative
• Abstract: Image Super-Resolution (ISR) has seen significant progress with the introduction of remarkable generative models. However, challenges such as the trade-off issues between fidelity and realism, as well as computational complexity, have also posed limitations on their application. Building upon the tremendous success of autoregressive models in the language domain, we propose \textbf{VARSR}, a novel visual autoregressive modeling for ISR framework with the form of next-scale prediction. To effectively integrate and preserve semantic information in low-resolution images, we propose using prefix tokens to incorporate the condition. Scale-aligned Rotary Positional Encodings are introduced to capture spatial structures and the diffusion refiner is utilized for modeling quantization residual loss to achieve pixel-level fidelity. Image-based Classifier-free Guidance is proposed to guide the generation of more realistic images. Furthermore, we collect large-scale data and design a training process to obtain robust generative priors. Quantitative and qualitative results show that VARSR is capable of generating high-fidelity and high-realism images with more efficiency than diffusion-based methods. Our codes will be released at this https URL.

Title: XRF V2: A Dataset for Action Summarization with Wi-Fi Signals, and IMUs in Phones, Watches, Earbuds, and Glasses

• Authors: Bo Lan, Pei Li, Jiaxi Yin, Yunpeng Song, Ge Wang, Han Ding, Jinsong Han, Fei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19034
• Pdf URL: https://arxiv.org/pdf/2501.19034
• Copy Paste: [[2501.19034]] XRF V2: A Dataset for Action Summarization with Wi-Fi Signals, and IMUs in Phones, Watches, Earbuds, and Glasses(https://arxiv.org/abs/2501.19034)
• Keywords: foundation model
• Abstract: Human Action Recognition (HAR) plays a crucial role in applications such as health monitoring, smart home automation, and human-computer interaction. While HAR has been extensively studied, action summarization, which involves identifying and summarizing continuous actions, remains an emerging task. This paper introduces the novel XRF V2 dataset, designed for indoor daily activity Temporal Action Localization (TAL) and action summarization. XRF V2 integrates multimodal data from Wi-Fi signals, IMU sensors (smartphones, smartwatches, headphones, and smart glasses), and synchronized video recordings, offering a diverse collection of indoor activities from 16 volunteers across three distinct environments. To tackle TAL and action summarization, we propose the XRFMamba neural network, which excels at capturing long-term dependencies in untrimmed sensory sequences and outperforms state-of-the-art methods, such as ActionFormer and WiFiTAD. We envision XRF V2 as a valuable resource for advancing research in human action localization, action forecasting, pose estimation, multimodal foundation models pre-training, synthetic data generation, and more.

Title: Self-Supervised Cross-Modal Text-Image Time Series Retrieval in Remote Sensing

• Authors: Genc Hoxha, Olivér Angyal, Begüm Demir
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19043
• Pdf URL: https://arxiv.org/pdf/2501.19043
• Copy Paste: [[2501.19043]] Self-Supervised Cross-Modal Text-Image Time Series Retrieval in Remote Sensing(https://arxiv.org/abs/2501.19043)
• Keywords: self-supervised
• Abstract: The development of image time series retrieval (ITSR) methods is a growing research interest in remote sensing (RS). Given a user-defined image time series (i.e., the query time series), the ITSR methods search and retrieve from large archives the image time series that have similar content to the query time series. The existing ITSR methods in RS are designed for unimodal retrieval problems, limiting their usability and versatility. To overcome this issue, as a first time in RS we introduce the task of cross-modal text-ITSR. In particular, we present a self-supervised cross-modal text-image time series retrieval (text-ITSR) method that enables the retrieval of image time series using text sentences as queries, and vice versa. In detail, we focus our attention on text-ITSR in pairs of images (i.e., bitemporal images). The proposed text-ITSR method consists of two key components: 1) modality-specific encoders to model the semantic content of bitemporal images and text sentences with discriminative features; and 2) modality-specific projection heads to align textual and image representations in a shared embedding space. To effectively model the temporal information within the bitemporal images, we introduce two fusion strategies: i) global feature fusion (GFF) strategy that combines global image features through simple yet effective operators; and ii) transformer-based feature fusion (TFF) strategy that leverages transformers for fine-grained temporal integration. Extensive experiments conducted on two benchmark RS archives demonstrate the effectiveness of the proposed method in accurately retrieving semantically relevant bitemporal images (or text sentences) to a query text sentence (or bitemporal image). The code of this work is publicly available at this https URL.

Title: Concept Steerers: Leveraging K-Sparse Autoencoders for Controllable Generations

• Authors: Dahye Kim, Deepti Ghadiyaram
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19066
• Pdf URL: https://arxiv.org/pdf/2501.19066
• Copy Paste: [[2501.19066]] Concept Steerers: Leveraging K-Sparse Autoencoders for Controllable Generations(https://arxiv.org/abs/2501.19066)
• Keywords: diffusion, generative
• Abstract: Despite the remarkable progress in text-to-image generative models, they are prone to adversarial attacks and inadvertently generate unsafe, unethical content. Existing approaches often rely on fine-tuning models to remove specific concepts, which is computationally expensive, lack scalability, and/or compromise generation quality. In this work, we propose a novel framework leveraging k-sparse autoencoders (k-SAEs) to enable efficient and interpretable concept manipulation in diffusion models. Specifically, we first identify interpretable monosemantic concepts in the latent space of text embeddings and leverage them to precisely steer the generation away or towards a given concept (e.g., nudity) or to introduce a new concept (e.g., photographic style). Through extensive experiments, we demonstrate that our approach is very simple, requires no retraining of the base model nor LoRA adapters, does not compromise the generation quality, and is robust to adversarial prompt manipulations. Our method yields an improvement of $\mathbf{20.01\%}$ in unsafe concept removal, is effective in style manipulation, and is $\mathbf{\sim5}$x faster than current state-of-the-art.

Title: A Bias-Correction Decentralized Stochastic Gradient Algorithm with Momentum Acceleration

• Authors: Yuchen Hu, Xi Chen, Weidong Liu, Xiaojun Mao
• Subjects: cs.LG, cs.DC, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.19082
• Pdf URL: https://arxiv.org/pdf/2501.19082
• Copy Paste: [[2501.19082]] A Bias-Correction Decentralized Stochastic Gradient Algorithm with Momentum Acceleration(https://arxiv.org/abs/2501.19082)
• Keywords: diffusion
• Abstract: Distributed stochastic optimization algorithms can handle large-scale data simultaneously and accelerate model training. However, the sparsity of distributed networks and the heterogeneity of data limit these advantages. This paper proposes a momentum-accelerated distributed stochastic gradient algorithm, referred to as Exact-Diffusion with Momentum (EDM), which can correct the bias caused by data heterogeneity and introduces the momentum method commonly used in deep learning to accelerate the convergence of the algorithm. We theoretically demonstrate that this algorithm converges to the neighborhood of the optimum sub-linearly irrelevant to data heterogeneity when applied to non-convex objective functions and linearly under the Polyak-Łojasiewicz condition (a weaker assumption than $\mu$-strongly convexity). Finally, we evaluate the performance of the proposed algorithm by simulation, comparing it with a range of existing decentralized optimization algorithms to demonstrate its effectiveness in addressing data heterogeneity and network sparsity.

Title: MotionPCM: Real-Time Motion Synthesis with Phased Consistency Model

• Authors: Lei Jiang, Ye Wei, Hao Ni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19083
• Pdf URL: https://arxiv.org/pdf/2501.19083
• Copy Paste: [[2501.19083]] MotionPCM: Real-Time Motion Synthesis with Phased Consistency Model(https://arxiv.org/abs/2501.19083)
• Keywords: diffusion, generative
• Abstract: Diffusion models have become a popular choice for human motion synthesis due to their powerful generative capabilities. However, their high computational complexity and large sampling steps pose challenges for real-time applications. Fortunately, the Consistency Model (CM) provides a solution to greatly reduce the number of sampling steps from hundreds to a few, typically fewer than four, significantly accelerating the synthesis of diffusion models. However, its application to text-conditioned human motion synthesis in latent space remains challenging. In this paper, we introduce \textbf{MotionPCM}, a phased consistency model-based approach designed to improve the quality and efficiency of real-time motion synthesis in latent space.

Title: Fairness Analysis of CLIP-Based Foundation Models for X-Ray Image Classification

• Authors: Xiangyu Sun, Xiaoguang Zou, Yuanquan Wu, Guotai Wang, Shaoting Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19086
• Pdf URL: https://arxiv.org/pdf/2501.19086
• Copy Paste: [[2501.19086]] Fairness Analysis of CLIP-Based Foundation Models for X-Ray Image Classification(https://arxiv.org/abs/2501.19086)
• Keywords: foundation model
• Abstract: X-ray imaging is pivotal in medical diagnostics, offering non-invasive insights into a range of health conditions. Recently, vision-language models, such as the Contrastive Language-Image Pretraining (CLIP) model, have demonstrated potential in improving diagnostic accuracy by leveraging large-scale image-text datasets. However, since CLIP was not initially designed for medical images, several CLIP-like models trained specifically on medical images have been developed. Despite their enhanced performance, issues of fairness - particularly regarding demographic attributes - remain largely unaddressed. In this study, we perform a comprehensive fairness analysis of CLIP-like models applied to X-ray image classification. We assess their performance and fairness across diverse patient demographics and disease categories using zero-shot inference and various fine-tuning techniques, including Linear Probing, Multilayer Perceptron (MLP), Low-Rank Adaptation (LoRA), and full fine-tuning. Our results indicate that while fine-tuning improves model accuracy, fairness concerns persist, highlighting the need for further fairness interventions in these foundational models.

Title: Ambient Denoising Diffusion Generative Adversarial Networks for Establishing Stochastic Object Models from Noisy Image Data

• Authors: Xichen Xu, Wentao Chen, Weimin Zhou
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.19094
• Pdf URL: https://arxiv.org/pdf/2501.19094
• Copy Paste: [[2501.19094]] Ambient Denoising Diffusion Generative Adversarial Networks for Establishing Stochastic Object Models from Noisy Image Data(https://arxiv.org/abs/2501.19094)
• Keywords: diffusion, generative
• Abstract: It is widely accepted that medical imaging systems should be objectively assessed via task-based image quality (IQ) measures that ideally account for all sources of randomness in the measured image data, including the variation in the ensemble of objects to be imaged. Stochastic object models (SOMs) that can randomly draw samples from the object distribution can be employed to characterize object variability. To establish realistic SOMs for task-based IQ analysis, it is desirable to employ experimental image data. However, experimental image data acquired from medical imaging systems are subject to measurement noise. Previous work investigated the ability of deep generative models (DGMs) that employ an augmented generative adversarial network (GAN), AmbientGAN, for establishing SOMs from noisy measured image data. Recently, denoising diffusion models (DDMs) have emerged as a leading DGM for image synthesis and can produce superior image quality than GANs. However, original DDMs possess a slow image-generation process because of the Gaussian assumption in the denoising steps. More recently, denoising diffusion GAN (DDGAN) was proposed to permit fast image generation while maintain high generated image quality that is comparable to the original DDMs. In this work, we propose an augmented DDGAN architecture, Ambient DDGAN (ADDGAN), for learning SOMs from noisy image data. Numerical studies that consider clinical computed tomography (CT) images and digital breast tomosynthesis (DBT) images are conducted. The ability of the proposed ADDGAN to learn realistic SOMs from noisy image data is demonstrated. It has been shown that the ADDGAN significantly outperforms the advanced AmbientGAN models for synthesizing high resolution medical images with complex textures.

Title: A theoretical framework for overfitting in energy-based modeling

• Authors: Giovanni Catania, Aurélien Decelle, Cyril Furtlehner, Beatriz Seoane
• Subjects: cs.LG, cond-mat.dis-nn, cond-mat.stat-mech
• Abstract URL: https://arxiv.org/abs/2501.19158
• Pdf URL: https://arxiv.org/pdf/2501.19158
• Copy Paste: [[2501.19158]] A theoretical framework for overfitting in energy-based modeling(https://arxiv.org/abs/2501.19158)
• Keywords: generative
• Abstract: We investigate the impact of limited data on training pairwise energy-based models for inverse problems aimed at identifying interaction networks. Utilizing the Gaussian model as testbed, we dissect training trajectories across the eigenbasis of the coupling matrix, exploiting the independent evolution of eigenmodes and revealing that the learning timescales are tied to the spectral decomposition of the empirical covariance matrix. We see that optimal points for early stopping arise from the interplay between these timescales and the initial conditions of training. Moreover, we show that finite data corrections can be accurately modeled through asymptotic random matrix theory calculations and provide the counterpart of generalized cross-validation in the energy based model context. Our analytical framework extends to binary-variable maximum-entropy pairwise models with minimal variations. These findings offer strategies to control overfitting in discrete-variable models through empirical shrinkage corrections, improving the management of overfitting in energy-based generative models.

Title: RMDM: Radio Map Diffusion Model with Physics Informed

• Authors: Haozhe Jia, Wenshuo Chen, Zhihui Huang, Hongru Xiao, Nanqian Jia, Keming Wu, Songning Lai, Yutao Yue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19160
• Pdf URL: https://arxiv.org/pdf/2501.19160
• Copy Paste: [[2501.19160]] RMDM: Radio Map Diffusion Model with Physics Informed(https://arxiv.org/abs/2501.19160)
• Keywords: diffusion
• Abstract: With the rapid development of wireless communication technology, the efficient utilization of spectrum resources, optimization of communication quality, and intelligent communication have become critical. Radio map reconstruction is essential for enabling advanced applications, yet challenges such as complex signal propagation and sparse data hinder accurate reconstruction. To address these issues, we propose the **Radio Map Diffusion Model (RMDM)**, a physics-informed framework that integrates **Physics-Informed Neural Networks (PINNs)** to incorporate constraints like the **Helmholtz equation**. RMDM employs a dual U-Net architecture: the first ensures physical consistency by minimizing PDE residuals, boundary conditions, and source constraints, while the second refines predictions via diffusion-based denoising. By leveraging physical laws, RMDM significantly enhances accuracy, robustness, and generalization. Experiments demonstrate that RMDM outperforms state-of-the-art methods, achieving **NMSE of 0.0031** and **RMSE of 0.0125** under the Static RM (SRM) setting, and **NMSE of 0.0047** and **RMSE of 0.0146** under the Dynamic RM (DRM) setting. These results establish a novel paradigm for integrating physics-informed and data-driven approaches in radio map reconstruction, particularly under sparse data conditions.

Title: PSyDUCK: Training-Free Steganography for Latent Diffusion

• Authors: Georgia Channing, Aqib Mahfuz, Mark van der Wilk, Philip Torr, Fabio Pizzati, Christian Schroeder de Witt
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.19172
• Pdf URL: https://arxiv.org/pdf/2501.19172
• Copy Paste: [[2501.19172]] PSyDUCK: Training-Free Steganography for Latent Diffusion(https://arxiv.org/abs/2501.19172)
• Keywords: diffusion, generative
• Abstract: Recent advances in AI-generated steganography highlight its potential for safeguarding the privacy of vulnerable democratic actors, including aid workers, journalists, and whistleblowers operating in oppressive regimes. In this work, we address current limitations and establish the foundations for large-throughput generative steganography. We introduce a novel approach that enables secure and efficient steganography within latent diffusion models. We show empirically that our methods perform well across a variety of open-source latent diffusion models, particularly in generative image and video tasks.

Title: No Foundations without Foundations -- Why semi-mechanistic models are essential for regulatory biology

• Authors: Luka Kovačević, Thomas Gaudelet, James Opzoomer, Hagen Triendl, John Whittaker, Caroline Uhler, Lindsay Edwards, Jake P. Taylor-King
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19178
• Pdf URL: https://arxiv.org/pdf/2501.19178
• Copy Paste: [[2501.19178]] No Foundations without Foundations -- Why semi-mechanistic models are essential for regulatory biology(https://arxiv.org/abs/2501.19178)
• Keywords: foundation model
• Abstract: Despite substantial efforts, deep learning has not yet delivered a transformative impact on elucidating regulatory biology, particularly in the realm of predicting gene expression profiles. Here, we argue that genuine "foundation models" of regulatory biology will remain out of reach unless guided by frameworks that integrate mechanistic insight with principled experimental design. We present one such ground-up, semi-mechanistic framework that unifies perturbation-based experimental designs across both in vitro and in vivo CRISPR screens, accounting for differentiating and non-differentiating cellular systems. By revealing previously unrecognised assumptions in published machine learning methods, our approach clarifies links with popular techniques such as variational autoencoders and structural causal models. In practice, this framework suggests a modified loss function that we demonstrate can improve predictive performance, and further suggests an error analysis that informs batching strategies. Ultimately, since cellular regulation emerges from innumerable interactions amongst largely uncharted molecular components, we contend that systems-level understanding cannot be achieved through structural biology alone. Instead, we argue that real progress will require a first-principles perspective on how experiments capture biological phenomena, how data are generated, and how these processes can be reflected in more faithful modelling architectures.

Title: A Comunication Framework for Compositional Generation

• Authors: Rafael Elberg, Mircea Petrache, Denis Parra
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19182
• Pdf URL: https://arxiv.org/pdf/2501.19182
• Copy Paste: [[2501.19182]] A Comunication Framework for Compositional Generation(https://arxiv.org/abs/2501.19182)
• Keywords: self-supervised, generative
• Abstract: Compositionality and compositional generalization--the ability to understand novel combinations of known concepts--are central characteristics of human language and are hypothesized to be essential for human cognition. In machine learning, the emergence of this property has been studied in a communication game setting, where independent agents (a sender and a receiver) converge to a shared encoding policy from a set of states to a space of discrete messages, where the receiver can correctly reconstruct the states observed by the sender using only the sender's messages. The use of communication games in generation tasks is still largely unexplored, with recent methods for compositional generation focusing mainly on the use of supervised guidance (either through class labels or text). In this work, we take the first steps to fill this gap, and we present a self-supervised generative communication game-based framework for creating compositional encodings in learned representations from pre-trained encoder-decoder models. In an Iterated Learning (IL) protocol involving a sender and a receiver, we apply alternating pressures for compression and diversity of encoded discrete messages, so that the protocol converges to an efficient but unambiguous encoding. Approximate message entropy regularization is used to favor compositional encodings. Our framework is based on rigorous justifications and proofs of defining and balancing the concepts of Eficiency, Unambiguity and Non-Holisticity in encoding. We test our method on the compositional image dataset Shapes3D, demonstrating robust performance in both reconstruction and compositionality metrics, surpassing other tested discrete message frameworks.

Title: A Variational Perspective on Generative Protein Fitness Optimization

• Authors: Lea Bogensperger, Dominik Narnhofer, Ahmed Allam, Konrad Schindler, Michael Krauthammer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19200
• Pdf URL: https://arxiv.org/pdf/2501.19200
• Copy Paste: [[2501.19200]] A Variational Perspective on Generative Protein Fitness Optimization(https://arxiv.org/abs/2501.19200)
• Keywords: generative
• Abstract: The goal of protein fitness optimization is to discover new protein variants with enhanced fitness for a given use. The vast search space and the sparsely populated fitness landscape, along with the discrete nature of protein sequences, pose significant challenges when trying to determine the gradient towards configurations with higher fitness. We introduce Variational Latent Generative Protein Optimization (VLGPO), a variational perspective on fitness optimization. Our method embeds protein sequences in a continuous latent space to enable efficient sampling from the fitness distribution and combines a (learned) flow matching prior over sequence mutations with a fitness predictor to guide optimization towards sequences with high fitness. VLGPO achieves state-of-the-art results on two different protein benchmarks of varying complexity. Moreover, the variational design with explicit prior and likelihood functions offers a flexible plug-and-play framework that can be easily customized to suit various protein design tasks.

Title: Accelerating Diffusion Transformer via Error-Optimized Cache

• Authors: Junxiang Qiu, Shuo Wang, Jinda Lu, Lin Liu, Houcheng Jiang, Yanbin Hao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19243
• Pdf URL: https://arxiv.org/pdf/2501.19243
• Copy Paste: [[2501.19243]] Accelerating Diffusion Transformer via Error-Optimized Cache(https://arxiv.org/abs/2501.19243)
• Keywords: diffusion
• Abstract: Diffusion Transformer (DiT) is a crucial method for content generation. However, it needs a lot of time to sample. Many studies have attempted to use caching to reduce the time consumption of sampling. Existing caching methods accelerate generation by reusing DiT features from the previous time step and skipping calculations in the next, but they tend to locate and cache low-error modules without focusing on reducing caching-induced errors, resulting in a sharp decline in generated content quality when increasing caching intensity. To solve this problem, we propose the Error-Optimized Cache (EOC). This method introduces three key improvements: (1) Prior knowledge extraction: Extract and process the caching differences; (2) A judgment method for cache optimization: Determine whether certain caching steps need to be optimized; (3) Cache optimization: reduce caching errors. Experiments show that this algorithm significantly reduces the error accumulation caused by caching (especially over-caching). On the ImageNet dataset, without significantly increasing the computational burden, this method improves the quality of the generated images under the over-caching, rule-based, and training-based methods. Specifically, the Fréchet Inception Distance (FID) values are improved as follows: from 6.857 to 5.821, from 3.870 to 3.692 and form 3.539 to 3.451 respectively.

Title: Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search

• Authors: Yuta Oshima, Masahiro Suzuki, Yutaka Matsuo, Hiroki Furuta
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19252
• Pdf URL: https://arxiv.org/pdf/2501.19252
• Copy Paste: [[2501.19252]] Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search(https://arxiv.org/abs/2501.19252)
• Keywords: diffusion
• Abstract: The remarkable progress in text-to-video diffusion models enables photorealistic generations, although the contents of the generated video often include unnatural movement or deformation, reverse playback, and motionless scenes. Recently, an alignment problem has attracted huge attention, where we steer the output of diffusion models based on some quantity on the goodness of the content. Because there is a large room for improvement of perceptual quality along the frame direction, we should address which metrics we should optimize and how we can optimize them in the video generation. In this paper, we propose diffusion latent beam search with lookahead estimator, which can select better diffusion latent to maximize a given alignment reward, at inference time. We then point out that the improvement of perceptual video quality considering the alignment to prompts requires reward calibration by weighting existing metrics. When evaluating outputs by using vision language models as a proxy of humans, many previous metrics to quantify the naturalness of video do not always correlate with evaluation and also depend on the degree of dynamic descriptions in evaluation prompts. We demonstrate that our method improves the perceptual quality based on the calibrated reward, without model parameter update, and outputs the best generation compared to greedy search and best-of-N sampling. We provide practical guidelines on which axes, among search budget, lookahead steps for reward estimate, and denoising steps, in the reverse diffusion process, we should allocate the inference-time computation.

Title: Medical Semantic Segmentation with Diffusion Pretrain

• Authors: David Li, Anvar Kurmukov, Mikhail Goncharov, Roman Sokolov, Mikhail Belyaev
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19265
• Pdf URL: https://arxiv.org/pdf/2501.19265
• Copy Paste: [[2501.19265]] Medical Semantic Segmentation with Diffusion Pretrain(https://arxiv.org/abs/2501.19265)
• Keywords: diffusion
• Abstract: Recent advances in deep learning have shown that learning robust feature representations is critical for the success of many computer vision tasks, including medical image segmentation. In particular, both transformer and convolutional-based architectures have benefit from leveraging pretext tasks for pretraining. However, the adoption of pretext tasks in 3D medical imaging has been less explored and remains a challenge, especially in the context of learning generalizable feature representations. We propose a novel pretraining strategy using diffusion models with anatomical guidance, tailored to the intricacies of 3D medical image data. We introduce an auxiliary diffusion process to pretrain a model that produce generalizable feature representations, useful for a variety of downstream segmentation tasks. We employ an additional model that predicts 3D universal body-part coordinates, providing guidance during the diffusion process and improving spatial awareness in generated representations. This approach not only aids in resolving localization inaccuracies but also enriches the model's ability to understand complex anatomical structures. Empirical validation on a 13-class organ segmentation task demonstrate the effectiveness of our pretraining technique. It surpasses existing restorative pretraining methods in 3D medical image segmentation by $7.5\%$, and is competitive with the state-of-the-art contrastive pretraining approach, achieving an average Dice coefficient of 67.8 in a non-linear evaluation scenario.

Title: Application of Generative Adversarial Network (GAN) for Synthetic Training Data Creation to improve performance of ANN Classifier for extracting Built-Up pixels from Landsat Satellite Imagery

• Authors: Amritendu Mukherjee, Dipanwita Sinha Mukherjee, Parthasarathy Ramachandran
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19283
• Pdf URL: https://arxiv.org/pdf/2501.19283
• Copy Paste: [[2501.19283]] Application of Generative Adversarial Network (GAN) for Synthetic Training Data Creation to improve performance of ANN Classifier for extracting Built-Up pixels from Landsat Satellite Imagery(https://arxiv.org/abs/2501.19283)
• Keywords: generative
• Abstract: Training a neural network for pixel based classification task using low resolution Landsat images is difficult as the size of the training data is usually small due to less number of available pixels that represent a single class without any mixing with other classes. Due to this scarcity of training data, neural network may not be able to attain expected level of accuracy. This limitation could be overcome using a generative network that aims to generate synthetic data having the same distribution as the sample data with which it is trained. In this work, we have proposed a methodology for improving the performance of ANN classifier to identify built-up pixels in the Landsat$7$ image with the help of developing a simple GAN architecture that could generate synthetic training pixels when trained using original set of sample built-up pixels. To ensure that the marginal and joint distributions of all the bands corresponding to the generated and original set of pixels are indistinguishable, non-parametric Kolmogorov Smirnov Test and Ball Divergence based Equality of Distributions Test have been performed respectively. It has been observed that the overall accuracy and kappa coefficient of the ANN model for built-up classification have continuously improved from $0.9331$ to $0.9983$ and $0.8277$ to $0.9958$ respectively, with the inclusion of generated sets of built-up pixels to the original one.

Title: Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs

• Authors: James Flemings, Haosheng Gan, Hongyi Li, Meisam Razaviyayn, Murali Annavaram
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19287
• Pdf URL: https://arxiv.org/pdf/2501.19287
• Copy Paste: [[2501.19287]] Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs(https://arxiv.org/abs/2501.19287)
• Keywords: in-context
• Abstract: In-context learning (ICL) has shown promising improvement in downstream task adaptation of LLMs by augmenting prompts with relevant input-output examples (demonstrations). However, the ICL demonstrations can contain privacy-sensitive information, which can be leaked and/or regurgitated by the LLM output. Differential Privacy (DP), a widely adopted privacy safeguard, has emerged to mitigate this privacy leakage, with recent work demonstrating strong privacy-utility tradeoffs in classification tasks for ICL. However, generation tasks for ICL are challenging due to the high-dimensional output space of open-ended generation. To this end, we propose $\texttt{dps-mozo}$, Differentially Private Sampling by Mixing One-shot with Zero-shot Outputs, a decoding framework that generates DP text by sampling from the product of multiple one-shot outputs mixed with a zero-shot output. This mixing effectively reduces the amount of information that can be leaked by each demonstration. By utilizing the inherent randomness in sampling from the mixed distributions, we can achieve DP without adding noise, thereby improving the privacy-utility tradeoff. Our experimental evaluations show $\texttt{dps-mozo}$ can achieve a strong privacy guarantee, $\epsilon=2$, with minimal utility degradation compared to non-private few-shot learning, $\textbf{0.3}$% ROUGE-L F1 score decrease on the SAMSum dataset with Gemma 2 2B.

Title: Noninterference Analysis of Irreversible or Reversible Systems with Nondeterminism and Probabilities

• Authors: Andrea Esposito, Alessandro Aldini, Marco Bernardo
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.19290
• Pdf URL: https://arxiv.org/pdf/2501.19290
• Copy Paste: [[2501.19290]] Noninterference Analysis of Irreversible or Reversible Systems with Nondeterminism and Probabilities(https://arxiv.org/abs/2501.19290)
• Keywords: generative
• Abstract: Noninterference theory supports the analysis of secure computations in multi-level security systems. Classical equivalence-based approaches to noninterference mainly rely on bisimilarity. In a nondeterministic setting, assessing noninterference through weak bisimilarity is adequate for irreversible systems, whereas for reversible ones branching bisimilarity has been recently proven to be more appropriate. In this paper we address the same two families of systems, with the difference that probabilities come into play in addition to nondeterminism. For irreversible systems we extend the results of Aldini, Bravetti, and Gorrieri developed in a generative-reactive probabilistic setting, while for reversible systems we extend the results of Esposito, Aldini, Bernardo, and Rossi developed in a purely nondeterministic setting. We recast noninterference properties by adopting probabilistic variants of weak and branching bisimilarities for irreversible and reversible systems respectively. Then we investigate a taxonomy of those properties as well as their preservation and compositionality aspects, along with a comparison with the nondeterministic taxonomy. The adequacy of the extended noninterference theory is illustrated via a probabilistic smart contract example.

Title: Consistent Video Colorization via Palette Guidance

• Authors: Han Wang, Yuang Zhang, Yuhong Zhang, Lingxiao Lu, Li Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19331
• Pdf URL: https://arxiv.org/pdf/2501.19331
• Copy Paste: [[2501.19331]] Consistent Video Colorization via Palette Guidance(https://arxiv.org/abs/2501.19331)
• Keywords: diffusion, generative
• Abstract: Colorization is a traditional computer vision task and it plays an important role in many time-consuming tasks, such as old film restoration. Existing methods suffer from unsaturated color and temporally inconsistency. In this paper, we propose a novel pipeline to overcome the challenges. We regard the colorization task as a generative task and introduce Stable Video Diffusion (SVD) as our base model. We design a palette-based color guider to assist the model in generating vivid and consistent colors. The color context introduced by the palette not only provides guidance for color generation, but also enhances the stability of the generated colors through a unified color context across multiple sequences. Experiments demonstrate that the proposed method can provide vivid and stable colors for videos, surpassing previous methods.

Title: PixelWorld: Towards Perceiving Everything as Pixels

• Authors: Zhiheng Lyu, Xueguang Ma, Wenhu Chen
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19339
• Pdf URL: https://arxiv.org/pdf/2501.19339
• Copy Paste: [[2501.19339]] PixelWorld: Towards Perceiving Everything as Pixels(https://arxiv.org/abs/2501.19339)
• Keywords: foundation model
• Abstract: Existing foundation models typically process visual input as pixels and textual input as tokens, a paradigm that contrasts with human perception, where both modalities are processed in a unified manner. With the rise of embodied and agentic AI, where inputs primarily come from camera pixels, the need for a unified perception framework becomes increasingly evident. In this paper, we propose to unify all modalities (text, tables, code, diagrams, images, etc) as pixel inputs, i.e. "Perceive Everything as Pixels" (PEAP). We introduce PixelWorld, a novel evaluation suite that unifies all the mentioned modalities into pixel space to gauge the existing models' performance. Our findings show that (1) PEAP outperforms baseline with token-based input in multimodal datasets, benefiting from unified input for better disambiguation, (2) significant declines in reasoning and coding capabilities across all models when processing pixel-based input, underscoring the need to enhance foundation models' perceptual abilities, (3) larger models can maintain strong performance on non-reasoning tasks under PEAP, while smaller models like Phi-3.5-V suffer significant performance degradation, (4) the attention pattern of PEAP is highly aligned with text token input, (5) PEAP can be accelerated significantly by exploiting the spatial sparsity. We conclude that the existing frontier models are competent in pixel perception, however, there is still headroom for improvement. Our code, dataset will be released upon acceptance.

Title: CoSTI: Consistency Models for (a faster) Spatio-Temporal Imputation

• Authors: Javier Solís-García, Belén Vega-Márquez, Juan A. Nepomuceno, Isabel A. Nepomuceno-Chamorro
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19364
• Pdf URL: https://arxiv.org/pdf/2501.19364
• Copy Paste: [[2501.19364]] CoSTI: Consistency Models for (a faster) Spatio-Temporal Imputation(https://arxiv.org/abs/2501.19364)
• Keywords: diffusion, generative
• Abstract: Multivariate Time Series Imputation (MTSI) is crucial for many applications, such as healthcare monitoring and traffic management, where incomplete data can compromise decision-making. Existing state-of-the-art methods, like Denoising Diffusion Probabilistic Models (DDPMs), achieve high imputation accuracy; however, they suffer from significant computational costs and are notably time-consuming due to their iterative nature. In this work, we propose CoSTI, an innovative adaptation of Consistency Models (CMs) for the MTSI domain. CoSTI employs Consistency Training to achieve comparable imputation quality to DDPMs while drastically reducing inference times, making it more suitable for real-time applications. We evaluate CoSTI across multiple datasets and missing data scenarios, demonstrating up to a 98% reduction in imputation time with performance on par with diffusion-based models. This work bridges the gap between efficiency and accuracy in generative imputation tasks, providing a scalable solution for handling missing data in critical spatio-temporal systems.

Title: Vintix: Action Model via In-Context Reinforcement Learning

• Authors: Andrey Polubarov, Nikita Lyubaykin, Alexander Derevyagin, Ilya Zisman, Denis Tarasov, Alexander Nikulin, Vladislav Kurenkov
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.19400
• Pdf URL: https://arxiv.org/pdf/2501.19400
• Copy Paste: [[2501.19400]] Vintix: Action Model via In-Context Reinforcement Learning(https://arxiv.org/abs/2501.19400)
• Keywords: in-context
• Abstract: In-Context Reinforcement Learning (ICRL) represents a promising paradigm for developing generalist agents that learn at inference time through trial-and-error interactions, analogous to how large language models adapt contextually, but with a focus on reward maximization. However, the scalability of ICRL beyond toy tasks and single-domain settings remains an open challenge. In this work, we present the first steps toward scaling ICRL by introducing a fixed, cross-domain model capable of learning behaviors through in-context reinforcement learning. Our results demonstrate that Algorithm Distillation, a framework designed to facilitate ICRL, offers a compelling and competitive alternative to expert distillation to construct versatile action models. These findings highlight the potential of ICRL as a scalable approach for generalist decision-making systems. Code to be released at this https URL