2025-05-16

Title: Analog Foundation Models

• Authors: Julian Büchel, Iason Chalas, Giovanni Acampa, An Chen, Omobayode Fagbohungbe, Sidney Tsai, Kaoutar El Maghraoui, Manuel Le Gallo, Abbas Rahimi, Abu Sebastian
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.09663
• Pdf URL: https://arxiv.org/pdf/2505.09663
• Copy Paste: [[2505.09663]] Analog Foundation Models(https://arxiv.org/abs/2505.09663)
• Keywords: foundation model
• Abstract: Analog in-memory computing (AIMC) is a promising compute paradigm to improve speed and power efficiency of neural network inference beyond the limits of conventional von Neumann-based architectures. However, AIMC introduces fundamental challenges such as noisy computations and strict constraints on input and output quantization. Because of these constraints and imprecisions, off-the-shelf LLMs are not able to achieve 4-bit-level performance when deployed on AIMC-based hardware. While researchers previously investigated recovering this accuracy gap on small, mostly vision-based models, a generic method applicable to LLMs pre-trained on trillions of tokens does not yet exist. In this work, we introduce a general and scalable method to robustly adapt LLMs for execution on noisy, low-precision analog hardware. Our approach enables state-of-the-art models $\unicode{x2013}$ including Phi-3-mini-4k-instruct and Llama-3.2-1B-Instruct $\unicode{x2013}$ to retain performance comparable to 4-bit weight, 8-bit activation baselines, despite the presence of analog noise and quantization constraints. Additionally, we show that as a byproduct of our training methodology, analog foundation models can be quantized for inference on low-precision digital hardware. Finally, we show that our models also benefit from test-time compute scaling, showing better scaling behavior than models trained with 4-bit weight and 8-bit static input quantization. Our work bridges the gap between high-capacity LLMs and efficient analog hardware, offering a path toward energy-efficient foundation models. Code is available at this https URL .

Title: A Generative Neural Annealer for Black-Box Combinatorial Optimization

• Authors: Yuan-Hang Zhang, Massimiliano Di Ventra
• Subjects: cs.LG, cond-mat.dis-nn, cond-mat.stat-mech, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2505.09742
• Pdf URL: https://arxiv.org/pdf/2505.09742
• Copy Paste: [[2505.09742]] A Generative Neural Annealer for Black-Box Combinatorial Optimization(https://arxiv.org/abs/2505.09742)
• Keywords: generative
• Abstract: We propose a generative, end-to-end solver for black-box combinatorial optimization that emphasizes both sample efficiency and solution quality on NP problems. Drawing inspiration from annealing-based algorithms, we treat the black-box objective as an energy function and train a neural network to model the associated Boltzmann distribution. By conditioning on temperature, the network captures a continuum of distributions--from near-uniform at high temperatures to sharply peaked around global optima at low temperatures--thereby learning the structure of the energy landscape and facilitating global optimization. When queries are expensive, the temperature-dependent distributions naturally enable data augmentation and improve sample efficiency. When queries are cheap but the problem remains hard, the model learns implicit variable interactions, effectively "opening" the black box. We validate our approach on challenging combinatorial tasks under both limited and unlimited query budgets, showing competitive performance against state-of-the-art black-box optimizers.

Title: Self-Consuming Generative Models with Adversarially Curated Data

• Authors: Xiukun Wei, Xueru Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.09768
• Pdf URL: https://arxiv.org/pdf/2505.09768
• Copy Paste: [[2505.09768]] Self-Consuming Generative Models with Adversarially Curated Data(https://arxiv.org/abs/2505.09768)
• Keywords: generative
• Abstract: Recent advances in generative models have made it increasingly difficult to distinguish real data from model-generated synthetic data. Using synthetic data for successive training of future model generations creates "self-consuming loops", which may lead to model collapse or training instability. Furthermore, synthetic data is often subject to human feedback and curated by users based on their preferences. Ferbach et al. (2024) recently showed that when data is curated according to user preferences, the self-consuming retraining loop drives the model to converge toward a distribution that optimizes those preferences. However, in practice, data curation is often noisy or adversarially manipulated. For example, competing platforms may recruit malicious users to adversarially curate data and disrupt rival models. In this paper, we study how generative models evolve under self-consuming retraining loops with noisy and adversarially curated data. We theoretically analyze the impact of such noisy data curation on generative models and identify conditions for the robustness of the retraining process. Building on this analysis, we design attack algorithms for competitive adversarial scenarios, where a platform with a limited budget employs malicious users to misalign a rival's model from actual user preferences. Experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed algorithms.

Title: Causal Predictive Optimization and Generation for Business AI

• Authors: Liyang Zhao, Olurotimi Seton, Himadeep Reddy Reddivari, Suvendu Jena, Shadow Zhao, Rachit Kumar, Changshuai Wei
• Subjects: cs.LG, cs.AI, cs.IR, stat.ML
• Abstract URL: https://arxiv.org/abs/2505.09847
• Pdf URL: https://arxiv.org/pdf/2505.09847
• Copy Paste: [[2505.09847]] Causal Predictive Optimization and Generation for Business AI(https://arxiv.org/abs/2505.09847)
• Keywords: generative
• Abstract: The sales process involves sales functions converting leads or opportunities to customers and selling more products to existing customers. The optimization of the sales process thus is key to success of any B2B business. In this work, we introduce a principled approach to sales optimization and business AI, namely the Causal Predictive Optimization and Generation, which includes three layers: 1) prediction layer with causal ML 2) optimization layer with constraint optimization and contextual bandit 3) serving layer with Generative AI and feedback-loop for system enhancement. We detail the implementation and deployment of the system in LinkedIn, showcasing significant wins over legacy systems and sharing learning and insight broadly applicable to this field.

Title: Predictability Shapes Adaptation: An Evolutionary Perspective on Modes of Learning in Transformers

• Authors: Alexander Y. Ku, Thomas L. Griffiths, Stephanie C.Y. Chan
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2505.09855
• Pdf URL: https://arxiv.org/pdf/2505.09855
• Copy Paste: [[2505.09855]] Predictability Shapes Adaptation: An Evolutionary Perspective on Modes of Learning in Transformers(https://arxiv.org/abs/2505.09855)
• Keywords: in-context
• Abstract: Transformer models learn in two distinct modes: in-weights learning (IWL), encoding knowledge into model weights, and in-context learning (ICL), adapting flexibly to context without weight modification. To better understand the interplay between these learning modes, we draw inspiration from evolutionary biology's analogous adaptive strategies: genetic encoding (akin to IWL, adapting over generations and fixed within an individual's lifetime) and phenotypic plasticity (akin to ICL, enabling flexible behavioral responses to environmental cues). In evolutionary biology, environmental predictability dictates the balance between these strategies: stability favors genetic encoding, while reliable predictive cues promote phenotypic plasticity. We experimentally operationalize these dimensions of predictability and systematically investigate their influence on the ICL/IWL balance in Transformers. Using regression and classification tasks, we show that high environmental stability decisively favors IWL, as predicted, with a sharp transition at maximal stability. Conversely, high cue reliability enhances ICL efficacy, particularly when stability is low. Furthermore, learning dynamics reveal task-contingent temporal evolution: while a canonical ICL-to-IWL shift occurs in some settings (e.g., classification with many classes), we demonstrate that scenarios with easier IWL (e.g., fewer classes) or slower ICL acquisition (e.g., regression) can exhibit an initial IWL phase later yielding to ICL dominance. These findings support a relative-cost hypothesis for explaining these learning mode transitions, establishing predictability as a critical factor governing adaptive strategies in Transformers, and offering novel insights for understanding ICL and guiding training methodologies.

Title: Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models

• Authors: Danush Kumar Venkatesh, Isabel Funke, Micha Pfeiffer, Fiona Kolbinger, Hanna Maria Schmeiser, Juergen Weitz, Marius Distler, Stefanie Speidel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.09858
• Pdf URL: https://arxiv.org/pdf/2505.09858
• Copy Paste: [[2505.09858]] Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models(https://arxiv.org/abs/2505.09858)
• Keywords: diffusion
• Abstract: Computer-assisted interventions can improve intra-operative guidance, particularly through deep learning methods that harness the spatiotemporal information in surgical videos. However, the severe data imbalance often found in surgical video datasets hinders the development of high-performing models. In this work, we aim to overcome the data imbalance by synthesizing surgical videos. We propose a unique two-stage, text-conditioned diffusion-based method to generate high-fidelity surgical videos for under-represented classes. Our approach conditions the generation process on text prompts and decouples spatial and temporal modeling by utilizing a 2D latent diffusion model to capture spatial content and then integrating temporal attention layers to ensure temporal consistency. Furthermore, we introduce a rejection sampling strategy to select the most suitable synthetic samples, effectively augmenting existing datasets to address class imbalance. We evaluate our method on two downstream tasks-surgical action recognition and intra-operative event prediction-demonstrating that incorporating synthetic videos from our approach substantially enhances model performance. We open-source our implementation at this https URL.

Title: Correlating Account on Ethereum Mixing Service via Domain-Invariant feature learning

• Authors: Zheng Che, Taoyu Li, Meng Shen, Hanbiao Du, Liehuang Zhu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2505.09892
• Pdf URL: https://arxiv.org/pdf/2505.09892
• Copy Paste: [[2505.09892]] Correlating Account on Ethereum Mixing Service via Domain-Invariant feature learning(https://arxiv.org/abs/2505.09892)
• Keywords: anomaly
• Abstract: The untraceability of transactions facilitated by Ethereum mixing services like Tornado Cash poses significant challenges to blockchain security and financial regulation. Existing methods for correlating mixing accounts suffer from limited labeled data and vulnerability to noisy annotations, which restrict their practical applicability. In this paper, we propose StealthLink, a novel framework that addresses these limitations through cross-task domain-invariant feature learning. Our key innovation lies in transferring knowledge from the well-studied domain of blockchain anomaly detection to the data-scarce task of mixing transaction tracing. Specifically, we design a MixFusion module that constructs and encodes mixing subgraphs to capture local transactional patterns, while introducing a knowledge transfer mechanism that aligns discriminative features across domains through adversarial discrepancy minimization. This dual approach enables robust feature learning under label scarcity and distribution shifts. Extensive experiments on real-world mixing transaction datasets demonstrate that StealthLink achieves state-of-the-art performance, with 96.98\% F1-score in 10-shot learning scenarios. Notably, our framework shows superior generalization capability in imbalanced data conditions than conventional supervised methods. This work establishes the first systematic approach for cross-domain knowledge transfer in blockchain forensics, providing a practical solution for combating privacy-enhanced financial crimes in decentralized ecosystems.

Title: PIG: Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimization

• Authors: Yidan Wang, Yanan Cao, Yubing Ren, Fang Fang, Zheng Lin, Binxing Fang
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2505.09921
• Pdf URL: https://arxiv.org/pdf/2505.09921
• Copy Paste: [[2505.09921]] PIG: Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimization(https://arxiv.org/abs/2505.09921)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) excel in various domains but pose inherent privacy risks. Existing methods to evaluate privacy leakage in LLMs often use memorized prefixes or simple instructions to extract data, both of which well-alignment models can easily block. Meanwhile, Jailbreak attacks bypass LLM safety mechanisms to generate harmful content, but their role in privacy scenarios remains underexplored. In this paper, we examine the effectiveness of jailbreak attacks in extracting sensitive information, bridging privacy leakage and jailbreak attacks in LLMs. Moreover, we propose PIG, a novel framework targeting Personally Identifiable Information (PII) and addressing the limitations of current jailbreak methods. Specifically, PIG identifies PII entities and their types in privacy queries, uses in-context learning to build a privacy context, and iteratively updates it with three gradient-based strategies to elicit target PII. We evaluate PIG and existing jailbreak methods using two privacy-related datasets. Experiments on four white-box and two black-box LLMs show that PIG outperforms baseline methods and achieves state-of-the-art (SoTA) results. The results underscore significant privacy risks in LLMs, emphasizing the need for stronger safeguards. Our code is availble at \href{this https URL}{this https URL}.

Title: Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity

• Authors: Zichen Liu, Wei Zhang, Tiejun Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.09922
• Pdf URL: https://arxiv.org/pdf/2505.09922
• Copy Paste: [[2505.09922]] Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity(https://arxiv.org/abs/2505.09922)
• Keywords: diffusion, generative
• Abstract: Euclidean diffusion models have achieved remarkable success in generative modeling across diverse domains, and they have been extended to manifold case in recent advances. Instead of explicitly utilizing the structure of special manifolds as studied in previous works, we investigate direct sampling of the Euclidean diffusion models for general manifold-constrained data in this paper. We reveal the multiscale singularity of the score function in the embedded space of manifold, which hinders the accuracy of diffusion-generated samples. We then present an elaborate theoretical analysis of the singularity structure of the score function by separating it along the tangential and normal directions of the manifold. To mitigate the singularity and improve the sampling accuracy, we propose two novel methods: (1) Niso-DM, which introduces non-isotropic noise along the normal direction to reduce scale discrepancies, and (2) Tango-DM, which trains only the tangential component of the score function using a tangential-only loss function. Numerical experiments demonstrate that our methods achieve superior performance on distributions over various manifolds with complex geometries.

Title: AdaptCLIP: Adapting CLIP for Universal Visual Anomaly Detection

• Authors: Bin-Bin Gao, Yue Zhu, Jiangtao Yan, Yuezhi Cai, Weixi Zhang, Meng Wang, Jun Liu, Yong Liu, Lei Wang, Chengjie Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.09926
• Pdf URL: https://arxiv.org/pdf/2505.09926
• Copy Paste: [[2505.09926]] AdaptCLIP: Adapting CLIP for Universal Visual Anomaly Detection(https://arxiv.org/abs/2505.09926)
• Keywords: anomaly
• Abstract: Universal visual anomaly detection aims to identify anomalies from novel or unseen vision domains without additional fine-tuning, which is critical in open scenarios. Recent studies have demonstrated that pre-trained vision-language models like CLIP exhibit strong generalization with just zero or a few normal images. However, existing methods struggle with designing prompt templates, complex token interactions, or requiring additional fine-tuning, resulting in limited flexibility. In this work, we present a simple yet effective method called AdaptCLIP based on two key insights. First, adaptive visual and textual representations should be learned alternately rather than jointly. Second, comparative learning between query and normal image prompt should incorporate both contextual and aligned residual features, rather than relying solely on residual features. AdaptCLIP treats CLIP models as a foundational service, adding only three simple adapters, visual adapter, textual adapter, and prompt-query adapter, at its input or output ends. AdaptCLIP supports zero-/few-shot generalization across domains and possesses a training-free manner on target domains once trained on a base dataset. AdaptCLIP achieves state-of-the-art performance on 12 anomaly detection benchmarks from industrial and medical domains, significantly outperforming existing competitive methods. We will make the code and model of AdaptCLIP available at this https URL.

Title: MambaControl: Anatomy Graph-Enhanced Mamba ControlNet with Fourier Refinement for Diffusion-Based Disease Trajectory Prediction

• Authors: Hao Yang, Tao Tan, Shuai Tan, Weiqin Yang, Kunyan Cai, Calvin Chen, Yue Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.09965
• Pdf URL: https://arxiv.org/pdf/2505.09965
• Copy Paste: [[2505.09965]] MambaControl: Anatomy Graph-Enhanced Mamba ControlNet with Fourier Refinement for Diffusion-Based Disease Trajectory Prediction(https://arxiv.org/abs/2505.09965)
• Keywords: diffusion
• Abstract: Modelling disease progression in precision medicine requires capturing complex spatio-temporal dynamics while preserving anatomical integrity. Existing methods often struggle with longitudinal dependencies and structural consistency in progressive disorders. To address these limitations, we introduce MambaControl, a novel framework that integrates selective state-space modelling with diffusion processes for high-fidelity prediction of medical image trajectories. To better capture subtle structural changes over time while maintaining anatomical consistency, MambaControl combines Mamba-based long-range modelling with graph-guided anatomical control to more effectively represent anatomical correlations. Furthermore, we introduce Fourier-enhanced spectral graph representations to capture spatial coherence and multiscale detail, enabling MambaControl to achieve state-of-the-art performance in Alzheimer's disease prediction. Quantitative and regional evaluations demonstrate improved progression prediction quality and anatomical fidelity, highlighting its potential for personalised prognosis and clinical decision support.

Title: Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Data

• Authors: Adel ElZemity, Budi Arief, Shujun Li
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.09974
• Pdf URL: https://arxiv.org/pdf/2505.09974
• Copy Paste: [[2505.09974]] Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Data(https://arxiv.org/abs/2505.09974)
• Keywords: generative
• Abstract: The integration of large language models (LLMs) into cyber security applications presents significant opportunities, such as enhancing threat analysis and malware detection, but can also introduce critical risks and safety concerns, including personal data leakage and automated generation of new malware. We present a systematic evaluation of safety risks in fine-tuned LLMs for cyber security applications. Using the OWASP Top 10 for LLM Applications framework, we assessed seven open-source LLMs: Phi 3 Mini 3.8B, Mistral 7B, Qwen 2.5 7B, Llama 3 8B, Llama 3.1 8B, Gemma 2 9B, and Llama 2 70B. Our evaluation shows that fine-tuning reduces safety resilience across all tested LLMs (e.g., the safety score of Llama 3.1 8B against prompt injection drops from 0.95 to 0.15). We propose and evaluate a safety alignment approach that carefully rewords instruction-response pairs to include explicit safety precautions and ethical considerations. This approach demonstrates that it is possible to maintain or even improve model safety while preserving technical utility, offering a practical path forward for developing safer fine-tuning methodologies. This work offers a systematic evaluation for safety risks in LLMs, enabling safer adoption of generative AI in sensitive domains, and contributing towards the development of secure, trustworthy, and ethically aligned LLMs.

Title: From Air to Wear: Personalized 3D Digital Fashion with AR/VR Immersive 3D Sketching

• Authors: Ying Zang, Yuanqi Hu, Xinyu Chen, Yuxia Xu, Suhui Wang, Chunan Yu, Lanyun Zhu, Deyi Ji, Xin Xu, Tianrun Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.09998
• Pdf URL: https://arxiv.org/pdf/2505.09998
• Copy Paste: [[2505.09998]] From Air to Wear: Personalized 3D Digital Fashion with AR/VR Immersive 3D Sketching(https://arxiv.org/abs/2505.09998)
• Keywords: diffusion
• Abstract: In the era of immersive consumer electronics, such as AR/VR headsets and smart devices, people increasingly seek ways to express their identity through virtual fashion. However, existing 3D garment design tools remain inaccessible to everyday users due to steep technical barriers and limited data. In this work, we introduce a 3D sketch-driven 3D garment generation framework that empowers ordinary users - even those without design experience - to create high-quality digital clothing through simple 3D sketches in AR/VR environments. By combining a conditional diffusion model, a sketch encoder trained in a shared latent space, and an adaptive curriculum learning strategy, our system interprets imprecise, free-hand input and produces realistic, personalized garments. To address the scarcity of training data, we also introduce KO3DClothes, a new dataset of paired 3D garments and user-created sketches. Extensive experiments and user studies confirm that our method significantly outperforms existing baselines in both fidelity and usability, demonstrating its promise for democratized fashion design on next-generation consumer platforms.

Title: ORL-LDM: Offline Reinforcement Learning Guided Latent Diffusion Model Super-Resolution Reconstruction

• Authors: Shijie Lyu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.10027
• Pdf URL: https://arxiv.org/pdf/2505.10027
• Copy Paste: [[2505.10027]] ORL-LDM: Offline Reinforcement Learning Guided Latent Diffusion Model Super-Resolution Reconstruction(https://arxiv.org/abs/2505.10027)
• Keywords: diffusion
• Abstract: With the rapid advancement of remote sensing technology, super-resolution image reconstruction is of great research and practical significance. Existing deep learning methods have made progress but still face limitations in handling complex scenes and preserving image details. This paper proposes a reinforcement learning-based latent diffusion model (LDM) fine-tuning method for remote sensing image super-resolution. The method constructs a reinforcement learning environment with states, actions, and rewards, optimizing decision objectives through proximal policy optimization (PPO) during the reverse denoising process of the LDM model. Experiments on the RESISC45 dataset show significant improvements over the baseline model in PSNR, SSIM, and LPIPS, with PSNR increasing by 3-4dB, SSIM improving by 0.08-0.11, and LPIPS reducing by 0.06-0.10, particularly in structured and complex natural scenes. The results demonstrate the method's effectiveness in enhancing super-resolution quality and adaptability across scenes.

Title: Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis

• Authors: Bingda Tang, Boyang Zheng, Xichen Pan, Sayak Paul, Saining Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10046
• Pdf URL: https://arxiv.org/pdf/2505.10046
• Copy Paste: [[2505.10046]] Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis(https://arxiv.org/abs/2505.10046)
• Keywords: diffusion
• Abstract: This paper does not describe a new method; instead, it provides a thorough exploration of an important yet understudied design space related to recent advances in text-to-image synthesis -- specifically, the deep fusion of large language models (LLMs) and diffusion transformers (DiTs) for multi-modal generation. Previous studies mainly focused on overall system performance rather than detailed comparisons with alternative methods, and key design details and training recipes were often left undisclosed. These gaps create uncertainty about the real potential of this approach. To fill these gaps, we conduct an empirical study on text-to-image generation, performing controlled comparisons with established baselines, analyzing important design choices, and providing a clear, reproducible recipe for training at scale. We hope this work offers meaningful data points and practical guidelines for future research in multi-modal generation.

Title: ChronoSteer: Bridging Large Language Model and Time Series Foundation Model via Synthetic Data

• Authors: Chengsen Wang, Qi Qi, Zhongwen Rao, Lujia Pan, Jingyu Wang, Jianxin Liao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.10083
• Pdf URL: https://arxiv.org/pdf/2505.10083
• Copy Paste: [[2505.10083]] ChronoSteer: Bridging Large Language Model and Time Series Foundation Model via Synthetic Data(https://arxiv.org/abs/2505.10083)
• Keywords: foundation model
• Abstract: Conventional forecasting methods rely on unimodal time series data, limiting their ability to exploit rich textual information. Recently, large language models (LLMs) and time series foundation models (TSFMs) have demonstrated powerful capability in textual reasoning and temporal modeling, respectively. Integrating the strengths of both to construct a multimodal model that concurrently leverages both temporal and textual information for future inference has emerged as a critical research challenge. To address the scarcity of event-series paired data, we propose a decoupled framework: an LLM is employed to transform textual events into revision instructions, which are then used to steer the output of TSFM. To implement this framework, we introduce ChronoSteer, a multimodal TSFM that can be steered through textual revision instructions, effectively bridging LLM and TSFM. Moreover, to mitigate the shortage of cross-modal instruction-series paired data, we devise a two-stage training strategy based on synthetic data. In addition, we also construct a high-quality multimodal time series forecasting benchmark to address the information leakage concerns during evaluation. After integrating with an LLM, ChronoSteer, which is trained exclusively on synthetic data, achieves a 25.7% improvement in prediction accuracy compared to the unimodal backbone and a 22.5% gain over the previous state-of-the-art multimodal method.

Title: ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization

• Authors: Wenhao Shen, Wanqi Yin, Xiaofeng Yang, Cheng Chen, Chaoyue Song, Zhongang Cai, Lei Yang, Hao Wang, Guosheng Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10250
• Pdf URL: https://arxiv.org/pdf/2505.10250
• Copy Paste: [[2505.10250]] ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization(https://arxiv.org/abs/2505.10250)
• Keywords: diffusion
• Abstract: Human mesh recovery (HMR) from a single image is inherently ill-posed due to depth ambiguity and occlusions. Probabilistic methods have tried to solve this by generating numerous plausible 3D human mesh predictions, but they often exhibit misalignment with 2D image observations and weak robustness to in-the-wild images. To address these issues, we propose ADHMR, a framework that Aligns a Diffusion-based HMR model in a preference optimization manner. First, we train a human mesh prediction assessment model, HMR-Scorer, capable of evaluating predictions even for in-the-wild images without 3D annotations. We then use HMR-Scorer to create a preference dataset, where each input image has a pair of winner and loser mesh predictions. This dataset is used to finetune the base model using direct preference optimization. Moreover, HMR-Scorer also helps improve existing HMR models by data cleaning, even with fewer training samples. Extensive experiments show that ADHMR outperforms current state-of-the-art methods. Code is available at: this https URL.

Title: The Evolving Landscape of Generative Large Language Models and Traditional Natural Language Processing in Medicine

• Authors: Rui Yang, Huitao Li, Matthew Yu Heng Wong, Yuhe Ke, Xin Li, Kunyu Yu, Jingchi Liao, Jonathan Chong Kai Liew, Sabarinath Vinod Nair, Jasmine Chiat Ling Ong, Irene Li, Douglas Teodoro, Chuan Hong, Daniel Shu Wei Ting, Nan Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.10261
• Pdf URL: https://arxiv.org/pdf/2505.10261
• Copy Paste: [[2505.10261]] The Evolving Landscape of Generative Large Language Models and Traditional Natural Language Processing in Medicine(https://arxiv.org/abs/2505.10261)
• Keywords: generative
• Abstract: Natural language processing (NLP) has been traditionally applied to medicine, and generative large language models (LLMs) have become prominent recently. However, the differences between them across different medical tasks remain underexplored. We analyzed 19,123 studies, finding that generative LLMs demonstrate advantages in open-ended tasks, while traditional NLP dominates in information extraction and analysis tasks. As these technologies advance, ethical use of them is essential to ensure their potential in medical applications.

Title: MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models

• Authors: Guillaume Balezo, Roger Trullo, Albert Pla Planas, Etienne Decenciere, Thomas Walter
• Subjects: cs.CV, q-bio.TO
• Abstract URL: https://arxiv.org/abs/2505.10294
• Pdf URL: https://arxiv.org/pdf/2505.10294
• Copy Paste: [[2505.10294]] MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models(https://arxiv.org/abs/2505.10294)
• Keywords: foundation model
• Abstract: Histopathological analysis is a cornerstone of cancer diagnosis, with Hematoxylin and Eosin (H&E) staining routinely acquired for every patient to visualize cell morphology and tissue architecture. On the other hand, multiplex immunofluorescence (mIF) enables more precise cell type identification via proteomic markers, but has yet to achieve widespread clinical adoption due to cost and logistical constraints. To bridge this gap, we introduce MIPHEI (Multiplex Immunofluorescence Prediction from H&E), a U-Net-inspired architecture that integrates state-of-the-art ViT foundation models as encoders to predict mIF signals from H&E images. MIPHEI targets a comprehensive panel of markers spanning nuclear content, immune lineages (T cells, B cells, myeloid), epithelium, stroma, vasculature, and proliferation. We train our model using the publicly available ORION dataset of restained H&E and mIF images from colorectal cancer tissue, and validate it on two independent datasets. MIPHEI achieves accurate cell-type classification from H&E alone, with F1 scores of 0.88 for Pan-CK, 0.57 for CD3e, 0.56 for SMA, 0.36 for CD68, and 0.30 for CD20, substantially outperforming both a state-of-the-art baseline and a random classifier for most markers. Our results indicate that our model effectively captures the complex relationships between nuclear morphologies in their tissue context, as visible in H&E images and molecular markers defining specific cell types. MIPHEI offers a promising step toward enabling cell-type-aware analysis of large-scale H&E datasets, in view of uncovering relationships between spatial cellular organization and patient outcomes.

Title: Defending the Edge: Representative-Attention for Mitigating Backdoor Attacks in Federated Learning

• Authors: Chibueze Peace Obioma, Youcheng Sun, Mustafa A. Mustafa
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2505.10297
• Pdf URL: https://arxiv.org/pdf/2505.10297
• Copy Paste: [[2505.10297]] Defending the Edge: Representative-Attention for Mitigating Backdoor Attacks in Federated Learning(https://arxiv.org/abs/2505.10297)
• Keywords: anomaly
• Abstract: Federated learning (FL) enhances privacy and reduces communication cost for resource-constrained edge clients by supporting distributed model training at the edge. However, the heterogeneous nature of such devices produces diverse, non-independent, and identically distributed (non-IID) data, making the detection of backdoor attacks more challenging. In this paper, we propose a novel federated representative-attention-based defense mechanism, named FeRA, that leverages cross-client attention over internal feature representations to distinguish benign from malicious clients. FeRA computes an anomaly score based on representation reconstruction errors, effectively identifying clients whose internal activations significantly deviate from the group consensus. Our evaluation demonstrates FeRA's robustness across various FL scenarios, including challenging non-IID data distributions typical of edge devices. Experimental results show that it effectively reduces backdoor attack success rates while maintaining high accuracy on the main task. The method is model-agnostic, attack-agnostic, and does not require labeled reference data, making it well suited to heterogeneous and resource-limited edge deployments.

Title: A Representation Learning Approach to Feature Drift Detection in Wireless Networks

• Authors: Athanasios Tziouvaras, Blaz Bertalanic, George Floros, Kostas Kolomvatsos, Panagiotis Sarigiannidis, Carolina Fortuna
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.10325
• Pdf URL: https://arxiv.org/pdf/2505.10325
• Copy Paste: [[2505.10325]] A Representation Learning Approach to Feature Drift Detection in Wireless Networks(https://arxiv.org/abs/2505.10325)
• Keywords: anomaly
• Abstract: AI is foreseen to be a centerpiece in next generation wireless networks enabling enabling ubiquitous communication as well as new services. However, in real deployment, feature distribution changes may degrade the performance of AI models and lead to undesired behaviors. To counter for undetected model degradation, we propose ALERT; a method that can detect feature distribution changes and trigger model re-training that works well on two wireless network use cases: wireless fingerprinting and link anomaly detection. ALERT includes three components: representation learning, statistical testing and utility assessment. We rely on MLP for designing the representation learning component, on Kolmogorov-Smirnov and Population Stability Index tests for designing the statistical testing and a new function for utility assessment. We show the superiority of the proposed method against ten standard drift detection methods available in the literature on two wireless network use cases.

Title: A Unified and Scalable Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability

• Authors: Jie Zhu, Jirong Zha, Ding Li, Leye Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10351
• Pdf URL: https://arxiv.org/pdf/2505.10351
• Copy Paste: [[2505.10351]] A Unified and Scalable Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability(https://arxiv.org/abs/2505.10351)
• Keywords: self-supervised
• Abstract: Self-supervised learning shows promise in harnessing extensive unlabeled data, but it also confronts significant privacy concerns, especially in vision. In this paper, we perform membership inference on visual self-supervised models in a more realistic setting: self-supervised training method and details are unknown for an adversary when attacking as he usually faces a black-box system in practice. In this setting, considering that self-supervised model could be trained by completely different self-supervised paradigms, e.g., masked image modeling and contrastive learning, with complex training details, we propose a unified membership inference method called PartCrop. It is motivated by the shared part-aware capability among models and stronger part response on the training data. Specifically, PartCrop crops parts of objects in an image to query responses within the image in representation space. We conduct extensive attacks on self-supervised models with different training protocols and structures using three widely used image datasets. The results verify the effectiveness and generalization of PartCrop. Moreover, to defend against PartCrop, we evaluate two common approaches, i.e., early stop and differential privacy, and propose a tailored method called shrinking crop scale range. The defense experiments indicate that all of them are effective. Finally, besides prototype testing on toy visual encoders and small-scale image datasets, we quantitatively study the impacts of scaling from both data and model aspects in a realistic scenario and propose a scalable PartCrop-v2 by introducing two structural improvements to PartCrop. Our code is at this https URL.

Title: Two-Stage Generative Model for Intracranial Aneurysm Meshes with Morphological Marker Conditioning

• Authors: Wenhao Ding, Choon Hwai Yap, Kangjun Ji, Simão Castro
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.10407
• Pdf URL: https://arxiv.org/pdf/2505.10407
• Copy Paste: [[2505.10407]] Two-Stage Generative Model for Intracranial Aneurysm Meshes with Morphological Marker Conditioning(https://arxiv.org/abs/2505.10407)
• Keywords: generative
• Abstract: A generative model for the mesh geometry of intracranial aneurysms (IA) is crucial for training networks to predict blood flow forces in real time, which is a key factor affecting disease progression. This need is necessitated by the absence of a large IA image datasets. Existing shape generation methods struggle to capture realistic IA features and ignore the relationship between IA pouches and parent vessels, limiting physiological realism and their generation cannot be controlled to have specific morphological measurements. We propose AneuG, a two-stage Variational Autoencoder (VAE)-based IA mesh generator. In the first stage, AneuG generates low-dimensional Graph Harmonic Deformation (GHD) tokens to encode and reconstruct aneurysm pouch shapes, constrained to morphing energy statistics truths. GHD enables more accurate shape encoding than alternatives. In the second stage, AneuG generates parent vessels conditioned on GHD tokens, by generating vascular centreline and propagating the cross-section. AneuG's IA shape generation can further be conditioned to have specific clinically relevant morphological measurements. This is useful for studies to understand shape variations represented by clinical measurements, and for flow simulation studies to understand effects of specific clinical shape parameters on fluid dynamics. Source code and implementation details are available at this https URL.

Title: Hierarchical Document Refinement for Long-context Retrieval-augmented Generation

• Authors: Jiajie Jin, Xiaoxi Li, Guanting Dong, Yuyao Zhang, Yutao Zhu, Yongkang Wu, Zhonghua Li, Qi Ye, Zhicheng Dou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.10413
• Pdf URL: https://arxiv.org/pdf/2505.10413
• Copy Paste: [[2505.10413]] Hierarchical Document Refinement for Long-context Retrieval-augmented Generation(https://arxiv.org/abs/2505.10413)
• Keywords: foundation model
• Abstract: Real-world RAG applications often encounter long-context input scenarios, where redundant information and noise results in higher inference costs and reduced performance. To address these challenges, we propose LongRefiner, an efficient plug-and-play refiner that leverages the inherent structural characteristics of long documents. LongRefiner employs dual-level query analysis, hierarchical document structuring, and adaptive refinement through multi-task learning on a single foundation model. Experiments on seven QA datasets demonstrate that LongRefiner achieves competitive performance in various scenarios while using 10x fewer computational costs and latency compared to the best baseline. Further analysis validates that LongRefiner is scalable, efficient, and effective, providing practical insights for real-world long-text RAG applications. Our code is available at this https URL.

Title: Score-based diffusion nowcasting of GOES imagery

• Authors: Randy J. Chase, Katherine Haynes, Lander Ver Hoef, Imme Ebert-Uphoff
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2505.10432
• Pdf URL: https://arxiv.org/pdf/2505.10432
• Copy Paste: [[2505.10432]] Score-based diffusion nowcasting of GOES imagery(https://arxiv.org/abs/2505.10432)
• Keywords: diffusion
• Abstract: Clouds and precipitation are important for understanding weather and climate. Simulating clouds and precipitation with traditional numerical weather prediction is challenging because of the sub-grid parameterizations required. Machine learning has been explored for forecasting clouds and precipitation, but early machine learning methods often created blurry forecasts. In this paper we explore a newer method, named score-based diffusion, to nowcast (zero to three hour forecast) clouds and precipitation. We discuss the background and intuition of score-based diffusion models - thus providing a starting point for the community - while exploring the methodology's use for nowcasting geostationary infrared imagery. We experiment with three main types of diffusion models: a standard score-based diffusion model (Diff); a residual correction diffusion model (CorrDiff); and a latent diffusion model (LDM). Our results show that the diffusion models are able to not only advect existing clouds, but also generate and decay clouds, including convective initiation. These results are surprising because the forecasts are initiated with only the past 20 mins of infrared satellite imagery. A case study qualitatively shows the preservation of high resolution features longer into the forecast than a conventional mean-squared error trained U-Net. The best of the three diffusion models tested was the CorrDiff approach, outperforming all other diffusion models, the traditional U-Net, and a persistence forecast by one to two kelvin on root mean squared error. The diffusion models also enable out-of-the-box ensemble generation, which shows skillful calibration, with the spread of the ensemble correlating well to the error.

Title: PIF: Anomaly detection via preference embedding

• Authors: Filippo Leveni, Luca Magri, Giacomo Boracchi, Cesare Alippi
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2505.10441
• Pdf URL: https://arxiv.org/pdf/2505.10441
• Copy Paste: [[2505.10441]] PIF: Anomaly detection via preference embedding(https://arxiv.org/abs/2505.10441)
• Keywords: anomaly
• Abstract: We address the problem of detecting anomalies with respect to structured patterns. To this end, we conceive a novel anomaly detection method called PIF, that combines the advantages of adaptive isolation methods with the flexibility of preference embedding. Specifically, we propose to embed the data in a high dimensional space where an efficient tree-based method, PI-Forest, is employed to compute an anomaly score. Experiments on synthetic and real datasets demonstrate that PIF favorably compares with state-of-the-art anomaly detection techniques, and confirm that PI-Forest is better at measuring arbitrary distances and isolate points in the preference space.

Title: Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models

• Authors: Zemin Huang, Zhiyang Chen, Zijun Wang, Tiancheng Li, Guo-Jun Qi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.10446
• Pdf URL: https://arxiv.org/pdf/2505.10446
• Copy Paste: [[2505.10446]] Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models(https://arxiv.org/abs/2505.10446)
• Keywords: diffusion
• Abstract: We introduce the \emph{Diffusion Chain of Lateral Thought (DCoLT)}, a reasoning framework for diffusion language models. DCoLT treats each intermediate step in the reverse diffusion process as a latent "thinking" action and optimizes the entire reasoning trajectory to maximize the reward on the correctness of the final answer with outcome-based Reinforcement Learning (RL). Unlike traditional Chain-of-Thought (CoT) methods that follow a causal, linear thinking process, DCoLT allows bidirectional, non-linear reasoning with no strict rule on grammatical correctness amid its intermediate steps of thought. We implement DCoLT on two representative Diffusion Language Models (DLMs). First, we choose SEDD as a representative continuous-time discrete diffusion model, where its concrete score derives a probabilistic policy to maximize the RL reward over the entire sequence of intermediate diffusion steps. We further consider the discrete-time masked diffusion language model -- LLaDA, and find that the order to predict and unmask tokens plays an essential role to optimize its RL action resulting from the ranking-based Unmasking Policy Module (UPM) defined by the Plackett-Luce model. Experiments on both math and code generation tasks show that using only public data and 16 H800 GPUs, DCoLT-reinforced DLMs outperform other DLMs trained by SFT or RL or even both. Notably, DCoLT-reinforced LLaDA boosts its reasoning accuracy by +9.8%, +5.7%, +11.4%, +19.5% on GSM8K, MATH, MBPP, and HumanEval.

Title: Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AI

• Authors: Agnik Saha, Victoria Churchill, Anny D. Rodriguez, Ugur Kursuncu, Muhammed Y. Idris
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.10472
• Pdf URL: https://arxiv.org/pdf/2505.10472
• Copy Paste: [[2505.10472]] Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AI(https://arxiv.org/abs/2505.10472)
• Keywords: generative
• Abstract: Effective communication about breast and cervical cancers remains a persistent health challenge, with significant gaps in public understanding of cancer prevention, screening, and treatment, potentially leading to delayed diagnoses and inadequate treatments. This study evaluates the capabilities and limitations of Large Language Models (LLMs) in generating accurate, safe, and accessible cancer-related information to support patient understanding. We evaluated five general-purpose and three medical LLMs using a mixed-methods evaluation framework across linguistic quality, safety and trustworthiness, and communication accessibility and affectiveness. Our approach utilized quantitative metrics, qualitative expert ratings, and statistical analysis using Welch's ANOVA, Games-Howell, and Hedges' g. Our results show that general-purpose LLMs produced outputs of higher linguistic quality and affectiveness, while medical LLMs demonstrate greater communication accessibility. However, medical LLMs tend to exhibit higher levels of potential harm, toxicity, and bias, reducing their performance in safety and trustworthiness. Our findings indicate a duality between domain-specific knowledge and safety in health communications. The results highlight the need for intentional model design with targeted improvements, particularly in mitigating harm and bias, and improving safety and affectiveness. This study provides a comprehensive evaluation of LLMs for cancer communication, offering critical insights for improving AI-generated health content and informing future development of accurate, safe, and accessible digital health tools.

Title: Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps

• Authors: Ningyuan Yang, Jiaxuan Gao, Feng Gao, Yi Wu, Chao Yu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.10482
• Pdf URL: https://arxiv.org/pdf/2505.10482
• Copy Paste: [[2505.10482]] Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps(https://arxiv.org/abs/2505.10482)
• Keywords: diffusion
• Abstract: Diffusion policies, widely adopted in decision-making scenarios such as robotics, gaming and autonomous driving, are capable of learning diverse skills from demonstration data due to their high representation power. However, the sub-optimal and limited coverage of demonstration data could lead to diffusion policies that generate sub-optimal trajectories and even catastrophic failures. While reinforcement learning (RL)-based fine-tuning has emerged as a promising solution to address these limitations, existing approaches struggle to effectively adapt Proximal Policy Optimization (PPO) to diffusion models. This challenge stems from the computational intractability of action likelihood estimation during the denoising process, which leads to complicated optimization objectives. In our experiments starting from randomly initialized policies, we find that online tuning of Diffusion Policies demonstrates much lower sample efficiency compared to directly applying PPO on MLP policies (MLP+PPO). To address these challenges, we introduce NCDPO, a novel framework that reformulates Diffusion Policy as a noise-conditioned deterministic policy. By treating each denoising step as a differentiable transformation conditioned on pre-sampled noise, NCDPO enables tractable likelihood evaluation and gradient backpropagation through all diffusion timesteps. Our experiments demonstrate that NCDPO achieves sample efficiency comparable to MLP+PPO when training from scratch, outperforming existing methods in both sample efficiency and final performance across diverse benchmarks, including continuous robot control and multi-agent game scenarios. Furthermore, our experimental results show that our method is robust to the number denoising timesteps in the Diffusion Policy.

Title: CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs

• Authors: Raman Dutt, Pedro Sanchez, Yongchen Yao, Steven McDonagh, Sotirios A. Tsaftaris, Timothy Hospedales
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10496
• Pdf URL: https://arxiv.org/pdf/2505.10496
• Copy Paste: [[2505.10496]] CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs(https://arxiv.org/abs/2505.10496)
• Keywords: generative
• Abstract: We introduce CheXGenBench, a rigorous and multifaceted evaluation framework for synthetic chest radiograph generation that simultaneously assesses fidelity, privacy risks, and clinical utility across state-of-the-art text-to-image generative models. Despite rapid advancements in generative AI for real-world imagery, medical domain evaluations have been hindered by methodological inconsistencies, outdated architectural comparisons, and disconnected assessment criteria that rarely address the practical clinical value of synthetic samples. CheXGenBench overcomes these limitations through standardised data partitioning and a unified evaluation protocol comprising over 20 quantitative metrics that systematically analyse generation quality, potential privacy vulnerabilities, and downstream clinical applicability across 11 leading text-to-image architectures. Our results reveal critical inefficiencies in the existing evaluation protocols, particularly in assessing generative fidelity, leading to inconsistent and uninformative comparisons. Our framework establishes a standardised benchmark for the medical AI community, enabling objective and reproducible comparisons while facilitating seamless integration of both existing and future generative models. Additionally, we release a high-quality, synthetic dataset, SynthCheX-75K, comprising 75K radiographs generated by the top-performing model (Sana 0.6B) in our benchmark to support further research in this critical domain. Through CheXGenBench, we establish a new state-of-the-art and release our framework, models, and SynthCheX-75K dataset at this https URL

Title: Multi-Token Prediction Needs Registers

• Authors: Anastasios Gerontopoulos, Spyros Gidaris, Nikos Komodakis
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.10518
• Pdf URL: https://arxiv.org/pdf/2505.10518
• Copy Paste: [[2505.10518]] Multi-Token Prediction Needs Registers(https://arxiv.org/abs/2505.10518)
• Keywords: generative
• Abstract: Multi-token prediction has emerged as a promising objective for improving language model pretraining, but its benefits have not consistently generalized to other settings such as fine-tuning. In this paper, we propose MuToR, a simple and effective approach to multi-token prediction that interleaves learnable register tokens into the input sequence, each tasked with predicting future targets. Compared to existing methods, MuToR offers several key advantages: it introduces only a negligible number of additional parameters, requires no architectural changes--ensuring compatibility with off-the-shelf pretrained language models--and remains aligned with the next-token pretraining objective, making it especially well-suited for supervised fine-tuning. Moreover, it naturally supports scalable prediction horizons. We demonstrate the effectiveness and versatility of MuToR across a range of use cases, including supervised fine-tuning, parameter-efficient fine-tuning (PEFT), and pretraining, on challenging generative tasks in both language and vision domains. Our code will be available at: this https URL.

Title: Pharmacophore-Conditioned Diffusion Model for Ligand-Based De Novo Drug Design

• Authors: Amira Alakhdar, Barnabas Poczos, Newell Washburn
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.10545
• Pdf URL: https://arxiv.org/pdf/2505.10545
• Copy Paste: [[2505.10545]] Pharmacophore-Conditioned Diffusion Model for Ligand-Based De Novo Drug Design(https://arxiv.org/abs/2505.10545)
• Keywords: diffusion, generative
• Abstract: Developing bioactive molecules remains a central, time- and cost-heavy challenge in drug discovery, particularly for novel targets lacking structural or functional data. Pharmacophore modeling presents an alternative for capturing the key features required for molecular bioactivity against a biological target. In this work, we present PharmaDiff, a pharmacophore-conditioned diffusion model for 3D molecular generation. PharmaDiff employs a transformer-based architecture to integrate an atom-based representation of the 3D pharmacophore into the generative process, enabling the precise generation of 3D molecular graphs that align with predefined pharmacophore hypotheses. Through comprehensive testing, PharmaDiff demonstrates superior performance in matching 3D pharmacophore constraints compared to ligand-based drug design methods. Additionally, it achieves higher docking scores across a range of proteins in structure-based drug design, without the need for target protein structures. By integrating pharmacophore modeling with 3D generative techniques, PharmaDiff offers a powerful and flexible framework for rational drug design.

Title: Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Data

• Authors: Yiwen Liu, Jessica Bader, Jae Myung Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.10551
• Pdf URL: https://arxiv.org/pdf/2505.10551
• Copy Paste: [[2505.10551]] Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Data(https://arxiv.org/abs/2505.10551)
• Keywords: diffusion
• Abstract: With the development of photorealistic diffusion models, models trained in part or fully on synthetic data achieve progressively better results. However, diffusion models still routinely generate images that would not exist in reality, such as a dog floating above the ground or with unrealistic texture artifacts. We define the concept of feasibility as whether attributes in a synthetic image could realistically exist in the real-world domain; synthetic images containing attributes that violate this criterion are considered infeasible. Intuitively, infeasible images are typically considered out-of-distribution; thus, training on such images is expected to hinder a model's ability to generalize to real-world data, and they should therefore be excluded from the training set whenever possible. However, does feasibility really matter? In this paper, we investigate whether enforcing feasibility is necessary when generating synthetic training data for CLIP-based classifiers, focusing on three target attributes: background, color, and texture. We introduce VariReal, a pipeline that minimally edits a given source image to include feasible or infeasible attributes given by the textual prompt generated by a large language model. Our experiments show that feasibility minimally affects LoRA-fine-tuned CLIP performance, with mostly less than 0.3% difference in top-1 accuracy across three fine-grained datasets. Also, the attribute matters on whether the feasible/infeasible images adversarially influence the classification performance. Finally, mixing feasible and infeasible images in training datasets does not significantly impact performance compared to using purely feasible or infeasible datasets.

Title: End-to-End Vision Tokenizer Tuning

• Authors: Wenxuan Wang, Fan Zhang, Yufeng Cui, Haiwen Diao, Zhuoyan Luo, Huchuan Lu, Jing Liu, Xinlong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10562
• Pdf URL: https://arxiv.org/pdf/2505.10562
• Copy Paste: [[2505.10562]] End-to-End Vision Tokenizer Tuning(https://arxiv.org/abs/2505.10562)
• Keywords: foundation model
• Abstract: Existing vision tokenization isolates the optimization of vision tokenizers from downstream training, implicitly assuming the visual tokens can generalize well across various tasks, e.g., image generation and visual question answering. The vision tokenizer optimized for low-level reconstruction is agnostic to downstream tasks requiring varied representations and semantics. This decoupled paradigm introduces a critical misalignment: The loss of the vision tokenization can be the representation bottleneck for target tasks. For example, errors in tokenizing text in a given image lead to poor results when recognizing or generating them. To address this, we propose ETT, an end-to-end vision tokenizer tuning approach that enables joint optimization between vision tokenization and target autoregressive tasks. Unlike prior autoregressive models that use only discrete indices from a frozen vision tokenizer, ETT leverages the visual embeddings of the tokenizer codebook, and optimizes the vision tokenizers end-to-end with both reconstruction and caption objectives. ETT can be seamlessly integrated into existing training pipelines with minimal architecture modifications. Our ETT is simple to implement and integrate, without the need to adjust the original codebooks or architectures of the employed large language models. Extensive experiments demonstrate that our proposed end-to-end vision tokenizer tuning unlocks significant performance gains, i.e., 2-6% for multimodal understanding and visual generation tasks compared to frozen tokenizer baselines, while preserving the original reconstruction capability. We hope this very simple and strong method can empower multimodal foundation models besides image generation and understanding.

Title: 3D-Fixup: Advancing Photo Editing with 3D Priors

• Authors: Yen-Chi Cheng, Krishna Kumar Singh, Jae Shin Yoon, Alex Schwing, Liangyan Gui, Matheus Gadelha, Paul Guerrero, Nanxuan Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10566
• Pdf URL: https://arxiv.org/pdf/2505.10566
• Copy Paste: [[2505.10566]] 3D-Fixup: Advancing Photo Editing with 3D Priors(https://arxiv.org/abs/2505.10566)
• Keywords: diffusion, generative
• Abstract: Despite significant advances in modeling image priors via diffusion models, 3D-aware image editing remains challenging, in part because the object is only specified via a single image. To tackle this challenge, we propose 3D-Fixup, a new framework for editing 2D images guided by learned 3D priors. The framework supports difficult editing situations such as object translation and 3D rotation. To achieve this, we leverage a training-based approach that harnesses the generative power of diffusion models. As video data naturally encodes real-world physical dynamics, we turn to video data for generating training data pairs, i.e., a source and a target frame. Rather than relying solely on a single trained model to infer transformations between source and target frames, we incorporate 3D guidance from an Image-to-3D model, which bridges this challenging task by explicitly projecting 2D information into 3D space. We design a data generation pipeline to ensure high-quality 3D guidance throughout training. Results show that by integrating these 3D priors, 3D-Fixup effectively supports complex, identity coherent 3D-aware edits, achieving high-quality results and advancing the application of diffusion models in realistic image manipulation. The code is provided at this https URL