2025-05-08

Title: Hierarchical Multi-Label Generation with Probabilistic Level-Constraint

• Authors: Linqing Chen, Weilei Wang, Wentao Wu, Hanmeng Zhong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.03775
• Pdf URL: https://arxiv.org/pdf/2505.03775
• Copy Paste: [[2505.03775]] Hierarchical Multi-Label Generation with Probabilistic Level-Constraint(https://arxiv.org/abs/2505.03775)
• Keywords: generative
• Abstract: Hierarchical Extreme Multi-Label Classification poses greater difficulties compared to traditional multi-label classification because of the intricate hierarchical connections of labels within a domain-specific taxonomy and the substantial number of labels. Some of the prior research endeavors centered on classifying text through several ancillary stages such as the cluster algorithm and multiphase classification. Others made attempts to leverage the assistance of generative methods yet were unable to properly control the output of the generative model. We redefine the task from hierarchical multi-Label classification to Hierarchical Multi-Label Generation (HMG) and employ a generative framework with Probabilistic Level Constraints (PLC) to generate hierarchical labels within a specific taxonomy that have complex hierarchical relationships. The approach we proposed in this paper enables the framework to generate all relevant labels across levels for each document without relying on preliminary operations like clustering. Meanwhile, it can control the model output precisely in terms of count, length, and level aspects. Experiments demonstrate that our approach not only achieves a new SOTA performance in the HMG task, but also has a much better performance in constrained the output of model than previous research work.

Title: A Time-Series Data Augmentation Model through Diffusion and Transformer Integration

• Authors: Yuren Zhang, Zhongnan Pu, Lei Jing
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.03790
• Pdf URL: https://arxiv.org/pdf/2505.03790
• Copy Paste: [[2505.03790]] A Time-Series Data Augmentation Model through Diffusion and Transformer Integration(https://arxiv.org/abs/2505.03790)
• Keywords: diffusion
• Abstract: With the development of Artificial Intelligence, numerous real-world tasks have been accomplished using technology integrated with deep learning. To achieve optimal performance, deep neural networks typically require large volumes of data for training. Although advances in data augmentation have facilitated the acquisition of vast datasets, most of this data is concentrated in domains like images and speech. However, there has been relatively less focus on augmenting time-series data. To address this gap and generate a substantial amount of time-series data, we propose a simple and effective method that combines the Diffusion and Transformer models. By utilizing an adjusted diffusion denoising model to generate a large volume of initial time-step action data, followed by employing a Transformer model to predict subsequent actions, and incorporating a weighted loss function to achieve convergence, the method demonstrates its effectiveness. Using the performance improvement of the model after applying augmented data as a benchmark, and comparing the results with those obtained without data augmentation or using traditional data augmentation methods, this approach shows its capability to produce high-quality augmented data.

Title: AI-Driven IRM: Transforming insider risk management with adaptive scoring and LLM-based threat detection

• Authors: Lokesh Koli, Shubham Kalra, Rohan Thakur, Anas Saifi, Karanpreet Singh
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.03796
• Pdf URL: https://arxiv.org/pdf/2505.03796
• Copy Paste: [[2505.03796]] AI-Driven IRM: Transforming insider risk management with adaptive scoring and LLM-based threat detection(https://arxiv.org/abs/2505.03796)
• Keywords: anomaly
• Abstract: Insider threats pose a significant challenge to organizational security, often evading traditional rule-based detection systems due to their subtlety and contextual nature. This paper presents an AI-powered Insider Risk Management (IRM) system that integrates behavioral analytics, dynamic risk scoring, and real-time policy enforcement to detect and mitigate insider threats with high accuracy and adaptability. We introduce a hybrid scoring mechanism - transitioning from the static PRISM model to an adaptive AI-based model utilizing an autoencoder neural network trained on expert-annotated user activity data. Through iterative feedback loops and continuous learning, the system reduces false positives by 59% and improves true positive detection rates by 30%, demonstrating substantial gains in detection precision. Additionally, the platform scales efficiently, processing up to 10 million log events daily with sub-300ms query latency, and supports automated enforcement actions for policy violations, reducing manual intervention. The IRM system's deployment resulted in a 47% reduction in incident response times, highlighting its operational impact. Future enhancements include integrating explainable AI, federated learning, graph-based anomaly detection, and alignment with Zero Trust principles to further elevate its adaptability, transparency, and compliance-readiness. This work establishes a scalable and proactive framework for mitigating emerging insider risks in both on-premises and hybrid environments.

Title: Position: Foundation Models Need Digital Twin Representations

• Authors: Yiqing Shen, Hao Ding, Lalithkumar Seenivasan, Tianmin Shu, Mathias Unberath
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.03798
• Pdf URL: https://arxiv.org/pdf/2505.03798
• Copy Paste: [[2505.03798]] Position: Foundation Models Need Digital Twin Representations(https://arxiv.org/abs/2505.03798)
• Keywords: foundation model
• Abstract: Current foundation models (FMs) rely on token representations that directly fragment continuous real-world multimodal data into discrete tokens. They limit FMs to learning real-world knowledge and relationships purely through statistical correlation rather than leveraging explicit domain knowledge. Consequently, current FMs struggle with maintaining semantic coherence across modalities, capturing fine-grained spatial-temporal dynamics, and performing causal reasoning. These limitations cannot be overcome by simply scaling up model size or expanding datasets. This position paper argues that the machine learning community should consider digital twin (DT) representations, which are outcome-driven digital representations that serve as building blocks for creating virtual replicas of physical processes, as an alternative to the token representation for building FMs. Finally, we discuss how DT representations can address these challenges by providing physically grounded representations that explicitly encode domain knowledge and preserve the continuous nature of real-world processes.

Title: Information Filtering Networks: Theoretical Foundations, Generative Methodologies, and Real-World Applications

• Authors: Tomaso Aste
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.03812
• Pdf URL: https://arxiv.org/pdf/2505.03812
• Copy Paste: [[2505.03812]] Information Filtering Networks: Theoretical Foundations, Generative Methodologies, and Real-World Applications(https://arxiv.org/abs/2505.03812)
• Keywords: generative
• Abstract: Information Filtering Networks (IFNs) provide a powerful framework for modeling complex systems through globally sparse yet locally dense and interpretable structures that capture multivariate dependencies. This review offers a comprehensive account of IFNs, covering their theoretical foundations, construction methodologies, and diverse applications. Tracing their origins from early network-based models to advanced formulations such as the Triangulated Maximally Filtered Graph (TMFG) and the Maximally Filtered Clique Forest (MFCF), the paper highlights how IFNs address key challenges in high-dimensional data-driven modeling. IFNs and their construction methodologies are intrinsically higher-order networks that generate simplicial complexes-structures that are only now becoming popular in the broader literature. Applications span fields including finance, biology, psychology, and artificial intelligence, where IFNs improve interpretability, computational efficiency, and predictive performance. Special attention is given to their role in graphical modeling, where IFNs enable the estimation of sparse inverse covariance matrices with greater accuracy and scalability than traditional approaches like Graphical LASSO. Finally, the review discusses recent developments that integrate IFNs with machine learning and deep learning, underscoring their potential not only to bridge classical network theory with contemporary data-driven paradigms, but also to shape the architectures of deep learning models themselves.

Title: Advanced Clustering Framework for Semiconductor Image Analytics Integrating Deep TDA with Self-Supervised and Transfer Learning Techniques

• Authors: Janhavi Giri, Attila Lengyel, Don Kent, Edward Kibardin
• Subjects: cs.CV, cs.AI, cs.ET, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.03848
• Pdf URL: https://arxiv.org/pdf/2505.03848
• Copy Paste: [[2505.03848]] Advanced Clustering Framework for Semiconductor Image Analytics Integrating Deep TDA with Self-Supervised and Transfer Learning Techniques(https://arxiv.org/abs/2505.03848)
• Keywords: self-supervised
• Abstract: Semiconductor manufacturing generates vast amounts of image data, crucial for defect identification and yield optimization, yet often exceeds manual inspection capabilities. Traditional clustering techniques struggle with high-dimensional, unlabeled data, limiting their effectiveness in capturing nuanced patterns. This paper introduces an advanced clustering framework that integrates deep Topological Data Analysis (TDA) with self-supervised and transfer learning techniques, offering a novel approach to unsupervised image clustering. TDA captures intrinsic topological features, while self-supervised learning extracts meaningful representations from unlabeled data, reducing reliance on labeled datasets. Transfer learning enhances the framework's adaptability and scalability, allowing fine-tuning to new datasets without retraining from scratch. Validated on synthetic and open-source semiconductor image datasets, the framework successfully identifies clusters aligned with defect patterns and process variations. This study highlights the transformative potential of combining TDA, self-supervised learning, and transfer learning, providing a scalable solution for proactive process monitoring and quality control in semiconductor manufacturing and other domains with large-scale image datasets.

Title: Machine Learning: a Lecture Note

• Authors: Kyunghyun Cho
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2505.03861
• Pdf URL: https://arxiv.org/pdf/2505.03861
• Copy Paste: [[2505.03861]] Machine Learning: a Lecture Note(https://arxiv.org/abs/2505.03861)
• Keywords: generative
• Abstract: This lecture note is intended to prepare early-year master's and PhD students in data science or a related discipline with foundational ideas in machine learning. It starts with basic ideas in modern machine learning with classification as a main target task. These basic ideas include loss formulation, backpropagation, stochastic gradient descent, generalization, model selection as well as fundamental blocks of artificial neural networks. Based on these basic ideas, the lecture note explores in depth the probablistic approach to unsupervised learning, covering directed latent variable models, product of experts, generative adversarial networks and autoregressive models. Finally, the note ends by covering a diverse set of further topics, such as reinforcement learning, ensemble methods and meta-learning. After reading this lecture note, a student should be ready to embark on studying and researching more advanced topics in machine learning and more broadly artificial intelligence.

Title: Explaining Anomalies with Tensor Networks

• Authors: Hans Hohenfeld, Marius Beuerle, Elie Mounzer
• Subjects: cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2505.03911
• Pdf URL: https://arxiv.org/pdf/2505.03911
• Copy Paste: [[2505.03911]] Explaining Anomalies with Tensor Networks(https://arxiv.org/abs/2505.03911)
• Keywords: anomaly
• Abstract: Tensor networks, a class of variational quantum many-body wave functions have attracted considerable research interest across many disciplines, including classical machine learning. Recently, Aizpurua et al. demonstrated explainable anomaly detection with matrix product states on a discrete-valued cyber-security task, using quantum-inspired methods to gain insight into the learned model and detected anomalies. Here, we extend this framework to real-valued data domains. We furthermore introduce tree tensor networks for the task of explainable anomaly detection. We demonstrate these methods with three benchmark problems, show adequate predictive performance compared to several baseline models and both tensor network architectures' ability to explain anomalous samples. We thereby extend the application of tensor networks to a broader class of potential problems and open a pathway for future extensions to more complex tensor network architectures.

Title: Diffusion Models are Secretly Exchangeable: Parallelizing DDPMs via Autospeculation

• Authors: Hengyuan Hu, Aniket Das, Dorsa Sadigh, Nima Anari
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.03983
• Pdf URL: https://arxiv.org/pdf/2505.03983
• Copy Paste: [[2505.03983]] Diffusion Models are Secretly Exchangeable: Parallelizing DDPMs via Autospeculation(https://arxiv.org/abs/2505.03983)
• Keywords: diffusion, generative
• Abstract: Denoising Diffusion Probabilistic Models (DDPMs) have emerged as powerful tools for generative modeling. However, their sequential computation requirements lead to significant inference-time bottlenecks. In this work, we utilize the connection between DDPMs and Stochastic Localization to prove that, under an appropriate reparametrization, the increments of DDPM satisfy an exchangeability property. This general insight enables near-black-box adaptation of various performance optimization techniques from autoregressive models to the diffusion setting. To demonstrate this, we introduce \emph{Autospeculative Decoding} (ASD), an extension of the widely used speculative decoding algorithm to DDPMs that does not require any auxiliary draft models. Our theoretical analysis shows that ASD achieves a $\tilde{O} (K^{\frac{1}{3}})$ parallel runtime speedup over the $K$ step sequential DDPM. We also demonstrate that a practical implementation of autospeculative decoding accelerates DDPM inference significantly in various domains.

Title: Action Spotting and Precise Event Detection in Sports: Datasets, Methods, and Challenges

• Authors: Hao Xu, Arbind Agrahari Baniya, Sam Well, Mohamed Reda Bouadjenek, Richard Dazeley, Sunil Aryal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.03991
• Pdf URL: https://arxiv.org/pdf/2505.03991
• Copy Paste: [[2505.03991]] Action Spotting and Precise Event Detection in Sports: Datasets, Methods, and Challenges(https://arxiv.org/abs/2505.03991)
• Keywords: self-supervised
• Abstract: Video event detection has become an essential component of sports analytics, enabling automated identification of key moments and enhancing performance analysis, viewer engagement, and broadcast efficiency. Recent advancements in deep learning, particularly Convolutional Neural Networks (CNNs) and Transformers, have significantly improved accuracy and efficiency in Temporal Action Localization (TAL), Action Spotting (AS), and Precise Event Spotting (PES). This survey provides a comprehensive overview of these three key tasks, emphasizing their differences, applications, and the evolution of methodological approaches. We thoroughly review and categorize existing datasets and evaluation metrics specifically tailored for sports contexts, highlighting the strengths and limitations of each. Furthermore, we analyze state-of-the-art techniques, including multi-modal approaches that integrate audio and visual information, methods utilizing self-supervised learning and knowledge distillation, and approaches aimed at generalizing across multiple sports. Finally, we discuss critical open challenges and outline promising research directions toward developing more generalized, efficient, and robust event detection frameworks applicable to diverse sports. This survey serves as a foundation for future research on efficient, generalizable, and multi-modal sports event detection.

Title: Natural Language Generation in Healthcare: A Review of Methods and Applications

• Authors: Mengxian Lyu, Xiaohan Li, Ziyi Chen, Jinqian Pan, Cheng Peng, Sankalp Talankar, Yonghui Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.04073
• Pdf URL: https://arxiv.org/pdf/2505.04073
• Copy Paste: [[2505.04073]] Natural Language Generation in Healthcare: A Review of Methods and Applications(https://arxiv.org/abs/2505.04073)
• Keywords: generative
• Abstract: Natural language generation (NLG) is the key technology to achieve generative artificial intelligence (AI). With the breakthroughs in large language models (LLMs), NLG has been widely used in various medical applications, demonstrating the potential to enhance clinical workflows, support clinical decision-making, and improve clinical documentation. Heterogeneous and diverse medical data modalities, such as medical text, images, and knowledge bases, are utilized in NLG. Researchers have proposed many generative models and applied them in a number of healthcare applications. There is a need for a comprehensive review of NLG methods and applications in the medical domain. In this study, we systematically reviewed 113 scientific publications from a total of 3,988 NLG-related articles identified using a literature search, focusing on data modality, model architecture, clinical applications, and evaluation methods. Following PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines, we categorize key methods, identify clinical applications, and assess their capabilities, limitations, and emerging challenges. This timely review covers the key NLG technologies and medical applications and provides valuable insights for future studies to leverage NLG to transform medical discovery and healthcare.

Title: MAISY: Motion-Aware Image SYnthesis for MedicalImage Motion Correction

• Authors: Andrew Zhang, Hao Wang, Shuchang Ye, Michael Fulham, Jinman Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.04105
• Pdf URL: https://arxiv.org/pdf/2505.04105
• Copy Paste: [[2505.04105]] MAISY: Motion-Aware Image SYnthesis for MedicalImage Motion Correction(https://arxiv.org/abs/2505.04105)
• Keywords: foundation model, generative
• Abstract: Patient motion during medical image acquisition causes blurring, ghosting, and distorts organs, which makes image interpretation this http URL state-of-the-art algorithms using Generative Adversarial Network (GAN)-based methods with their ability to learn the mappings between corrupted images and their ground truth via Structural Similarity Index Measure (SSIM) loss effectively generate motion-free images. However, we identified the following limitations: (i) they mainly focus on global structural characteristics and therefore overlook localized features that often carry critical pathological information, and (ii) the SSIM loss function struggles to handle images with varying pixel intensities, luminance factors, and variance. In this study, we propose Motion-Aware Image SYnthesis (MAISY) which initially characterize motion and then uses it for correction by: (a) leveraging the foundation model Segment Anything Model (SAM), to dynamically learn spatial patterns along anatomical boundaries where motion artifacts are most pronounced and, (b) introducing the Variance-Selective SSIM (VS-SSIM) loss which adaptively emphasizes spatial regions with high pixel variance to preserve essential anatomical details during artifact correction. Experiments on chest and head CT datasets demonstrate that our model outperformed the state-of-the-art counterparts, with Peak Signal-to-Noise Ratio (PSNR) increasing by 40%, SSIM by 10%, and Dice by 16%.

Title: DiffPattern-Flex: Efficient Layout Pattern Generation via Discrete Diffusion

• Authors: Zixiao Wang, Wenqian Zhao, Yunheng Shen, Yang Bai, Guojin Chen, Farzan Farnia, Bei Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.04173
• Pdf URL: https://arxiv.org/pdf/2505.04173
• Copy Paste: [[2505.04173]] DiffPattern-Flex: Efficient Layout Pattern Generation via Discrete Diffusion(https://arxiv.org/abs/2505.04173)
• Keywords: diffusion, generative
• Abstract: Recent advancements in layout pattern generation have been dominated by deep generative models. However, relying solely on neural networks for legality guarantees raises concerns in many practical applications. In this paper, we present \tool{DiffPattern}-Flex, a novel approach designed to generate reliable layout patterns efficiently. \tool{DiffPattern}-Flex incorporates a new method for generating diverse topologies using a discrete diffusion model while maintaining a lossless and compute-efficient layout representation. To ensure legal pattern generation, we employ {an} optimization-based, white-box pattern assessment process based on specific design rules. Furthermore, fast sampling and efficient legalization technologies are employed to accelerate the generation process. Experimental results across various benchmarks demonstrate that \tool{DiffPattern}-Flex significantly outperforms existing methods and excels at producing reliable layout patterns.

Title: A Large Language Model for Feasible and Diverse Population Synthesis

• Authors: Sung Yoo Lim, Hyunsoo Yun, Prateek Bansal, Dong-Kyu Kim, Eui-Jin Kim
• Subjects: cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2505.04196
• Pdf URL: https://arxiv.org/pdf/2505.04196
• Copy Paste: [[2505.04196]] A Large Language Model for Feasible and Diverse Population Synthesis(https://arxiv.org/abs/2505.04196)
• Keywords: generative
• Abstract: Generating a synthetic population that is both feasible and diverse is crucial for ensuring the validity of downstream activity schedule simulation in activity-based models (ABMs). While deep generative models (DGMs), such as variational autoencoders and generative adversarial networks, have been applied to this task, they often struggle to balance the inclusion of rare but plausible combinations (i.e., sampling zeros) with the exclusion of implausible ones (i.e., structural zeros). To improve feasibility while maintaining diversity, we propose a fine-tuning method for large language models (LLMs) that explicitly controls the autoregressive generation process through topological orderings derived from a Bayesian Network (BN). Experimental results show that our hybrid LLM-BN approach outperforms both traditional DGMs and proprietary LLMs (e.g., ChatGPT-4o) with few-shot learning. Specifically, our approach achieves approximately 95% feasibility, significantly higher than the ~80% observed in DGMs, while maintaining comparable diversity, making it well-suited for practical applications. Importantly, the method is based on a lightweight open-source LLM, enabling fine-tuning and inference on standard personal computing environments. This makes the approach cost-effective and scalable for large-scale applications, such as synthesizing populations in megacities, without relying on expensive infrastructure. By initiating the ABM pipeline with high-quality synthetic populations, our method improves overall simulation reliability and reduces downstream error propagation. The source code for these methods is available for research and practical application.

Title: Cyber Security Data Science: Machine Learning Methods and their Performance on Imbalanced Datasets

• Authors: Mateo Lopez-Ledezma, Gissel Velarde
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.04204
• Pdf URL: https://arxiv.org/pdf/2505.04204
• Copy Paste: [[2505.04204]] Cyber Security Data Science: Machine Learning Methods and their Performance on Imbalanced Datasets(https://arxiv.org/abs/2505.04204)
• Keywords: anomaly
• Abstract: Cybersecurity has become essential worldwide and at all levels, concerning individuals, institutions, and governments. A basic principle in cybersecurity is to be always alert. Therefore, automation is imperative in processes where the volume of daily operations is large. Several cybersecurity applications can be addressed as binary classification problems, including anomaly detection, fraud detection, intrusion detection, spam detection, or malware detection. We present three experiments. In the first experiment, we evaluate single classifiers including Random Forests, Light Gradient Boosting Machine, eXtreme Gradient Boosting, Logistic Regression, Decision Tree, and Gradient Boosting Decision Tree. In the second experiment, we test different sampling techniques including over-sampling, under-sampling, Synthetic Minority Over-sampling Technique, and Self-Paced Ensembling. In the last experiment, we evaluate Self-Paced Ensembling and its number of base classifiers. We found that imbalance learning techniques had positive and negative effects, as reported in related studies. Thus, these techniques should be applied with caution. Besides, we found different best performers for each dataset. Therefore, we recommend testing single classifiers and imbalance learning techniques for each new dataset and application involving imbalanced datasets as is the case in several cyber security applications.

Title: Technology prediction of a 3D model using Neural Network

• Authors: Grzegorz Miebs, Rafał A. Bachorz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.04241
• Pdf URL: https://arxiv.org/pdf/2505.04241
• Copy Paste: [[2505.04241]] Technology prediction of a 3D model using Neural Network(https://arxiv.org/abs/2505.04241)
• Keywords: generative
• Abstract: Accurate estimation of production times is critical for effective manufacturing scheduling, yet traditional methods relying on expert analysis or historical data often fall short in dynamic or customized production environments. This paper introduces a data-driven approach that predicts manufacturing steps and their durations directly from a product's 3D model. By rendering the model into multiple 2D images and leveraging a neural network inspired by the Generative Query Network, the method learns to map geometric features into time estimates for predefined production steps enabling scalable, adaptive, and precise process planning across varied product types.

Title: Bridging Geometry-Coherent Text-to-3D Generation with Multi-View Diffusion Priors and Gaussian Splatting

• Authors: Feng Yang, Wenliang Qian, Wangmeng Zuo, Hui Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.04262
• Pdf URL: https://arxiv.org/pdf/2505.04262
• Copy Paste: [[2505.04262]] Bridging Geometry-Coherent Text-to-3D Generation with Multi-View Diffusion Priors and Gaussian Splatting(https://arxiv.org/abs/2505.04262)
• Keywords: diffusion
• Abstract: Score Distillation Sampling (SDS) leverages pretrained 2D diffusion models to advance text-to-3D generation but neglects multi-view correlations, being prone to geometric inconsistencies and multi-face artifacts in the generated 3D content. In this work, we propose Coupled Score Distillation (CSD), a framework that couples multi-view joint distribution priors to ensure geometrically consistent 3D generation while enabling the stable and direct optimization of 3D Gaussian Splatting. Specifically, by reformulating the optimization as a multi-view joint optimization problem, we derive an effective optimization rule that effectively couples multi-view priors to guide optimization across different viewpoints while preserving the diversity of generated 3D assets. Additionally, we propose a framework that directly optimizes 3D Gaussian Splatting (3D-GS) with random initialization to generate geometrically consistent 3D content. We further employ a deformable tetrahedral grid, initialized from 3D-GS and refined through CSD, to produce high-quality, refined meshes. Quantitative and qualitative experimental results demonstrate the efficiency and competitive quality of our approach.

Title: Physics-Informed DeepONets for drift-diffusion on metric graphs: simulation and parameter identification

• Authors: Jan Blechschmidt, Tom-Christian Riemer, Max Winkler, Martin Stoll, Jan-F. Pietschmann
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2505.04263
• Pdf URL: https://arxiv.org/pdf/2505.04263
• Copy Paste: [[2505.04263]] Physics-Informed DeepONets for drift-diffusion on metric graphs: simulation and parameter identification(https://arxiv.org/abs/2505.04263)
• Keywords: diffusion
• Abstract: We develop a novel physics informed deep learning approach for solving nonlinear drift-diffusion equations on metric graphs. These models represent an important model class with a large number of applications in areas ranging from transport in biological cells to the motion of human crowds. While traditional numerical schemes require a large amount of tailoring, especially in the case of model design or parameter identification problems, physics informed deep operator networks (DeepONet) have emerged as a versatile tool for the solution of partial differential equations with the particular advantage that they easily incorporate parameter identification questions. We here present an approach where we first learn three DeepONet models for representative inflow, inner and outflow edges, resp., and then subsequently couple these models for the solution of the drift-diffusion metric graph problem by relying on an edge-based domain decomposition approach. We illustrate that our framework is applicable for the accurate evaluation of graph-coupled physics models and is well suited for solving optimization or inverse problems on these coupled networks.

Title: HDiffTG: A Lightweight Hybrid Diffusion-Transformer-GCN Architecture for 3D Human Pose Estimation

• Authors: Yajie Fu, Chaorui Huang, Junwei Li, Hui Kong, Yibin Tian, Huakang Li, Zhiyuan Zhang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2505.04276
• Pdf URL: https://arxiv.org/pdf/2505.04276
• Copy Paste: [[2505.04276]] HDiffTG: A Lightweight Hybrid Diffusion-Transformer-GCN Architecture for 3D Human Pose Estimation(https://arxiv.org/abs/2505.04276)
• Keywords: diffusion
• Abstract: We propose HDiffTG, a novel 3D Human Pose Estimation (3DHPE) method that integrates Transformer, Graph Convolutional Network (GCN), and diffusion model into a unified framework. HDiffTG leverages the strengths of these techniques to significantly improve pose estimation accuracy and robustness while maintaining a lightweight design. The Transformer captures global spatiotemporal dependencies, the GCN models local skeletal structures, and the diffusion model provides step-by-step optimization for fine-tuning, achieving a complementary balance between global and local features. This integration enhances the model's ability to handle pose estimation under occlusions and in complex scenarios. Furthermore, we introduce lightweight optimizations to the integrated model and refine the objective function design to reduce computational overhead without compromising performance. Evaluation results on the Human3.6M and MPI-INF-3DHP datasets demonstrate that HDiffTG achieves state-of-the-art (SOTA) performance on the MPI-INF-3DHP dataset while excelling in both accuracy and computational efficiency. Additionally, the model exhibits exceptional robustness in noisy and occluded environments. Source codes and models are available at this https URL

Title: Non-stationary Diffusion For Probabilistic Time Series Forecasting

• Authors: Weiwei Ye, Zhuopeng Xu, Ning Gui
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.04278
• Pdf URL: https://arxiv.org/pdf/2505.04278
• Copy Paste: [[2505.04278]] Non-stationary Diffusion For Probabilistic Time Series Forecasting(https://arxiv.org/abs/2505.04278)
• Keywords: diffusion, generative
• Abstract: Due to the dynamics of underlying physics and external influences, the uncertainty of time series often varies over time. However, existing Denoising Diffusion Probabilistic Models (DDPMs) often fail to capture this non-stationary nature, constrained by their constant variance assumption from the additive noise model (ANM). In this paper, we innovatively utilize the Location-Scale Noise Model (LSNM) to relax the fixed uncertainty assumption of ANM. A diffusion-based probabilistic forecasting framework, termed Non-stationary Diffusion (NsDiff), is designed based on LSNM that is capable of modeling the changing pattern of uncertainty. Specifically, NsDiff combines a denoising diffusion-based conditional generative model with a pre-trained conditional mean and variance estimator, enabling adaptive endpoint distribution modeling. Furthermore, we propose an uncertainty-aware noise schedule, which dynamically adjusts the noise levels to accurately reflect the data uncertainty at each step and integrates the time-varying variances into the diffusion process. Extensive experiments conducted on nine real-world and synthetic datasets demonstrate the superior performance of NsDiff compared to existing approaches. Code is available at this https URL.

Title: TS-Diff: Two-Stage Diffusion Model for Low-Light RAW Image Enhancement

• Authors: Yi Li, Zhiyuan Zhang, Jiangnan Xia, Jianghan Cheng, Qilong Wu, Junwei Li, Yibin Tian, Hui Kong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.04281
• Pdf URL: https://arxiv.org/pdf/2505.04281
• Copy Paste: [[2505.04281]] TS-Diff: Two-Stage Diffusion Model for Low-Light RAW Image Enhancement(https://arxiv.org/abs/2505.04281)
• Keywords: diffusion
• Abstract: This paper presents a novel Two-Stage Diffusion Model (TS-Diff) for enhancing extremely low-light RAW images. In the pre-training stage, TS-Diff synthesizes noisy images by constructing multiple virtual cameras based on a noise space. Camera Feature Integration (CFI) modules are then designed to enable the model to learn generalizable features across diverse virtual cameras. During the aligning stage, CFIs are averaged to create a target-specific CFI$^T$, which is fine-tuned using a small amount of real RAW data to adapt to the noise characteristics of specific cameras. A structural reparameterization technique further simplifies CFI$^T$ for efficient deployment. To address color shifts during the diffusion process, a color corrector is introduced to ensure color consistency by dynamically adjusting global color distributions. Additionally, a novel dataset, QID, is constructed, featuring quantifiable illumination levels and a wide dynamic range, providing a comprehensive benchmark for training and evaluation under extreme low-light conditions. Experimental results demonstrate that TS-Diff achieves state-of-the-art performance on multiple datasets, including QID, SID, and ELD, excelling in denoising, generalization, and color consistency across various cameras and illumination levels. These findings highlight the robustness and versatility of TS-Diff, making it a practical solution for low-light imaging applications. Source codes and models are available at this https URL

Title: MoDE: Mixture of Diffusion Experts for Any Occluded Face Recognition

• Authors: Qiannan Fan, Zhuoyang Li, Jitong Li, Chenyang Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.04306
• Pdf URL: https://arxiv.org/pdf/2505.04306
• Copy Paste: [[2505.04306]] MoDE: Mixture of Diffusion Experts for Any Occluded Face Recognition(https://arxiv.org/abs/2505.04306)
• Keywords: diffusion, generative
• Abstract: With the continuous impact of epidemics, people have become accustomed to wearing masks. However, most current occluded face recognition (OFR) algorithms lack prior knowledge of occlusions, resulting in poor performance when dealing with occluded faces of varying types and severity in reality. Recognizing occluded faces is still a significant challenge, which greatly affects the convenience of people's daily lives. In this paper, we propose an identity-gated mixture of diffusion experts (MoDE) for OFR. Each diffusion-based generative expert estimates one possible complete image for occluded faces. Considering the random sampling process of the diffusion model, which introduces inevitable differences and variations between the inpainted faces and the real ones. To ensemble effective information from multi-reconstructed faces, we introduce an identity-gating network to evaluate the contribution of each reconstructed face to the identity and adaptively integrate the predictions in the decision space. Moreover, our MoDE is a plug-and-play module for most existing face recognition models. Extensive experiments on three public face datasets and two datasets in the wild validate our advanced performance for various occlusions in comparison with the competing methods.

Title: Riemannian Denoising Diffusion Probabilistic Models

• Authors: Zichen Liu, Wei Zhang, Christof Schütte, Tiejun Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.04338
• Pdf URL: https://arxiv.org/pdf/2505.04338
• Copy Paste: [[2505.04338]] Riemannian Denoising Diffusion Probabilistic Models(https://arxiv.org/abs/2505.04338)
• Keywords: diffusion, generative
• Abstract: We propose Riemannian Denoising Diffusion Probabilistic Models (RDDPMs) for learning distributions on submanifolds of Euclidean space that are level sets of functions, including most of the manifolds relevant to applications. Existing methods for generative modeling on manifolds rely on substantial geometric information such as geodesic curves or eigenfunctions of the Laplace-Beltrami operator and, as a result, they are limited to manifolds where such information is available. In contrast, our method, built on a projection scheme, can be applied to more general manifolds, as it only requires being able to evaluate the value and the first order derivatives of the function that defines the submanifold. We provide a theoretical analysis of our method in the continuous-time limit, which elucidates the connection between our RDDPMs and score-based generative models on manifolds. The capability of our method is demonstrated on datasets from previous studies and on new datasets sampled from two high-dimensional manifolds, i.e. $\mathrm{SO}(10)$ and the configuration space of molecular system alanine dipeptide with fixed dihedral angle.

Title: CountDiffusion: Text-to-Image Synthesis with Training-Free Counting-Guidance Diffusion

• Authors: Yanyu Li, Pencheng Wan, Liang Han, Yaowei Wang, Liqiang Nie, Min Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.04347
• Pdf URL: https://arxiv.org/pdf/2505.04347
• Copy Paste: [[2505.04347]] CountDiffusion: Text-to-Image Synthesis with Training-Free Counting-Guidance Diffusion(https://arxiv.org/abs/2505.04347)
• Keywords: diffusion
• Abstract: Stable Diffusion has advanced text-to-image synthesis, but training models to generate images with accurate object quantity is still difficult due to the high computational cost and the challenge of teaching models the abstract concept of quantity. In this paper, we propose CountDiffusion, a training-free framework aiming at generating images with correct object quantity from textual descriptions. CountDiffusion consists of two stages. In the first stage, an intermediate denoising result is generated by the diffusion model to predict the final synthesized image with one-step denoising, and a counting model is used to count the number of objects in this image. In the second stage, a correction module is used to correct the object quantity by changing the attention map of the object with universal guidance. The proposed CountDiffusion can be plugged into any diffusion-based text-to-image (T2I) generation models without further training. Experiment results demonstrate the superiority of our proposed CountDiffusion, which improves the accurate object quantity generation ability of T2I models by a large margin.

Title: Predicting Road Surface Anomalies by Visual Tracking of a Preceding Vehicle

• Authors: Petr Jahoda, Jan Cech
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.04392
• Pdf URL: https://arxiv.org/pdf/2505.04392
• Copy Paste: [[2505.04392]] Predicting Road Surface Anomalies by Visual Tracking of a Preceding Vehicle(https://arxiv.org/abs/2505.04392)
• Keywords: anomaly
• Abstract: A novel approach to detect road surface anomalies by visual tracking of a preceding vehicle is proposed. The method is versatile, predicting any kind of road anomalies, such as potholes, bumps, debris, etc., unlike direct observation methods that rely on training visual detectors of those cases. The method operates in low visibility conditions or in dense traffic where the anomaly is occluded by a preceding vehicle. Anomalies are detected predictively, i.e., before a vehicle encounters them, which allows to pre-configure low-level vehicle systems (such as chassis) or to plan an avoidance maneuver in case of autonomous driving. A challenge is that the signal coming from camera-based tracking of a preceding vehicle may be weak and disturbed by camera ego motion due to vibrations affecting the ego vehicle. Therefore, we propose an efficient method to compensate camera pitch rotation by an iterative robust estimator. Our experiments on both controlled setup and normal traffic conditions show that road anomalies can be detected reliably at a distance even in challenging cases where the ego vehicle traverses imperfect road surfaces. The method is effective and performs in real time on standard consumer hardware.

Title: DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception

• Authors: Junjie Wang, Bin Chen, Yulin Li, Bin Kang, Yichi Chen, Zhuotao Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.04410
• Pdf URL: https://arxiv.org/pdf/2505.04410
• Copy Paste: [[2505.04410]] DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception(https://arxiv.org/abs/2505.04410)
• Keywords: foundation model
• Abstract: Dense visual prediction tasks have been constrained by their reliance on predefined categories, limiting their applicability in real-world scenarios where visual concepts are unbounded. While Vision-Language Models (VLMs) like CLIP have shown promise in open-vocabulary tasks, their direct application to dense prediction often leads to suboptimal performance due to limitations in local feature representation. In this work, we present our observation that CLIP's image tokens struggle to effectively aggregate information from spatially or semantically related regions, resulting in features that lack local discriminability and spatial consistency. To address this issue, we propose DeCLIP, a novel framework that enhances CLIP by decoupling the self-attention module to obtain ``content'' and ``context'' features respectively. The ``content'' features are aligned with image crop representations to improve local discriminability, while ``context'' features learn to retain the spatial correlations under the guidance of vision foundation models, such as DINO. Extensive experiments demonstrate that DeCLIP significantly outperforms existing methods across multiple open-vocabulary dense prediction tasks, including object detection and semantic segmentation. Code is available at \textcolor{magenta}{this https URL}.

Title: Localized Diffusion Models for High Dimensional Distributions Generation

• Authors: Georg A. Gottwald, Shuigen Liu, Youssef Marzouk, Sebastian Reich, Xin T. Tong
• Subjects: cs.LG, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2505.04417
• Pdf URL: https://arxiv.org/pdf/2505.04417
• Copy Paste: [[2505.04417]] Localized Diffusion Models for High Dimensional Distributions Generation(https://arxiv.org/abs/2505.04417)
• Keywords: diffusion, generative
• Abstract: Diffusion models are the state-of-the-art tools for various generative tasks. However, estimating high-dimensional score functions makes them potentially suffer from the curse of dimensionality (CoD). This underscores the importance of better understanding and exploiting low-dimensional structure in the target distribution. In this work, we consider locality structure, which describes sparse dependencies between model components. Under locality structure, the score function is effectively low-dimensional, so that it can be estimated by a localized neural network with significantly reduced sample complexity. This motivates the localized diffusion model, where a localized score matching loss is used to train the score function within a localized hypothesis space. We prove that such localization enables diffusion models to circumvent CoD, at the price of additional localization error. Under realistic sample size scaling, we show both theoretically and numerically that a moderate localization radius can balance the statistical and localization error, leading to a better overall performance. The localized structure also facilitates parallel training of diffusion models, making it potentially more efficient for large-scale applications.

Title: CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation

• Authors: Jiahao Li, Weijian Ma, Xueyang Li, Yunzhong Lou, Guichun Zhou, Xiangdong Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.04481
• Pdf URL: https://arxiv.org/pdf/2505.04481
• Copy Paste: [[2505.04481]] CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation(https://arxiv.org/abs/2505.04481)
• Keywords: generative
• Abstract: Recently, Large Language Models (LLMs) have achieved significant success, prompting increased interest in expanding their generative capabilities beyond general text into domain-specific areas. This study investigates the generation of parametric sequences for computer-aided design (CAD) models using LLMs. This endeavor represents an initial step towards creating parametric 3D shapes with LLMs, as CAD model parameters directly correlate with shapes in three-dimensional space. Despite the formidable generative capacities of LLMs, this task remains challenging, as these models neither encounter parametric sequences during their pretraining phase nor possess direct awareness of 3D structures. To address this, we present CAD-Llama, a framework designed to enhance pretrained LLMs for generating parametric 3D CAD models. Specifically, we develop a hierarchical annotation pipeline and a code-like format to translate parametric 3D CAD command sequences into Structured Parametric CAD Code (SPCC), incorporating hierarchical semantic descriptions. Furthermore, we propose an adaptive pretraining approach utilizing SPCC, followed by an instruction tuning process aligned with CAD-specific guidelines. This methodology aims to equip LLMs with the spatial knowledge inherent in parametric sequences. Experimental results demonstrate that our framework significantly outperforms prior autoregressive methods and existing LLM baselines.

Title: Efficient Flow Matching using Latent Variables

• Authors: Anirban Samaddar, Yixuan Sun, Viktor Nilsson, Sandeep Madireddy
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.04486
• Pdf URL: https://arxiv.org/pdf/2505.04486
• Copy Paste: [[2505.04486]] Efficient Flow Matching using Latent Variables(https://arxiv.org/abs/2505.04486)
• Keywords: diffusion, generative
• Abstract: Flow matching models have shown great potential in image generation tasks among probabilistic generative models. Building upon the ideas of continuous normalizing flows, flow matching models generalize the transport path of the diffusion models from a simple prior distribution to the data. Most flow matching models in the literature do not explicitly model the underlying structure/manifold in the target data when learning the flow from a simple source distribution like the standard Gaussian. This leads to inefficient learning, especially for many high-dimensional real-world datasets, which often reside in a low-dimensional manifold. Existing strategies of incorporating manifolds, including data with underlying multi-modal distribution, often require expensive training and hence frequently lead to suboptimal performance. To this end, we present \texttt{Latent-CFM}, which provides simplified training/inference strategies to incorporate multi-modal data structures using pretrained deep latent variable models. Through experiments on multi-modal synthetic data and widely used image benchmark datasets, we show that \texttt{Latent-CFM} exhibits improved generation quality with significantly less training ($\sim 50\%$ less in some cases) and computation than state-of-the-art flow matching models. Using a 2d Darcy flow dataset, we demonstrate that our approach generates more physically accurate samples than competitive approaches. In addition, through latent space analysis, we demonstrate that our approach can be used for conditional image generation conditioned on latent features.

Title: Defining and Quantifying Creative Behavior in Popular Image Generators

• Authors: Aditi Ramaswamy
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.04497
• Pdf URL: https://arxiv.org/pdf/2505.04497
• Copy Paste: [[2505.04497]] Defining and Quantifying Creative Behavior in Popular Image Generators(https://arxiv.org/abs/2505.04497)
• Keywords: generative
• Abstract: Creativity of generative AI models has been a subject of scientific debate in the last years, without a conclusive answer. In this paper, we study creativity from a practical perspective and introduce quantitative measures that help the user to choose a suitable AI model for a given task. We evaluated our measures on a number of popular image-to-image generation models, and the results of this suggest that our measures conform to human intuition.

Title: Detecting Spelling and Grammatical Anomalies in Russian Poetry Texts

• Authors: Ilya Koziev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.04507
• Pdf URL: https://arxiv.org/pdf/2505.04507
• Copy Paste: [[2505.04507]] Detecting Spelling and Grammatical Anomalies in Russian Poetry Texts(https://arxiv.org/abs/2505.04507)
• Keywords: generative, anomaly
• Abstract: The quality of natural language texts in fine-tuning datasets plays a critical role in the performance of generative models, particularly in computational creativity tasks such as poem or song lyric generation. Fluency defects in generated poems significantly reduce their value. However, training texts are often sourced from internet-based platforms without stringent quality control, posing a challenge for data engineers to manage defect levels effectively. To address this issue, we propose the use of automated linguistic anomaly detection to identify and filter out low-quality texts from training datasets for creative models. In this paper, we present a comprehensive comparison of unsupervised and supervised text anomaly detection approaches, utilizing both synthetic and human-labeled datasets. We also introduce the RUPOR dataset, a collection of Russian-language human-labeled poems designed for cross-sentence grammatical error detection, and provide the full evaluation code. Our work aims to empower the community with tools and insights to improve the quality of training datasets for generative models in creative domains.

Title: Text2CT: Towards 3D CT Volume Generation from Free-text Descriptions Using Diffusion Model

• Authors: Pengfei Guo, Can Zhao, Dong Yang, Yufan He, Vishwesh Nath, Ziyue Xu, Pedro R. A. S. Bassi, Zongwei Zhou, Benjamin D. Simon, Stephanie Anne Harmon, Baris Turkbey, Daguang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.04522
• Pdf URL: https://arxiv.org/pdf/2505.04522
• Copy Paste: [[2505.04522]] Text2CT: Towards 3D CT Volume Generation from Free-text Descriptions Using Diffusion Model(https://arxiv.org/abs/2505.04522)
• Keywords: diffusion
• Abstract: Generating 3D CT volumes from descriptive free-text inputs presents a transformative opportunity in diagnostics and research. In this paper, we introduce Text2CT, a novel approach for synthesizing 3D CT volumes from textual descriptions using the diffusion model. Unlike previous methods that rely on fixed-format text input, Text2CT employs a novel prompt formulation that enables generation from diverse, free-text descriptions. The proposed framework encodes medical text into latent representations and decodes them into high-resolution 3D CT scans, effectively bridging the gap between semantic text inputs and detailed volumetric representations in a unified 3D framework. Our method demonstrates superior performance in preserving anatomical fidelity and capturing intricate structures as described in the input text. Extensive evaluations show that our approach achieves state-of-the-art results, offering promising potential applications in diagnostics, and data augmentation.

Title: Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review

• Authors: Josh McGiff, Nikola S. Nikolov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.04531
• Pdf URL: https://arxiv.org/pdf/2505.04531
• Copy Paste: [[2505.04531]] Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review(https://arxiv.org/abs/2505.04531)
• Keywords: generative
• Abstract: Generative language modelling has surged in popularity with the emergence of services such as ChatGPT and Google Gemini. While these models have demonstrated transformative potential in productivity and communication, they overwhelmingly cater to high-resource languages like English. This has amplified concerns over linguistic inequality in natural language processing (NLP). This paper presents the first systematic review focused specifically on strategies to address data scarcity in generative language modelling for low-resource languages (LRL). Drawing from 54 studies, we identify, categorise and evaluate technical approaches, including monolingual data augmentation, back-translation, multilingual training, and prompt engineering, across generative tasks. We also analyse trends in architecture choices, language family representation, and evaluation methods. Our findings highlight a strong reliance on transformer-based models, a concentration on a small subset of LRLs, and a lack of consistent evaluation across studies. We conclude with recommendations for extending these methods to a wider range of LRLs and outline open challenges in building equitable generative language systems. Ultimately, this review aims to support researchers and developers in building inclusive AI tools for underrepresented languages, a necessary step toward empowering LRL speakers and the preservation of linguistic diversity in a world increasingly shaped by large-scale language technologies.

Title: OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning

• Authors: Xianhang Li, Yanqing Liu, Haoqin Tu, Hongru Zhu, Cihang Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.04601
• Pdf URL: https://arxiv.org/pdf/2505.04601
• Copy Paste: [[2505.04601]] OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning(https://arxiv.org/abs/2505.04601)
• Keywords: foundation model
• Abstract: OpenAI's CLIP, released in early 2021, have long been the go-to choice of vision encoder for building multimodal foundation models. Although recent alternatives such as SigLIP have begun to challenge this status quo, to our knowledge none are fully open: their training data remains proprietary and/or their training recipes are not released. This paper fills this gap with OpenVision, a fully-open, cost-effective family of vision encoders that match or surpass the performance of OpenAI's CLIP when integrated into multimodal frameworks like LLaVA. OpenVision builds on existing works -- e.g., CLIPS for training framework and Recap-DataComp-1B for training data -- while revealing multiple key insights in enhancing encoder quality and showcasing practical benefits in advancing multimodal models. By releasing vision encoders spanning from 5.9M to 632.1M parameters, OpenVision offers practitioners a flexible trade-off between capacity and efficiency in building multimodal models: larger models deliver enhanced multimodal performance, while smaller versions enable lightweight, edge-ready multimodal deployments.

Title: On Path to Multimodal Generalist: General-Level and General-Bench

• Authors: Hao Fei, Yuan Zhou, Juncheng Li, Xiangtai Li, Qingshan Xu, Bobo Li, Shengqiong Wu, Yaoting Wang, Junbao Zhou, Jiahao Meng, Qingyu Shi, Zhiyuan Zhou, Liangtao Shi, Minghe Gao, Daoan Zhang, Zhiqi Ge, Weiming Wu, Siliang Tang, Kaihang Pan, Yaobo Ye, Haobo Yuan, Tao Zhang, Tianjie Ju, Zixiang Meng, Shilin Xu, Liyu Jia, Wentao Hu, Meng Luo, Jiebo Luo, Tat-Seng Chua, Shuicheng Yan, Hanwang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.04620
• Pdf URL: https://arxiv.org/pdf/2505.04620
• Copy Paste: [[2505.04620]] On Path to Multimodal Generalist: General-Level and General-Bench(https://arxiv.org/abs/2505.04620)
• Keywords: foundation model
• Abstract: The Multimodal Large Language Model (MLLM) is currently experiencing rapid growth, driven by the advanced capabilities of LLMs. Unlike earlier specialists, existing MLLMs are evolving towards a Multimodal Generalist paradigm. Initially limited to understanding multiple modalities, these models have advanced to not only comprehend but also generate across modalities. Their capabilities have expanded from coarse-grained to fine-grained multimodal understanding and from supporting limited modalities to arbitrary ones. While many benchmarks exist to assess MLLMs, a critical question arises: Can we simply assume that higher performance across tasks indicates a stronger MLLM capability, bringing us closer to human-level AI? We argue that the answer is not as straightforward as it seems. This project introduces General-Level, an evaluation framework that defines 5-scale levels of MLLM performance and generality, offering a methodology to compare MLLMs and gauge the progress of existing systems towards more robust multimodal generalists and, ultimately, towards AGI. At the core of the framework is the concept of Synergy, which measures whether models maintain consistent capabilities across comprehension and generation, and across multiple modalities. To support this evaluation, we present General-Bench, which encompasses a broader spectrum of skills, modalities, formats, and capabilities, including over 700 tasks and 325,800 instances. The evaluation results that involve over 100 existing state-of-the-art MLLMs uncover the capability rankings of generalists, highlighting the challenges in reaching genuine AI. We expect this project to pave the way for future research on next-generation multimodal foundation models, providing a robust infrastructure to accelerate the realization of AGI. Project page: this https URL