2025-05-01

Title: Model Connectomes: A Generational Approach to Data-Efficient Language Models

• Authors: Klemen Kotar, Greta Tuckute
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21047
• Pdf URL: https://arxiv.org/pdf/2504.21047
• Copy Paste: [[2504.21047]] Model Connectomes: A Generational Approach to Data-Efficient Language Models(https://arxiv.org/abs/2504.21047)
• Keywords: generation
• Abstract: Biological neural networks are shaped both by evolution across generations and by individual learning within an organism's lifetime, whereas standard artificial neural networks undergo a single, large training procedure without inherited constraints. In this preliminary work, we propose a framework that incorporates this crucial generational dimension - an "outer loop" of evolution that shapes the "inner loop" of learning - so that artificial networks better mirror the effects of evolution and individual learning in biological organisms. Focusing on language, we train a model that inherits a "model connectome" from the outer evolution loop before exposing it to a developmental-scale corpus of 100M tokens. Compared with two closely matched control models, we show that the connectome model performs better or on par on natural language processing tasks as well as alignment to human behavior and brain data. These findings suggest that a model connectome serves as an efficient prior for learning in low-data regimes - narrowing the gap between single-generation artificial models and biologically evolved neural networks.
• 摘要：生物神经网络既通过一代人的进化和生物体中的个人学习来塑造，而标准的人工神经网络则经历了单一的大型训练程序，而没有遗传的约束。在这项初步工作中，我们提出了一个框架，该框架结合了这种关键的世代维度 - 进化的“外部环”，它塑造了学习的“内部环” - 以便人造网络更好地反映了生物生物体进化和个体学习的影响。专注于语言，我们训练一个模型，该模型在将其暴露于100m代币的发育规模语料库之前，从外部进化环中继承了“模型connectome”。与两个紧密匹配的对照模型相比，我们表明Connectome模型在自然语言处理任务以及对人类行为和大脑数据的一致性方面的表现更好或相当。这些发现表明，模型Connectome是在低数据制度中学习的有效先验 - 缩小了单一代人工模型与生物学演化的神经网络之间的差距。

Title: A 3D pocket-aware and affinity-guided diffusion model for lead optimization

• Authors: Anjie Qiao, Junjie Xie, Weifeng Huang, Hao Zhang, Jiahua Rao, Shuangjia Zheng, Yuedong Yang, Zhen Wang, Guo-Bo Li, Jinping Lei
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21065
• Pdf URL: https://arxiv.org/pdf/2504.21065
• Copy Paste: [[2504.21065]] A 3D pocket-aware and affinity-guided diffusion model for lead optimization(https://arxiv.org/abs/2504.21065)
• Keywords: generation, generative
• Abstract: Molecular optimization, aimed at improving binding affinity or other molecular properties, is a crucial task in drug discovery that often relies on the expertise of medicinal chemists. Recently, deep learning-based 3D generative models showed promise in enhancing the efficiency of molecular optimization. However, these models often struggle to adequately consider binding affinities with protein targets during lead optimization. Herein, we propose a 3D pocket-aware and affinity-guided diffusion model, named Diffleop, to optimize molecules with enhanced binding affinity. The model explicitly incorporates the knowledge of protein-ligand binding affinity to guide the denoising sampling for molecule generation with high affinity. The comprehensive evaluations indicated that Diffleop outperforms baseline models across multiple metrics, especially in terms of binding affinity.
• 摘要：旨在改善结合亲和力或其他分子特性的分子优化是药物发现的至关重要的任务，通常依赖于药物化学家的专业知识。最近，基于深度学习的3D生成模型在提高分子优化的效率方面表现出了希望。但是，在铅优化过程中，这些模型通常很难考虑与蛋白质靶标的结合亲和力。本文中，我们提出了一个名为Diffleop的3D袖珍感知和亲和力引导的扩散模型，以优化具有增强结合亲和力的分子。该模型明确地结合了蛋白质 - 配体结合亲和力的知识，以指导具有高亲和力的分子产生的降解采样。综合评估表明，差异型的表现优于多个指标的基线模型，尤其是在结合亲和力方面。

Title: Artificial Intelligence for Personalized Prediction of Alzheimer's Disease Progression: A Survey of Methods, Data Challenges, and Future Directions

• Authors: Gulsah Hancerliogullari Koksalmis, Bulent Soykan, Laura J. Brattain, Hsin-Hsiung Huang
• Subjects: cs.LG, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2504.21189
• Pdf URL: https://arxiv.org/pdf/2504.21189
• Copy Paste: [[2504.21189]] Artificial Intelligence for Personalized Prediction of Alzheimer's Disease Progression: A Survey of Methods, Data Challenges, and Future Directions(https://arxiv.org/abs/2504.21189)
• Keywords: generation, generative
• Abstract: Alzheimer's Disease (AD) is marked by significant inter-individual variability in its progression, complicating accurate prognosis and personalized care planning. This heterogeneity underscores the critical need for predictive models capable of forecasting patient-specific disease trajectories. Artificial Intelligence (AI) offers powerful tools to address this challenge by analyzing complex, multi-modal, and longitudinal patient data. This paper provides a comprehensive survey of AI methodologies applied to personalized AD progression prediction. We review key approaches including state-space models for capturing temporal dynamics, deep learning techniques like Recurrent Neural Networks for sequence modeling, Graph Neural Networks (GNNs) for leveraging network structures, and the emerging concept of AI-driven digital twins for individualized simulation. Recognizing that data limitations often impede progress, we examine common challenges such as high dimensionality, missing data, and dataset imbalance. We further discuss AI-driven mitigation strategies, with a specific focus on synthetic data generation using Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) to augment and balance datasets. The survey synthesizes the strengths and limitations of current approaches, emphasizing the trend towards multimodal integration and the persistent need for model interpretability and generalizability. Finally, we identify critical open challenges, including robust external validation, clinical integration, and ethical considerations, and outline promising future research directions such as hybrid models, causal inference, and federated learning. This review aims to consolidate current knowledge and guide future efforts in developing clinically relevant AI tools for personalized AD prognostication.
• 摘要：阿尔茨海默氏病（AD）的特征是其进展方面的显着性差异，使准确的预后和个性化护理计划变得复杂。这种异质性强调了能够预测患者特异性疾病轨迹的预测模型的关键需求。人工智能（AI）提供了强大的工具来通过分析复杂，多模式和纵向患者数据来应对这一挑战。本文对应用于个性化AD进程预测的AI方法进行了全面的调查。我们审查关键方法，包括用于捕获时间动力学的状态空间模型，诸如用于序列建模的复发神经网络之类的深度学习技术，用于利用网络结构的图形神经网络（GNN）以及用于个体化模拟的AI-Digital Twins的新兴数字概念。认识到数据限制通常会阻碍进度，我们研究了常见的挑战，例如高维度，丢失的数据和数据集不平衡。我们进一步讨论了AI驱动的缓解策略，特别关注使用变异自动编码器（VAE）和生成对抗网络（GAN）来生成合成数据，以增强和平衡数据集。该调查综合了当前方法的优势和局限性，强调了多模式整合的趋势以及对模型可解释性和概括性的持续需求。最后，我们确定了关键的公开挑战，包括强大的外部验证，临床整合和道德考虑，并概述了有希望的未来研究方向，例如混合模型，因果推理和联合学习。这篇评论旨在巩固当前的知识，并指导未来的努力，以开发临床相关的AI工具进行个性化的AD预测。

Title: Graph Synthetic Out-of-Distribution Exposure with Large Language Models

• Authors: Haoyan Xu, Zhengtao Yao, Ziyi Wang, Zhan Cheng, Xiyang Hu, Mengyuan Li, Yue Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.21198
• Pdf URL: https://arxiv.org/pdf/2504.21198
• Copy Paste: [[2504.21198]] Graph Synthetic Out-of-Distribution Exposure with Large Language Models(https://arxiv.org/abs/2504.21198)
• Keywords: generation
• Abstract: Out-of-distribution (OOD) detection in graphs is critical for ensuring model robustness in open-world and safety-sensitive applications. Existing approaches to graph OOD detection typically involve training an in-distribution (ID) classifier using only ID data, followed by the application of post-hoc OOD scoring techniques. Although OOD exposure - introducing auxiliary OOD samples during training - has proven to be an effective strategy for enhancing detection performance, current methods in the graph domain generally assume access to a set of real OOD nodes. This assumption, however, is often impractical due to the difficulty and cost of acquiring representative OOD samples. In this paper, we introduce GOE-LLM, a novel framework that leverages Large Language Models (LLMs) for OOD exposure in graph OOD detection without requiring real OOD nodes. GOE-LLM introduces two pipelines: (1) identifying pseudo-OOD nodes from the initially unlabeled graph using zero-shot LLM annotations, and (2) generating semantically informative synthetic OOD nodes via LLM-prompted text generation. These pseudo-OOD nodes are then used to regularize the training of the ID classifier for improved OOD awareness. We evaluate our approach across multiple benchmark datasets, showing that GOE-LLM significantly outperforms state-of-the-art graph OOD detection methods that do not use OOD exposure and achieves comparable performance to those relying on real OOD data.
• 摘要：图表中的分布（OOD）检测对于确保开放世界和安全敏感应用中的模型鲁棒性至关重要。现有的图形OOD检测方法通常涉及仅使用ID数据培训分布（ID）分类器，然后应用事后OOD评分技术。尽管事实证明，在训练期间引入辅助OOD样品的OOD暴露是提高检测性能的有效策略，但图域中的当前方法通常假设访问一组实际的OOD节点。但是，由于获得代表性OOD样本的困难和成本，这种假设通常是不切实际的。在本文中，我们介绍了Goe-Llm，这是一个新型框架，该框架利用大型语言模型（LLMS）在图中检测中的OOD暴露，而无需真正的OOD节点。 GOE-LLM介绍了两个管道：（1）使用零摄像机LLM注释从最初未标记的图中识别伪-OON节点，以及（2）通过LLM-prompted文本生成生成语义上有用的合成OOD节点。然后，这些伪-OOD节点用于将ID分类器的培训正规化，以提高OOD意识。我们在多个基准数据集中评估了我们的方法，这表明GOE-LLM显着胜过不使用OOD曝光并与依赖实际OOD数据的人的最先进的OOD检测方法。

Title: Multi-Domain Causal Discovery in Bijective Causal Models

• Authors: Kasra Jalaldoust, Saber Salehkaleybar, Negar Kiyavash
• Subjects: cs.LG, cs.AI, stat.ME
• Abstract URL: https://arxiv.org/abs/2504.21261
• Pdf URL: https://arxiv.org/pdf/2504.21261
• Copy Paste: [[2504.21261]] Multi-Domain Causal Discovery in Bijective Causal Models(https://arxiv.org/abs/2504.21261)
• Keywords: generation
• Abstract: We consider the problem of causal discovery (a.k.a., causal structure learning) in a multi-domain setting. We assume that the causal functions are invariant across the domains, while the distribution of the exogenous noise may vary. Under causal sufficiency (i.e., no confounders exist), we show that the causal diagram can be discovered under less restrictive functional assumptions compared to previous work. What enables causal discovery in this setting is bijective generation mechanisms (BGM), which ensures that the functional relation between the exogenous noise $E$ and the endogenous variable $Y$ is bijective and differentiable in both directions at every level of the cause variable $X = x$. BGM generalizes a variety of models including additive noise model, LiNGAM, post-nonlinear model, and location-scale noise model. Further, we derive a statistical test to find the parents set of the target variable. Experiments on various synthetic and real-world datasets validate our theoretical findings.
• 摘要：我们考虑在多域环境中的因果发现（又称因果结构学习）的问题。我们假设因果函数在整个域之间是不变的，而外源噪声的分布可能会有所不同。在因果充足的情况下（即不存在混杂因素），我们表明，与以前的工作相比，在限制性较低的功能假设下可以发现因果图。在这种情况下，可以使因果发现的因果发现的是肉类生成机制（BGM），它确保了外源性噪声$ e $与内源性变量$ y $之间的功能关系在CASION变量的各个级别上都是两种方向上的五个方向上的差异，$ x = x $。 BGM概括了各种模型，包括添加噪声模型，lingam，非线性后模型和位置尺度噪声模型。此外，我们得出了统计检验，以找到目标变量的父母集。各种合成和现实世界数据集的实验验证了我们的理论发现。

Title: Can We Achieve Efficient Diffusion without Self-Attention? Distilling Self-Attention into Convolutions

• Authors: ZiYi Dong, Chengxing Zhou, Weijian Deng, Pengxu Wei, Xiangyang Ji, Liang Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21292
• Pdf URL: https://arxiv.org/pdf/2504.21292
• Copy Paste: [[2504.21292]] Can We Achieve Efficient Diffusion without Self-Attention? Distilling Self-Attention into Convolutions(https://arxiv.org/abs/2504.21292)
• Keywords: generation, generative
• Abstract: Contemporary diffusion models built upon U-Net or Diffusion Transformer (DiT) architectures have revolutionized image generation through transformer-based attention mechanisms. The prevailing paradigm has commonly employed self-attention with quadratic computational complexity to handle global spatial relationships in complex images, thereby synthesizing high-fidelity images with coherent visual this http URL to conventional wisdom, our systematic layer-wise analysis reveals an interesting discrepancy: self-attention in pre-trained diffusion models predominantly exhibits localized attention patterns, closely resembling convolutional inductive biases. This suggests that global interactions in self-attention may be less critical than commonly this http URL by this, we propose \(\Delta\)ConvFusion to replace conventional self-attention modules with Pyramid Convolution Blocks (\(\Delta\)ConvBlocks).By distilling attention patterns into localized convolutional operations while keeping other components frozen, \(\Delta\)ConvFusion achieves performance comparable to transformer-based counterparts while reducing computational cost by 6929$\times$ and surpassing LinFusion by 5.42$\times$ in efficiency--all without compromising generative fidelity.
• 摘要：建立在U-NET或扩散变压器（DIT）架构上的当代扩散模型通过基于变压器的注意机制彻底改变了图像的产生。 The prevailing paradigm has commonly employed self-attention with quadratic computational complexity to handle global spatial relationships in complex images, thereby synthesizing high-fidelity images with coherent visual this http URL to conventional wisdom, our systematic layer-wise analysis reveals an interesting discrepancy: self-attention in pre-trained diffusion models predominantly exhibits localized attention patterns, closely resembling convolutional inductive偏见。这表明，自我注意事件中的全球互动可能不如这是HTTP URL的普遍互动，我们提出\（\ delta \）的说明，用金字塔卷积块（\（\ delta \ \）convercons替换常规的自我发入模块，以替代杂交的汇总，同时将其他散发性的组合组合在一起，同时又保持了汇总。 \(\Delta\)ConvFusion achieves performance comparable to transformer-based counterparts while reducing computational cost by 6929$\times$ and surpassing LinFusion by 5.42$\times$ in efficiency--all without compromising generative fidelity.

Title: Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teaming

• Authors: Nanxu Gong, Xinyuan Wang, Wangyang Ying, Haoyue Bai, Sixun Dong, Haifeng Chen, Yanjie Fu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.21304
• Pdf URL: https://arxiv.org/pdf/2504.21304
• Copy Paste: [[2504.21304]] Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teaming(https://arxiv.org/abs/2504.21304)
• Keywords: generation
• Abstract: Feature transformation involves generating a new set of features from the original dataset to enhance the data's utility. In certain domains like material performance screening, dimensionality is large and collecting labels is expensive and lengthy. It highly necessitates transforming feature spaces efficiently and without supervision to enhance data readiness and AI utility. However, existing methods fall short in efficient navigation of a vast space of feature combinations, and are mostly designed for supervised settings. To fill this gap, our unique perspective is to leverage a generator-critic duet-play teaming framework using LLM agents and in-context learning to derive pseudo-supervision from unsupervised data. The framework consists of three interconnected steps: (1) Critic agent diagnoses data to generate actionable advice, (2) Generator agent produces tokenized feature transformations guided by the critic's advice, and (3) Iterative refinement ensures continuous improvement through feedback between agents. The generator-critic framework can be generalized to human-agent collaborative generation, by replacing the critic agent with human experts. Extensive experiments demonstrate that the proposed framework outperforms even supervised baselines in feature transformation efficiency, robustness, and practical applicability across diverse datasets.
• 摘要：功能转换涉及从原始数据集生成一组新功能，以增强数据的实用程序。在某些领域，例如材料性能筛选，维度很大，收集标签昂贵且冗长。它高度需要有效地转换特征空间，没有监督来增强数据准备和AI实用性。但是，现有方法在有效的特征组合空间有效导航方面缺乏，并且主要是为监督设置而设计的。为了填补这一差距，我们独特的观点是利用LLM代理和内部文化学习来利用发电机 - 批判性二重奏组合框架，从无监督的数据中得出伪耶稣。该框架由三个相互联系的步骤组成：（1）评论家诊断数据以生成可行的建议，（2）发电机代理会产生以评论家建议指导的令牌化特征转换，以及（3）迭代改进可确保通过代理之间的反馈来确保持续改进。通过用人类专家代替批评者，可以将生成者批评的框架推广到人类代理的合作生成。广泛的实验表明，所提出的框架在特征转换效率，鲁棒性和各种数据集的实际适用性方面的表现甚至优于监督基线。

Title: AGHI-QA: A Subjective-Aligned Dataset and Metric for AI-Generated Human Images

• Authors: Yunhao Li, Sijing Wu, Wei Sun, Zhichao Zhang, Yucheng Zhu, Zicheng Zhang, Huiyu Duan, Xiongkuo Min, Guangtao Zhai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21308
• Pdf URL: https://arxiv.org/pdf/2504.21308
• Copy Paste: [[2504.21308]] AGHI-QA: A Subjective-Aligned Dataset and Metric for AI-Generated Human Images(https://arxiv.org/abs/2504.21308)
• Keywords: generation, quality assessment
• Abstract: The rapid development of text-to-image (T2I) generation approaches has attracted extensive interest in evaluating the quality of generated images, leading to the development of various quality assessment methods for general-purpose T2I outputs. However, existing image quality assessment (IQA) methods are limited to providing global quality scores, failing to deliver fine-grained perceptual evaluations for structurally complex subjects like humans, which is a critical challenge considering the frequent anatomical and textural distortions in AI-generated human images (AGHIs). To address this gap, we introduce AGHI-QA, the first large-scale benchmark specifically designed for quality assessment of AGHIs. The dataset comprises 4,000 images generated from 400 carefully crafted text prompts using 10 state of-the-art T2I models. We conduct a systematic subjective study to collect multidimensional annotations, including perceptual quality scores, text-image correspondence scores, visible and distorted body part labels. Based on AGHI-QA, we evaluate the strengths and weaknesses of current T2I methods in generating human images from multiple dimensions. Furthermore, we propose AGHI-Assessor, a novel quality metric that integrates the large multimodal model (LMM) with domain-specific human features for precise quality prediction and identification of visible and distorted body parts in AGHIs. Extensive experimental results demonstrate that AGHI-Assessor showcases state-of-the-art performance, significantly outperforming existing IQA methods in multidimensional quality assessment and surpassing leading LMMs in detecting structural distortions in AGHIs.
• 摘要：文本对图像（T2I）生成方法的快速发展引起了人们对评估生成图像质量的广泛兴趣，从而开发了通用T2I输出的各种质量评估方法。但是，现有的图像质量评估（IQA）方法仅限于提供全球质量评分，无法为像人类这样的结构复杂受试者提供精细的感知评估，考虑到AI生成的人类图像（AGHIS）的频繁解剖和质地扭曲，这是一个关键的挑战。为了解决这一差距，我们介绍了AGHI-QA，这是第一个专门针对Aghis质量评估的大型基准。该数据集包含400张使用10个T2I状态模型的精心制作的文本提示产生的图像。我们进行了一项系统的主观研究，以收集多维注释，包括感知质量得分，文本图像对应得分，可见和扭曲的身体部位标签。基于AGHI-QA，我们评估了当前T2i方法从多个维度产生人类图像的优势和缺点。此外，我们提出了Aghi-Assessor，这是一种新型的质量指标，将大型多模型（LMM）与特定领域的人类特征集成在一起，以精确的质量预测和鉴定Aghis中可见和扭曲的身体部位。广泛的实验结果表明，Aghi-Assessor展示了最先进的性能，在多维质量评估中的现有IQA方法显着超过了现有的IQA方法，并且超过了领先的LMM在检测AGHIS中的结构扭曲方面。

Title: Capturing Conditional Dependence via Auto-regressive Diffusion Models

• Authors: Xunpeng Huang, Yujin Han, Difan Zou, Yian Ma, Tong Zhang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2504.21314
• Pdf URL: https://arxiv.org/pdf/2504.21314
• Copy Paste: [[2504.21314]] Capturing Conditional Dependence via Auto-regressive Diffusion Models(https://arxiv.org/abs/2504.21314)
• Keywords: generation
• Abstract: Diffusion models have demonstrated appealing performance in both image and video generation. However, many works discover that they struggle to capture important, high-level relationships that are present in the real world. For example, they fail to learn physical laws from data, and even fail to understand that the objects in the world exist in a stable fashion. This is due to the fact that important conditional dependence structures are not adequately captured in the vanilla diffusion models. In this work, we initiate an in-depth study on strengthening the diffusion model to capture the conditional dependence structures in the data. In particular, we examine the efficacy of the auto-regressive (AR) diffusion models for such purpose and develop the first theoretical results on the sampling error of AR diffusion models under (possibly) the mildest data assumption. Our theoretical findings indicate that, compared with typical diffusion models, the AR variant produces samples with a reduced gap in approximating the data conditional distribution. On the other hand, the overall inference time of the AR-diffusion models is only moderately larger than that for the vanilla diffusion models, making them still practical for large scale applications. We also provide empirical results showing that when there is clear conditional dependence structure in the data, the AR diffusion models captures such structure, whereas vanilla DDPM fails to do so. On the other hand, when there is no obvious conditional dependence across patches of the data, AR diffusion does not outperform DDPM.
• 摘要：扩散模型在图像和视频生成中都表现出了吸引人的性能。但是，许多作品发现他们努力捕捉现实世界中存在的重要高级关系。例如，他们无法从数据中学习物理定律，甚至无法理解世界上的对象以稳定的方式存在。这是由于以下事实：在香草扩散模型中，重要的条件依赖性结构未充分捕获。在这项工作中，我们启动了一项有关加强扩散模型以捕获数据中有条件依赖性结构的深入研究。特别是，我们研究了自动回归（AR）扩散模型的功效，并为此目的开发了第一个理论结果，即（可能）（可能）最轻度的数据假设下的AR扩散模型的采样误差。我们的理论发现表明，与典型的扩散模型相比，AR变体在近似数据条件分布时会产生差异降低的样品。另一方面，AR扩散模型的总体推理时间仅比香草扩散模型大的推理时间中等程度，这使得它们对于大型应用程序仍然很可行。我们还提供了经验结果，表明当数据中有明确的条件依赖性结构时，AR扩散模型会捕获这种结构，而香草DDPM则无法做到。另一方面，当数据斑块之间没有明显的条件依赖性时，AR扩散不会超过DDPM。

Title: Text-Conditioned Diffusion Model for High-Fidelity Korean Font Generation

• Authors: Abdul Sami, Avinash Kumar, Irfanullah Memon, Youngwon Jo, Muhammad Rizwan, Jaeyoung Choi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21325
• Pdf URL: https://arxiv.org/pdf/2504.21325
• Copy Paste: [[2504.21325]] Text-Conditioned Diffusion Model for High-Fidelity Korean Font Generation(https://arxiv.org/abs/2504.21325)
• Keywords: generation, generative
• Abstract: Automatic font generation (AFG) is the process of creating a new font using only a few examples of the style images. Generating fonts for complex languages like Korean and Chinese, particularly in handwritten styles, presents significant challenges. Traditional AFGs, like Generative adversarial networks (GANs) and Variational Auto-Encoders (VAEs), are usually unstable during training and often face mode collapse problems. They also struggle to capture fine details within font images. To address these problems, we present a diffusion-based AFG method which generates high-quality, diverse Korean font images using only a single reference image, focusing on handwritten and printed styles. Our approach refines noisy images incrementally, ensuring stable training and visually appealing results. A key innovation is our text encoder, which processes phonetic representations to generate accurate and contextually correct characters, even for unseen characters. We used a pre-trained style encoder from DG FONT to effectively and accurately encode the style images. To further enhance the generation quality, we used perceptual loss that guides the model to focus on the global style of generated images. Experimental results on over 2000 Korean characters demonstrate that our model consistently generates accurate and detailed font images and outperforms benchmark methods, making it a reliable tool for generating authentic Korean fonts across different styles.
• 摘要：自动字体生成（AFG）是仅使用样式图像的几个示例创建新字体的过程。为韩文和中文（尤其是手写风格）产生复杂语言的字体提出了重大挑战。传统的AFG，例如生成对抗网络（GAN）和变异自动编码器（VAE），通常在训练中不稳定，并且通常会面临模式崩溃问题。他们还难以在字体图像中捕获细节。为了解决这些问题，我们提出了一种基于扩散的AFG方法，该方法仅使用单个参考图像生成高质量的韩国字体图像，重点是手写和印刷样式。我们的方法会逐步完善嘈杂的图像，确保稳定的培训和视觉吸引力的结果。一个关键的创新是我们的文本编码器，它处理语音表示以生成准确且上下文正确的字符，即使对于看不见的字符也是如此。我们使用了从DG字体的预训练样式编码器来有效，准确地编码样式图像。为了进一步提高发电质量，我们使用了感知损失，该损失指导该模型专注于全球生成的图像风格。超过2000个韩国角色的实验结果表明，我们的模型始终生成准确，详细的字体图像和优于基准方法，使其成为在不同样式上生成真实的韩国字体的可靠工具。

Title: Simple Visual Artifact Detection in Sora-Generated Videos

• Authors: Misora Sugiyama, Hirokatsu Kataoka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21334
• Pdf URL: https://arxiv.org/pdf/2504.21334
• Copy Paste: [[2504.21334]] Simple Visual Artifact Detection in Sora-Generated Videos(https://arxiv.org/abs/2504.21334)
• Keywords: generation
• Abstract: The December 2024 release of OpenAI's Sora, a powerful video generation model driven by natural language prompts, highlights a growing convergence between large language models (LLMs) and video synthesis. As these multimodal systems evolve into video-enabled LLMs (VidLLMs), capable of interpreting, generating, and interacting with visual content, understanding their limitations and ensuring their safe deployment becomes essential. This study investigates visual artifacts frequently found and reported in Sora-generated videos, which can compromise quality, mislead viewers, or propagate disinformation. We propose a multi-label classification framework targeting four common artifact label types: label 1: boundary / edge defects, label 2: texture / noise issues, label 3: movement / joint anomalies, and label 4: object mismatches / disappearances. Using a dataset of 300 manually annotated frames extracted from 15 Sora-generated videos, we trained multiple 2D CNN architectures (ResNet-50, EfficientNet-B3 / B4, ViT-Base). The best-performing model trained by ResNet-50 achieved an average multi-label classification accuracy of 94.14%. This work supports the broader development of VidLLMs by contributing to (1) the creation of datasets for video quality evaluation, (2) interpretable artifact-based analysis beyond language metrics, and (3) the identification of visual risks relevant to factuality and safety.
• 摘要：2024年12月的Openai's Sora是由自然语言提示驱动的强大视频生成模型的发行，突出了大语言模型（LLM）和视频合成之间日益增长的收敛性。随着这些多模式系统能够演变为具有视频的LLM（Vidllms），能够解释，生成和与视觉内容进行交互，了解它们的局限性并确保其安全部署变得必不可少。这项研究调查了在Sora生成的视频中经常发现和报告的视觉伪像，这可能会损害质量，误导观众或传播虚假信息。我们提出了一个针对四种常见人工制品标签类型的多标签分类框架：标签1：边界 /边缘缺陷，标签2：纹理 /噪声问题，标签3：运动 /关节异常和标签4：对象不匹配 /消失。使用从15个Sora生成的视频中提取的300个手动注释框架的数据集，我们训练了多个2D CNN体系结构（Resnet-50，EdgitionNet-B3 / b4，vit-base）。由Resnet-50训练的最佳表现模型的平均多标签分类精度为94.14％。这项工作通过（1）创建用于视频质量评估的数据集，（2）基于语言指标的可解释的基于人工制品的分析以及（3）识别与事实和安全有关的视觉风险的识别。

Title: UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretation

• Authors: Linshan Wu, Yuxiang Nie, Sunan He, Jiaxin Zhuang, Hao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21336
• Pdf URL: https://arxiv.org/pdf/2504.21336
• Copy Paste: [[2504.21336]] UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretation(https://arxiv.org/abs/2504.21336)
• Keywords: generation
• Abstract: Multi-modal interpretation of biomedical images opens up novel opportunities in biomedical image analysis. Conventional AI approaches typically rely on disjointed training, i.e., Large Language Models (LLMs) for clinical text generation and segmentation models for target extraction, which results in inflexible real-world deployment and a failure to leverage holistic biomedical information. To this end, we introduce UniBiomed, the first universal foundation model for grounded biomedical image interpretation. UniBiomed is based on a novel integration of Multi-modal Large Language Model (MLLM) and Segment Anything Model (SAM), which effectively unifies the generation of clinical texts and the segmentation of corresponding biomedical objects for grounded interpretation. In this way, UniBiomed is capable of tackling a wide range of biomedical tasks across ten diverse biomedical imaging modalities. To develop UniBiomed, we curate a large-scale dataset comprising over 27 million triplets of images, annotations, and text descriptions across ten imaging modalities. Extensive validation on 84 internal and external datasets demonstrated that UniBiomed achieves state-of-the-art performance in segmentation, disease recognition, region-aware diagnosis, visual question answering, and report generation. Moreover, unlike previous models that rely on clinical experts to pre-diagnose images and manually craft precise textual or visual prompts, UniBiomed can provide automated and end-to-end grounded interpretation for biomedical image analysis. This represents a novel paradigm shift in clinical workflows, which will significantly improve diagnostic efficiency. In summary, UniBiomed represents a novel breakthrough in biomedical AI, unlocking powerful grounded interpretation capabilities for more accurate and efficient biomedical image analysis.
• 摘要：生物医学图像的多模式解释为生物医学图像分析打开了新的机会。常规的AI方法通常依赖于脱节的培训，即用于临床文本生成和分割模型的大型语言模型（LLMS）用于目标提取，这导致了不灵活的现实世界部署，并且未能利用整体生物医学信息。为此，我们介绍了Unibiomed，这是第一个用于基础生物医学图像解释的通用基础模型。 Unibiomed基于多模式大语言模型（MLLM）和分段的任何模型（SAM）的新型整合，该模型有效地统一了临床文本的产生以及相应的生物医学对象的分割以进行接地解释。这样，Unibiomed能够解决十种不同的生物医学成像方式的广泛生物医学任务。为了开发Unibiomed，我们策划了一个大规模的数据集，其中包含超过2700万三联的图像，注释和文本描述。对84个内部和外部数据集进行了广泛的验证，表明，单位在细分，疾病识别，区域意识诊断，视觉问题答案和报告生成方面取得了最先进的表现。此外，与以前依靠临床专家预先诊断图像并手动制作精确的文本或视觉提示的模型不同，Unibiomed可以为生物医学图像分析提供自动化和端到端的基础解释。这代表了临床工作流程的新型范式转移，这将显着提高诊断效率。总而言之，Unibiomed代表了生物医学AI的新突破，可以解锁强大的接地解释能力，以更准确，有效的生物医学图像分析。

Title: Generative QoE Modeling: A Lightweight Approach for Telecom Networks

• Authors: Vinti Nayar, Kanica Sachdev, Brejesh Lall
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2504.21353
• Pdf URL: https://arxiv.org/pdf/2504.21353
• Copy Paste: [[2504.21353]] Generative QoE Modeling: A Lightweight Approach for Telecom Networks(https://arxiv.org/abs/2504.21353)
• Keywords: generative
• Abstract: Quality of Experience (QoE) prediction plays a crucial role in optimizing resource management and enhancing user satisfaction across both telecommunication and OTT services. While recent advances predominantly rely on deep learning models, this study introduces a lightweight generative modeling framework that balances computational efficiency, interpretability, and predictive accuracy. By validating the use of Vector Quantization (VQ) as a preprocessing technique, continuous network features are effectively transformed into discrete categorical symbols, enabling integration with a Hidden Markov Model (HMM) for temporal sequence modeling. This VQ-HMM pipeline enhances the model's capacity to capture dynamic QoE patterns while supporting probabilistic inference on new and unseen data. Experimental results on publicly available time-series datasets incorporating both objective indicators and subjective QoE scores demonstrate the viability of this approach in real-time and resource-constrained environments, where inference latency is also critical. The framework offers a scalable alternative to complex deep learning methods, particularly in scenarios with limited computational resources or where latency constraints are critical.
• 摘要：经验质量（QOE）预测在优化资源管理和增强电信和OTT服务的用户满意度方面起着至关重要的作用。尽管最近的进步主要取决于深度学习模型，但本研究引入了轻巧的生成建模框架，该框架平衡了计算效率，可解释性和预测精度。通过验证矢量量化（VQ）作为预处理技术的使用，有效地将连续网络特征转化为离散的分类符号，从而可以使用隐藏的Markov模型（HMM）进行时间序列建模。该VQ-HMM管道增强了模型捕获动态QoE模式的能力，同时支持对新数据和看不见的数据的概率推断。关于包含客观指标和主观QOE分数的公开时间序列数据集的实验结果证明了这种方法在实时和资源约束环境中的生存能力，在这些环境中，推理潜伏期也至关重要。该框架为复杂的深度学习方法提供了可扩展的替代方法，尤其是在计算资源有限或延迟约束至关重要的情况下。

Title: Nexus-Gen: A Unified Model for Image Understanding, Generation, and Editing

• Authors: Hong Zhang, Zhongjie Duan, Xingjun Wang, Yingda Chen, Yuze Zhao, Yu Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21356
• Pdf URL: https://arxiv.org/pdf/2504.21356
• Copy Paste: [[2504.21356]] Nexus-Gen: A Unified Model for Image Understanding, Generation, and Editing(https://arxiv.org/abs/2504.21356)
• Keywords: generation
• Abstract: Unified multimodal large language models (MLLMs) aim to integrate multimodal understanding and generation abilities through a single framework. Despite their versatility, existing open-source unified models exhibit performance gaps against domain-specific architectures. To bridge this gap, we present Nexus-Gen, a unified model that synergizes the language reasoning capabilities of LLMs with the image synthesis power of diffusion models. To align the embedding space of the LLM and diffusion model, we conduct a dual-phase alignment training process. (1) The autoregressive LLM learns to predict image embeddings conditioned on multimodal inputs, while (2) the vision decoder is trained to reconstruct high-fidelity images from these embeddings. During training the LLM, we identified a critical discrepancy between the autoregressive paradigm's training and inference phases, where error accumulation in continuous embedding space severely degrades generation quality. To avoid this issue, we introduce a prefilled autoregression strategy that prefills input sequence with position-embedded special tokens instead of continuous embeddings. Through dual-phase training, Nexus-Gen has developed the integrated capability to comprehensively address the image understanding, generation and editing tasks. All models, datasets, and codes are published at this https URL to facilitate further advancements across the field.
• 摘要：统一的多模式大语模型（MLLM）旨在通过一个框架整合多模式的理解和发电能力。尽管它们具有多功能性，但现有的开源统一模型仍针对特定领域的体系结构表现出性能差距。为了弥合这一差距，我们提出了Nexus-Gen，这是一个统一的模型，它通过扩散模型的图像合成能力协同LLM的语言推理能力。为了使LLM和扩散模型的嵌入空间保持一致，我们进行了双相比对训练过程。 （1）自回归的LLM学会预测以多模式输入为条件的图像嵌入，而（2）视觉解码器经过训练以重建这些嵌入的高保真图像。在训练LLM期间，我们确定了自回旋范式的训练和推理阶段之间的严重差异，在这种范围内，在连续嵌入空间中累积的错误会严重降低发电质量。为了避免此问题，我们引入了预填充的自动化策略，该策略将输入序列与位置所包含的特殊令牌而不是连续嵌入。通过双相培训，Nexus-Gen开发了综合能力，可以全面解决图像理解，生成和编辑任务。所有模型，数据集和代码均在此HTTPS URL上发布，以促进整个现场的进一步进步。

Title: Sparse-to-Sparse Training of Diffusion Models

• Authors: Inês Cardoso Oliveira, Decebal Constantin Mocanu, Luis A. Leiva
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21380
• Pdf URL: https://arxiv.org/pdf/2504.21380
• Copy Paste: [[2504.21380]] Sparse-to-Sparse Training of Diffusion Models(https://arxiv.org/abs/2504.21380)
• Keywords: generation, generative
• Abstract: Diffusion models (DMs) are a powerful type of generative models that have achieved state-of-the-art results in various image synthesis tasks and have shown potential in other domains, such as natural language processing and temporal data modeling. Despite their stable training dynamics and ability to produce diverse high-quality samples, DMs are notorious for requiring significant computational resources, both in the training and inference stages. Previous work has focused mostly on increasing the efficiency of model inference. This paper introduces, for the first time, the paradigm of sparse-to-sparse training to DMs, with the aim of improving both training and inference efficiency. We focus on unconditional generation and train sparse DMs from scratch (Latent Diffusion and ChiroDiff) on six datasets using three different methods (Static-DM, RigL-DM, and MagRan-DM) to study the effect of sparsity in model performance. Our experiments show that sparse DMs are able to match and often outperform their Dense counterparts, while substantially reducing the number of trainable parameters and FLOPs. We also identify safe and effective values to perform sparse-to-sparse training of DMs.
• 摘要：扩散模型（DMS）是一种强大的生成模型，这些模型已实现了最新的模型，从而导致了各种图像合成任务，并在其他域中显示了潜力，例如自然语言处理和时间数据建模。尽管具有稳定的训练动力和生产各种高质量样本的能力，但在培训和推理阶段都需要大量的计算资源，DMS还是臭名昭著的。先前的工作主要集中在提高模型推理的效率上。本文首次介绍了向DMS稀疏到较小的培训的范式，目的是提高培训和推理效率。我们使用三种不同的方法（static-dm，rigl-dm和magran-dm）在六个数据集上从头开始（潜在扩散和chirodiff）的无条件生成和火车稀疏的DMS来研究稀疏对模型性能的影响。我们的实验表明，稀疏的DMS能够匹配并经常胜过其密集的对应物，同时大大减少了可训练的参数和FLOP的数量。我们还确定了安全有效的值，以对DMS进行稀疏到相对的培训。

Title: Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision

• Authors: Weicai Yan, Wang Lin, Zirun Guo, Ye Wang, Fangming Feng, Xiaoda Yang, Zehan Wang, Tao Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21423
• Pdf URL: https://arxiv.org/pdf/2504.21423
• Copy Paste: [[2504.21423]] Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision(https://arxiv.org/abs/2504.21423)
• Keywords: generation, generative
• Abstract: Prompt learning has demonstrated promising results in fine-tuning pre-trained multimodal models. However, the performance improvement is limited when applied to more complex and fine-grained tasks. The reason is that most existing methods directly optimize the parameters involved in the prompt generation process through loss backpropagation, which constrains the richness and specificity of the prompt representations. In this paper, we propose Diffusion-Driven Prompt Generator (Diff-Prompt), aiming to use the diffusion model to generate rich and fine-grained prompt information for complex downstream tasks. Specifically, our approach consists of three stages. In the first stage, we train a Mask-VAE to compress the masks into latent space. In the second stage, we leverage an improved Diffusion Transformer (DiT) to train a prompt generator in the latent space, using the masks for supervision. In the third stage, we align the denoising process of the prompt generator with the pre-trained model in the semantic space, and use the generated prompts to fine-tune the model. We conduct experiments on a complex pixel-level downstream task, referring expression comprehension, and compare our method with various parameter-efficient fine-tuning approaches. Diff-Prompt achieves a maximum improvement of 8.87 in R@1 and 14.05 in R@5 compared to the foundation model and also outperforms other state-of-the-art methods across multiple metrics. The experimental results validate the effectiveness of our approach and highlight the potential of using generative models for prompt generation. Code is available at this https URL.
• 摘要：迅速学习在微调预训练的多模型模型中表现出了令人鼓舞的结果。但是，当应用于更复杂和细粒度的任务时，性能改善受到限制。原因是大多数现有方法通过损失反向传播直接优化及时生成过程中涉及的参数，这限制了及时表示的丰富性和特异性。在本文中，我们提出了扩散驱动的提示生成器（DIFF-PROMPT），旨在使用扩散模型为复杂的下游任务生成丰富且细粒的及时信息。具体而言，我们的方法包括三个阶段。在第一阶段，我们训练面具vae将面具压缩到潜在空间中。在第二阶段，我们利用改进的扩散变压器（DIT）在潜在空间中训练一个提示发电机，并使用口罩进行监督。在第三阶段，我们将提示生成器的固定过程与语义空间中的预训练模型保持一致，并使用生成的提示来微调模型。我们对复杂的像素级下游任务进行实验，参考表达理解，并将我们的方法与各种参数有效的微调方法进行比较。与基础模型相比，DIFF-PROMPT在R@1和R@5中的最大提高为8.87，在R@5中的最大改善，并且在多个指标上的其他最新方法都胜过其他最先进的方法。实验结果验证了我们方法的有效性，并突出了将生成模型用于及时生成的潜力。代码可在此HTTPS URL上找到。

Title: Multiview Point Cloud Registration via Optimization in an Autoencoder Latent Space

• Authors: Luc Vedrenne, Sylvain Faisan, Denis Fortun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21467
• Pdf URL: https://arxiv.org/pdf/2504.21467
• Copy Paste: [[2504.21467]] Multiview Point Cloud Registration via Optimization in an Autoencoder Latent Space(https://arxiv.org/abs/2504.21467)
• Keywords: generative
• Abstract: Point cloud rigid registration is a fundamental problem in 3D computer vision. In the multiview case, we aim to find a set of 6D poses to align a set of objects. Methods based on pairwise registration rely on a subsequent synchronization algorithm, which makes them poorly scalable with the number of views. Generative approaches overcome this limitation, but are based on Gaussian Mixture Models and use an Expectation-Maximization algorithm. Hence, they are not well suited to handle large transformations. Moreover, most existing methods cannot handle high levels of degradations. In this paper, we introduce POLAR (POint cloud LAtent Registration), a multiview registration method able to efficiently deal with a large number of views, while being robust to a high level of degradations and large initial angles. To achieve this, we transpose the registration problem into the latent space of a pretrained autoencoder, design a loss taking degradations into account, and develop an efficient multistart optimization strategy. Our proposed method significantly outperforms state-of-the-art approaches on synthetic and real data. POLAR is available at this http URL or as a standalone package which can be installed with pip install polaregistration.
• 摘要：点云刚性注册是3D计算机视觉中的一个基本问题。在多视案例中，我们旨在找到一组6D姿势以对齐一组对象。基于成对注册的方法依赖于随后的同步算法，这使其与视图数量的可扩展性差。生成方法克服了这一限制，但基于高斯混合模型，并使用预期最大化算法。因此，它们不适合处理大型转换。此外，大多数现有方法无法处理高水平的降解。在本文中，我们介绍了Polar（点云潜在注册），这是一种能够有效处理大量视图的多视图注册方法，同时可以强大地降低降级和大的初始角度。为了实现这一目标，我们将注册问题转换为预验证的自动编码器的潜在空间，设计损失考虑降解，并制定有效的多设备优化策略。我们提出的方法在合成和真实数据上的最先进方法大大优于最先进的方法。 Polar可在此HTTP URL或可以通过PIP Install PolareGistration安装的独立包装上找到。

Title: GarmentDiffusion: 3D Garment Sewing Pattern Generation with Multimodal Diffusion Transformers

• Authors: Xinyu Li, Qi Yao, Yuanda Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21476
• Pdf URL: https://arxiv.org/pdf/2504.21476
• Copy Paste: [[2504.21476]] GarmentDiffusion: 3D Garment Sewing Pattern Generation with Multimodal Diffusion Transformers(https://arxiv.org/abs/2504.21476)
• Keywords: generation, generative
• Abstract: Garment sewing patterns are fundamental design elements that bridge the gap between design concepts and practical manufacturing. The generative modeling of sewing patterns is crucial for creating diversified garments. However, existing approaches are limited either by reliance on a single input modality or by suboptimal generation efficiency. In this work, we present \textbf{\textit{GarmentDiffusion}}, a new generative model capable of producing centimeter-precise, vectorized 3D sewing patterns from multimodal inputs (text, image, and incomplete sewing pattern). Our method efficiently encodes 3D sewing pattern parameters into compact edge token representations, achieving a sequence length that is $\textbf{10}\times$ shorter than that of the autoregressive SewingGPT in DressCode. By employing a diffusion transformer, we simultaneously denoise all edge tokens along the temporal axis, while maintaining a constant number of denoising steps regardless of dataset-specific edge and panel statistics. With all combination of designs of our model, the sewing pattern generation speed is accelerated by $\textbf{100}\times$ compared to SewingGPT. We achieve new state-of-the-art results on DressCodeData, as well as on the largest sewing pattern dataset, namely GarmentCodeData. The project website is available at this https URL.
• 摘要：服装缝纫模式是弥合设计概念和实用制造之间差距的基本设计元素。缝纫模式的生成建模对于创建多样化的服装至关重要。但是，现有方法受到依赖单个输入模式或次优产生效率的限制。在这项工作中，我们提出了\ textbf {\ textit {garmentDiffusion}}，这是一种能够生成厘米专家的新生成模型，来自多模式输入（文本，图像和不完整的缝纫模式）的矢量化3D缝制模式。我们的方法有效地将3D缝纫模式参数编码为紧凑的边缘令牌表示形式，实现了$ \ textbf {10} \ times $ $ $ $短于dresscode中的自动式缝纫的序列。通过采用扩散变压器，我们同时沿时间轴同时将所有边缘令牌变形，同时保持恒定数量的Denoising步骤，无论数据集特异性边缘和面板统计量如何。通过我们模型的所有设计组合，与Sewinggpt相比，$ \ textbf {100} \ times $加速了缝纫模式生成速度。我们在DressCodedata以及最大的缝纫模式数据集（即GarmentCodedata）上获得了新的最新结果。该项目网站可在此HTTPS URL上找到。

Title: DGSolver: Diffusion Generalist Solver with Universal Posterior Sampling for Image Restoration

• Authors: Hebaixu Wang, Jing Zhang, Haonan Guo, Di Wang, Jiayi Ma, Bo Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21487
• Pdf URL: https://arxiv.org/pdf/2504.21487
• Copy Paste: [[2504.21487]] DGSolver: Diffusion Generalist Solver with Universal Posterior Sampling for Image Restoration(https://arxiv.org/abs/2504.21487)
• Keywords: restoration
• Abstract: Diffusion models have achieved remarkable progress in universal image restoration. While existing methods speed up inference by reducing sampling steps, substantial step intervals often introduce cumulative errors. Moreover, they struggle to balance the commonality of degradation representations and restoration quality. To address these challenges, we introduce \textbf{DGSolver}, a diffusion generalist solver with universal posterior sampling. We first derive the exact ordinary differential equations for generalist diffusion models and tailor high-order solvers with a queue-based accelerated sampling strategy to improve both accuracy and efficiency. We then integrate universal posterior sampling to better approximate manifold-constrained gradients, yielding a more accurate noise estimation and correcting errors in inverse inference. Extensive experiments show that DGSolver outperforms state-of-the-art methods in restoration accuracy, stability, and scalability, both qualitatively and quantitatively. Code and models will be available at this https URL.
• 摘要：扩散模型在通用图像恢复中取得了显着进步。尽管现有方法通过减少采样步骤加快推断，但大量步骤间隔通常会引入累积错误。此外，他们努力平衡退化表示和恢复质量的共同点。为了应对这些挑战，我们介绍了\ textbf {dgsolver}，这是一种具有通用后验采样的扩散通用求解器。我们首先通过基于队列的加速采样策略来得出通才扩散模型的确切普通微分方程，并量身定制高阶求解器，以提高准确性和效率。然后，我们将通用的后验采样集成到更好的近似歧管约束梯度，从而产生更准确的噪声估计并纠正反推理中的误差。广泛的实验表明，在定性和定量上，DGSolver在恢复精度，稳定性和可扩展性方面的最先进方法均优于最先进的方法。代码和型号将在此HTTPS URL上可用。

Title: Consistency-aware Fake Videos Detection on Short Video Platforms

• Authors: Junxi Wang, Jize liu, Na Zhang, Yaxiong Wang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2504.21495
• Pdf URL: https://arxiv.org/pdf/2504.21495
• Copy Paste: [[2504.21495]] Consistency-aware Fake Videos Detection on Short Video Platforms(https://arxiv.org/abs/2504.21495)
• Keywords: generation
• Abstract: This paper focuses to detect the fake news on the short video platforms. While significant research efforts have been devoted to this task with notable progress in recent years, current detection accuracy remains suboptimal due to the rapid evolution of content manipulation and generation technologies. Existing approaches typically employ a cross-modal fusion strategy that directly combines raw video data with metadata inputs before applying a classification layer. However, our empirical observations reveal a critical oversight: manipulated content frequently exhibits inter-modal inconsistencies that could serve as valuable discriminative features, yet remain underutilized in contemporary detection frameworks. Motivated by this insight, we propose a novel detection paradigm that explicitly identifies and leverages cross-modal contradictions as discriminative cues. Our approach consists of two core modules: Cross-modal Consistency Learning (CMCL) and Multi-modal Collaborative Diagnosis (MMCD). CMCL includes Pseudo-label Generation (PLG) and Cross-modal Consistency Diagnosis (CMCD). In PLG, a Multimodal Large Language Model is used to generate pseudo-labels for evaluating cross-modal semantic consistency. Then, CMCD extracts [CLS] tokens and computes cosine loss to quantify cross-modal inconsistencies. MMCD further integrates multimodal features through Multimodal Feature Fusion (MFF) and Probability Scores Fusion (PSF). MFF employs a co-attention mechanism to enhance semantic interactions across different modalities, while a Transformer is utilized for comprehensive feature fusion. Meanwhile, PSF further integrates the fake news probability scores obtained in the previous step. Extensive experiments on established benchmarks (FakeSV and FakeTT) demonstrate our model exhibits outstanding performance in Fake videos detection.
• 摘要：本文重点是在短视频平台上检测假新闻。虽然近年来一直致力于这项任务，但由于内容操纵和发电技术的迅速发展，目前的检测准确性仍然不错。现有方法通常采用跨模式融合策略，该策略将原始视频数据与元数据输入直接结合在一起，然后再应用分类层。但是，我们的经验观察结果揭示了一个关键的监督：操纵的内容经常表现出模式间的不一致之处，这些不一致可能是有价值的歧视性特征，但在当代检测框架中仍未充分利用。在这种见解的推动下，我们提出了一个新颖的检测范式，该范式明确地识别和利用了跨模式矛盾作为歧视性提示。我们的方法由两个核心模块组成：跨模式一致性学习（CMCL）和多模式协作诊断（MMCD）。 CMCL包括伪标记生成（PLG）和跨模式一致性诊断（CMCD）。在PLG中，使用多模式的大语模型来生成伪标记，以评估跨模式语义一致性。然后，CMCD提取[CLS]令牌并计算余弦损失以量化跨模式不一致。 MMCD通过多模式特征融合（MFF）和概率得分融合（PSF）进一步整合了多模式特征。 MFF采用共同注意机制来增强不同方式之间的语义相互作用，而变压器则用于全面的特征融合。同时，PSF进一步整合了上一步中获得的虚假新闻概率分数。关于既定基准（Fakesv和Fakett）的广泛实验表明，我们的模型在假视频检测中表现出了出色的性能。

Title: MagicPortrait: Temporally Consistent Face Reenactment with 3D Geometric Guidance

• Authors: Mengting Wei, Yante Li, Tuomas Varanka, Yan Jiang, Licai Sun, Guoying Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21497
• Pdf URL: https://arxiv.org/pdf/2504.21497
• Copy Paste: [[2504.21497]] MagicPortrait: Temporally Consistent Face Reenactment with 3D Geometric Guidance(https://arxiv.org/abs/2504.21497)
• Keywords: generation
• Abstract: In this paper, we propose a method for video face reenactment that integrates a 3D face parametric model into a latent diffusion framework, aiming to improve shape consistency and motion control in existing video-based face generation approaches. Our approach employs the FLAME (Faces Learned with an Articulated Model and Expressions) model as the 3D face parametric representation, providing a unified framework for modeling face expressions and head pose. This enables precise extraction of detailed face geometry and motion features from driving videos. Specifically, we enhance the latent diffusion model with rich 3D expression and detailed pose information by incorporating depth maps, normal maps, and rendering maps derived from FLAME sequences. A multi-layer face movements fusion module with integrated self-attention mechanisms is used to combine identity and motion latent features within the spatial domain. By utilizing the 3D face parametric model as motion guidance, our method enables parametric alignment of face identity between the reference image and the motion captured from the driving video. Experimental results on benchmark datasets show that our method excels at generating high-quality face animations with precise expression and head pose variation modeling. In addition, it demonstrates strong generalization performance on out-of-domain images. Code is publicly available at this https URL.
• 摘要：在本文中，我们提出了一种视频面部重演的方法，该方法将3D面部参数模型集成到潜在的扩散框架中，旨在在现有的基于视频的面部生成方法中提高形状一致性和运动控制。我们的方法采用火焰（以铰接模型和表达方式学习的面孔为3D面部参数表示，为建模面部表情和头部姿势提供了统一的框架。这可以精确提取详细的面部几何形状和运动特征从驱动视频中。具体而言，我们通过结合了从火焰序列中得出的深度图，正常地图和渲染图来增强具有丰富3D表达和详细姿势信息的潜在扩散模型。具有集成的自我发项机制的多层脸部运动融合模块用于将身份和运动潜在特征结合在空间结构域中。通过利用3D脸参数模型作为运动指导，我们的方法启用了参考图像与从驾驶视频捕获的运动之间的面部身份的参数对齐。基准数据集的实验结果表明，我们的方法在生成具有精确表达和头部姿势变化建模的高质量面动画方面表现出色。此外，它在室外图像上表现出强烈的概括性能。代码在此HTTPS URL上公开可用。

Title: Diffusion-based Adversarial Identity Manipulation for Facial Privacy Protection

• Authors: Liqin Wang, Qianyue Hu, Wei Lu, Xiangyang Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21646
• Pdf URL: https://arxiv.org/pdf/2504.21646
• Copy Paste: [[2504.21646]] Diffusion-based Adversarial Identity Manipulation for Facial Privacy Protection(https://arxiv.org/abs/2504.21646)
• Keywords: generation
• Abstract: The success of face recognition (FR) systems has led to serious privacy concerns due to potential unauthorized surveillance and user tracking on social networks. Existing methods for enhancing privacy fail to generate natural face images that can protect facial privacy. In this paper, we propose diffusion-based adversarial identity manipulation (DiffAIM) to generate natural and highly transferable adversarial faces against malicious FR systems. To be specific, we manipulate facial identity within the low-dimensional latent space of a diffusion model. This involves iteratively injecting gradient-based adversarial identity guidance during the reverse diffusion process, progressively steering the generation toward the desired adversarial faces. The guidance is optimized for identity convergence towards a target while promoting semantic divergence from the source, facilitating effective impersonation while maintaining visual naturalness. We further incorporate structure-preserving regularization to preserve facial structure consistency during manipulation. Extensive experiments on both face verification and identification tasks demonstrate that compared with the state-of-the-art, DiffAIM achieves stronger black-box attack transferability while maintaining superior visual quality. We also demonstrate the effectiveness of the proposed approach for commercial FR APIs, including Face++ and Aliyun.
• 摘要：由于潜在的未经授权的监视和社交网络上的用户跟踪，面部识别（FR）系统的成功导致了严重的隐私问题。现有的增强隐私的方法无法产生可以保护面部隐私的自然面部图像。在本文中，我们提出了基于扩散的对抗身份操纵（DIFFAIM），以生成针对恶意FR系统的天然且高度可转移的对抗性面孔。具体而言，我们在扩散模型的低维潜在空间内操纵面部身份。这涉及在反向扩散过程中迭代注射基于梯度的对抗身份指南，从而逐步将一代转向所需的对抗面。该指南是针对目标融合到目标的同时优化的，同时促进语义差异与来源的差异，从而促进有效的模仿，同时保持视觉自然性。我们进一步结合了结构性的正则化，以保持操纵过程中的面部结构一致性。关于面部验证和识别任务的广泛实验表明，与最先进的Diffaim相比，Diffaim可实现更强的黑盒攻击传递性，同时保持出色的视觉质量。我们还证明了拟议的商业FRAPI的有效性，包括Face ++和Aliyun。

Title: HoloTime: Taming Video Diffusion Models for Panoramic 4D Scene Generation

• Authors: Haiyang Zhou, Wangbo Yu, Jiawen Guan, Xinhua Cheng, Yonghong Tian, Li Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21650
• Pdf URL: https://arxiv.org/pdf/2504.21650
• Copy Paste: [[2504.21650]] HoloTime: Taming Video Diffusion Models for Panoramic 4D Scene Generation(https://arxiv.org/abs/2504.21650)
• Keywords: generation
• Abstract: The rapid advancement of diffusion models holds the promise of revolutionizing the application of VR and AR technologies, which typically require scene-level 4D assets for user experience. Nonetheless, existing diffusion models predominantly concentrate on modeling static 3D scenes or object-level dynamics, constraining their capacity to provide truly immersive experiences. To address this issue, we propose HoloTime, a framework that integrates video diffusion models to generate panoramic videos from a single prompt or reference image, along with a 360-degree 4D scene reconstruction method that seamlessly transforms the generated panoramic video into 4D assets, enabling a fully immersive 4D experience for users. Specifically, to tame video diffusion models for generating high-fidelity panoramic videos, we introduce the 360World dataset, the first comprehensive collection of panoramic videos suitable for downstream 4D scene reconstruction tasks. With this curated dataset, we propose Panoramic Animator, a two-stage image-to-video diffusion model that can convert panoramic images into high-quality panoramic videos. Following this, we present Panoramic Space-Time Reconstruction, which leverages a space-time depth estimation method to transform the generated panoramic videos into 4D point clouds, enabling the optimization of a holistic 4D Gaussian Splatting representation to reconstruct spatially and temporally consistent 4D scenes. To validate the efficacy of our method, we conducted a comparative analysis with existing approaches, revealing its superiority in both panoramic video generation and 4D scene reconstruction. This demonstrates our method's capability to create more engaging and realistic immersive environments, thereby enhancing user experiences in VR and AR applications.
• 摘要：扩散模型的快速发展具有彻底改变VR和AR技术的应用的希望，VR和AR技术的应用通常需要场景级别的4D资产来获得用户体验。但是，现有的扩散模型主要集中于建模静态3D场景或对象级动力学，从而限制了其提供真正沉浸式体验的能力。为了解决这个问题，我们提出了整个框架，该框架集成了视频扩散模型，以从单个提示或参考图像中生成全景视频，以及360度的4D场景重建方法，将生成的全景视频无缝地转换为4D资产，从而为用户提供完全沉浸式的4D体验。具体来说，要驯服用于生成高保真全景视频的视频扩散模型，我们介绍了360world数据集，这是适合下游4D场景重建任务的全景集合的第一个全面集合。借助此策划的数据集，我们提出了全景动画师，这是一个两阶段的图像到视频扩散模型，可以将全景图像转换为高质量的全景视频。在此之后，我们提出了全景时空重建，该重建利用了时空深度估计方法将生成的全景视频转换为4D点云，从而使整体4D高斯碎片表示可以优化空间和时间一致的4D场景。为了验证我们方法的疗效，我们对现有方法进行了比较分析，揭示了其在全景视频生成和4D场景重建中的优势。这证明了我们的方法能够创建更具吸引力和现实的沉浸式环境，从而增强了VR和AR应用程序中的用户体验。

Title: Visual Text Processing: A Comprehensive Review and Unified Evaluation

• Authors: Yan Shu, Weichao Zeng, Fangmin Zhao, Zeyu Chen, Zhenhang Li, Xiaomeng Yang, Yu Zhou, Paolo Rota, Xiang Bai, Lianwen Jin, Xu-Cheng Yin, Nicu Sebe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21682
• Pdf URL: https://arxiv.org/pdf/2504.21682
• Copy Paste: [[2504.21682]] Visual Text Processing: A Comprehensive Review and Unified Evaluation(https://arxiv.org/abs/2504.21682)
• Keywords: quality assessment
• Abstract: Visual text is a crucial component in both document and scene images, conveying rich semantic information and attracting significant attention in the computer vision community. Beyond traditional tasks such as text detection and recognition, visual text processing has witnessed rapid advancements driven by the emergence of foundation models, including text image reconstruction and text image manipulation. Despite significant progress, challenges remain due to the unique properties that differentiate text from general objects. Effectively capturing and leveraging these distinct textual characteristics is essential for developing robust visual text processing models. In this survey, we present a comprehensive, multi-perspective analysis of recent advancements in visual text processing, focusing on two key questions: (1) What textual features are most suitable for different visual text processing tasks? (2) How can these distinctive text features be effectively incorporated into processing frameworks? Furthermore, we introduce VTPBench, a new benchmark that encompasses a broad range of visual text processing datasets. Leveraging the advanced visual quality assessment capabilities of multimodal large language models (MLLMs), we propose VTPScore, a novel evaluation metric designed to ensure fair and reliable evaluation. Our empirical study with more than 20 specific models reveals substantial room for improvement in the current techniques. Our aim is to establish this work as a fundamental resource that fosters future exploration and innovation in the dynamic field of visual text processing. The relevant repository is available at this https URL.
• 摘要：视觉文本是文档和场景图像中的关键组成部分，传达了丰富的语义信息并引起了计算机视觉社区的重大关注。除了文本检测和识别之类的传统任务外，视觉文本处理还目睹了基础模型的出现（包括文本图像重建和文本图像操纵）所驱动的快速进步。尽管取得了重大进展，但由于将文本与一般对象区分开的独特属性，挑战仍然存在。有效地捕获和利用这些独特的文本特征对于开发强大的视觉文本处理模型至关重要。在这项调查中，我们对视觉文本处理的最新进步进行了全面的，多观点的分析，重点介绍了两个关键问题：（1）哪些文本功能最适合不同的视觉文本处理任务？ （2）如何有效地将这些独特的文本特征纳入处理框架中？此外，我们推出了VTPBench，这是一种新的基准测试，该基准包括广泛的视觉文本处理数据集。利用多模式大语言模型（MLLM）的先进视觉质量评估功能，我们提出了VTPSCore，这是一种新颖的评估度量，旨在确保公平可靠的评估。我们对20多个特定模型的实证研究揭示了当前技术改善的大量空间。我们的目的是将这项工作确立为基本资源，以促进视觉文本处理动态领域的未来探索和创新。相关存储库可在此HTTPS URL上找到。

Title: Anatomical Similarity as a New Metric to Evaluate Brain Generative Models

• Authors: Bahram Jafrasteh, Wei Peng, Cheng Wan, Yimin Luo, Ehsan Adeli, Qingyu Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21771
• Pdf URL: https://arxiv.org/pdf/2504.21771
• Copy Paste: [[2504.21771]] Anatomical Similarity as a New Metric to Evaluate Brain Generative Models(https://arxiv.org/abs/2504.21771)
• Keywords: generative
• Abstract: Generative models enhance neuroimaging through data augmentation, quality improvement, and rare condition studies. Despite advances in realistic synthetic MRIs, evaluations focus on texture and perception, lacking sensitivity to crucial anatomical fidelity. This study proposes a new metric, called WASABI (Wasserstein-Based Anatomical Brain Index), to assess the anatomical realism of synthetic brain MRIs. WASABI leverages \textit{SynthSeg}, a deep learning-based brain parcellation tool, to derive volumetric measures of brain regions in each MRI and uses the multivariate Wasserstein distance to compare distributions between real and synthetic anatomies. Based on controlled experiments on two real datasets and synthetic MRIs from five generative models, WASABI demonstrates higher sensitivity in quantifying anatomical discrepancies compared to traditional image-level metrics, even when synthetic images achieve near-perfect visual quality. Our findings advocate for shifting the evaluation paradigm beyond visual inspection and conventional metrics, emphasizing anatomical fidelity as a crucial benchmark for clinically meaningful brain MRI synthesis. Our code is available at this https URL.
• 摘要：生成模型通过数据增强，质量改善和罕见状况研究来增强神经影像学。尽管逼真的合成MRI进步，但评估的重点是纹理和感知，对关键的解剖学忠诚度缺乏敏感性。这项研究提出了一种新的度量标准，称为Wasabi（基于Wasserstein的解剖学脑指数），以评估合成脑MRIS的解剖学现实主义。 Wasabi利用\ TextIt {SynthSeg}，一种基于深度学习的大脑拟合工具，用于得出每个MRI中大脑区域的体积测量，并使用多变量的Wasserstein距离来比较真实和合成解剖学之间的分布。基于与传统图像级指标相比，基于五个生成模型的两个真实数据集和五个生成模型的合成MRI的受控实验，在量化解剖学差异方面也表现出更高的灵敏度，即使合成图像达到了近乎完美的视觉质量。我们的发现主张将评估范式转移到视觉检查和常规指标之外，强调解剖学忠诚度是临床上有意义的大脑MRI合成的关键基准。我们的代码可在此HTTPS URL上找到。

Title: Why Compress What You Can Generate? When GPT-4o Generation Ushers in Image Compression Fields

• Authors: Yixin Gao, Xiaohan Pan, Xin Li, Zhibo Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21814
• Pdf URL: https://arxiv.org/pdf/2504.21814
• Copy Paste: [[2504.21814]] Why Compress What You Can Generate? When GPT-4o Generation Ushers in Image Compression Fields(https://arxiv.org/abs/2504.21814)
• Keywords: generation, generative
• Abstract: The rapid development of AIGC foundation models has revolutionized the paradigm of image compression, which paves the way for the abandonment of most pixel-level transform and coding, compelling us to ask: why compress what you can generate if the AIGC foundation model is powerful enough to faithfully generate intricate structure and fine-grained details from nothing more than some compact descriptors, i.e., texts, or cues. Fortunately, recent GPT-4o image generation of OpenAI has achieved impressive cross-modality generation, editing, and design capabilities, which motivates us to answer the above question by exploring its potential in image compression fields. In this work, we investigate two typical compression paradigms: textual coding and multimodal coding (i.e., text + extremely low-resolution image), where all/most pixel-level information is generated instead of compressing via the advanced GPT-4o image generation function. The essential challenge lies in how to maintain semantic and structure consistency during the decoding process. To overcome this, we propose a structure raster-scan prompt engineering mechanism to transform the image into textual space, which is compressed as the condition of GPT-4o image generation. Extensive experiments have shown that the combination of our designed structural raster-scan prompts and GPT-4o's image generation function achieved the impressive performance compared with recent multimodal/generative image compression at ultra-low bitrate, further indicating the potential of AIGC generation in image compression fields.
• 摘要：The rapid development of AIGC foundation models has revolutionized the paradigm of image compression, which paves the way for the abandonment of most pixel-level transform and coding, compelling us to ask: why compress what you can generate if the AIGC foundation model is powerful enough to faithfully generate intricate structure and fine-grained details from nothing more than some compact descriptors, i.e., texts, or cues.幸运的是，最近的GPT-4O图像生成的OpenAI产生了令人印象深刻的跨模式生成，编辑和设计功能，从而促使我们通过在图像压缩领域探索其潜力来回答上述问题。在这项工作中，我们研究了两个典型的压缩范式：文本编码和多模式编码（即文本 +极低分辨率图像），其中所有/大多数像素级信息都是生成的，而不是通过高级的GPT-4O图像生成函数来压缩。基本挑战在于如何在解码过程中维持语义和结构一致性。为了克服这一点，我们提出了一种结构栅格扫描及时工程机制，以将图像转换为文本空间，该图像被压缩为GPT-4O图像生成的条件。广泛的实验表明，与最近的多模式/生成图像压缩相比，我们设计的结构栅格扫描提示和GPT-4O的图像生成功能的组合达到了令人印象深刻的性能，进一步表明了AIGC在图像压缩领域的潜力。

Title: 3D Stylization via Large Reconstruction Model

• Authors: Ipek Oztas, Duygu Ceylan, Aysegul Dundar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21836
• Pdf URL: https://arxiv.org/pdf/2504.21836
• Copy Paste: [[2504.21836]] 3D Stylization via Large Reconstruction Model(https://arxiv.org/abs/2504.21836)
• Keywords: generation
• Abstract: With the growing success of text or image guided 3D generators, users demand more control over the generation process, appearance stylization being one of them. Given a reference image, this requires adapting the appearance of a generated 3D asset to reflect the visual style of the reference while maintaining visual consistency from multiple viewpoints. To tackle this problem, we draw inspiration from the success of 2D stylization methods that leverage the attention mechanisms in large image generation models to capture and transfer visual style. In particular, we probe if large reconstruction models, commonly used in the context of 3D generation, has a similar capability. We discover that the certain attention blocks in these models capture the appearance specific features. By injecting features from a visual style image to such blocks, we develop a simple yet effective 3D appearance stylization method. Our method does not require training or test time optimization. Through both quantitative and qualitative evaluations, we demonstrate that our approach achieves superior results in terms of 3D appearance stylization, significantly improving efficiency while maintaining high-quality visual outcomes.
• 摘要：随着文本或图像引导的3D发电机的越来越多的成功，用户要求对生成过程进行更多的控制，外观风格是其中之一。给定参考图像，这需要调整生成的3D资产的外观，以反映参考的视觉样式，同时从多个角度保持视觉一致性。为了解决这个问题，我们从2D风格化方法的成功中汲取灵感，这些方法利用了大型图像生成模型中的注意机制来捕获和传输视觉样式。特别是，如果在3D生成的上下文中使用的大型重建模型具有相似的功能，我们会探测。我们发现这些模型中的某些注意力块捕获了外观特定的特征。通过从视觉样式图像到此类块的注入功能，我们开发了一种简单而有效的3D外观样式化方法。我们的方法不需要培训或测试时间优化。通过定量和定性评估，我们证明了我们的方法在3D外观风格方面取得了卓越的成果，在维持高质量的视觉结果的同时显着提高了效率。

Title: A Survey of Interactive Generative Video

• Authors: Jiwen Yu, Yiran Qin, Haoxuan Che, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, Hao Chen, Xihui Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21853
• Pdf URL: https://arxiv.org/pdf/2504.21853
• Copy Paste: [[2504.21853]] A Survey of Interactive Generative Video(https://arxiv.org/abs/2504.21853)
• Keywords: generation, generative
• Abstract: Interactive Generative Video (IGV) has emerged as a crucial technology in response to the growing demand for high-quality, interactive video content across various domains. In this paper, we define IGV as a technology that combines generative capabilities to produce diverse high-quality video content with interactive features that enable user engagement through control signals and responsive feedback. We survey the current landscape of IGV applications, focusing on three major domains: 1) gaming, where IGV enables infinite exploration in virtual worlds; 2) embodied AI, where IGV serves as a physics-aware environment synthesizer for training agents in multimodal interaction with dynamically evolving scenes; and 3) autonomous driving, where IGV provides closed-loop simulation capabilities for safety-critical testing and validation. To guide future development, we propose a comprehensive framework that decomposes an ideal IGV system into five essential modules: Generation, Control, Memory, Dynamics, and Intelligence. Furthermore, we systematically analyze the technical challenges and future directions in realizing each component for an ideal IGV system, such as achieving real-time generation, enabling open-domain control, maintaining long-term coherence, simulating accurate physics, and integrating causal reasoning. We believe that this systematic analysis will facilitate future research and development in the field of IGV, ultimately advancing the technology toward more sophisticated and practical applications.
• 摘要：互动生成视频（IGV）已成为一种至关重要的技术，响应着对各个领域的高质量，交互式视频内容的需求的增长。在本文中，我们将IGV定义为一种结合生成能力的技术，可以生成各种高质量的视频内容和交互式功能，从而通过控制信号和响应反馈来使用户参与度。我们调查了IGV应用程序的当前格局，重点关注三个主要领域：1）游戏，IGV可以在虚拟世界中实现无限探索； 2）体现了AI，其中IGV充当了与动态不断发展的场景的多模式相互作用的训练剂的物理学环境合成器； 3）自动驾驶，其中IGV提供了闭环模拟功能，可用于安全至关重要的测试和验证。为了指导未来的发展，我们提出了一个综合框架，将理想的IGV系统分解为五个基本模块：生成，控制，记忆，动态和智能。此外，我们在实现理想IGV系统的每个组件时系统地分析了技术挑战和未来的方向，例如实现实时生成，实现开放域控制，保持长期连贯性，模拟准确的物理学以及整​​合出因果推理。我们认为，这种系统的分析将促进IGV领域的未来研究和发展，最终将技术推向了更复杂和更实际的应用。

Title: ReVision: High-Quality, Low-Cost Video Generation with Explicit 3D Physics Modeling for Complex Motion and Interaction

• Authors: Qihao Liu, Ju He, Qihang Yu, Liang-Chieh Chen, Alan Yuille
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.21855
• Pdf URL: https://arxiv.org/pdf/2504.21855
• Copy Paste: [[2504.21855]] ReVision: High-Quality, Low-Cost Video Generation with Explicit 3D Physics Modeling for Complex Motion and Interaction(https://arxiv.org/abs/2504.21855)
• Keywords: generation
• Abstract: In recent years, video generation has seen significant advancements. However, challenges still persist in generating complex motions and interactions. To address these challenges, we introduce ReVision, a plug-and-play framework that explicitly integrates parameterized 3D physical knowledge into a pretrained conditional video generation model, significantly enhancing its ability to generate high-quality videos with complex motion and interactions. Specifically, ReVision consists of three stages. First, a video diffusion model is used to generate a coarse video. Next, we extract a set of 2D and 3D features from the coarse video to construct a 3D object-centric representation, which is then refined by our proposed parameterized physical prior model to produce an accurate 3D motion sequence. Finally, this refined motion sequence is fed back into the same video diffusion model as additional conditioning, enabling the generation of motion-consistent videos, even in scenarios involving complex actions and interactions. We validate the effectiveness of our approach on Stable Video Diffusion, where ReVision significantly improves motion fidelity and coherence. Remarkably, with only 1.5B parameters, it even outperforms a state-of-the-art video generation model with over 13B parameters on complex video generation by a substantial margin. Our results suggest that, by incorporating 3D physical knowledge, even a relatively small video diffusion model can generate complex motions and interactions with greater realism and controllability, offering a promising solution for physically plausible video generation.
• 摘要：近年来，视频产生取得了重大进步。但是，挑战仍然存在于产生复杂的动作和互动。为了应对这些挑战，我们引入了修订版，这是一个插件框架，将参数化的3D物理知识明确整合到预算的条件视频生成模型中，从而显着增强了其具有复杂运动和相互作用的高质量视频的能力。具体而言，修订包括三个阶段。首先，使用视频扩散模型来生成粗糙的视频。接下来，我们从粗视频中提取一组2D和3D特征来构建一个以3D对象为中心的表示，然后通过我们提出的参数化物理先验模型来完善，以产生准确的3D运动序列。最后，这种精致的运动序列被反馈到与其他条件相同的视频扩散模型中，即使在涉及复杂的动作和相互作用的情况下，也能够产生与运动一致的视频。我们验证了方法对稳定视频扩散的有效性，其中修订可显着提高运动保真度和连贯性。值得注意的是，只有1.5b参数，它甚至超过了最先进的视频生成模型，其复杂视频生成的13B参数超过了一个大幅度。我们的结果表明，通过合并3D物理知识，即使是相对较小的视频扩散模型也可以通过更大的现实主义和可控性产生复杂的动作和相互作用，从而为物理上合理的视频生成提供了有希望的解决方案。