2025-05-16

Title: Robust Federated Learning with Confidence-Weighted Filtering and GAN-Based Completion under Noisy and Incomplete Data

• Authors: Alpaslan Gokcen, Ali Boyaci
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.09733
• Pdf URL: https://arxiv.org/pdf/2505.09733
• Copy Paste: [[2505.09733]] Robust Federated Learning with Confidence-Weighted Filtering and GAN-Based Completion under Noisy and Incomplete Data(https://arxiv.org/abs/2505.09733)
• Keywords: generation
• Abstract: Federated learning (FL) presents an effective solution for collaborative model training while maintaining data privacy across decentralized client datasets. However, data quality issues such as noisy labels, missing classes, and imbalanced distributions significantly challenge its effectiveness. This study proposes a federated learning methodology that systematically addresses data quality issues, including noise, class imbalance, and missing labels. The proposed approach systematically enhances data integrity through adaptive noise cleaning, collaborative conditional GAN-based synthetic data generation, and robust federated model training. Experimental evaluations conducted on benchmark datasets (MNIST and Fashion-MNIST) demonstrate significant improvements in federated model performance, particularly macro-F1 Score, under varying noise and class imbalance conditions. Additionally, the proposed framework carefully balances computational feasibility and substantial performance gains, ensuring practicality for resource constrained edge devices while rigorously maintaining data privacy. Our results indicate that this method effectively mitigates common data quality challenges, providing a robust, scalable, and privacy compliant solution suitable for diverse real-world federated learning scenarios.
• 摘要：联合学习（FL）为协作模型培训提供了一个有效的解决方案，同时在分散的客户数据集中保持数据隐私。但是，诸如嘈杂标签，缺失类和不平衡分布之类的数据质量问题大大挑战了其有效性。这项研究提出了一种联合学习方法，该方法系统地解决了数据质量问题，包括噪声，阶级失衡和缺失标签。提出的方法通过自适应噪声清洁，基于有条件的GAN合成数据生成以及强大的联合模型培训来系统地增强数据完整性。在不同的噪声和类不平衡条件下，在基准数据集（MNIST和时尚界）上进行的实验评估表明，在不同的噪声和类不平衡条件下，联合模型性能，尤其是Macro-F1得分的显着改善。此外，提议的框架仔细平衡了计算可行性和实质性的增长，从而确保了资源约束的边缘设备的实用性，同时严格地维护数据隐私。我们的结果表明，这种方法有效地减轻了共同的数据质量挑战，提供了适合各种现实世界联合学习方案的强大，可扩展和隐私的解决方案。

Title: A Generative Neural Annealer for Black-Box Combinatorial Optimization

• Authors: Yuan-Hang Zhang, Massimiliano Di Ventra
• Subjects: cs.LG, cond-mat.dis-nn, cond-mat.stat-mech, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2505.09742
• Pdf URL: https://arxiv.org/pdf/2505.09742
• Copy Paste: [[2505.09742]] A Generative Neural Annealer for Black-Box Combinatorial Optimization(https://arxiv.org/abs/2505.09742)
• Keywords: generative
• Abstract: We propose a generative, end-to-end solver for black-box combinatorial optimization that emphasizes both sample efficiency and solution quality on NP problems. Drawing inspiration from annealing-based algorithms, we treat the black-box objective as an energy function and train a neural network to model the associated Boltzmann distribution. By conditioning on temperature, the network captures a continuum of distributions--from near-uniform at high temperatures to sharply peaked around global optima at low temperatures--thereby learning the structure of the energy landscape and facilitating global optimization. When queries are expensive, the temperature-dependent distributions naturally enable data augmentation and improve sample efficiency. When queries are cheap but the problem remains hard, the model learns implicit variable interactions, effectively "opening" the black box. We validate our approach on challenging combinatorial tasks under both limited and unlimited query budgets, showing competitive performance against state-of-the-art black-box optimizers.
• 摘要：我们提出了一个用于黑盒组合优化的生成端到端求解器，该求解器强调了NP问题的样本效率和解决方案质量。从基于退火的算法中汲取灵感，我们将黑盒目标视为能量功能，并训练神经网络以建模相关的玻尔兹曼分布。通过在温度下进行调节，该网络捕获了分布的连续体 - 在高温下接近均匀的分布，在低温下全球最佳峰达到高峰 - 在此学习能量景观的结构并促进全球优化。当查询昂贵时，与温度相关的分布自然可以增强数据并提高样品效率。当查询价格便宜但问题仍然很困难时，该模型就会学习隐式变量交互，从而有效地“打开”黑匣子。我们验证了在有限和无限查询预算下挑战组合任务的方法，对最先进的黑盒优化者的竞争性能表现出了竞争性能。

Title: Self-Consuming Generative Models with Adversarially Curated Data

• Authors: Xiukun Wei, Xueru Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.09768
• Pdf URL: https://arxiv.org/pdf/2505.09768
• Copy Paste: [[2505.09768]] Self-Consuming Generative Models with Adversarially Curated Data(https://arxiv.org/abs/2505.09768)
• Keywords: generation, generative
• Abstract: Recent advances in generative models have made it increasingly difficult to distinguish real data from model-generated synthetic data. Using synthetic data for successive training of future model generations creates "self-consuming loops", which may lead to model collapse or training instability. Furthermore, synthetic data is often subject to human feedback and curated by users based on their preferences. Ferbach et al. (2024) recently showed that when data is curated according to user preferences, the self-consuming retraining loop drives the model to converge toward a distribution that optimizes those preferences. However, in practice, data curation is often noisy or adversarially manipulated. For example, competing platforms may recruit malicious users to adversarially curate data and disrupt rival models. In this paper, we study how generative models evolve under self-consuming retraining loops with noisy and adversarially curated data. We theoretically analyze the impact of such noisy data curation on generative models and identify conditions for the robustness of the retraining process. Building on this analysis, we design attack algorithms for competitive adversarial scenarios, where a platform with a limited budget employs malicious users to misalign a rival's model from actual user preferences. Experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed algorithms.
• 摘要：生成模型的最新进展使得将实际数据与模型生成的合成数据区分开变得越来越困难。使用合成数据连续训练未来的模型一代会产生“自我消耗循环”，这可能导致模型崩溃或训练不稳定。此外，合成数据通常受到人类反馈的约束，并根据用户的偏好策划。 Ferbach等。 （2024）最近表明，当数据根据用户偏好策划时，自我消耗的重新训练循环驱动该模型趋向于优化这些偏好的分布。但是，实际上，数据策划通常是嘈杂或对抗性操纵的。例如，竞争平台可能会招募恶意用户来对抗数据并破坏竞争对手模型。在本文中，我们研究了生成模型如何在具有嘈杂和对抗性策划数据的自我消耗的重新训练环下发展。我们理论上分析了这种嘈杂数据策展对生成模型的影响，并确定了重新培训过程的鲁棒性的条件。在此分析的基础上，我们为竞争性的对抗场景设计了攻击算法，其中有限预算的平台雇用恶意用户将竞争对手的模型从实际用户的偏好中误归一致。对合成和现实世界数据集的实验证明了所提出的算法的有效性。

Title: Causal Predictive Optimization and Generation for Business AI

• Authors: Liyang Zhao, Olurotimi Seton, Himadeep Reddy Reddivari, Suvendu Jena, Shadow Zhao, Rachit Kumar, Changshuai Wei
• Subjects: cs.LG, cs.AI, cs.IR, stat.ML
• Abstract URL: https://arxiv.org/abs/2505.09847
• Pdf URL: https://arxiv.org/pdf/2505.09847
• Copy Paste: [[2505.09847]] Causal Predictive Optimization and Generation for Business AI(https://arxiv.org/abs/2505.09847)
• Keywords: generation, generative
• Abstract: The sales process involves sales functions converting leads or opportunities to customers and selling more products to existing customers. The optimization of the sales process thus is key to success of any B2B business. In this work, we introduce a principled approach to sales optimization and business AI, namely the Causal Predictive Optimization and Generation, which includes three layers: 1) prediction layer with causal ML 2) optimization layer with constraint optimization and contextual bandit 3) serving layer with Generative AI and feedback-loop for system enhancement. We detail the implementation and deployment of the system in LinkedIn, showcasing significant wins over legacy systems and sharing learning and insight broadly applicable to this field.
• 摘要：销售过程涉及将潜在客户或机会转换为客户并向现有客户出售更多产品的销售功能。因此，销售过程的优化是任何B2B业务成功的关键。在这项工作中，我们引入了一种原则性的销售优化和业务AI方法，即因果预测优化和生成，其中包括三层：1）具有因果ML的预测层2）具有约束优化和上下文的优化层的优化层3）具有生成AI和反馈的系统增强系统。我们详细介绍了该系统在LinkedIn中的实施和部署，展示了对旧系统的重大胜利，并共享学习和洞察力广泛适用于该领域。

Title: Predictability Shapes Adaptation: An Evolutionary Perspective on Modes of Learning in Transformers

• Authors: Alexander Y. Ku, Thomas L. Griffiths, Stephanie C.Y. Chan
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2505.09855
• Pdf URL: https://arxiv.org/pdf/2505.09855
• Copy Paste: [[2505.09855]] Predictability Shapes Adaptation: An Evolutionary Perspective on Modes of Learning in Transformers(https://arxiv.org/abs/2505.09855)
• Keywords: generation
• Abstract: Transformer models learn in two distinct modes: in-weights learning (IWL), encoding knowledge into model weights, and in-context learning (ICL), adapting flexibly to context without weight modification. To better understand the interplay between these learning modes, we draw inspiration from evolutionary biology's analogous adaptive strategies: genetic encoding (akin to IWL, adapting over generations and fixed within an individual's lifetime) and phenotypic plasticity (akin to ICL, enabling flexible behavioral responses to environmental cues). In evolutionary biology, environmental predictability dictates the balance between these strategies: stability favors genetic encoding, while reliable predictive cues promote phenotypic plasticity. We experimentally operationalize these dimensions of predictability and systematically investigate their influence on the ICL/IWL balance in Transformers. Using regression and classification tasks, we show that high environmental stability decisively favors IWL, as predicted, with a sharp transition at maximal stability. Conversely, high cue reliability enhances ICL efficacy, particularly when stability is low. Furthermore, learning dynamics reveal task-contingent temporal evolution: while a canonical ICL-to-IWL shift occurs in some settings (e.g., classification with many classes), we demonstrate that scenarios with easier IWL (e.g., fewer classes) or slower ICL acquisition (e.g., regression) can exhibit an initial IWL phase later yielding to ICL dominance. These findings support a relative-cost hypothesis for explaining these learning mode transitions, establishing predictability as a critical factor governing adaptive strategies in Transformers, and offering novel insights for understanding ICL and guiding training methodologies.
• 摘要：变压器模型以两种不同的模式学习：重量学习（IWL），将知识编码为模型权重，以及内在的学习（ICL），灵活地适应上下文，而无需修改权重。为了更好地理解这些学习模式之间的相互作用，我们从进化生物学的类似自适应策略中汲取灵感：遗传编码（类似于IWL，适应世代相传并固定在个人的寿命中）和表型可塑性（类似于ICL，与ICL相似，对环境线索产生灵活的行为响应）。在进化生物学中，环境可预测性决定了这些策略之间的平衡：稳定性有利于遗传编码，而可靠的预测提示则促进了表型可塑性。我们通过实验性地操作可预测性的这些维度，并系统地研究它们对变形金刚中ICL/IWL平衡的影响。使用回归和分类任务，我们表明，如所预测的那样，高环境稳定性果断地有利于IWL的IWL，并且在最大稳定性下急剧过渡。相反，高提示可靠性增强了ICL功效，尤其是在稳定性较低时。此外，学习动力学揭示了任务执行时间的演变：虽然在某些设置（例如，与许多类别的分类）中发生了规范的ICL到IWL转移，但我们证明，具有更轻松的IWL（例如，较少的类）或较慢的ICL习得（例如，回归）可以表现出最初的IW级时iCL ICL统治的情况（例如，较少的类）或较慢的ICL获取（例如，更较少的ICL）可以表现出。这些发现支持一个相对成本假设，用于解释这些学习模式过渡，将可预测性作为控制变压器中适应性策略的关键因素，并为理解ICL和指导培训方法提供新颖的见解。

Title: Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models

• Authors: Danush Kumar Venkatesh, Isabel Funke, Micha Pfeiffer, Fiona Kolbinger, Hanna Maria Schmeiser, Juergen Weitz, Marius Distler, Stefanie Speidel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.09858
• Pdf URL: https://arxiv.org/pdf/2505.09858
• Copy Paste: [[2505.09858]] Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models(https://arxiv.org/abs/2505.09858)
• Keywords: generation
• Abstract: Computer-assisted interventions can improve intra-operative guidance, particularly through deep learning methods that harness the spatiotemporal information in surgical videos. However, the severe data imbalance often found in surgical video datasets hinders the development of high-performing models. In this work, we aim to overcome the data imbalance by synthesizing surgical videos. We propose a unique two-stage, text-conditioned diffusion-based method to generate high-fidelity surgical videos for under-represented classes. Our approach conditions the generation process on text prompts and decouples spatial and temporal modeling by utilizing a 2D latent diffusion model to capture spatial content and then integrating temporal attention layers to ensure temporal consistency. Furthermore, we introduce a rejection sampling strategy to select the most suitable synthetic samples, effectively augmenting existing datasets to address class imbalance. We evaluate our method on two downstream tasks-surgical action recognition and intra-operative event prediction-demonstrating that incorporating synthetic videos from our approach substantially enhances model performance. We open-source our implementation at this https URL.
• 摘要：计算机辅助的干预措施可以改善术中的指导，尤其是通过利用外科视频中时空信息的深度学习方法。但是，在手术视频数据集中经常发现的严重数据失衡阻碍了高性能模型的发展。在这项工作中，我们旨在通过综合手术视频来克服数据失衡。我们提出了一种独特的两阶段，基于文本的扩散方法，以生成用于代表性不足类的高保真手术视频。我们的方法通过利用2D潜在扩散模型来捕获空间内容，然后集成时间关注层以确保时间一致性，从而在文本提示上的生成过程和脱离空间和时间建模。此外，我们引入了一种拒绝采样策略，以选择最合适的合成样本，从而有效增加现有数据集以解决类不平衡。我们在两个下游任务识别和术中事件预测中评估我们的方法，从我们方法中纳入合成视频可以大大提高模型性能。我们在此HTTPS URL上开源我们的实现。

Title: Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity

• Authors: Zichen Liu, Wei Zhang, Tiejun Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.09922
• Pdf URL: https://arxiv.org/pdf/2505.09922
• Copy Paste: [[2505.09922]] Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity(https://arxiv.org/abs/2505.09922)
• Keywords: generation, generative
• Abstract: Euclidean diffusion models have achieved remarkable success in generative modeling across diverse domains, and they have been extended to manifold case in recent advances. Instead of explicitly utilizing the structure of special manifolds as studied in previous works, we investigate direct sampling of the Euclidean diffusion models for general manifold-constrained data in this paper. We reveal the multiscale singularity of the score function in the embedded space of manifold, which hinders the accuracy of diffusion-generated samples. We then present an elaborate theoretical analysis of the singularity structure of the score function by separating it along the tangential and normal directions of the manifold. To mitigate the singularity and improve the sampling accuracy, we propose two novel methods: (1) Niso-DM, which introduces non-isotropic noise along the normal direction to reduce scale discrepancies, and (2) Tango-DM, which trains only the tangential component of the score function using a tangential-only loss function. Numerical experiments demonstrate that our methods achieve superior performance on distributions over various manifolds with complex geometries.
• 摘要：欧几里得扩散模型在跨不同领域的生成建模方面取得了巨大的成功，并且在最近的进步中已扩展到歧视案例。在本文中，我们研究了欧几里得扩散模型的直接采样，而不是明确利用特殊歧管的结构，用于本文中的一般歧管受限数据。我们揭示了分数函数在歧管的嵌入式空间中的多尺度奇异性，这阻碍了扩散生成的样品的准确性。然后，我们通过沿歧管的切向和正常方向分离得分函数的奇异性结构进行了详尽的理论分析。为了减轻奇异性并提高采样精度，我们提出了两种新颖的方法：（1）NISO-DM，该方法沿正常方向引入了非异向噪声以减少比例差异，以及（2）Tango-DM，该探戈DM仅使用切线量损失函数来训练得分函数的切向组件。数值实验表明，我们的方法比具有复杂几何形状的各种歧管上的分布实现了卓越的性能。

Title: DDFP: Data-dependent Frequency Prompt for Source Free Domain Adaptation of Medical Image Segmentation

• Authors: Siqi Yin, Shaolei Liu, Manning Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.09927
• Pdf URL: https://arxiv.org/pdf/2505.09927
• Copy Paste: [[2505.09927]] DDFP: Data-dependent Frequency Prompt for Source Free Domain Adaptation of Medical Image Segmentation(https://arxiv.org/abs/2505.09927)
• Keywords: generation
• Abstract: Domain adaptation addresses the challenge of model performance degradation caused by domain gaps. In the typical setup for unsupervised domain adaptation, labeled data from a source domain and unlabeled data from a target domain are used to train a target model. However, access to labeled source domain data, particularly in medical datasets, can be restricted due to privacy policies. As a result, research has increasingly shifted to source-free domain adaptation (SFDA), which requires only a pretrained model from the source domain and unlabeled data from the target domain data for adaptation. Existing SFDA methods often rely on domain-specific image style translation and self-supervision techniques to bridge the domain gap and train the target domain model. However, the quality of domain-specific style-translated images and pseudo-labels produced by these methods still leaves room for improvement. Moreover, training the entire model during adaptation can be inefficient under limited supervision. In this paper, we propose a novel SFDA framework to address these challenges. Specifically, to effectively mitigate the impact of domain gap in the initial training phase, we introduce preadaptation to generate a preadapted model, which serves as an initialization of target model and allows for the generation of high-quality enhanced pseudo-labels without introducing extra parameters. Additionally, we propose a data-dependent frequency prompt to more effectively translate target domain images into a source-like style. To further enhance adaptation, we employ a style-related layer fine-tuning strategy, specifically designed for SFDA, to train the target model using the prompted target domain images and pseudo-labels. Extensive experiments on cross-modality abdominal and cardiac SFDA segmentation tasks demonstrate that our proposed method outperforms existing state-of-the-art methods.
• 摘要：域的适应性解决了由域间隙引起的模型性能降解的挑战。在无监督域适应的典型设置中，使用来自源域和来自目标域的未标记数据的标记数据用于训练目标模型。但是，由于隐私政策，访问标记的源域数据，尤其是在医疗数据集中，可能会受到限制。结果，研究越来越多地转移到无源域的适应性（SFDA），该域仅需要从源域中审慎的模型，而来自目标域数据的未标记数据才能进行适应。现有的SFDA方法通常依赖于特定于域的图像样式翻译和自我划分技术来弥合域间隙并训练目标域模型。但是，这些方法生产的域特异性翻译图像和伪标签的质量仍然留出了改进的空间。此外，在有限的监督下，在适应过程中培训整个模型可能会降低。在本文中，我们提出了一个新颖的SFDA框架来应对这些挑战。具体而言，为了有效地减轻域间隙在初始训练阶段的影响，我们引入了预先限制以生成预先启发的模型，该模型可作为目标模型的初始化，并允许在不引入额外参数的情况下生成高质量增强的伪标签。此外，我们提出了一个与数据有关的频率提示，以更有效地将目标域图像转化为类似源的样式。为了进一步增强适应性，我们采用了一种与样式相关的层微观调整策略，专门为SFDA设计，以使用促进的目标域图像和伪标签来训练目标模型。关于腹膜和心脏SFDA分割任务的广泛实验表明，我们所提出的方法的表现优于现有的最新方法。

Title: TransPL: VQ-Code Transition Matrices for Pseudo-Labeling of Time Series Unsupervised Domain Adaptation

• Authors: Jaeho Kim, Seulki Lee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.09955
• Pdf URL: https://arxiv.org/pdf/2505.09955
• Copy Paste: [[2505.09955]] TransPL: VQ-Code Transition Matrices for Pseudo-Labeling of Time Series Unsupervised Domain Adaptation(https://arxiv.org/abs/2505.09955)
• Keywords: generation
• Abstract: Unsupervised domain adaptation (UDA) for time series data remains a critical challenge in deep learning, with traditional pseudo-labeling strategies failing to capture temporal patterns and channel-wise shifts between domains, producing sub-optimal pseudo-labels. As such, we introduce TransPL, a novel approach that addresses these limitations by modeling the joint distribution $P(\mathbf{X}, y)$ of the source domain through code transition matrices, where the codes are derived from vector quantization (VQ) of time series patches. Our method constructs class- and channel-wise code transition matrices from the source domain and employs Bayes' rule for target domain adaptation, generating pseudo-labels based on channel-wise weighted class-conditional likelihoods. TransPL offers three key advantages: explicit modeling of temporal transitions and channel-wise shifts between different domains, versatility towards different UDA scenarios (e.g., weakly-supervised UDA), and explainable pseudo-label generation. We validate TransPL's effectiveness through extensive analysis on four time series UDA benchmarks and confirm that it consistently outperforms state-of-the-art pseudo-labeling methods by a strong margin (6.1% accuracy improvement, 4.9% F1 improvement), while providing interpretable insights into the domain adaptation process through its learned code transition matrices.
• 摘要：时间序列数据的无监督域适应性（UDA）在深度学习中仍然是一个关键挑战，传统的伪标记策略未能捕获域之间的时间模式和渠道的转移，从而产生了次优伪标签。因此，我们介绍了Transpl，这是一种新颖的方法，通过对源域的源域的联合分布$ p（\ mathbf {x}，y）进行建模，通过代码过渡矩阵进行建模，其中代码来自时间序列贴片的向量量化（VQ）。我们的方法从源域构建了类和通道的代码过渡矩阵，并采用了贝叶斯的目标域适应性规则，从而基于渠道加权的类条件可能性生成伪标签。 Transpl提供了三个关键优势：对不同域之间的时间过渡和渠道转移的明确建模，对不同UDA场景的多功能性（例如，弱监督UDA）以及可解释的伪标签生成。我们通过对四个时间序列UDA基准测试的广泛分析来验证Transpl的有效性，并确认它始终优于最先进的伪标记方法，通过强差（6.1％的准确性提高，4.9％的F1提高，提高4.9％F1），同时通过其通过其学识渊博的域代码提供了对域的适应过程的可解释见解。

Title: Sybil-based Virtual Data Poisoning Attacks in Federated Learning

• Authors: Changxun Zhu, Qilong Wu, Lingjuan Lyu, Shibei Xue
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.09983
• Pdf URL: https://arxiv.org/pdf/2505.09983
• Copy Paste: [[2505.09983]] Sybil-based Virtual Data Poisoning Attacks in Federated Learning(https://arxiv.org/abs/2505.09983)
• Keywords: generation
• Abstract: Federated learning is vulnerable to poisoning attacks by malicious adversaries. Existing methods often involve high costs to achieve effective attacks. To address this challenge, we propose a sybil-based virtual data poisoning attack, where a malicious client generates sybil nodes to amplify the poisoning model's impact. To reduce neural network computational complexity, we develop a virtual data generation method based on gradient matching. We also design three schemes for target model acquisition, applicable to online local, online global, and offline scenarios. In simulation, our method outperforms other attack algorithms since our method can obtain a global target model under non-independent uniformly distributed data.
• 摘要：联邦学习容易受到恶意对手的中毒攻击。现有方法通常涉及高成本以实现有效的攻击。为了应对这一挑战，我们提出了一种基于Sybil的虚拟数据中毒攻击，恶意客户生成Sybil节点以扩大中毒模型的影响。为了降低神经网络计算复杂性，我们基于梯度匹配开发了虚拟数据生成方法。我们还设计了三个用于目标模型获取的方案，适用于在线本地，在线全局和离线方案。在仿真中，我们的方法优于其他攻击算法，因为我们的方法可以在非独立统一分布的数据下获得全局目标模型。

Title: From Air to Wear: Personalized 3D Digital Fashion with AR/VR Immersive 3D Sketching

• Authors: Ying Zang, Yuanqi Hu, Xinyu Chen, Yuxia Xu, Suhui Wang, Chunan Yu, Lanyun Zhu, Deyi Ji, Xin Xu, Tianrun Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.09998
• Pdf URL: https://arxiv.org/pdf/2505.09998
• Copy Paste: [[2505.09998]] From Air to Wear: Personalized 3D Digital Fashion with AR/VR Immersive 3D Sketching(https://arxiv.org/abs/2505.09998)
• Keywords: generation
• Abstract: In the era of immersive consumer electronics, such as AR/VR headsets and smart devices, people increasingly seek ways to express their identity through virtual fashion. However, existing 3D garment design tools remain inaccessible to everyday users due to steep technical barriers and limited data. In this work, we introduce a 3D sketch-driven 3D garment generation framework that empowers ordinary users - even those without design experience - to create high-quality digital clothing through simple 3D sketches in AR/VR environments. By combining a conditional diffusion model, a sketch encoder trained in a shared latent space, and an adaptive curriculum learning strategy, our system interprets imprecise, free-hand input and produces realistic, personalized garments. To address the scarcity of training data, we also introduce KO3DClothes, a new dataset of paired 3D garments and user-created sketches. Extensive experiments and user studies confirm that our method significantly outperforms existing baselines in both fidelity and usability, demonstrating its promise for democratized fashion design on next-generation consumer platforms.
• 摘要：在沉浸式消费电子产品（例如AR/VR耳机和智能设备）的时代，人们越来越多地寻求通过虚拟方式表达其身份的方法。但是，由于陡峭的技术障碍和有限的数据，现有的3D服装设计工具仍然无法使用日常用户。在这项工作中，我们介绍了一个3D素描驱动的3D服装生成框架，该框架使普通用户（即使是没有设计经验的人）可以通过AR/VR环境中的简单3D草图创建高质量的数字服装。通过结合有条件的扩散模型，在共享潜在空间中训练的草图编码器以及一种自适应课程学习策略，我们的系统可以解释不精确，自由手的输入并产生现实的个性化服装。为了解决培训数据的稀缺性，我们还介绍了KO3DCLOTHES，这是一个新的配对3D服装和用户创建的草图的数据集。广泛的实验和用户研究证实，我们的方法在忠诚度和可用性方面都大大优于现有基准，这表明了其对下一代消费者平台上民主化时装设计的希望。

Title: ORL-LDM: Offline Reinforcement Learning Guided Latent Diffusion Model Super-Resolution Reconstruction

• Authors: Shijie Lyu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.10027
• Pdf URL: https://arxiv.org/pdf/2505.10027
• Copy Paste: [[2505.10027]] ORL-LDM: Offline Reinforcement Learning Guided Latent Diffusion Model Super-Resolution Reconstruction(https://arxiv.org/abs/2505.10027)
• Keywords: super-resolution
• Abstract: With the rapid advancement of remote sensing technology, super-resolution image reconstruction is of great research and practical significance. Existing deep learning methods have made progress but still face limitations in handling complex scenes and preserving image details. This paper proposes a reinforcement learning-based latent diffusion model (LDM) fine-tuning method for remote sensing image super-resolution. The method constructs a reinforcement learning environment with states, actions, and rewards, optimizing decision objectives through proximal policy optimization (PPO) during the reverse denoising process of the LDM model. Experiments on the RESISC45 dataset show significant improvements over the baseline model in PSNR, SSIM, and LPIPS, with PSNR increasing by 3-4dB, SSIM improving by 0.08-0.11, and LPIPS reducing by 0.06-0.10, particularly in structured and complex natural scenes. The results demonstrate the method's effectiveness in enhancing super-resolution quality and adaptability across scenes.
• 摘要：随着遥感技术的快速发展，超分辨率图像重建具有巨大的研究和实际意义。现有的深度学习方法取得了进步，但仍面临着处理复杂场景并保留图像细节的局限性。本文提出了一种基于增强学习的潜在扩散模型（LDM）微分辨率的微分调用方法。该方法通过状态，行动和奖励构建强化学习环境，在LDM模型的反向denoising过程中通过近端策略优化（PPO）优化决策目标。 RESISC45数据集的实验表明，PSNR，SSIM和LPIPS中基线模型的显着改善，PSNR增加了3-4DB，SSIM提高了0.08-0.11，而LPIPS则在0.06-0.10降低了0.06-0.10，尤其是在结构和复杂的自然场景中。结果证明了该方法在增强场景的超分辨率质量和适应性方面的有效性。

Title: Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis

• Authors: Bingda Tang, Boyang Zheng, Xichen Pan, Sayak Paul, Saining Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10046
• Pdf URL: https://arxiv.org/pdf/2505.10046
• Copy Paste: [[2505.10046]] Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis(https://arxiv.org/abs/2505.10046)
• Keywords: generation
• Abstract: This paper does not describe a new method; instead, it provides a thorough exploration of an important yet understudied design space related to recent advances in text-to-image synthesis -- specifically, the deep fusion of large language models (LLMs) and diffusion transformers (DiTs) for multi-modal generation. Previous studies mainly focused on overall system performance rather than detailed comparisons with alternative methods, and key design details and training recipes were often left undisclosed. These gaps create uncertainty about the real potential of this approach. To fill these gaps, we conduct an empirical study on text-to-image generation, performing controlled comparisons with established baselines, analyzing important design choices, and providing a clear, reproducible recipe for training at scale. We hope this work offers meaningful data points and practical guidelines for future research in multi-modal generation.
• 摘要：本文没有描述一种新方法。取而代之的是，它对与文本对图像合成的最新进展相关的重要而研究的设计空间进行了彻底的探索 - 特别是大型语言模型（LLMS）和扩散变压器（DITS）的深层融合。先前的研究主要集中在整体系统性能上，而不是与替代方法的详细比较，关键的设计细节和培训食谱通常始终不公开。这些差距对这种方法的真正潜力产生了不确定性。为了填补这些空白，我们对文本对图像生成进行了一项实证研究，与已建立的基准进行了受控的比较，分析重要的设计选择，并提供清晰，可重复的食谱以进行大规模培训。我们希望这项工作为多模式生成的未来研究提供有意义的数据点和实用指南。

Title: ToonifyGB: StyleGAN-based Gaussian Blendshapes for 3D Stylized Head Avatars

• Authors: Rui-Yang Ju, Sheng-Yen Huang, Yi-Ping Hung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10072
• Pdf URL: https://arxiv.org/pdf/2505.10072
• Copy Paste: [[2505.10072]] ToonifyGB: StyleGAN-based Gaussian Blendshapes for 3D Stylized Head Avatars(https://arxiv.org/abs/2505.10072)
• Keywords: generation
• Abstract: The introduction of 3D Gaussian blendshapes has enabled the real-time reconstruction of animatable head avatars from monocular video. Toonify, a StyleGAN-based framework, has become widely used for facial image stylization. To extend Toonify for synthesizing diverse stylized 3D head avatars using Gaussian blendshapes, we propose an efficient two-stage framework, ToonifyGB. In Stage 1 (stylized video generation), we employ an improved StyleGAN to generate the stylized video from the input video frames, which addresses the limitation of cropping aligned faces at a fixed resolution as preprocessing for normal StyleGAN. This process provides a more stable video, which enables Gaussian blendshapes to better capture the high-frequency details of the video frames, and efficiently generate high-quality animation in the next stage. In Stage 2 (Gaussian blendshapes synthesis), we learn a stylized neutral head model and a set of expression blendshapes from the generated video. By combining the neutral head model with expression blendshapes, ToonifyGB can efficiently render stylized avatars with arbitrary expressions. We validate the effectiveness of ToonifyGB on the benchmark dataset using two styles: Arcane and Pixar.
• 摘要：3D高斯混合形的引入使单眼视频的动画头像实时重建。基于Stylegan的框架Toonify已被广泛用于面部图像样式化。为了扩展使用高斯杂物形状合成多样化的样式化3D头像的TOONFY，我们提出了一个有效的两阶段框架，即Toonifygb。在第1阶段（风格化的视频生成）中，我们采用了改进的StyleGAN来生成来自输入视频框架的风格化视频，该视频框架以固定分辨率作为正常样式Gangan的固定分辨率来解决裁剪的面孔的限制。此过程提供了一个更稳定的视频，该视频使高斯杂物形状能够更好地捕获视频帧的高频细节，并在下一阶段有效地生成高质量的动画。在第2阶段（高斯混合形成合成）中，我们从生成的视频中学习了一个风格化的中性头模型和一组表达搅拌形状。通过将中性头模型与表达搅拌形态相结合，Toonifygb可以有效地呈现具有任意表达式的样式化的化身。我们使用两种样式在基准数据集上验证ToonifyGB的有效性：Arcane和Pixar。

Title: VolE: A Point-cloud Framework for Food 3D Reconstruction and Volume Estimation

• Authors: Umair Haroon, Ahmad AlMughrabi, Thanasis Zoumpekas, Ricardo Marques, Petia Radeva
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10205
• Pdf URL: https://arxiv.org/pdf/2505.10205
• Copy Paste: [[2505.10205]] VolE: A Point-cloud Framework for Food 3D Reconstruction and Volume Estimation(https://arxiv.org/abs/2505.10205)
• Keywords: generation
• Abstract: Accurate food volume estimation is crucial for medical nutrition management and health monitoring applications, but current food volume estimation methods are often limited by mononuclear data, leveraging single-purpose hardware such as 3D scanners, gathering sensor-oriented information such as depth information, or relying on camera calibration using a reference object. In this paper, we present VolE, a novel framework that leverages mobile device-driven 3D reconstruction to estimate food volume. VolE captures images and camera locations in free motion to generate precise 3D models, thanks to AR-capable mobile devices. To achieve real-world measurement, VolE is a reference- and depth-free framework that leverages food video segmentation for food mask generation. We also introduce a new food dataset encompassing the challenging scenarios absent in the previous benchmarks. Our experiments demonstrate that VolE outperforms the existing volume estimation techniques across multiple datasets by achieving 2.22 % MAPE, highlighting its superior performance in food volume estimation.
• 摘要：准确的食品量估计对于医疗营养管理和健康监测应用至关重要，但是当前的食物量估计方法通常受单核数据的限制，利用单用途硬件（例如3D扫描仪），收集面向传感器的信息，例如深度信息，例如使用参考对象，依靠相机校准。在本文中，我们提出了Vole，这是一个新型框架，利用移动设备驱动的3D重建来估计食品量。 Vole捕获自由运动的图像和相机位置，以生成精确的3D型号，这要归功于具有AR的移动设备。为了实现现实世界的测量，Vole是一个无参考和深度的框架，它利用食物视频分割来生成食物面具。我们还介绍了一个新的食物数据集，其中包括以前的基准测试中缺乏的具有挑战性的情况。我们的实验表明，Vole通过达到2.22％的MAPE来优于多个数据集的现有体积估计技术，从而强调了其在食品体积估计中的卓越性能。

Title: ComplexFormer: Disruptively Advancing Transformer Inference Ability via Head-Specific Complex Vector Attention

• Authors: Jintian Shao, Hongyi Huang, Jiayi Wu, Beiwen Zhang, ZhiYu Wu, You Shan, MingKai Zheng
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2505.10222
• Pdf URL: https://arxiv.org/pdf/2505.10222
• Copy Paste: [[2505.10222]] ComplexFormer: Disruptively Advancing Transformer Inference Ability via Head-Specific Complex Vector Attention(https://arxiv.org/abs/2505.10222)
• Keywords: generation
• Abstract: Transformer models rely on self-attention to capture token dependencies but face challenges in effectively integrating positional information while allowing multi-head attention (MHA) flexibility. Prior methods often model semantic and positional differences disparately or apply uniform positional adjustments across heads, potentially limiting representational capacity. This paper introduces ComplexFormer, featuring Complex Multi-Head Attention-CMHA. CMHA empowers each head to independently model semantic and positional differences unified within the complex plane, representing interactions as rotations and scaling. ComplexFormer incorporates two key improvements: (1) a per-head Euler transformation, converting real-valued query/key projections into polar-form complex vectors for head-specific complex subspace operation; and (2) a per-head adaptive differential rotation mechanism, exp[i(Adapt(ASmn,i) + Delta(Pmn),i)], allowing each head to learn distinct strategies for integrating semantic angle differences (ASmn,i) with relative positional encodings (Delta(Pmn),i). Extensive experiments on language modeling, text generation, code generation, and mathematical reasoning show ComplexFormer achieves superior performance, significantly lower generation perplexity , and improved long-context coherence compared to strong baselines like RoPE-Transformers. ComplexFormer demonstrates strong parameter efficiency, offering a more expressive, adaptable attention mechanism.
• 摘要：变压器模型依靠自我注意力来捕获令牌依赖性，但在有效整合位置信息的同时，在允许多头关注（MHA）灵活性时面临挑战。先前的方法通常对语义和位置差异进行模拟，或者在跨头部进行统一的位置调整，从而可能限制代表能力。本文介绍了复杂形式，具有复杂的多头注意-CMHA。 CMHA赋予每个头部独立模型的语义和位置差异，并在复杂平面内统一，代表相互作用作为旋转和缩放。复合物结合了两个关键改进：（1）每个头欧拉转换，将实值查询/钥匙投影转换为极性复杂的矢量，以进行特定于特异性的复杂子空间操作； （2）exp [i（ASMN，I） + delta（pmn），i）]的每头自适应差异机制，允许每个头部学习与相对位置编码（delta（delta（pmn），i），I）的不同语义角度差异（ASMN，i）的不同策略。与绳索转换器（如绳索转换器）相比，有关语言建模，文本生成，代码生成和数学推理的广泛实验表明，复杂的表现可以达到卓越的性能，显着降低了发电性的困惑，并提高了长期影响的连贯性。 ComplexFormer表现出强大的参数效率，提供了更具表现力，适应性的注意机制。

Title: MTVCrafter: 4D Motion Tokenization for Open-World Human Image Animation

• Authors: Yanbo Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10238
• Pdf URL: https://arxiv.org/pdf/2505.10238
• Copy Paste: [[2505.10238]] MTVCrafter: 4D Motion Tokenization for Open-World Human Image Animation(https://arxiv.org/abs/2505.10238)
• Keywords: generation
• Abstract: Human image animation has gained increasing attention and developed rapidly due to its broad applications in digital humans. However, existing methods rely largely on 2D-rendered pose images for motion guidance, which limits generalization and discards essential 3D information for open-world animation. To tackle this problem, we propose MTVCrafter (Motion Tokenization Video Crafter), the first framework that directly models raw 3D motion sequences (i.e., 4D motion) for human image animation. Specifically, we introduce 4DMoT (4D motion tokenizer) to quantize 3D motion sequences into 4D motion tokens. Compared to 2D-rendered pose images, 4D motion tokens offer more robust spatio-temporal cues and avoid strict pixel-level alignment between pose image and character, enabling more flexible and disentangled control. Then, we introduce MV-DiT (Motion-aware Video DiT). By designing unique motion attention with 4D positional encodings, MV-DiT can effectively leverage motion tokens as 4D compact yet expressive context for human image animation in the complex 3D world. Hence, it marks a significant step forward in this field and opens a new direction for pose-guided human video generation. Experiments show that our MTVCrafter achieves state-of-the-art results with an FID-VID of 6.98, surpassing the second-best by 65%. Powered by robust motion tokens, MTVCrafter also generalizes well to diverse open-world characters (single/multiple, full/half-body) across various styles and scenarios. Our video demos and code are provided in the supplementary material and at this anonymous GitHub link: this https URL.
• 摘要：由于其在数字人类中的广泛应用，人类形象动画已经越来越多地引起人们的关注并迅速发展。但是，现有方法在很大程度上依赖于2D渲染的姿势图像进行运动指导，该图像限制了概括并丢弃开放世界动画的基本3D信息。为了解决这个问题，我们提出了MTVCrafter（运动令牌化视频Crafter），这是第一个直接建模为人类图像动画的原始3D运动序列（即4D运动）的框架。具体而言，我们引入4DMOT（4D运动令牌）将3D运动序列量化为4D运动令牌。与2D渲染的姿势图像相比，4D运动令牌提供了更强大的时空提示，并避免了姿势图像和性格之间严格的像素级对齐，从而实现了更灵活和更不合格的控制。然后，我们介绍MV-DIT（运动吸引视频DIT）。通过使用4D位置编码设计独特的运动注意力，MV-DIT可以有效利用运动令牌作为4D紧凑而表达的上下文，用于复杂的3D世界中人类图像动画。因此，它标志着这一领域迈出的重要一步，并为姿势引导的人类视频产生打开了一个新的方向。实验表明，我们的MTVCrafter以6.98的FID-VID实现了最先进的结果，超过了第二好的65％。 MTVCrafter由强大的运动代币提供支持，还可以很好地推广到各种样式和场景的各种开放世界角色（单/多，全/半身）。我们的视频演示和代码在补充材料中以及此匿名github链接中提供：此HTTPS URL。

Title: StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation

• Authors: Daniel A. P. Oliveira, David Martins de Matos
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2505.10292
• Pdf URL: https://arxiv.org/pdf/2505.10292
• Copy Paste: [[2505.10292]] StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation(https://arxiv.org/abs/2505.10292)
• Keywords: generation
• Abstract: Visual storytelling systems struggle to maintain character identity across frames and link actions to appropriate subjects, frequently leading to referential hallucinations. These issues can be addressed through grounding of characters, objects, and other entities on the visual elements. We propose StoryReasoning, a dataset containing 4,178 stories derived from 52,016 movie images, with both structured scene analyses and grounded stories. Each story maintains character and object consistency across frames while explicitly modeling multi-frame relationships through structured tabular representations. Our approach features cross-frame object re-identification using visual similarity and face recognition, chain-of-thought reasoning for explicit narrative modeling, and a grounding scheme that links textual elements to visual entities across multiple frames. We establish baseline performance by fine-tuning Qwen2.5-VL 7B, creating Qwen Storyteller, which performs end-to-end object detection, re-identification, and landmark detection while maintaining consistent object references throughout the story. Evaluation demonstrates a reduction from 4.06 to 3.56 (-12.3%) hallucinations on average per story when compared to a non-fine-tuned model.
• 摘要：视觉讲故事的系统难以维持跨框架的性格身份，并将动作链接到适当的主题，经常导致参考幻觉。这些问题可以通过视觉元素上的字符，对象和其他实体的接地来解决。我们提出了故事策划，这是一个数据集，其中包含来自52,016部电影图像的4,178个故事，并进行了结构化场景分析和扎根的故事。每个故事都保持跨帧的性格和对象一致性，同时通过结构化表达表示多框架关系。我们的方法具有跨框架对象的重新识别，该对象使用视觉相似性和面部识别，对明确叙事建模的经过想法的推理以及将文本元素链接到跨多个帧的视觉实体的基础方案。我们通过微调QWEN2.5-VL 7B来建立基线性能，创建QWEN讲故事的人，该Qwen讲故事的人执行端到端对象检测，重新识别和具有里程碑意义的检测，同时在整个故事中保持一致的对象参考。评估表明，与非最佳模型相比，每个故事的平均幻觉从4.06（-12.3％）降低到3.56（-12.3％）。

Title: A Representation Learning Approach to Feature Drift Detection in Wireless Networks

• Authors: Athanasios Tziouvaras, Blaz Bertalanic, George Floros, Kostas Kolomvatsos, Panagiotis Sarigiannidis, Carolina Fortuna
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.10325
• Pdf URL: https://arxiv.org/pdf/2505.10325
• Copy Paste: [[2505.10325]] A Representation Learning Approach to Feature Drift Detection in Wireless Networks(https://arxiv.org/abs/2505.10325)
• Keywords: generation
• Abstract: AI is foreseen to be a centerpiece in next generation wireless networks enabling enabling ubiquitous communication as well as new services. However, in real deployment, feature distribution changes may degrade the performance of AI models and lead to undesired behaviors. To counter for undetected model degradation, we propose ALERT; a method that can detect feature distribution changes and trigger model re-training that works well on two wireless network use cases: wireless fingerprinting and link anomaly detection. ALERT includes three components: representation learning, statistical testing and utility assessment. We rely on MLP for designing the representation learning component, on Kolmogorov-Smirnov and Population Stability Index tests for designing the statistical testing and a new function for utility assessment. We show the superiority of the proposed method against ten standard drift detection methods available in the literature on two wireless network use cases.
• 摘要：AI可以预见到下一代无线网络的核心，从而实现无处不在的通信以及新服务。但是，在实际部署中，特征分布更改可能会降低AI模型的性能并导致不希望的行为。为了抵制未发现的模型退化，我们提出警报；一种可以检测特征分布变化并触发模型重新训练的方法，该方法在两个无线网络用例中效果很好：无线指纹和链接异常检测。警报包括三个组件：表示学习，统计测试和公用事业评估。我们依靠MLP来设计表示表示组件，Kolmogorov-Smirnov和人群稳定指数测试，用于设计统计测试和用于实用程序评估的新功能。我们展示了所提出的方法的优越性，即有关两个无线网络用例的文献中可用的十种标准漂移检测方法。

Title: FactsR: A Safer Method for Producing High Quality Healthcare Documentation

• Authors: Victor Petrén Bach Hansen, Lasse Krogsbøll, Jonas Lyngsø, Mathias Baltzersen, Andreas Motzfeldt, Kevin Pelgrims, Lars Maaløe
• Subjects: cs.LG, cs.AI, stat.AP
• Abstract URL: https://arxiv.org/abs/2505.10360
• Pdf URL: https://arxiv.org/pdf/2505.10360
• Copy Paste: [[2505.10360]] FactsR: A Safer Method for Producing High Quality Healthcare Documentation(https://arxiv.org/abs/2505.10360)
• Keywords: generation
• Abstract: There are now a multitude of AI-scribing solutions for healthcare promising the utilization of large language models for ambient documentation. However, these AI scribes still rely on one-shot, or few-shot prompts for generating notes after the consultation has ended, employing little to no reasoning. This risks long notes with an increase in hallucinations, misrepresentation of the intent of the clinician, and reliance on the proofreading of the clinician to catch errors. A dangerous combination for patient safety if vigilance is compromised by workload and fatigue. In this paper, we introduce a method for extracting salient clinical information in real-time alongside the healthcare consultation, denoted Facts, and use that information recursively to generate the final note. The FactsR method results in more accurate and concise notes by placing the clinician-in-the-loop of note generation, while opening up new use cases within real-time decision support.
• 摘要：现在有多种AI标记解决方案用于医疗保健，有望利用大型语言模型用于环境文档。但是，这些AI抄写员仍然依靠单枪或很少的提示来在咨询结束后生成笔记，几乎没有理由。这有可能长期注意到幻觉的增加，对临床医生意图的虚假陈述以及依靠临床医生遇到错误的校对。如果工作量和疲劳损害了警惕性，则危险的组合，以实现患者的安全。在本文中，我们介绍了一种与医疗保健咨询一起实时提取显着临床信息的方法，表示事实，并递归使用该信息来生成最终音符。 FACTSR方法通过将纸币生成的临床医生放置在实时决策支持中，从而产生更准确和简洁的说明。

Title: Two-Stage Generative Model for Intracranial Aneurysm Meshes with Morphological Marker Conditioning

• Authors: Wenhao Ding, Choon Hwai Yap, Kangjun Ji, Simão Castro
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.10407
• Pdf URL: https://arxiv.org/pdf/2505.10407
• Copy Paste: [[2505.10407]] Two-Stage Generative Model for Intracranial Aneurysm Meshes with Morphological Marker Conditioning(https://arxiv.org/abs/2505.10407)
• Keywords: generation, generative
• Abstract: A generative model for the mesh geometry of intracranial aneurysms (IA) is crucial for training networks to predict blood flow forces in real time, which is a key factor affecting disease progression. This need is necessitated by the absence of a large IA image datasets. Existing shape generation methods struggle to capture realistic IA features and ignore the relationship between IA pouches and parent vessels, limiting physiological realism and their generation cannot be controlled to have specific morphological measurements. We propose AneuG, a two-stage Variational Autoencoder (VAE)-based IA mesh generator. In the first stage, AneuG generates low-dimensional Graph Harmonic Deformation (GHD) tokens to encode and reconstruct aneurysm pouch shapes, constrained to morphing energy statistics truths. GHD enables more accurate shape encoding than alternatives. In the second stage, AneuG generates parent vessels conditioned on GHD tokens, by generating vascular centreline and propagating the cross-section. AneuG's IA shape generation can further be conditioned to have specific clinically relevant morphological measurements. This is useful for studies to understand shape variations represented by clinical measurements, and for flow simulation studies to understand effects of specific clinical shape parameters on fluid dynamics. Source code and implementation details are available at this https URL.
• 摘要：颅内动脉瘤（IA）网格几何形状的生成模型对于训练网络实时预测血流量至关重要，这是影响疾病进展的关键因素。由于缺乏大型IA图像数据集，需要这一需求。现有的形状生成方法难以捕获现实的IA特征，并忽略IA袋与母船之间的关系，限制生理现实主义及其代表无法控制特定的形态测量。我们提出了基于两阶段的自动编码器（VAE）的IA网状发电机Aneug。在第一阶段，Aneug产生了低维图谐波变形（GHD）令牌以编码和重建动脉瘤袋形状，这些囊囊形状受到变形能量统计的真相。 GHD比替代方案实现更准确的形状编码。在第二阶段，Aneug通过产生血管中心线并传播横截面来生成以GHD令牌为条件的母体。可以进一步调节Aneug的IA形状产生，以进行特定的临床相关形态测量。这对于了解以临床测量为代表的形状变化以及流动模拟研究以了解特定临床形状参数对流体动力学的影响很有用。源代码和实现详细信息可在此HTTPS URL上找到。

Title: Score-based diffusion nowcasting of GOES imagery

• Authors: Randy J. Chase, Katherine Haynes, Lander Ver Hoef, Imme Ebert-Uphoff
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2505.10432
• Pdf URL: https://arxiv.org/pdf/2505.10432
• Copy Paste: [[2505.10432]] Score-based diffusion nowcasting of GOES imagery(https://arxiv.org/abs/2505.10432)
• Keywords: generation
• Abstract: Clouds and precipitation are important for understanding weather and climate. Simulating clouds and precipitation with traditional numerical weather prediction is challenging because of the sub-grid parameterizations required. Machine learning has been explored for forecasting clouds and precipitation, but early machine learning methods often created blurry forecasts. In this paper we explore a newer method, named score-based diffusion, to nowcast (zero to three hour forecast) clouds and precipitation. We discuss the background and intuition of score-based diffusion models - thus providing a starting point for the community - while exploring the methodology's use for nowcasting geostationary infrared imagery. We experiment with three main types of diffusion models: a standard score-based diffusion model (Diff); a residual correction diffusion model (CorrDiff); and a latent diffusion model (LDM). Our results show that the diffusion models are able to not only advect existing clouds, but also generate and decay clouds, including convective initiation. These results are surprising because the forecasts are initiated with only the past 20 mins of infrared satellite imagery. A case study qualitatively shows the preservation of high resolution features longer into the forecast than a conventional mean-squared error trained U-Net. The best of the three diffusion models tested was the CorrDiff approach, outperforming all other diffusion models, the traditional U-Net, and a persistence forecast by one to two kelvin on root mean squared error. The diffusion models also enable out-of-the-box ensemble generation, which shows skillful calibration, with the spread of the ensemble correlating well to the error.
• 摘要：云和降水对于理解天气和气候很重要。由于所需的子网格参数化，通过传统的数值天气预测模拟云和降水是具有挑战性的。已经探索了用于预测云和降水的机器学习，但是早期的机器学习方法通​​常会产生模糊的预测。在本文中，我们探讨了一种较新的方法，即基于得分的扩散，以将云和降水量（零至三小时）和降水。我们讨论了基于得分的扩散模型的背景和直觉，从而为社区提供了一个起点 - 同时探索了方法论对现实的地理红外图像的使用。我们尝试三种主要类型的扩散模型：基于标准的基于得分的扩散模型（DIFF）；残留校正扩散模型（CORRDIFF）；和潜在扩散模型（LDM）。我们的结果表明，扩散模型不仅能够排除现有云​​，还可以产生和衰减云，包括对流启动。这些结果令人惊讶，因为预测仅在过去20分钟的红外卫星图像中启动。案例研究定性地表明，与常规的于点误差训练的U-NET相比，预测中的高分辨率特征的保存更长。在测试的三个扩散模型中，最好的是Corrdiff方法，表现优于所有其他扩散模型，传统的U-NET，并且在根平方误差上，一到两个Kelvin的持久性预测。扩散模型还启用了开箱即用的集合生成，该集合的生成显示了熟练的校准，合奏的传播与误差息息相关。

Title: Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AI

• Authors: Agnik Saha, Victoria Churchill, Anny D. Rodriguez, Ugur Kursuncu, Muhammed Y. Idris
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.10472
• Pdf URL: https://arxiv.org/pdf/2505.10472
• Copy Paste: [[2505.10472]] Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AI(https://arxiv.org/abs/2505.10472)
• Keywords: generative
• Abstract: Effective communication about breast and cervical cancers remains a persistent health challenge, with significant gaps in public understanding of cancer prevention, screening, and treatment, potentially leading to delayed diagnoses and inadequate treatments. This study evaluates the capabilities and limitations of Large Language Models (LLMs) in generating accurate, safe, and accessible cancer-related information to support patient understanding. We evaluated five general-purpose and three medical LLMs using a mixed-methods evaluation framework across linguistic quality, safety and trustworthiness, and communication accessibility and affectiveness. Our approach utilized quantitative metrics, qualitative expert ratings, and statistical analysis using Welch's ANOVA, Games-Howell, and Hedges' g. Our results show that general-purpose LLMs produced outputs of higher linguistic quality and affectiveness, while medical LLMs demonstrate greater communication accessibility. However, medical LLMs tend to exhibit higher levels of potential harm, toxicity, and bias, reducing their performance in safety and trustworthiness. Our findings indicate a duality between domain-specific knowledge and safety in health communications. The results highlight the need for intentional model design with targeted improvements, particularly in mitigating harm and bias, and improving safety and affectiveness. This study provides a comprehensive evaluation of LLMs for cancer communication, offering critical insights for improving AI-generated health content and informing future development of accurate, safe, and accessible digital health tools.
• 摘要：关于乳腺癌和宫颈癌的有效沟通仍然是一项持续的健康挑战，在公众对预防癌症，筛查和治疗方面的理解差距很大，可能导致诊断延迟和治疗不足。这项研究评估了大语言模型（LLM）的能力和局限性，以产生准确，安全和可访问的癌症相关信息以支持患者的理解。我们使用混合方法评估框架评估了五个通用和三个医学LLM，涉及语言质量，安全性和可信赖性以及通信可及性和情感性。我们的方法利用了使用Welch的ANOVA，Games-Howell和Hedges'G使用定量指标，定性专家评级和统计分析。我们的结果表明，通用LLMS产生了更高语言质量和情感性的产出，而医学LLM则表现出更大的通信可访问性。但是，医学LLM倾向于表现出更高水平的潜在伤害，毒性和偏见，从而降低了其在安全性和可信度方面的表现。我们的发现表明，在健康通信中，特定于领域的知识与安全性之间的二元性。结果突出了有针对性改进的有意模型设计的需求，尤其是在减轻危害和偏见方面，并提高安全性和情感。这项研究提供了对癌症交流LLM的全面评估，为改善AI生成的健康内容提供了关键见解，并为未来的准确，安全且可访问的数字健康工具提供了未来的开发。

Title: UniEval: Unified Holistic Evaluation for Unified Multimodal Understanding and Generation

• Authors: Yi Li, Haonan Wang, Qixiang Zhang, Boyu Xiao, Chenchang Hu, Hualiang Wang, Xiaomeng Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.10483
• Pdf URL: https://arxiv.org/pdf/2505.10483
• Copy Paste: [[2505.10483]] UniEval: Unified Holistic Evaluation for Unified Multimodal Understanding and Generation(https://arxiv.org/abs/2505.10483)
• Keywords: generation
• Abstract: The emergence of unified multimodal understanding and generation models is rapidly attracting attention because of their ability to enhance instruction-following capabilities while minimizing model redundancy. However, there is a lack of a unified evaluation framework for these models, which would enable an elegant, simplified, and overall evaluation. Current models conduct evaluations on multiple task-specific benchmarks, but there are significant limitations, such as the lack of overall results, errors from extra evaluation models, reliance on extensive labeled images, benchmarks that lack diversity, and metrics with limited capacity for instruction-following evaluation. To tackle these challenges, we introduce UniEval, the first evaluation framework designed for unified multimodal models without extra models, images, or annotations. This facilitates a simplified and unified evaluation process. The UniEval framework contains a holistic benchmark, UniBench (supports both unified and visual generation models), along with the corresponding UniScore metric. UniBench includes 81 fine-grained tags contributing to high diversity. Experimental results indicate that UniBench is more challenging than existing benchmarks, and UniScore aligns closely with human evaluations, surpassing current metrics. Moreover, we extensively evaluated SoTA unified and visual generation models, uncovering new insights into Univeral's unique values.
• 摘要：统一的多模式理解和生成模型的出现引起了人们的关注，因为它们具有增强指导遵循能力的能力，同时最大程度地减少模型冗余。但是，对于这些模型缺乏统一的评估框架，这将使优雅，简化和整体评估能够进行。当前的模型对多个特定于任务的基准进行了评估，但是存在重大局限性，例如缺乏总体结果，额外评估模型的错误，依赖广泛标记的图像，缺乏多样性的基准和指标能力有限的指导依据索引评估。为了应对这些挑战，我们引入了Unieval，这是为统一的多模型设计的第一个评估框架，而没有额外的模型，图像或注释。这有助于简化而统一的评估过程。 Unieval框架包含一个整体基准，Unibench（支持统一和视觉生成模型），以及相应的Uniscore度量。 Unibench包括81个促成高度多样性的细粒标签。实验结果表明，Unibench比现有基准比现有基准更具挑战性，并且联合国与人类评估紧密保持一致，超过了当前指标。此外，我们广泛评估了SOTA统一和视觉生成模型，发现了对Univeral独特价值的新见解。

Title: RouteNator: A Router-Based Multi-Modal Architecture for Generating Synthetic Training Data for Function Calling LLMs

• Authors: Vibha Belavadi, Tushar Vatsa, Dewang Sultania, Suhas Suresha, Ishita Verma, Cheng Chen, Tracy Holloway King, Michael Friedrich
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2505.10495
• Pdf URL: https://arxiv.org/pdf/2505.10495
• Copy Paste: [[2505.10495]] RouteNator: A Router-Based Multi-Modal Architecture for Generating Synthetic Training Data for Function Calling LLMs(https://arxiv.org/abs/2505.10495)
• Keywords: generation
• Abstract: This paper addresses fine-tuning Large Language Models (LLMs) for function calling tasks when real user interaction data is unavailable. In digital content creation tools, where users express their needs through natural language queries that must be mapped to API calls, the lack of real-world task-specific data and privacy constraints for training on it necessitate synthetic data generation. Existing approaches to synthetic data generation fall short in diversity and complexity, failing to replicate real-world data distributions and leading to suboptimal performance after LLM fine-tuning. We present a novel router-based architecture that leverages domain resources like content metadata and structured knowledge graphs, along with text-to-text and vision-to-text language models to generate high-quality synthetic training data. Our architecture's flexible routing mechanism enables synthetic data generation that matches observed real-world distributions, addressing a fundamental limitation of traditional approaches. Evaluation on a comprehensive set of real user queries demonstrates significant improvements in both function classification accuracy and API parameter selection. Models fine-tuned with our synthetic data consistently outperform traditional approaches, establishing new benchmarks for function calling tasks.
• 摘要：本文在不可用的实际用户交互数据时介绍了用于函数调用任务的微型语言模型（LLMS）。在数字内容创建工具中，用户通过必须映射到API呼叫的自然语言查询表达需求，缺乏针对特定任务的数据和对其进行培训的隐私限制，需要综合数据生成。现有的综合数据生成方法的多样性和复杂性缺乏，未能复制现实世界的数据分布，并在LLM微调后导致次优性能。我们提出了一种基于路由器的新型体系结构，该体系结构利用内容元数据和结构化知识图，以及文本对文本和视觉对文本语言模型，以生成高质量的合成训练数据。我们体系结构的灵活路由机制可实现与观察到的现实世界分布相匹配的综合数据生成，从而解决了传统方法的基本限制。对一组真实用户查询的评估表明，功能分类精度和API参数选择都有显着改善。使用我们的合成数据微调的模型始终超过传统方法，建立了用于函数调用任务的新基准。

Title: CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs

• Authors: Raman Dutt, Pedro Sanchez, Yongchen Yao, Steven McDonagh, Sotirios A. Tsaftaris, Timothy Hospedales
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10496
• Pdf URL: https://arxiv.org/pdf/2505.10496
• Copy Paste: [[2505.10496]] CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs(https://arxiv.org/abs/2505.10496)
• Keywords: generation, generative
• Abstract: We introduce CheXGenBench, a rigorous and multifaceted evaluation framework for synthetic chest radiograph generation that simultaneously assesses fidelity, privacy risks, and clinical utility across state-of-the-art text-to-image generative models. Despite rapid advancements in generative AI for real-world imagery, medical domain evaluations have been hindered by methodological inconsistencies, outdated architectural comparisons, and disconnected assessment criteria that rarely address the practical clinical value of synthetic samples. CheXGenBench overcomes these limitations through standardised data partitioning and a unified evaluation protocol comprising over 20 quantitative metrics that systematically analyse generation quality, potential privacy vulnerabilities, and downstream clinical applicability across 11 leading text-to-image architectures. Our results reveal critical inefficiencies in the existing evaluation protocols, particularly in assessing generative fidelity, leading to inconsistent and uninformative comparisons. Our framework establishes a standardised benchmark for the medical AI community, enabling objective and reproducible comparisons while facilitating seamless integration of both existing and future generative models. Additionally, we release a high-quality, synthetic dataset, SynthCheX-75K, comprising 75K radiographs generated by the top-performing model (Sana 0.6B) in our benchmark to support further research in this critical domain. Through CheXGenBench, we establish a new state-of-the-art and release our framework, models, and SynthCheX-75K dataset at this https URL
• 摘要：我们介绍了Chexgenbench，这是一个严格且多方面的评估框架，用于合成胸部X光片生成，同时评估了最先进的文本对图像生成模型的保真度，隐私风险和临床实用性。尽管用于现实世界图像的生成AI的快速发展，但医学领域的评估受到方法论上的不一致，过时的结构比较和断开评估标准的阻碍，这些评估标准很少解决合成样本的实际临床价值。 Chexgenbench通过标准化的数据分配和一个统一的评估协议来克服这些局限性，其中包括20个定量指标，这些定量指标可以系统地分析发电质量，潜在的隐私脆弱性以及在11个领先的文本形象到图像架构中的下游临床适用性。我们的结果揭示了现有评估方案中的关键效率低下，尤其是在评估生成忠诚度时，导致不一致且不了解的比较。我们的框架为医疗AI社区建立了标准化的基准，在促进现有生成模型和未来生成模型的无缝集成的同时，进行了客观和可重复的比较。此外，我们在我们的基准测试中发布了由最佳模型（SANA 0.6B）生成的75K X光片，释放高质量的合成数据集，合成数据集，包括75K X光片，以支持该关键领域的进一步研究。通过ChexGenbench，我们在此HTTPS URL上建立了一个新的最新技术，并发布我们的框架，模型和Synthchex-75K数据集

Title: Pharmacophore-Conditioned Diffusion Model for Ligand-Based De Novo Drug Design

• Authors: Amira Alakhdar, Barnabas Poczos, Newell Washburn
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.10545
• Pdf URL: https://arxiv.org/pdf/2505.10545
• Copy Paste: [[2505.10545]] Pharmacophore-Conditioned Diffusion Model for Ligand-Based De Novo Drug Design(https://arxiv.org/abs/2505.10545)
• Keywords: generation, generative
• Abstract: Developing bioactive molecules remains a central, time- and cost-heavy challenge in drug discovery, particularly for novel targets lacking structural or functional data. Pharmacophore modeling presents an alternative for capturing the key features required for molecular bioactivity against a biological target. In this work, we present PharmaDiff, a pharmacophore-conditioned diffusion model for 3D molecular generation. PharmaDiff employs a transformer-based architecture to integrate an atom-based representation of the 3D pharmacophore into the generative process, enabling the precise generation of 3D molecular graphs that align with predefined pharmacophore hypotheses. Through comprehensive testing, PharmaDiff demonstrates superior performance in matching 3D pharmacophore constraints compared to ligand-based drug design methods. Additionally, it achieves higher docking scores across a range of proteins in structure-based drug design, without the need for target protein structures. By integrating pharmacophore modeling with 3D generative techniques, PharmaDiff offers a powerful and flexible framework for rational drug design.
• 摘要：在药物发现中，开发生物活性分子仍然是一个核心，时间和成本较高的挑战，特别是对于缺乏结构或功能数据的新靶标。药效团建模为捕获针对生物学靶标的分子生物活性所需的关键特征提供了替代方法。在这项工作中，我们提出了Pharmadiff，这是一种用于3D分子生成的药效团条件扩散模型。 Pharmadiff采用基于变压器的结构来将基于原子的3D药剂团的基于原子的表示形式集成到生成过程中，从而使3D分子图的精确生成与预定义的药物团假设相一致。通过全面的测试，与基于配体的药物设计方法相比，药物库在匹配3D药效团限制方面表现出卓越的性能。此外，它在基于结构的药物设计中的一系列蛋白质中达到了较高的对接得分，而无需目标蛋白质结构。通过将药效团建模与3D生成技术集成，药物提供了一个强大而灵活的框架，用于理性药物设计。

Title: End-to-End Vision Tokenizer Tuning

• Authors: Wenxuan Wang, Fan Zhang, Yufeng Cui, Haiwen Diao, Zhuoyan Luo, Huchuan Lu, Jing Liu, Xinlong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10562
• Pdf URL: https://arxiv.org/pdf/2505.10562
• Copy Paste: [[2505.10562]] End-to-End Vision Tokenizer Tuning(https://arxiv.org/abs/2505.10562)
• Keywords: generation
• Abstract: Existing vision tokenization isolates the optimization of vision tokenizers from downstream training, implicitly assuming the visual tokens can generalize well across various tasks, e.g., image generation and visual question answering. The vision tokenizer optimized for low-level reconstruction is agnostic to downstream tasks requiring varied representations and semantics. This decoupled paradigm introduces a critical misalignment: The loss of the vision tokenization can be the representation bottleneck for target tasks. For example, errors in tokenizing text in a given image lead to poor results when recognizing or generating them. To address this, we propose ETT, an end-to-end vision tokenizer tuning approach that enables joint optimization between vision tokenization and target autoregressive tasks. Unlike prior autoregressive models that use only discrete indices from a frozen vision tokenizer, ETT leverages the visual embeddings of the tokenizer codebook, and optimizes the vision tokenizers end-to-end with both reconstruction and caption objectives. ETT can be seamlessly integrated into existing training pipelines with minimal architecture modifications. Our ETT is simple to implement and integrate, without the need to adjust the original codebooks or architectures of the employed large language models. Extensive experiments demonstrate that our proposed end-to-end vision tokenizer tuning unlocks significant performance gains, i.e., 2-6% for multimodal understanding and visual generation tasks compared to frozen tokenizer baselines, while preserving the original reconstruction capability. We hope this very simple and strong method can empower multimodal foundation models besides image generation and understanding.
• 摘要：现有的视觉令牌化将视力令牌从下游训练中隔离开来，隐含地假设视觉令牌可以很好地跨越各种任务，例如图像产生和视觉问题答案。为低水平重建优化的视觉令牌剂对需要各种表示和语义的下游任务不可知。这种解耦范式引入了严重的错位：视力令牌化的丧失可能是目标任务的表示瓶颈。例如，在给定图像中的文本中，识别或生成它们时会导致结果不佳。为了解决这个问题，我们提出了ETT，这是一种端到端的视觉令牌调整方法，可以在视觉令牌和目标自回归任务之间进行联合优化。与以前仅使用来自冷冻视觉令牌的离散索引的自回旋模型不同，ETT利用了令牌码书的视觉嵌入，并通过重建和字幕目标优化了视觉令牌的端到端。可以将ETT无缝集成到具有最小架构修改的现有培训管道中。我们的ETT易于实现和集成，而无需调整所采用的大语言模型的原始代码手册或架构。广泛的实验表明，我们提出的端到端视力令牌器调整可解锁显着的性能增长，即与冷冻的令牌基线相比，多模式理解和视觉生成任务为2-6％，同时保留原始的重建能力。我们希望这种非常简单且强大的方法可以增强除图像生成和理解外的多模式基础模型。

Title: Depth Anything with Any Prior

• Authors: Zehan Wang, Siyu Chen, Lihe Yang, Jialei Wang, Ziang Zhang, Hengshuang Zhao, Zhou Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10565
• Pdf URL: https://arxiv.org/pdf/2505.10565
• Copy Paste: [[2505.10565]] Depth Anything with Any Prior(https://arxiv.org/abs/2505.10565)
• Keywords: super-resolution
• Abstract: This work presents Prior Depth Anything, a framework that combines incomplete but precise metric information in depth measurement with relative but complete geometric structures in depth prediction, generating accurate, dense, and detailed metric depth maps for any scene. To this end, we design a coarse-to-fine pipeline to progressively integrate the two complementary depth sources. First, we introduce pixel-level metric alignment and distance-aware weighting to pre-fill diverse metric priors by explicitly using depth prediction. It effectively narrows the domain gap between prior patterns, enhancing generalization across varying scenarios. Second, we develop a conditioned monocular depth estimation (MDE) model to refine the inherent noise of depth priors. By conditioning on the normalized pre-filled prior and prediction, the model further implicitly merges the two complementary depth sources. Our model showcases impressive zero-shot generalization across depth completion, super-resolution, and inpainting over 7 real-world datasets, matching or even surpassing previous task-specific methods. More importantly, it performs well on challenging, unseen mixed priors and enables test-time improvements by switching prediction models, providing a flexible accuracy-efficiency trade-off while evolving with advancements in MDE models.
• 摘要：这项工作介绍了事先深度，该框架将不完整但精确的度量信息深入测量与相对但完整的几何结构进行了深入预测，从而为任何场景生成了准确，密集和详细的度量深度图。为此，我们设计了一条粗到精细的管道，以逐步整合两个互补的深度源。首先，我们通过使用深度预测明确地介绍了像素级度量对准和距离感知权重，以明确地进行预填充多样的度量先验。它有效地缩小了先前模式之间的域间隙，从而增强了各种情况的概括。其次，我们开发了条件的单眼深度估计（MDE）模型，以完善深度先验的固有噪声。通过对标准化的预先填充的先验和预测进行调节，该模型进一步隐含了两个互补的深度来源。我们的模型在深度完成，超分辨率和在7个现实世界数据集上的介绍中展示了令人印象深刻的零击概括，匹配甚至超过了以前的特定于任务的方法。更重要的是，它在具有挑战性的，看不见的混合先验方面表现良好，并通过切换预测模型来实现测试时间的改进，从而在MDE模型中的进步中提供了灵活的准确性效率折衷。

Title: 3D-Fixup: Advancing Photo Editing with 3D Priors

• Authors: Yen-Chi Cheng, Krishna Kumar Singh, Jae Shin Yoon, Alex Schwing, Liangyan Gui, Matheus Gadelha, Paul Guerrero, Nanxuan Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.10566
• Pdf URL: https://arxiv.org/pdf/2505.10566
• Copy Paste: [[2505.10566]] 3D-Fixup: Advancing Photo Editing with 3D Priors(https://arxiv.org/abs/2505.10566)
• Keywords: generation, generative
• Abstract: Despite significant advances in modeling image priors via diffusion models, 3D-aware image editing remains challenging, in part because the object is only specified via a single image. To tackle this challenge, we propose 3D-Fixup, a new framework for editing 2D images guided by learned 3D priors. The framework supports difficult editing situations such as object translation and 3D rotation. To achieve this, we leverage a training-based approach that harnesses the generative power of diffusion models. As video data naturally encodes real-world physical dynamics, we turn to video data for generating training data pairs, i.e., a source and a target frame. Rather than relying solely on a single trained model to infer transformations between source and target frames, we incorporate 3D guidance from an Image-to-3D model, which bridges this challenging task by explicitly projecting 2D information into 3D space. We design a data generation pipeline to ensure high-quality 3D guidance throughout training. Results show that by integrating these 3D priors, 3D-Fixup effectively supports complex, identity coherent 3D-aware edits, achieving high-quality results and advancing the application of diffusion models in realistic image manipulation. The code is provided at this https URL
• 摘要：尽管通过扩散模型对图像先验进行建模方面取得了重大进展，但3D感知图像编辑仍然具有挑战性，部分原因是该对象仅通过单个图像指定。为了应对这一挑战，我们提出了3D-fixup，这是一个新的框架，用于编辑由学习的3D先验指导的2D图像。该框架支持困难的编辑情况，例如对象翻译和3D旋转。为了实现这一目标，我们利用一种基于培训的方法来利用扩散模型的生成力量。当视频数据自然编码现实世界的物理动力学时，我们转向视频数据来生成培训数据对，即源和目标框架。我们不仅依靠一个受过训练的模型来推断源和目标框架之间的转换，而是从图像到3D模型中纳入了3D指导，该指导通过将2D信息明确投影到3D空间来弥合这项具有挑战性的任务。我们设计了数据生成管道，以确保在整个培训中确保高质量的3D指导。结果表明，通过整合这些3D先验，3D-FixUp有效地支持复杂的身份相干3D感知的编辑，从而实现了高质量的结果，并推进了扩散模型在现实图像操纵中的应用。该代码在此HTTPS URL上提供