2025-04-03

Title: Gaze-Guided 3D Hand Motion Prediction for Detecting Intent in Egocentric Grasping Tasks

• Authors: Yufei He, Xucong Zhang, Arno H. A. Stienen
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2504.01024
• Pdf URL: https://arxiv.org/pdf/2504.01024
• Copy Paste: [[2504.01024]] Gaze-Guided 3D Hand Motion Prediction for Detecting Intent in Egocentric Grasping Tasks(https://arxiv.org/abs/2504.01024)
• Keywords: generative
• Abstract: Human intention detection with hand motion prediction is critical to drive the upper-extremity assistive robots in neurorehabilitation applications. However, the traditional methods relying on physiological signal measurement are restrictive and often lack environmental context. We propose a novel approach that predicts future sequences of both hand poses and joint positions. This method integrates gaze information, historical hand motion sequences, and environmental object data, adapting dynamically to the assistive needs of the patient without prior knowledge of the intended object for grasping. Specifically, we use a vector-quantized variational autoencoder for robust hand pose encoding with an autoregressive generative transformer for effective hand motion sequence prediction. We demonstrate the usability of these novel techniques in a pilot study with healthy subjects. To train and evaluate the proposed method, we collect a dataset consisting of various types of grasp actions on different objects from multiple subjects. Through extensive experiments, we demonstrate that the proposed method can successfully predict sequential hand movement. Especially, the gaze information shows significant enhancements in prediction capabilities, particularly with fewer input frames, highlighting the potential of the proposed method for real-world applications.
• 摘要：通过手运动预测的人类意图检测对于推动神经康复应用中的高级辅助机器人至关重要。但是，依赖生理信号测量的传统方法是限制性的，通常缺乏环境环境。我们提出了一种新的方法，可以预测手姿势和关节位置的未来序列。此方法将目光信息，历史手运动序列和环境对象数据集成在一起，并在没有预先了解预期的对象的情况下，动态地适应了患者的辅助需求。具体而言，我们使用矢量定量的变分自动编码器来用自回旋生成变压器编码稳健的手姿势，以进行有效的手运动序列预测。我们在一项具有健康受试者的试点研究中证明了这些新技术的可用性。为了训练和评估所提出的方法，我们收集了一个数据集，该数据集由来自多个受试者的不同对象的各种类型的掌握操作组成。通过广泛的实验，我们证明了所提出的方法可以成功预测顺序手动运动。尤其是，目光信息显示出预测能力的显着增强，尤其是在输入框架较少的情况下，突出了所提出的实际应用方法的潜力。

Title: Coarse-to-Fine Learning for Multi-Pipette Localisation in Robot-Assisted In Vivo Patch-Clamp

• Authors: Lan Wei, Gema Vera Gonzalez, Phatsimo Kgwarae, Alexander Timms, Denis Zahorovsky, Simon Schultz, Dandan Zhang
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2504.01044
• Pdf URL: https://arxiv.org/pdf/2504.01044
• Copy Paste: [[2504.01044]] Coarse-to-Fine Learning for Multi-Pipette Localisation in Robot-Assisted In Vivo Patch-Clamp(https://arxiv.org/abs/2504.01044)
• Keywords: generative
• Abstract: In vivo image-guided multi-pipette patch-clamp is essential for studying cellular interactions and network dynamics in neuroscience. However, current procedures mainly rely on manual expertise, which limits accessibility and scalability. Robotic automation presents a promising solution, but achieving precise real-time detection of multiple pipettes remains a challenge. Existing methods focus on ex vivo experiments or single pipette use, making them inadequate for in vivo multi-pipette scenarios. To address these challenges, we propose a heatmap-augmented coarse-to-fine learning technique to facilitate multi-pipette real-time localisation for robot-assisted in vivo patch-clamp. More specifically, we introduce a Generative Adversarial Network (GAN)-based module to remove background noise and enhance pipette visibility. We then introduce a two-stage Transformer model that starts with predicting the coarse heatmap of the pipette tips, followed by the fine-grained coordination regression module for precise tip localisation. To ensure robust training, we use the Hungarian algorithm for optimal matching between the predicted and actual locations of tips. Experimental results demonstrate that our method achieved > 98% accuracy within 10 {\mu}m, and > 89% accuracy within 5 {\mu}m for the localisation of multi-pipette tips. The average MSE is 2.52 {\mu}m.
• 摘要：体内图像引导的多页板贴片钳对于研究神经科学中的细胞相互作用和网络动态至关重要。但是，当前程序主要依赖于手动专业知识，这限制了可访问性和可伸缩性。机器人自动化提出了一个有希望的解决方案，但是实现多个移液器的精确实时检测仍然是一个挑战。现有的方法着眼于离体实验或单一移液管的使用，使其不足以进行体内多页板场景。为了应对这些挑战，我们提出了一项由热图调节的粗到精细学习技术，以促进用于机器人辅助体内贴片钳的多页板实时定位。更具体地说，我们引入了基于生成的对抗网络（GAN）的模块，以消除背景噪声并增强移液管的可见性。然后，我们引入了一个两阶段的变压器模型，该模型从预测移液器尖端的粗热图开始，然后是细粒的配位回归模块，以进行精确的尖端定位。为了确保良好的训练，我们使用匈牙利算法在预测位置和实际位置之间进行最佳匹配。实验结果表明，我们的方法在10 {\ MU} M内达到了> 98％的精度，并且在5 {\ MU} M内的精度> 89％，用于定位多页板尖端。平均MSE为2.52 {\ MU} m。

Title: ShieldGemma 2: Robust and Tractable Image Content Moderation

• Authors: Wenjun Zeng, Dana Kurniawan, Ryan Mullins, Yuchi Liu, Tamoghna Saha, Dirichi Ike-Njoku, Jindong Gu, Yiwen Song, Cai Xu, Jingjing Zhou, Aparna Joshi, Shravan Dheep, Mani Malek, Hamid Palangi, Joon Baek, Rick Pereira, Karthik Narasimhan
• Subjects: cs.CV, cs.CL, eess.IV
• Abstract URL: https://arxiv.org/abs/2504.01081
• Pdf URL: https://arxiv.org/pdf/2504.01081
• Copy Paste: [[2504.01081]] ShieldGemma 2: Robust and Tractable Image Content Moderation(https://arxiv.org/abs/2504.01081)
• Keywords: generation
• Abstract: We introduce ShieldGemma 2, a 4B parameter image content moderation model built on Gemma 3. This model provides robust safety risk predictions across the following key harm categories: Sexually Explicit, Violence \& Gore, and Dangerous Content for synthetic images (e.g. output of any image generation model) and natural images (e.g. any image input to a Vision-Language Model). We evaluated on both internal and external benchmarks to demonstrate state-of-the-art performance compared to LlavaGuard \citep{helff2024llavaguard}, GPT-4o mini \citep{hurst2024gpt}, and the base Gemma 3 model \citep{gemma_2025} based on our policies. Additionally, we present a novel adversarial data generation pipeline which enables a controlled, diverse, and robust image generation. ShieldGemma 2 provides an open image moderation tool to advance multimodal safety and responsible AI development.
• 摘要：我们介绍了建立在Gemma 3上的Shieldgemma 2，一个4B参数图像内容调节模型。该模型在以下关键危害类别中提供了强大的安全风险预测：性明确，暴力\＆Gore \＆Gore，以及合成图像的危险内容（例如，任何图像生成模型的输出）和自然图像（例如，对视觉模型的任何图像输入）。我们对内部和外部基准测试进行了评估，以证明与llavaguard \ citep \ citep {helff2024llavaguard}，GPT-4O Mini \ citep \ citep {hurst2024gpt}相比，我们证明了最先进的性能。此外，我们提出了一种新颖的对抗数据生成管道，该管道能够产生受控，多样化和健壮的图像生成。 ShieldGemma 2提供了一种开放的图像审核工具，可提高多模式安全性和负责任的AI开发。

Title: Performative Drift Resistant Classification Using Generative Domain Adversarial Networks

• Authors: Maciej Makowski, Brandon Gower-Winter, Georg Krempl
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.01135
• Pdf URL: https://arxiv.org/pdf/2504.01135
• Copy Paste: [[2504.01135]] Performative Drift Resistant Classification Using Generative Domain Adversarial Networks(https://arxiv.org/abs/2504.01135)
• Keywords: generative
• Abstract: Performative Drift is a special type of Concept Drift that occurs when a model's predictions influence the future instances the model will encounter. In these settings, retraining is not always feasible. In this work, we instead focus on drift understanding as a method for creating drift-resistant classifiers. To achieve this, we introduce the Generative Domain Adversarial Network (GDAN) which combines both Domain and Generative Adversarial Networks. Using GDAN, domain-invariant representations of incoming data are created and a generative network is used to reverse the effects of performative drift. Using semi-real and synthetic data generators, we empirically evaluate GDAN's ability to provide drift-resistant classification. Initial results are promising with GDAN limiting performance degradation over several timesteps. Additionally, GDAN's generative network can be used in tandem with other models to limit their performance degradation in the presence of performative drift. Lastly, we highlight the relationship between model retraining and the unpredictability of performative drift, providing deeper insights into the challenges faced when using traditional Concept Drift mitigation strategies in the performative setting.
• 摘要：表演性漂移是一种特殊的概念漂移类型，当模型的预测影响模型将遇到的未来实例时，会发生这种特殊的概念漂移。在这些环境中，再训练并不总是可行的。在这项工作中，我们将专注于漂移理解作为创建耐漂移分类器的一种方法。为此，我们介绍了结合域和生成对抗网络的生成域对抗网络（GDAN）。使用GDAN，创建了传入数据的域不变表示，并使用生成网络来扭转性能漂移的影响。使用半真实和合成数据生成器，我们从经验上评估了GDAN提供抗漂移分类的能力。最初的结果是有希望的，因为GDAN限制了几个时间段的性能降解。此外，GDAN的生成网络可与其他模型同时使用，以限制其在表演性漂移存在下的性能降解。最后，我们强调了模型再培训与表演性漂移的不可预测性之间的关系，从而更深入地了解在表演环境中使用传统概念漂移策略时所面临的挑战。

Title: Prompting Forgetting: Unlearning in GANs via Textual Guidance

• Authors: Piyush Nagasubramaniam (1), Neeraj Karamchandani (1), Chen Wu (2), Sencun Zhu (1) ((1) The Pennsylvania State University, (2) Meta)
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01218
• Pdf URL: https://arxiv.org/pdf/2504.01218
• Copy Paste: [[2504.01218]] Prompting Forgetting: Unlearning in GANs via Textual Guidance(https://arxiv.org/abs/2504.01218)
• Keywords: generation, generative
• Abstract: State-of-the-art generative models exhibit powerful image-generation capabilities, introducing various ethical and legal challenges to service providers hosting these models. Consequently, Content Removal Techniques (CRTs) have emerged as a growing area of research to control outputs without full-scale retraining. Recent work has explored the use of Machine Unlearning in generative models to address content removal. However, the focus of such research has been on diffusion models, and unlearning in Generative Adversarial Networks (GANs) has remained largely unexplored. We address this gap by proposing Text-to-Unlearn, a novel framework that selectively unlearns concepts from pre-trained GANs using only text prompts, enabling feature unlearning, identity unlearning, and fine-grained tasks like expression and multi-attribute removal in models trained on human faces. Leveraging natural language descriptions, our approach guides the unlearning process without requiring additional datasets or supervised fine-tuning, offering a scalable and efficient solution. To evaluate its effectiveness, we introduce an automatic unlearning assessment method adapted from state-of-the-art image-text alignment metrics, providing a comprehensive analysis of the unlearning methodology. To our knowledge, Text-to-Unlearn is the first cross-modal unlearning framework for GANs, representing a flexible and efficient advancement in managing generative model behavior.
• 摘要：最先进的生成模型具有强大的图像生成能力，向主持这些模型的服务提供商引入了各种道德和法律挑战。因此，内容清除技术（CRT）已成为一个不断增长的研究领域，可以控制输出而无需全面的再训练。最近的工作探索了在生成模型中使用机器未学习来解决内容清除的方法。但是，此类研究的重点是扩散模型，并且在生成对抗网络（GAN）中进行了学习，但基本上仍未开发。我们通过提出文本到Unlearnn来解决这一差距，这是一个新颖的框架，仅使用文本提示从预先训练的gan中选择性地学习概念，从而使功能不学习，身份学习和精细粒度的任务，例如表达方式和在人脸上受过训练的模型中的多粒子删除。利用自然语言描述，我们的方法指导了学习过程，而无需其他数据集或监督微调，提供了可扩展和高效的解决方案。为了评估其有效性，我们引入了一种自动学习评估方法，该方法是根据最先进的图像文本对准指标进行的，从而对未学习方法进行了全面分析。据我们所知，文本到未学习是gan的第一个跨模式学习框架，代表了管理生成模型行为方面灵活而有效的进步。

Title: rPPG-SysDiaGAN: Systolic-Diastolic Feature Localization in rPPG Using Generative Adversarial Network with Multi-Domain Discriminator

• Authors: Banafsheh Adami, Nima Karimian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01220
• Pdf URL: https://arxiv.org/pdf/2504.01220
• Copy Paste: [[2504.01220]] rPPG-SysDiaGAN: Systolic-Diastolic Feature Localization in rPPG Using Generative Adversarial Network with Multi-Domain Discriminator(https://arxiv.org/abs/2504.01220)
• Keywords: generative
• Abstract: Remote photoplethysmography (rPPG) offers a novel approach to noninvasive monitoring of vital signs, such as respiratory rate, utilizing a camera. Although several supervised and self-supervised methods have been proposed, they often fail to accurately reconstruct the PPG signal, particularly in distinguishing between systolic and diastolic components. Their primary focus tends to be solely on extracting heart rate, which may not accurately represent the complete PPG signal. To address this limitation, this paper proposes a novel deep learning architecture using Generative Adversarial Networks by introducing multi-discriminators to extract rPPG signals from facial videos. These discriminators focus on the time domain, the frequency domain, and the second derivative of the original time domain signal. The discriminator integrates four loss functions: variance loss to mitigate local minima caused by noise; dynamic time warping loss to address local minima induced by alignment and sequences of variable lengths; Sparsity Loss for heart rate adjustment, and Variance Loss to ensure a uniform distribution across the desired frequency domain and time interval between systolic and diastolic phases of the PPG signal.
• 摘要：远程照相体积学（RPPG）提供了一种新型的方法，可以利用相机对生命体征的无创监测，例如呼吸率。尽管已经提出了几种监督和自我监督的方法，但它们通常无法准确地重建PPG信号，尤其是在区分收缩压和舒张期成分时。它们的主要重点往往仅仅是提取心率，这可能不能准确地代表完整的PPG信号。为了解决这一限制，本文提出了一种新颖的深度学习架构，使用生成的对抗网络引入多歧视器来从面部视频中提取RPPG信号。这些歧视因子专注于时间域，频域和原始时域信号的第二个导数。鉴别器集成了四个损失函数：减轻噪声引起的局部最小值的方差损失；动态时间扭曲损失，以解决由比对引起的局部最小值和可变长度的序列；心率调整的稀疏性损失以及差异损失，以确保PPG信号的收缩期和舒张期之间所需的频域和时间间隔的均匀分布。

Title: Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding

• Authors: Sakhinana Sagar Srinivas, Venkataramana Runkana
• Subjects: cs.LG, cs.AI, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2504.01281
• Pdf URL: https://arxiv.org/pdf/2504.01281
• Copy Paste: [[2504.01281]] Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding(https://arxiv.org/abs/2504.01281)
• Keywords: generation
• Abstract: We present a comprehensive framework for enhancing Retrieval-Augmented Generation (RAG) systems through dynamic retrieval strategies and reinforcement fine-tuning. This approach significantly improves large language models on knowledge-intensive tasks, including opendomain question answering and complex reasoning. Our framework integrates two complementary techniques: Policy-Optimized RetrievalAugmented Generation (PORAG), which optimizes the use of retrieved information, and Adaptive Token-Layer Attention Scoring (ATLAS), which dynamically determines retrieval timing and content based on contextual needs. Together, these techniques enhance both the utilization and relevance of retrieved content, improving factual accuracy and response quality. Designed as a lightweight solution compatible with any Transformer-based LLM without requiring additional training, our framework excels in knowledge-intensive tasks, boosting output accuracy in RAG settings. We further propose CRITIC, a novel method to selectively compress key-value caches by token importance, mitigating memory bottlenecks in long-context applications. The framework also incorporates test-time scaling techniques to dynamically balance reasoning depth and computational resources, alongside optimized decoding strategies for faster inference. Experiments on benchmark datasets show that our framework reduces hallucinations, strengthens domain-specific reasoning, and achieves significant efficiency and scalability gains over traditional RAG systems. This integrated approach advances the development of robust, efficient, and scalable RAG systems across diverse applications.
• 摘要：我们提出了一个全面的框架，用于通过动态检索策略和加强微调来增强检索功能的生成（RAG）系统。这种方法可大大改善知识密集型任务的大型语言模型，包括opendomain问题答案和复杂的推理。我们的框架集成了两种互补技术：策略优化检索的生成（PORAG），它优化了检索信息的使用以及自适应令牌的注意力评分（ATLAS），该评分（ATLAS）动态地决定了基于上下文需求的检索时间和内容。这些技术共同增强了检索内容的利用率和相关性，从而提高了事实准确性和响应质量。我们的框架设计为与任何基于变压器的LLM兼容的轻量级解决方案，而无需额外的培训，我们的框架在知识密集型任务中表现出色，从而提高了抹布设置中的输出精度。我们进一步提出了评论家，这是一种新颖的方法，可以通过令牌的重要性选择性地压缩键值缓存，从而减轻长篇文化应用程序中的记忆瓶颈。该框架还结合了测试时间缩放技术，以动态平衡推理深度和计算资源，以及优化的解码策略，以更快地推断。基准数据集上的实验表明，我们的框架减少了幻觉，增强特定于域的推理，并在传统的抹布系统上实现了显着的效率和可伸缩性。这种综合方法推动了各种应用程序跨不同应用程序的强大，高效和可扩展的破布系统的发展。

Title: De Novo Molecular Design Enabled by Direct Preference Optimization and Curriculum Learning

• Authors: Junyu Hou
• Subjects: cs.LG, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2504.01389
• Pdf URL: https://arxiv.org/pdf/2504.01389
• Copy Paste: [[2504.01389]] De Novo Molecular Design Enabled by Direct Preference Optimization and Curriculum Learning(https://arxiv.org/abs/2504.01389)
• Keywords: generation
• Abstract: De novo molecular design has extensive applications in drug discovery and materials science. The vast chemical space renders direct molecular searches computationally prohibitive, while traditional experimental screening is both time- and labor-intensive. Efficient molecular generation and screening methods are therefore essential for accelerating drug discovery and reducing costs. Although reinforcement learning (RL) has been applied to optimize molecular properties via reward mechanisms, its practical utility is limited by issues in training efficiency, convergence, and stability. To address these challenges, we adopt Direct Preference Optimization (DPO) from NLP, which uses molecular score-based sample pairs to maximize the likelihood difference between high- and low-quality molecules, effectively guiding the model toward better compounds. Moreover, integrating curriculum learning further boosts training efficiency and accelerates convergence. A systematic evaluation of the proposed method on the GuacaMol Benchmark yielded excellent scores. For instance, the method achieved a score of 0.883 on the Perindopril MPO task, representing a 6\% improvement over competing models. And subsequent target protein binding experiments confirmed its practical efficacy. These results demonstrate the strong potential of DPO for molecular design tasks and highlight its effectiveness as a robust and efficient solution for data-driven drug discovery.
• 摘要：从头分子设计在药物发现和材料科学中有广泛的应用。庞大的化学空间使直接的分子搜索在计算上具有过时的态度，而传统的实验筛查既是时间和劳动力密集的。因此，有效的分子产生和筛选方法对于加速药物发现和降低成本至关重要。尽管增强学习（RL）已应用于通过奖励机制优化分子特性，但其实际效用受训练效率，收敛和稳定性问题的限制。为了应对这些挑战，我们采用了NLP的直接偏好优化（DPO），该优化使用基于分子得分的样品对来最大程度地提高高质量分子和低质量分子之间的可能性差异，从而有效地指导模型朝着更好的化合物指导。此外，整合课程学习进一步提高了培训效率并加速了融合。对鳄梨调味酱基准提出的方法的系统评估产生了出色的分数。例如，该方法在Perindopril MPO任务上达到了0.883的得分，代表比竞争模型的6 \％改进。随后的靶蛋白结合实验证实了其实际功效。这些结果证明了DPO在分子设计任务中的强大潜力，并强调了其有效性作为数据驱动药物发现的强大而有效的解决方案。

Title: All Patches Matter, More Patches Better: Enhance AI-Generated Image Detection via Panoptic Patch Learning

• Authors: Zheng Yang, Ruoxin Chen, Zhiyuan Yan, Ke-Yue Zhang, Xinghe Fu, Shuang Wu, Xiujun Shu, Taiping Yao, Junchi Yan, Shouhong Ding, Xi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01396
• Pdf URL: https://arxiv.org/pdf/2504.01396
• Copy Paste: [[2504.01396]] All Patches Matter, More Patches Better: Enhance AI-Generated Image Detection via Panoptic Patch Learning(https://arxiv.org/abs/2504.01396)
• Keywords: generation
• Abstract: The exponential growth of AI-generated images (AIGIs) underscores the urgent need for robust and generalizable detection methods. In this paper, we establish two key principles for AIGI detection through systematic analysis: \textbf{(1) All Patches Matter:} Unlike conventional image classification where discriminative features concentrate on object-centric regions, each patch in AIGIs inherently contains synthetic artifacts due to the uniform generation process, suggesting that every patch serves as an important artifact source for detection. \textbf{(2) More Patches Better}: Leveraging distributed artifacts across more patches improves detection robustness by capturing complementary forensic evidence and reducing over-reliance on specific patches, thereby enhancing robustness and generalization. However, our counterfactual analysis reveals an undesirable phenomenon: naively trained detectors often exhibit a \textbf{Few-Patch Bias}, discriminating between real and synthetic images based on minority patches. We identify \textbf{Lazy Learner} as the root cause: detectors preferentially learn conspicuous artifacts in limited patches while neglecting broader artifact distributions. To address this bias, we propose the \textbf{P}anoptic \textbf{P}atch \textbf{L}earning (PPL) framework, involving: (1) Random Patch Replacement that randomly substitutes synthetic patches with real counterparts to compel models to identify artifacts in underutilized regions, encouraging the broader use of more patches; (2) Patch-wise Contrastive Learning that enforces consistent discriminative capability across all patches, ensuring uniform utilization of all patches. Extensive experiments across two different settings on several benchmarks verify the effectiveness of our approach.
• 摘要：AI生成的图像（AIGI）的指数增长强调了迫切需要强大而可推广的检测方法。在本文中，我们通过系统分析建立了AIGI检测的两个关键原理：\ textbf {（1）所有贴片都很重要：}与常规图像分类不同，在传统的图像分类中，宽大型特征集中在对象中心区域上，每个斑块都固有地包含由于统一的生成过程而固有的合成伪像，建议每个贴片都可以用作重要的零件，以供应。 \ textbf {（2）更多的补丁更好}：通过捕获补充法医证据并减少对特定补丁的过度依赖，从而利用更多补丁的分布式伪像可以提高检测鲁棒性，从而增强了鲁棒性和概括。但是，我们的反事实分析揭示了一种不良现象：训练有素的探测器通常表现出\ textbf {几个斑点偏见}，根据少数斑块区分了真实和合成图像。我们将\ textbf {懒惰学习者}识别为根本原因：检测器优先在有限的补丁中学习明显的人工制品，同时忽略更广泛的人工制品分布。 To address this bias, we propose the \textbf{P}anoptic \textbf{P}atch \textbf{L}earning (PPL) framework, involving: (1) Random Patch Replacement that randomly substitutes synthetic patches with real counterparts to compel models to identify artifacts in underutilized regions, encouraging the broader use of more patches; （2）通过斑块的对比度学习，可以在所有斑块上执行一致的判别能力，从而确保所有斑块的均匀利用。在几个基准上的两个不同设置进行的广泛实验验证了我们方法的有效性。

Title: A Prefixed Patch Time Series Transformer for Two-Point Boundary Value Problems in Three-Body Problems

• Authors: Akira Hatakeyama, Shota Ito, Toshihiko Yanase, Naoya Ozaki
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.01464
• Pdf URL: https://arxiv.org/pdf/2504.01464
• Copy Paste: [[2504.01464]] A Prefixed Patch Time Series Transformer for Two-Point Boundary Value Problems in Three-Body Problems(https://arxiv.org/abs/2504.01464)
• Keywords: generative
• Abstract: Two-point boundary value problems for cislunar trajectories present significant challenges in circler restricted three body problem, making traditional analytical methods like Lambert's problem inapplicable. This study proposes a novel approach using a prefixed patch time series Transformer model that automates the solution of two-point boundary value problems from lunar flyby to arbitrary terminal conditions. Using prefix tokens of terminal conditions in our deep generative model enables solving boundary value problems in three-body dynamics. The training dataset consists of trajectories obtained through forward propagation rather than solving boundary value problems directly. The model demonstrates potential practical utility for preliminary trajectory design in cislunar mission scenarios.
• 摘要：Cislunar轨迹的两点边界价值问题在圆圈中提出了重大挑战，这限制了三个身体问题，这使得像兰伯特的问题这样的传统分析方法不可应用。这项研究提出了一种使用前缀贴片时间序列变压器模型的新方法，该模型可以自动化从月球飞行到任意终端条件的两点边界值问题的解决方案。在我们的深生成模型中，使用终端条件的前缀令牌可以解决三体动力学中的边界价值问题。训练数据集由通过正向传播获得的轨迹组成，而不是直接解决边界价值问题。该模型显示了Cislunar任务方案中初步轨迹设计的潜在实用性。

Title: ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction

• Authors: Yuejiao Su, Yi Wang, Qiongyang Hu, Chuang Yang, Lap-Pui Chau
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01472
• Pdf URL: https://arxiv.org/pdf/2504.01472
• Copy Paste: [[2504.01472]] ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction(https://arxiv.org/abs/2504.01472)
• Keywords: generation
• Abstract: Egocentric interaction perception is one of the essential branches in investigating human-environment interaction, which lays the basis for developing next-generation intelligent systems. However, existing egocentric interaction understanding methods cannot yield coherent textual and pixel-level responses simultaneously according to user queries, which lacks flexibility for varying downstream application requirements. To comprehend egocentric interactions exhaustively, this paper presents a novel task named Egocentric Interaction Reasoning and pixel Grounding (Ego-IRG). Taking an egocentric image with the query as input, Ego-IRG is the first task that aims to resolve the interactions through three crucial steps: analyzing, answering, and pixel grounding, which results in fluent textual and fine-grained pixel-level responses. Another challenge is that existing datasets cannot meet the conditions for the Ego-IRG task. To address this limitation, this paper creates the Ego-IRGBench dataset based on extensive manual efforts, which includes over 20k egocentric images with 1.6 million queries and corresponding multimodal responses about interactions. Moreover, we design a unified ANNEXE model to generate text- and pixel-level outputs utilizing multimodal large language models, which enables a comprehensive interpretation of egocentric interactions. The experiments on the Ego-IRGBench exhibit the effectiveness of our ANNEXE model compared with other works.
• 摘要：以自我为中心的互动感知是研究人类环境相互作用的重要分支之一，这为开发下一代智能系统的基础奠定了基础。但是，根据用户查询，现有的以自我为中心的互动理解方法不能同时产生连贯的文本和像素级响应，因为用户查询缺乏改变下游应用程序要求的灵活性。为了详尽地理解以自我为中心的互动，本文提出了一项名为中心的互动推理和像素接地（EGO-IRG）的新任务。以查询为输入的以自我为中心的图像是旨在通过三个关键步骤来解决相互作用的第一个任务：分析，答案和像素接地，从而导致流利的文本和细粒度和细粒度的像素级响应。另一个挑战是现有数据集无法满足自我IRG任务的条件。为了解决这一限制，本文基于大量的手动工作创建了自我-IRGBENCH数据集，其中包括超过20K的以160万个查询的egentric图像，以及相应的关于交互的多模式响应。此外，我们设计了一个统一的附件模型，以生成使用多模式大语言模型的文本和像素级输出，从而可以全面解释以自我为中心的互动。与其他作品相比，对自我irgbench的实验表现出我们的附件模型的有效性。

Title: High-fidelity 3D Object Generation from Single Image with RGBN-Volume Gaussian Reconstruction Model

• Authors: Yiyang Shen, Kun Zhou, He Wang, Yin Yang, Tianjia Shao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01512
• Pdf URL: https://arxiv.org/pdf/2504.01512
• Copy Paste: [[2504.01512]] High-fidelity 3D Object Generation from Single Image with RGBN-Volume Gaussian Reconstruction Model(https://arxiv.org/abs/2504.01512)
• Keywords: generation
• Abstract: Recently single-view 3D generation via Gaussian splatting has emerged and developed quickly. They learn 3D Gaussians from 2D RGB images generated from pre-trained multi-view diffusion (MVD) models, and have shown a promising avenue for 3D generation through a single image. Despite the current progress, these methods still suffer from the inconsistency jointly caused by the geometric ambiguity in the 2D images, and the lack of structure of 3D Gaussians, leading to distorted and blurry 3D object generation. In this paper, we propose to fix these issues by GS-RGBN, a new RGBN-volume Gaussian Reconstruction Model designed to generate high-fidelity 3D objects from single-view images. Our key insight is a structured 3D representation can simultaneously mitigate the afore-mentioned two issues. To this end, we propose a novel hybrid Voxel-Gaussian representation, where a 3D voxel representation contains explicit 3D geometric information, eliminating the geometric ambiguity from 2D images. It also structures Gaussians during learning so that the optimization tends to find better local optima. Our 3D voxel representation is obtained by a fusion module that aligns RGB features and surface normal features, both of which can be estimated from 2D images. Extensive experiments demonstrate the superiority of our methods over prior works in terms of high-quality reconstruction results, robust generalization, and good efficiency.
• 摘要：最近，通过高斯脱落的单视图3D生成已经出现并迅速发展。他们从预先训练的多视图扩散（MVD）模型中生成的2D RGB图像中学习了3D高斯人，并通过单个图像显示了3D生成的有希望的途径。尽管目前取得了进展，但这些方法仍然遭受由2D图像中的几何歧义共同引起的不一致，以及缺乏3D高斯人的结构，导致3D对象产生扭曲和模糊。在本文中，我们建议通过GS-RGBN修复这些问题，GS-RGBN是一种新的rgbn-volume高斯重建模型，旨在从单视图图像中生成高保真3D对象。我们的主要见解是结构化的3D表示可以同时减轻上述两个问题。为此，我们提出了一种新型的混合体素 - 高斯表示，其中3D体素表示包含显式3D几何信息，从而消除了来自2D图像的几何歧义。它还在学习过程中构造高斯人，以便优化倾向于找到更好的本地优点。我们的3D体素表示是通过融合模块对齐RGB特征和表面正常特征的融合模块获得的，这两者都可以从2D图像估算。广泛的实验证明了我们的方法优于先前的工作，从高质量的重建结果，稳定的概括和良好的效率方面。

Title: Training-free Dense-Aligned Diffusion Guidance for Modular Conditional Image Synthesis

• Authors: Zixuan Wang, Duo Peng, Feng Chen, Yuwei Yang, Yinjie Lei
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.01515
• Pdf URL: https://arxiv.org/pdf/2504.01515
• Copy Paste: [[2504.01515]] Training-free Dense-Aligned Diffusion Guidance for Modular Conditional Image Synthesis(https://arxiv.org/abs/2504.01515)
• Keywords: generation, generative
• Abstract: Conditional image synthesis is a crucial task with broad applications, such as artistic creation and virtual reality. However, current generative methods are often task-oriented with a narrow scope, handling a restricted condition with constrained applicability. In this paper, we propose a novel approach that treats conditional image synthesis as the modular combination of diverse fundamental condition units. Specifically, we divide conditions into three primary units: text, layout, and drag. To enable effective control over these conditions, we design a dedicated alignment module for each. For the text condition, we introduce a Dense Concept Alignment (DCA) module, which achieves dense visual-text alignment by drawing on diverse textual concepts. For the layout condition, we propose a Dense Geometry Alignment (DGA) module to enforce comprehensive geometric constraints that preserve the spatial configuration. For the drag condition, we introduce a Dense Motion Alignment (DMA) module to apply multi-level motion regularization, ensuring that each pixel follows its desired trajectory without visual artifacts. By flexibly inserting and combining these alignment modules, our framework enhances the model's adaptability to diverse conditional generation tasks and greatly expands its application range. Extensive experiments demonstrate the superior performance of our framework across a variety of conditions, including textual description, segmentation mask (bounding box), drag manipulation, and their combinations. Code is available at this https URL.
• 摘要：有条件的图像合成是具有广泛应用的至关重要的任务，例如艺术创作和虚拟现实。但是，当前的生成方法通常以狭窄的范围以任务为导向，并以受限的适用性处理受限条件。在本文中，我们提出了一种新颖的方法，将条件图像合成视为各种基本条件单位的模块化组合。具体而言，我们将条件分为三个主要单元：文本，布局和拖动。为了实现对这些条件的有效控制，我们为每个条件设计一个专用的对齐模块。对于文本条件，我们引入了一个密集的概念对准（DCA）模块，该模块通过利用各种文本概念来实现密集的视觉文本对齐。对于布局条件，我们提出了一个密集的几何比对（DGA）模块，以强制执行保留空间配置的综合几何约束。对于阻力条件，我们引入了一个密集的运动对准（DMA）模块以应用多级运动正则化，以确保每个像素均遵循其所需的轨迹，而无需视觉伪影。通过灵活地插入和组合这些对齐模块，我们的框架可以增强该模型对各种有条件生成任务的适应性，并大大扩展了其应用程序范围。广泛的实验表明，我们的框架在各种条件下的出色性能，包括文本描述，分割面罩（边界框），拖动操纵及其组合。代码可在此HTTPS URL上找到。

Title: Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model

• Authors: Jincheng Zhong, Xiangcheng Zhang, Jianmin Wang, Mingsheng Long
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01521
• Pdf URL: https://arxiv.org/pdf/2504.01521
• Copy Paste: [[2504.01521]] Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model(https://arxiv.org/abs/2504.01521)
• Keywords: generation, generative
• Abstract: Recent advancements in diffusion models have revolutionized generative modeling. However, the impressive and vivid outputs they produce often come at the cost of significant model scaling and increased computational demands. Consequently, building personalized diffusion models based on off-the-shelf models has emerged as an appealing alternative. In this paper, we introduce a novel perspective on conditional generation for transferring a pre-trained model. From this viewpoint, we propose *Domain Guidance*, a straightforward transfer approach that leverages pre-trained knowledge to guide the sampling process toward the target domain. Domain Guidance shares a formulation similar to advanced classifier-free guidance, facilitating better domain alignment and higher-quality generations. We provide both empirical and theoretical analyses of the mechanisms behind Domain Guidance. Our experimental results demonstrate its substantial effectiveness across various transfer benchmarks, achieving over a 19.6% improvement in FID and a 23.4% improvement in FD$_\text{DINOv2}$ compared to standard fine-tuning. Notably, existing fine-tuned models can seamlessly integrate Domain Guidance to leverage these benefits, without additional training.
• 摘要：扩散模型的最新进展已彻底改变了生成建模。但是，它们产生的令人印象深刻和生动的产出通常是以显着的模型缩放和增加计算需求的成本。因此，建立基于现成模型的个性化扩散模型已成为一种吸引人的选择。在本文中，我们介绍了有关传递预训练模型的条件生成的新观点。从这个角度来看，我们提出了 *域指导 *，这是一种简单的转移方法，利用预训练的知识来指导采样过程朝着目标域。域指南具有类似于无高级分类器指导的公式，促进了更好的领域对准和更高质量的世代。我们提供了领域指导背后机制的经验和理论分析。我们的实验结果表明，其在各种转移基准测试基准中的实质性有效性，与标准微调相比，FID提高了19.6％，FD $ _ \ text {dinov2} $提高了23.4％。值得注意的是，现有的微调模型可以无缝整合域指南以利用这些好处，而无需其他培训。

Title: Semi-Supervised Biomedical Image Segmentation via Diffusion Models and Teacher-Student Co-Training

• Authors: Luca Ciampi, Gabriele Lagani, Giuseppe Amato, Fabrizio Falchi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01547
• Pdf URL: https://arxiv.org/pdf/2504.01547
• Copy Paste: [[2504.01547]] Semi-Supervised Biomedical Image Segmentation via Diffusion Models and Teacher-Student Co-Training(https://arxiv.org/abs/2504.01547)
• Keywords: generation, generative
• Abstract: Supervised deep learning for semantic segmentation has achieved excellent results in accurately identifying anatomical and pathological structures in medical images. However, it often requires large annotated training datasets, which limits its scalability in clinical settings. To address this challenge, semi-supervised learning is a well-established approach that leverages both labeled and unlabeled data. In this paper, we introduce a novel semi-supervised teacher-student framework for biomedical image segmentation, inspired by the recent success of generative models. Our approach leverages denoising diffusion probabilistic models (DDPMs) to generate segmentation masks by progressively refining noisy inputs conditioned on the corresponding images. The teacher model is first trained in an unsupervised manner using a cycle-consistency constraint based on noise-corrupted image reconstruction, enabling it to generate informative semantic masks. Subsequently, the teacher is integrated into a co-training process with a twin-student network. The student learns from ground-truth labels when available and from teacher-generated pseudo-labels otherwise, while the teacher continuously improves its pseudo-labeling capabilities. Finally, to further enhance performance, we introduce a multi-round pseudo-label generation strategy that iteratively improves the pseudo-labeling process. We evaluate our approach on multiple biomedical imaging benchmarks, spanning multiple imaging modalities and segmentation tasks. Experimental results show that our method consistently outperforms state-of-the-art semi-supervised techniques, highlighting its effectiveness in scenarios with limited annotated data. The code to replicate our experiments can be found at this https URL
• 摘要：对语义细分的监督深度学习在准确识别医学图像中的解剖学和病理结构方面取得了出色的结果。但是，它通常需要大量的注释培训数据集，这限制了其在临床环境中的可扩展性。为了应对这一挑战，半监督学习是一种既有标记和未标记数据的良好方法。在本文中，我们介绍了一个新型的半监督教师研究框架，用于生物医学图像分割，灵感来自生成模型的成功。我们的方法利用扩散概率模型（DDPM）来利用通过在相应图像上进行条件的嘈杂输入来生成分割掩码。首先，使用基于噪声浪费的图像重建的循环一致性约束以无监督的方式训练教师模型，从而使其能够生成内容丰富的语义掩码。随后，教师将与双胞胎网络集成到共同培训过程中。该学生在可用的情况下从地面真实标签中学习，否则从教师生成的伪标签中学习，而教师不断提高其伪标记的功能。最后，为了进一步提高性能，我们引入了多轮标签生成策略，迭代地改善了伪标记的过程。我们在多个生物医学成像基准上评估了我们的方法，涵盖了多种成像方式和分割任务。实验结果表明，我们的方法始终优于最先进的半监督技术，从而强调了其在有限的注释数据的情况下的有效性。可以在此HTTPS URL上找到复制我们实验的代码

Title: Representation Bending for Large Language Model Safety

• Authors: Ashkan Yousefpour, Taeheon Kim, Ryan S. Kwon, Seungbeen Lee, Wonje Jeung, Seungju Han, Alvin Wan, Harrison Ngan, Youngjae Yu, Jonghyun Choi
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2504.01550
• Pdf URL: https://arxiv.org/pdf/2504.01550
• Copy Paste: [[2504.01550]] Representation Bending for Large Language Model Safety(https://arxiv.org/abs/2504.01550)
• Keywords: generation
• Abstract: Large Language Models (LLMs) have emerged as powerful tools, but their inherent safety risks - ranging from harmful content generation to broader societal harms - pose significant challenges. These risks can be amplified by the recent adversarial attacks, fine-tuning vulnerabilities, and the increasing deployment of LLMs in high-stakes environments. Existing safety-enhancing techniques, such as fine-tuning with human feedback or adversarial training, are still vulnerable as they address specific threats and often fail to generalize across unseen attacks, or require manual system-level defenses. This paper introduces RepBend, a novel approach that fundamentally disrupts the representations underlying harmful behaviors in LLMs, offering a scalable solution to enhance (potentially inherent) safety. RepBend brings the idea of activation steering - simple vector arithmetic for steering model's behavior during inference - to loss-based fine-tuning. Through extensive evaluation, RepBend achieves state-of-the-art performance, outperforming prior methods such as Circuit Breaker, RMU, and NPO, with up to 95% reduction in attack success rates across diverse jailbreak benchmarks, all with negligible reduction in model usability and general capabilities.
• 摘要：大型语言模型（LLM）已成为强大的工具，但是它们固有的安全风险（从有害内容产生到更广泛的社会危害）构成了重大挑战。这些风险可以通过最近的对抗性攻击，微调漏洞以及在高风险环境中的LLM部署增加来扩大这些风险。现有的安全增强技术，例如对人类反馈或对抗性培训进行微调，因为它们应对特定的威胁，并且通常无法在看不见的攻击中概括或需要手动系统级防御。本文介绍了Repbend，这是一种新颖的方法，它从根本上破坏了LLMS中有害行为的代表，从而提供了可扩展的解决方案来增强（潜在固有的）安全性。 repbend带来了激活转向的概念 - 在推理过程中指导模型的行为的简单矢量算术 - 基于基于损失的微调。通过广泛的评估，Repbend实现了最先进的性能，胜过诸如断路器，RMU和NPO等先前方法，在各种越狱基准的攻击成功率下降了95％，所有这些基准的攻击成功率都降低了，所有这些基准都可以降低模型可用性和一般能力。

Title: DEPTHOR: Depth Enhancement from a Practical Light-Weight dToF Sensor and RGB Image

• Authors: Jijun Xiang, Xuan Zhu, Xianqi Wang, Yu Wang, Hong Zhang, Fei Guo, Xin Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01596
• Pdf URL: https://arxiv.org/pdf/2504.01596
• Copy Paste: [[2504.01596]] DEPTHOR: Depth Enhancement from a Practical Light-Weight dToF Sensor and RGB Image(https://arxiv.org/abs/2504.01596)
• Keywords: super-resolution
• Abstract: Depth enhancement, which uses RGB images as guidance to convert raw signals from dToF into high-precision, dense depth maps, is a critical task in computer vision. Although existing super-resolution-based methods show promising results on public datasets, they often rely on idealized assumptions like accurate region correspondences and reliable dToF inputs, overlooking calibration errors that cause misalignment and anomaly signals inherent to dToF imaging, limiting real-world applicability. To address these challenges, we propose a novel completion-based method, named DEPTHOR, featuring advances in both the training strategy and model architecture. First, we propose a method to simulate real-world dToF data from the accurate ground truth in synthetic datasets to enable noise-robust training. Second, we design a novel network that incorporates monocular depth estimation (MDE), leveraging global depth relationships and contextual information to improve prediction in challenging regions. On the ZJU-L5 dataset, our training strategy significantly enhances depth completion models, achieving results comparable to depth super-resolution methods, while our model achieves state-of-the-art results, improving Rel and RMSE by 27% and 18%, respectively. On a more challenging set of dToF samples we collected, our method outperforms SOTA methods on preliminary stereo-based GT, improving Rel and RMSE by 23% and 22%, respectively. Our Code is available at this https URL
• 摘要：深度增强使用RGB图像作为指导，将原始信号从DTOF转换为高精度，密集的深度图，这是计算机视觉中的关键任务。尽管现有的基于超分辨率的方法在公共数据集上显示出令人鼓舞的结果，但它们通常依赖于理想化的假设，例如准确的区域对应关系和可靠的DTOF输入，俯瞰导致未对准的校准错误和DTOF成像固有的异常信号，从而限制了现实世界的适用性。为了应对这些挑战，我们提出了一种新颖的基于完成的方法，名为Depthor，以培训策略和模型体系结构的进步为特征。首先，我们提出了一种方法，以模拟合成数据集中准确地面真实的现实世界DTOF数据，以实现噪声训练。其次，我们设计了一个新颖的网络，该网络结合了单眼深度估计（MDE），利用全球深度关系和上下文信息，以改善挑战区域的预测。在ZJU-L5数据集上，我们的培训策略显着增强了深度完成模型，实现了与深度超分辨率方法相当的结果，而我们的模型可实现最先进的结果，将REL和RMSE分别提高了27％和18％。在我们收集的一组更具挑战性的DTOF样品中，我们的方法的表现优于基于初步立体声的GT的SOTA方法，分别提高了RER和RMSE的23％和22％。我们的代码可在此HTTPS URL上找到

Title: 3DBonsai: Structure-Aware Bonsai Modeling Using Conditioned 3D Gaussian Splatting

• Authors: Hao Wu, Hao Wang, Ruochong Li, Xuran Ma, Hui Xiong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01619
• Pdf URL: https://arxiv.org/pdf/2504.01619
• Copy Paste: [[2504.01619]] 3DBonsai: Structure-Aware Bonsai Modeling Using Conditioned 3D Gaussian Splatting(https://arxiv.org/abs/2504.01619)
• Keywords: generation
• Abstract: Recent advancements in text-to-3D generation have shown remarkable results by leveraging 3D priors in combination with 2D diffusion. However, previous methods utilize 3D priors that lack detailed and complex structural information, limiting them to generating simple objects and presenting challenges for creating intricate structures such as bonsai. In this paper, we propose 3DBonsai, a novel text-to-3D framework for generating 3D bonsai with complex structures. Technically, we first design a trainable 3D space colonization algorithm to produce bonsai structures, which are then enhanced through random sampling and point cloud augmentation to serve as the 3D Gaussian priors. We introduce two bonsai generation pipelines with distinct structural levels: fine structure conditioned generation, which initializes 3D Gaussians using a 3D structure prior to produce detailed and complex bonsai, and coarse structure conditioned generation, which employs a multi-view structure consistency module to align 2D and 3D structures. Moreover, we have compiled a unified 2D and 3D Chinese-style bonsai dataset. Our experimental results demonstrate that 3DBonsai significantly outperforms existing methods, providing a new benchmark for structure-aware 3D bonsai generation.
• 摘要：文本到3D代的最新进展通过利用3D先验与2D扩散相结合，显示出显着的结果。但是，以前的方法利用了缺乏详细且复杂的结构信息的3D先验，将它们限制在生成简单的对象并提出创建复杂结构（例如盆景）的挑战。在本文中，我们提出了3DBonsai，这是一个新型的文本到3D框架，用于生成具有复杂结构的3D盆景。从技术上讲，我们首先设计了一种可训练的3D空间定植算法来生产盆景结构，然后通过随机采样和点云增强来增强该结构，以作为3D高斯先验。我们介绍了两个具有不同结构水平的盆景生成管道：精细的结构条件生成，它在生产详细且复杂的盆景之前使用3D结构初始化了3D高斯，并且具有粗糙的结构条件生成，该结构使用多视图结构一致性模块，以使2D和3D结构对齐。此外，我们编制了一个统一的2D和3D中国风格的盆景数据集。我们的实验结果表明，3Dbonsai明显胜过现有方法，为结构感知3D盆景的生成提供了新的基准。

Title: FlowR: Flowing from Sparse to Dense 3D Reconstructions

• Authors: Tobias Fischer, Samuel Rota Bulò, Yung-Hsu Yang, Nikhil Varma Keetha, Lorenzo Porzi, Norman Müller, Katja Schwarz, Jonathon Luiten, Marc Pollefeys, Peter Kontschieder
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01647
• Pdf URL: https://arxiv.org/pdf/2504.01647
• Copy Paste: [[2504.01647]] FlowR: Flowing from Sparse to Dense 3D Reconstructions(https://arxiv.org/abs/2504.01647)
• Keywords: generation, generative
• Abstract: 3D Gaussian splatting enables high-quality novel view synthesis (NVS) at real-time frame rates. However, its quality drops sharply as we depart from the training views. Thus, dense captures are needed to match the high-quality expectations of some applications, e.g. Virtual Reality (VR). However, such dense captures are very laborious and expensive to obtain. Existing works have explored using 2D generative models to alleviate this requirement by distillation or generating additional training views. These methods are often conditioned only on a handful of reference input views and thus do not fully exploit the available 3D information, leading to inconsistent generation results and reconstruction artifacts. To tackle this problem, we propose a multi-view, flow matching model that learns a flow to connect novel view renderings from possibly sparse reconstructions to renderings that we expect from dense reconstructions. This enables augmenting scene captures with novel, generated views to improve reconstruction quality. Our model is trained on a novel dataset of 3.6M image pairs and can process up to 45 views at 540x960 resolution (91K tokens) on one H100 GPU in a single forward pass. Our pipeline consistently improves NVS in sparse- and dense-view scenarios, leading to higher-quality reconstructions than prior works across multiple, widely-used NVS benchmarks.
• 摘要：3D高斯脱落可以实时帧速率以高质量的新型视图合成（NVS）。但是，当我们偏离训练观点时，其质量急剧下降。因此，需要密集的捕获以匹配某些应用的高质量期望，例如虚拟现实（VR）。但是，这种密集的捕获非常费力且昂贵。现有作品已经使用2D生成模型探索了通过蒸馏或产生其他培训视图来减轻此需求。这些方法通常仅以少数参考输入视图来调节，因此不能完全利用可用的3D信息，从而导致产生不一致的结果和重建伪像。为了解决这个问题，我们提出了一个多视图，流匹配模型，该模型学习了一个流程，将新型视图渲染从可能的稀疏重建到我们期望从密集重建中获得的渲染。这使增强场景通过新颖，生成的视图捕获以提高重建质量。我们的模型经过360万映像对的新型数据集进行了训练，可以在单个正向通行证中以一个H100 GPU在540x960分辨率（91K令牌）上处理多达45个视图。我们的管道始终改善稀疏和密集视图方案中的NV，从而比在多个广泛使用的NVS基准中的先前作品相比，导致更高质量的重建。

Title: Q-Adapt: Adapting LMM for Visual Quality Assessment with Progressive Instruction Tuning

• Authors: Yiting Lu, Xin Li, Haoning Wu, Bingchen Li, Weisi Lin, Zhibo Chen
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2504.01655
• Pdf URL: https://arxiv.org/pdf/2504.01655
• Copy Paste: [[2504.01655]] Q-Adapt: Adapting LMM for Visual Quality Assessment with Progressive Instruction Tuning(https://arxiv.org/abs/2504.01655)
• Keywords: quality assessment
• Abstract: The rapid advancement of Large Multi-modal Foundation Models (LMM) has paved the way for the possible Explainable Image Quality Assessment (EIQA) with instruction tuning from two perspectives: overall quality explanation, and attribute-wise perception answering. However, existing works usually overlooked the conflicts between these two types of perception explanations during joint instruction tuning, leading to insufficient perception understanding. To mitigate this, we propose a new paradigm for perception-oriented instruction tuning, i.e., Q-Adapt, which aims to eliminate the conflicts and achieve the synergy between these two EIQA tasks when adapting LMM, resulting in enhanced multi-faceted explanations of IQA. Particularly, we propose a progressive instruction tuning strategy by dividing the adaption process of LMM for EIQA into two stages, where the first stage empowers the LMM with universal perception knowledge tailored for two tasks using an efficient transfer learning strategy, i.e., LoRA, and the second stage introduces the instruction-adaptive visual prompt tuning to dynamically adapt visual features for the different instructions from two tasks. In this way, our proposed Q-Adapt can achieve a lightweight visual quality evaluator, demonstrating comparable performance and, in some instances, superior results across perceptual-related benchmarks and commonly-used IQA databases. The source code is publicly available at this https URL.
• 摘要：大型多模式基础模型（LMM）的快速发展为可能解释的图像质量评估（EIQA）铺平了道路，并从两个角度进行了教学调整：整体质量解释和属性的感知回答。但是，现有的作品通常忽略了这两种类型的感知解释之间的冲突，从而导致理解不足。为了减轻这种情况，我们提出了一种新的面向感知指令调整的范式，即Q-Adapt，旨在消除冲突并在适应LMM时实现这两个EIQA任务之间的协同作用，从而导致IQA的增强的多面解释IQA。特别是，我们通过将EIQA的LMM的适应过程分为两个阶段，提出了一种渐进的教学调整策略，在该阶段中，使用有效的转移学习策略（即Lora，Lora，Lora，第二阶段）介绍了指导性的视觉效果，从而使LMM量身定制了为两个任务量身定制的通用感知知识。通过这种方式，我们提出的Q-Adapt可以实现轻巧的视觉质量评估器，表现出可比的性能，并且在某些情况下，在与感知相关的基准和常用的IQA数据库中相比，较高的结果。源代码可在此HTTPS URL上公开可用。

Title: Robust Unsupervised Domain Adaptation for 3D Point Cloud Segmentation Under Source Adversarial Attacks

• Authors: Haosheng Li, Yuecong Xu, Junjie Chen, Kemi Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01659
• Pdf URL: https://arxiv.org/pdf/2504.01659
• Copy Paste: [[2504.01659]] Robust Unsupervised Domain Adaptation for 3D Point Cloud Segmentation Under Source Adversarial Attacks(https://arxiv.org/abs/2504.01659)
• Keywords: generation
• Abstract: Unsupervised domain adaptation (UDA) frameworks have shown good generalization capabilities for 3D point cloud semantic segmentation models on clean data. However, existing works overlook adversarial robustness when the source domain itself is compromised. To comprehensively explore the robustness of the UDA frameworks, we first design a stealthy adversarial point cloud generation attack that can significantly contaminate datasets with only minor perturbations to the point cloud surface. Based on that, we propose a novel dataset, AdvSynLiDAR, comprising synthesized contaminated LiDAR point clouds. With the generated corrupted data, we further develop the Adversarial Adaptation Framework (AAF) as the countermeasure. Specifically, by extending the key point sensitive (KPS) loss towards the Robust Long-Tail loss (RLT loss) and utilizing a decoder branch, our approach enables the model to focus on long-tail classes during the pre-training phase and leverages high-confidence decoded point cloud information to restore point cloud structures during the adaptation phase. We evaluated our AAF method on the AdvSynLiDAR dataset, where the results demonstrate that our AAF method can mitigate performance degradation under source adversarial perturbations for UDA in the 3D point cloud segmentation application.
• 摘要：无监督的域适应（UDA）框架已显示出良好的概括能力，可在干净的数据上使用3D点云语义分割模型。但是，当源域本身受到损害时，现有作品忽略了对抗性的鲁棒性。为了全面地探索UDA框架的鲁棒性，我们首先设计了一个隐秘的对抗点云生成攻击，该攻击只能对点云表面造成少量扰动，从而显着污染数据集。基于此，我们提出了一个新型数据集Advsynlidar，其中包括合成的受污染的LiDAR点云。通过生成的损坏数据，我们进一步开发了对抗适应框架（AAF）作为对策。具体而言，通过将关键点敏感（KPS）损失扩展到可靠的长尾损失（RLT损失）并利用解码器分支，我们的方法使该模型能够在训练前阶段专注于长尾类，并利用高信心解码的点云信息在适应阶段恢复了点云信息。我们在Advsynlidar数据集上评估了我们的AAF方法，其中结果表明我们的AAF方法可以在3D点云分割应用程序中对UDA的源对抗扰动下的性能降解。

Title: {GSR4B}: Biomass Map Super-Resolution with Sentinel-1/2 Guidance

• Authors: Kaan Karaman, Yuchang Jiang, Damien Robert, Vivien Sainte Fare Garnot, Maria João Santos, Jan Dirk Wegner
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01722
• Pdf URL: https://arxiv.org/pdf/2504.01722
• Copy Paste: [[2504.01722]] {GSR4B}: Biomass Map Super-Resolution with Sentinel-1/2 Guidance(https://arxiv.org/abs/2504.01722)
• Keywords: super-resolution
• Abstract: Accurate Above-Ground Biomass (AGB) mapping at both large scale and high spatio-temporal resolution is essential for applications ranging from climate modeling to biodiversity assessment, and sustainable supply chain monitoring. At present, fine-grained AGB mapping relies on costly airborne laser scanning acquisition campaigns usually limited to regional scales. Initiatives such as the ESA CCI map attempt to generate global biomass products from diverse spaceborne sensors but at a coarser resolution. To enable global, high-resolution (HR) mapping, several works propose to regress AGB from HR satellite observations such as ESA Sentinel-1/2 images. We propose a novel way to address HR AGB estimation, by leveraging both HR satellite observations and existing low-resolution (LR) biomass products. We cast this problem as Guided Super-Resolution (GSR), aiming at upsampling LR biomass maps (sources) from $100$ to $10$ m resolution, using auxiliary HR co-registered satellite images (guides). We compare super-resolving AGB maps with and without guidance, against direct regression from satellite images, on the public BioMassters dataset. We observe that Multi-Scale Guidance (MSG) outperforms direct regression both for regression ($-780$ t/ha RMSE) and perception ($+2.0$ dB PSNR) metrics, and better captures high-biomass values, without significant computational overhead. Interestingly, unlike the RGB+Depth setting they were originally designed for, our best-performing AGB GSR approaches are those that most preserve the guide image texture. Our results make a strong case for adopting the GSR framework for accurate HR biomass mapping at scale. Our code and model weights are made publicly available (this https URL).
• 摘要：准确的地面生物量（AGB）映射在大规模和高时空分辨率上均对从气候建模到生物多样性评估以及可持续供应链监测的应用至关重要。目前，细粒度的AGB映射依赖于昂贵的机载激光扫描收购活动，通常仅限于区域尺度。 ESA CCI地图之类的计划试图从不同的太空传感器中生成全球生物量产品，但以更粗的分辨率生成。为了实现全局高分辨率（HR）映射，几项作品建议从HR卫星观测值（例如ESA Sentinel-1/2图像）中回归AGB。我们提出了一种通过利用HR卫星观测和现有的低分辨率（LR）生物量产品来解决HR AGB估计的新型方法。我们使用辅助HR共同注册的卫星图像（Guides）（Guides），我们将这个问题作为指导性超分辨率（GSR），目的是将LR生物量地图（来源）从$ 100 $分辨率提高到$ 10 $ M分辨率。我们将带有和没有指导的超级分辨AGB地图与公共生物堆数据集中的卫星图像进行直接回归。我们观察到，多尺度指南（MSG）的回归直接回归（$ -780 $ t/ha rmse）和感知（$++++2.0 $ db psnr）指标，并且更好地捕获了高生物量的值，而没有大量的计算机。有趣的是，与最初为RGB+深度设置不同，我们表现最好的AGB GSR方法是最保留指南图像纹理的方法。我们的结果为采用GSR框架以进行精确的HR生物量映射提供了有力的理由。我们的代码和模型权重公开可用（此HTTPS URL）。

Title: DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidance

• Authors: Yuxuan Luo, Zhengkun Rong, Lizhen Wang, Longhao Zhang, Tianshu Hu, Yongming Zhu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.01724
• Pdf URL: https://arxiv.org/pdf/2504.01724
• Copy Paste: [[2504.01724]] DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidance(https://arxiv.org/abs/2504.01724)
• Keywords: generation
• Abstract: While recent image-based human animation methods achieve realistic body and facial motion synthesis, critical gaps remain in fine-grained holistic controllability, multi-scale adaptability, and long-term temporal coherence, which leads to their lower expressiveness and robustness. We propose a diffusion transformer (DiT) based framework, DreamActor-M1, with hybrid guidance to overcome these limitations. For motion guidance, our hybrid control signals that integrate implicit facial representations, 3D head spheres, and 3D body skeletons achieve robust control of facial expressions and body movements, while producing expressive and identity-preserving animations. For scale adaptation, to handle various body poses and image scales ranging from portraits to full-body views, we employ a progressive training strategy using data with varying resolutions and scales. For appearance guidance, we integrate motion patterns from sequential frames with complementary visual references, ensuring long-term temporal coherence for unseen regions during complex movements. Experiments demonstrate that our method outperforms the state-of-the-art works, delivering expressive results for portraits, upper-body, and full-body generation with robust long-term consistency. Project Page: this https URL.
• 摘要：虽然最近基于图像的人类动画方法实现了现实的身体和面部运动综合，但临界差距仍保持精细的整体可控性，多尺度适应性和长期的时间连贯性，从而导致其较低的表现力和稳健性。我们提出了一个基于扩散变压器（DIT）的框架Dreamactor-M1，并通过混合指导来克服这些限制。为了进行运动引导，我们的混合控制信号集成了隐式面部表示，3D头部球和3D身体骨架，可实现对面部表情和身体运动的强大控制，同时产生表达和具有身份的动画。为了适应秤，要处理从肖像到全身视图的各种身体姿势和图像秤，我们使用具有不同分辨率和秤的数据采用了渐进式培训策略。对于外观引导，我们将来自顺序框架的运动模式与互补的视觉参考结合在一起，从而确保复杂运动过程中看不见区域的长期时间连贯性。实验表明，我们的方法的表现优于最先进的作品，为肖像，上身和全身产生带来了富有稳定的长期一致性的表现力结果。项目页面：此HTTPS URL。

Title: Understanding Cross-Model Perceptual Invariances Through Ensemble Metamers

• Authors: Lukas Boehm, Jonas Leo Mueller, Christoffer Loeffler, Leo Schwinn, Bjoern Eskofier, Dario Zanca
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01739
• Pdf URL: https://arxiv.org/pdf/2504.01739
• Copy Paste: [[2504.01739]] Understanding Cross-Model Perceptual Invariances Through Ensemble Metamers(https://arxiv.org/abs/2504.01739)
• Keywords: generation
• Abstract: Understanding the perceptual invariances of artificial neural networks is essential for improving explainability and aligning models with human vision. Metamers - stimuli that are physically distinct yet produce identical neural activations - serve as a valuable tool for investigating these invariances. We introduce a novel approach to metamer generation by leveraging ensembles of artificial neural networks, capturing shared representational subspaces across diverse architectures, including convolutional neural networks and vision transformers. To characterize the properties of the generated metamers, we employ a suite of image-based metrics that assess factors such as semantic fidelity and naturalness. Our findings show that convolutional neural networks generate more recognizable and human-like metamers, while vision transformers produce realistic but less transferable metamers, highlighting the impact of architectural biases on representational invariances.
• 摘要：了解人工神经网络的感知不变，对于提高解释性和与人类视野的模型保持一致至关重要。 METAMERS-物理上不同但会产生相同神经激活的刺激 - 是研究这些不断增长的宝贵工具。我们通过利用人工神经网络的合奏来介绍一种新的Metamer生成方法，捕获了包括卷积神经网络和视觉变压器在内的各种体系结构之间的共享代表性子空间。为了表征生成的metamer的性质，我们采用了一套基于图像的指标来评估语义忠诚度和自然性等因素。我们的发现表明，卷积神经网络会产生更具识别性和类似人类的变形物，而视觉变压器产生了现实但不可传递的元素，强调了建筑偏见对代表性不可分裂的影响。

Title: Bridge the Gap between SNN and ANN for Image Restoration

• Authors: Xin Su, Chen Wu, Zhuoran Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01755
• Pdf URL: https://arxiv.org/pdf/2504.01755
• Copy Paste: [[2504.01755]] Bridge the Gap between SNN and ANN for Image Restoration(https://arxiv.org/abs/2504.01755)
• Keywords: restoration
• Abstract: Models of dense prediction based on traditional Artificial Neural Networks (ANNs) require a lot of energy, especially for image restoration tasks. Currently, neural networks based on the SNN (Spiking Neural Network) framework are beginning to make their mark in the field of image restoration, especially as they typically use less than 10\% of the energy of ANNs with the same architecture. However, training an SNN is much more expensive than training an ANN, due to the use of the heuristic gradient descent strategy. In other words, the process of SNN's potential membrane signal changing from sparse to dense is very slow, which affects the convergence of the whole this http URL tackle this problem, we propose a novel distillation technique, called asymmetric framework (ANN-SNN) distillation, in which the teacher is an ANN and the student is an SNN. Specifically, we leverage the intermediate features (feature maps) learned by the ANN as hints to guide the training process of the SNN. This approach not only accelerates the convergence of the SNN but also improves its final performance, effectively bridging the gap between the efficiency of the SNN and the superior learning capabilities of ANN. Extensive experimental results show that our designed SNN-based image restoration model, which has only 1/300 the number of parameters of the teacher network and 1/50 the energy consumption of the teacher network, is as good as the teacher network in some denoising tasks.
• 摘要：基于传统人工神经网络（ANN）的密集预测模型需要大量能量，尤其是对于图像恢复任务。当前，基于SNN（尖峰神经网络）框架的神经网络正在开始在图像恢复领域中标记，尤其是因为它们通常使用相同体系结构的ANN能量的10 \％。但是，由于使用了启发式梯度下降策略，训练SNN比训练ANN要贵得多。换句话说，SNN潜在的膜信号从稀疏变为密集的过程非常慢，这会影响整个HTTP URL的整体收敛性解决这个问题，我们提出了一种新型的蒸馏技术，称为不对称框架（ANN-SNN）蒸馏，其中老师是ANN，学生是SNN。具体来说，我们利用ANN学到的中间特征（特征地图）作为指导SNN的训练过程的提示。这种方法不仅加速了SNN的收敛性，而且可以提高其最终性能，从而有效地弥合了SNN效率与ANN卓越学习能力之间的差距。广泛的实验结果表明，我们设计的基于SNN的图像恢复模型只有1/300的教师网络参数数量和1/50的教师网络的能耗，在某些DeNoSising任务中与教师网络一样好。

Title: Prompting Medical Vision-Language Models to Mitigate Diagnosis Bias by Generating Realistic Dermoscopic Images

• Authors: Nusrat Munia, Abdullah-Al-Zubaer Imran
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01838
• Pdf URL: https://arxiv.org/pdf/2504.01838
• Copy Paste: [[2504.01838]] Prompting Medical Vision-Language Models to Mitigate Diagnosis Bias by Generating Realistic Dermoscopic Images(https://arxiv.org/abs/2504.01838)
• Keywords: generative
• Abstract: Artificial Intelligence (AI) in skin disease diagnosis has improved significantly, but a major concern is that these models frequently show biased performance across subgroups, especially regarding sensitive attributes such as skin color. To address these issues, we propose a novel generative AI-based framework, namely, Dermatology Diffusion Transformer (DermDiT), which leverages text prompts generated via Vision Language Models and multimodal text-image learning to generate new dermoscopic images. We utilize large vision language models to generate accurate and proper prompts for each dermoscopic image which helps to generate synthetic images to improve the representation of underrepresented groups (patient, disease, etc.) in highly imbalanced datasets for clinical diagnoses. Our extensive experimentation showcases the large vision language models providing much more insightful representations, that enable DermDiT to generate high-quality images. Our code is available at this https URL
• 摘要：皮肤疾病诊断中的人工智能（AI）有了显着改善，但主要问题是，这些模型经常表现出跨亚组的偏见性能，尤其是关于诸如肤色之类的敏感属性。为了解决这些问题，我们提出了一个基于AI的新颖性框架，即皮肤科扩散变压器（Dermdit），该框架利用文本提示通过视觉语言模型和多模式文本图像学习生成的文本以生成新的皮肤镜图像。我们利用大型视觉语言模型为每个皮肤镜图像生成准确和适当的提示，这有助于产生合成图像，以改善高度不平衡的数据集中代表不足的组（患者，疾病等）以进行临床诊断。我们广泛的实验展示了大型视觉语言模型，提供了更具洞察力的表示，使Dermdit能够产生高质量的图像。我们的代码可在此HTTPS URL上找到

Title: Multi-fidelity Parameter Estimation Using Conditional Diffusion Models

• Authors: Caroline Tatsuoka, Minglei Yang, Dongbin Xiu, Guannan Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.01894
• Pdf URL: https://arxiv.org/pdf/2504.01894
• Copy Paste: [[2504.01894]] Multi-fidelity Parameter Estimation Using Conditional Diffusion Models(https://arxiv.org/abs/2504.01894)
• Keywords: generative
• Abstract: We present a multi-fidelity method for uncertainty quantification of parameter estimates in complex systems, leveraging generative models trained to sample the target conditional distribution. In the Bayesian inference setting, traditional parameter estimation methods rely on repeated simulations of potentially expensive forward models to determine the posterior distribution of the parameter values, which may result in computationally intractable workflows. Furthermore, methods such as Markov Chain Monte Carlo (MCMC) necessitate rerunning the entire algorithm for each new data observation, further increasing the computational burden. Hence, we propose a novel method for efficiently obtaining posterior distributions of parameter estimates for high-fidelity models given data observations of interest. The method first constructs a low-fidelity, conditional generative model capable of amortized Bayesian inference and hence rapid posterior density approximation over a wide-range of data observations. When higher accuracy is needed for a specific data observation, the method employs adaptive refinement of the density approximation. It uses outputs from the low-fidelity generative model to refine the parameter sampling space, ensuring efficient use of the computationally expensive high-fidelity solver. Subsequently, a high-fidelity, unconditional generative model is trained to achieve greater accuracy in the target posterior distribution. Both low- and high- fidelity generative models enable efficient sampling from the target posterior and do not require repeated simulation of the high-fidelity forward model. We demonstrate the effectiveness of the proposed method on several numerical examples, including cases with multi-modal densities, as well as an application in plasma physics for a runaway electron simulation model.
• 摘要：我们提出了一种多保真方法，用于对复杂系统中参数估计值的不确定性定量，利用了经过训练的生成模型来采样目标条件分布。在贝叶斯推理环境中，传统的参数估计方法依赖于对潜在昂贵的远期模型的重复模拟来确定参数值的后验分布，这可能导致计算上棘手的工作流程。此外，诸如马尔可夫链蒙特卡洛（MCMC）之类的方法需要为每个新的数据观察重新研究整个算法，从而进一步增加计算负担。因此，我们提出了一种新的方法，用于有效地获得高保真模型的参数估计值的后验分布，并给定关注的数据观察。该方法首先构建了能够在广泛的数据观察结果上构建能够摊销贝叶斯推断的低保真性，有条件的生成模型，并因此构建了摊销的贝叶斯推理以及后期的快速后密度近似。当特定数据观察需要更高的精度时，该方法采用了密度近似的自适应改进。它使用低保真生成模型的输出来完善参数采样空间，以确保有效利用计算昂贵的高保真求解器。随后，对高保真性，无条件生成模型进行了训练，以在目标后分布中获得更高的精度。低和高保真生成模型都可以从目标后部进行有效的采样，并且不需要重复对高保真正向模型进行模拟。我们在几个数值示例中证明了该方法的有效性，包括具有多模式密度的病例，以及在血浆物理中应用于失控的电子仿真模型。

Title: Benchmarking Synthetic Tabular Data: A Multi-Dimensional Evaluation Framework

• Authors: Andrey Sidorenko, Michael Platzer, Mario Scriminaci, Paul Tiwald
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.01908
• Pdf URL: https://arxiv.org/pdf/2504.01908
• Copy Paste: [[2504.01908]] Benchmarking Synthetic Tabular Data: A Multi-Dimensional Evaluation Framework(https://arxiv.org/abs/2504.01908)
• Keywords: generation
• Abstract: Evaluating the quality of synthetic data remains a key challenge for ensuring privacy and utility in data-driven research. In this work, we present an evaluation framework that quantifies how well synthetic data replicates original distributional properties while ensuring privacy. The proposed approach employs a holdout-based benchmarking strategy that facilitates quantitative assessment through low- and high-dimensional distribution comparisons, embedding-based similarity measures, and nearest-neighbor distance metrics. The framework supports various data types and structures, including sequential and contextual information, and enables interpretable quality diagnostics through a set of standardized metrics. These contributions aim to support reproducibility and methodological consistency in benchmarking of synthetic data generation techniques. The code of the framework is available at this https URL.
• 摘要：评估合成数据的质量仍然是确保数据驱动研究中隐私和实用性的关键挑战。在这项工作中，我们提出了一个评估框架，该框架可以量化合成数据在确保隐私的同时复制原始分布属性的程度。提出的方法采用基于固定的基准测试策略，通过低维分布比较，基于嵌入的相似性度量和最近的邻居距离指标来促进定量评估。该框架支持各种数据类型和结构，包括顺序和上下文信息，并通过一组标准化指标启用可解释的质量诊断。这些贡献旨在支持合成数据生成技术基准测试的可重复性和方法学的一致性。该框架的代码可在此HTTPS URL上获得。

Title: FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs

• Authors: Mothilal Asokan, Kebin Wu, Fatima Albreiki
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2504.01916
• Pdf URL: https://arxiv.org/pdf/2504.01916
• Copy Paste: [[2504.01916]] FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs(https://arxiv.org/abs/2504.01916)
• Keywords: generation
• Abstract: As a pioneering vision-language model, CLIP (Contrastive Language-Image Pre-training) has achieved significant success across various domains and a wide range of downstream vision-language tasks. However, the text encoders in popular CLIP models are limited to processing only 77 text tokens, which constrains their ability to effectively handle longer, detail-rich captions. Additionally, CLIP models often struggle to effectively capture detailed visual and textual information, which hampers their performance on tasks that require fine-grained analysis. To address these limitations, we present a novel approach, \textbf{FineLIP}, that extends the capabilities of CLIP. FineLIP enhances cross-modal text-image mapping by incorporating \textbf{Fine}-grained alignment with \textbf{L}onger text input within the CL\textbf{IP}-style framework. FineLIP first extends the positional embeddings to handle longer text, followed by the dynamic aggregation of local image and text tokens. The aggregated results are then used to enforce fine-grained token-to-token cross-modal alignment. We validate our model on datasets with long, detailed captions across two tasks: zero-shot cross-modal retrieval and text-to-image generation. Quantitative and qualitative experimental results demonstrate the effectiveness of FineLIP, outperforming existing state-of-the-art approaches. Furthermore, comprehensive ablation studies validate the benefits of key design elements within FineLIP.
• 摘要：作为开创性的视觉语言模型，剪辑（对比语言图像预训练）在各个领域和广泛的下游视觉 - 语言任务上取得了巨大的成功。但是，流行剪辑模型中的文本编码仅限于处理77个文本令牌，这会限制其有效处理更长，细节丰富的字幕的能力。此外，剪辑模型通常很难有效地捕获详细的视觉和文本信息，从而阻碍了他们在需要细粒分析的任务上的性能。为了解决这些局限性，我们提出了一种新颖的方法\ textbf {finelip}，该方法扩展了剪辑的功能。 Finelip通过将\ textbf {fine}对齐与\ textbf {l} cl \ textbf {ip} - 式式框架中的\ textbf {l}结合到\ textbf {l} onger文本输入来增强跨模式文本图像映射。 Finelip首先扩展了位置嵌入以处理更长的文本，然后是本地图像和文本令牌的动态聚合。然后，汇总结果用于强制执行细粒度的令牌到框的交叉模式对齐。我们在数据集上验证了我们的模型，并在两个任务中具有长而详细的字幕：零射击横向模式检索和文本对图像生成。定量和定性实验结果证明了Finelip的有效性，表现优于现有的最新方法。此外，全面的消融研究验证了Finelip中关键设计元素的好处。

Title: ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement

• Authors: Runhui Huang, Chunwei Wang, Junwei Yang, Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, Hang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01934
• Pdf URL: https://arxiv.org/pdf/2504.01934
• Copy Paste: [[2504.01934]] ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement(https://arxiv.org/abs/2504.01934)
• Keywords: super-resolution, generation
• Abstract: We present ILLUME+ that leverages dual visual tokenization and a diffusion decoder to improve both deep semantic understanding and high-fidelity image generation. Existing unified models have struggled to simultaneously handle the three fundamental capabilities in a unified model: understanding, generation, and editing. Models like Chameleon and EMU3 utilize VQGAN for image discretization, due to the lack of deep semantic interaction, they lag behind specialist models like LLaVA in visual understanding tasks. To mitigate this, LaViT and ILLUME employ semantic encoders for tokenization, but they struggle with image editing due to poor texture preservation. Meanwhile, Janus series decouples the input and output image representation, limiting their abilities to seamlessly handle interleaved image-text understanding and generation. In contrast, ILLUME+ introduces a unified dual visual tokenizer, DualViTok, which preserves both fine-grained textures and text-aligned semantics while enabling a coarse-to-fine image representation strategy for multimodal understanding and generation. Additionally, we employ a diffusion model as the image detokenizer for enhanced generation quality and efficient super-resolution. ILLUME+ follows a continuous-input, discrete-output scheme within the unified MLLM and adopts a progressive training procedure that supports dynamic resolution across the vision tokenizer, MLLM, and diffusion decoder. This design allows for flexible and efficient context-aware image editing and generation across diverse tasks. ILLUME+ (3B) exhibits competitive performance against existing unified MLLMs and specialized models across multimodal understanding, generation, and editing benchmarks. With its strong performance, ILLUME+ provides a scalable and versatile foundation for future multimodal applications. Project Page: this https URL.
• 摘要：我们提出了利用双重视觉令牌化和扩散解码器来改善深层语义理解和高保真图像产生的iLlume+。现有的统一模型一直在努力在统一模型中同时处理三个基本功能：理解，生成和编辑。 Chameleon和Emu3等模型将VQGAN用于图像离散化，由于缺乏深层语义相互作用，它们落后于Llava等专家模型，例如在视觉理解任务中。为了减轻这种情况，Lavit和Illume采用语义编码器进行令牌化，但由于质地保存不佳，它们在图像编辑方面挣扎。同时，Janus系列将输入和输出图像表示形式分解，从而限制了它们无缝处理交织的图像文本理解和生成的能力。相比之下，Illume+引入了统一的双重视觉令牌dualVitok，该统一的纹理和文本一致的语义既保留了细粒的纹理，同时启用了多模式理解和产生的粗到1个图像表示策略。此外，我们采用扩散模型作为图像偏言，以增强发电质量和有效的超分辨率。 Illume+遵循统一MLLM内的连续输入，离散输出方案，并采用了渐进式训练程序，该程序支持整个视觉令牌，MLLM和扩散解码器的动态分辨率。这种设计允许跨不同任务进行灵活有效的上下文感知图像编辑和生成。 Illume+（3b）在多模式理解，生成和编辑基准的现有统一MLLM和专业模型上表现出竞争性能。 Illume+凭借其出色的性能为将来的多模式应用提供了可扩展和多功能的基础。项目页面：此HTTPS URL。

Title: A Unified Approach to Analysis and Design of Denoising Markov Models

• Authors: Yinuo Ren, Grant M. Rotskoff, Lexing Ying
• Subjects: cs.LG, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2504.01938
• Pdf URL: https://arxiv.org/pdf/2504.01938
• Copy Paste: [[2504.01938]] A Unified Approach to Analysis and Design of Denoising Markov Models(https://arxiv.org/abs/2504.01938)
• Keywords: generative
• Abstract: Probabilistic generative models based on measure transport, such as diffusion and flow-based models, are often formulated in the language of Markovian stochastic dynamics, where the choice of the underlying process impacts both algorithmic design choices and theoretical analysis. In this paper, we aim to establish a rigorous mathematical foundation for denoising Markov models, a broad class of generative models that postulate a forward process transitioning from the target distribution to a simple, easy-to-sample distribution, alongside a backward process particularly constructed to enable efficient sampling in the reverse direction. Leveraging deep connections with nonequilibrium statistical mechanics and generalized Doob's $h$-transform, we propose a minimal set of assumptions that ensure: (1) explicit construction of the backward generator, (2) a unified variational objective directly minimizing the measure transport discrepancy, and (3) adaptations of the classical score-matching approach across diverse dynamics. Our framework unifies existing formulations of continuous and discrete diffusion models, identifies the most general form of denoising Markov models under certain regularity assumptions on forward generators, and provides a systematic recipe for designing denoising Markov models driven by arbitrary Lévy-type processes. We illustrate the versatility and practical effectiveness of our approach through novel denoising Markov models employing geometric Brownian motion and jump processes as forward dynamics, highlighting the framework's potential flexibility and capability in modeling complex distributions.
• 摘要：基于测量传输的概率生成模型，例如扩散和基于流量的模型，通常以马尔可夫随机动力学的语言制定，其中基础过程的选择都会影响算法设计选择和理论分析。在本文中，我们旨在为Denoising Markov模型建立一个严格的数学基础，Markov模型是一类广泛的生成模型，假定从目标分布到简单，易于样本的分布的前进过程过渡，并与构建的向后过程特别构建，以在反向方向上启用有效的采样。利用非平衡统计力学和广义的DOOB的$ H $转换，我们提出了一组最小的假设，以确保：（1）显式构造向后发电机，（2）一个统一的变分目标直接直接使量度运输差异和（3）适应跨越类别的分数动态的统一变异目标。我们的框架统一了连续和离散扩散模型的现有公式，它标识了在正向发生器上的某些规律性假设下最通用的Markov模型的形式，并提供了一个系统的配方，用于设计由任意lévyVyVy-Lévy-type流程驱动的DeNo Markov模型。我们通过新颖的DeNoing Markov模型来说明我们的方法的多功能性和实际有效性，该模型采用几何布朗运动和跳跃过程作为前向动力学，突出了该框架在建模复杂分布时的潜在灵活性和能力。

Title: VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step

• Authors: Hanyang Wang, Fangfu Liu, Jiawei Chi, Yueqi Duan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01956
• Pdf URL: https://arxiv.org/pdf/2504.01956
• Copy Paste: [[2504.01956]] VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step(https://arxiv.org/abs/2504.01956)
• Keywords: generation, generative
• Abstract: Recovering 3D scenes from sparse views is a challenging task due to its inherent ill-posed problem. Conventional methods have developed specialized solutions (e.g., geometry regularization or feed-forward deterministic model) to mitigate the issue. However, they still suffer from performance degradation by minimal overlap across input views with insufficient visual information. Fortunately, recent video generative models show promise in addressing this challenge as they are capable of generating video clips with plausible 3D structures. Powered by large pretrained video diffusion models, some pioneering research start to explore the potential of video generative prior and create 3D scenes from sparse views. Despite impressive improvements, they are limited by slow inference time and the lack of 3D constraint, leading to inefficiencies and reconstruction artifacts that do not align with real-world geometry structure. In this paper, we propose VideoScene to distill the video diffusion model to generate 3D scenes in one step, aiming to build an efficient and effective tool to bridge the gap from video to 3D. Specifically, we design a 3D-aware leap flow distillation strategy to leap over time-consuming redundant information and train a dynamic denoising policy network to adaptively determine the optimal leap timestep during inference. Extensive experiments demonstrate that our VideoScene achieves faster and superior 3D scene generation results than previous video diffusion models, highlighting its potential as an efficient tool for future video to 3D applications. Project Page: this https URL
• 摘要：从稀疏视图中恢复3D场景是一项具有挑战性的任务，因为它固有的问题固有的问题。常规方法开发了专门的解决方案（例如几何正规化或馈送前馈确定性模型）来减轻问题。但是，他们仍然因在视图上的最小重叠而遭受性能降解，并且视觉信息不足。幸运的是，最近的视频生成模型在应对这一挑战方面有望，因为它们能够生成具有合理的3D结构的视频剪辑。一些开创性的研究由大型的视频扩散模型提供支持，开始探索视频生成之前的潜力，并从稀疏视图中创建3D场景。尽管有令人印象深刻的改进，但它们受到推理时间缓慢和缺乏3D约束的限制，导致效率低下和与现实世界几何结构不符的重建伪像。在本文中，我们提出视频苏芬，以提炼视频扩散模型，以一步生成3D场景，旨在建立一个有效的有效工具，以弥合从视频到3D的差距。具体而言，我们设计了3D感知的LEAP流动蒸馏策略，以跨越耗时的冗余信息并训练动态的DeNoising策略网络，以适应推理期间的最佳LEAP TIMESTEP。广泛的实验表明，与以前的视频扩散模型相比，我们的视频兴趣烯能够更快，更高的3D场景生成结果，从而强调了其作为将来对3D应用程序的未来视频的有效工具的潜力。项目页面：此HTTPS URL