2025-04-02

Title: LayerCraft: Enhancing Text-to-Image Generation with CoT Reasoning and Layered Object Integration

• Authors: Yuyao Zhang, Jinghao Li, Yu-Wing Tai
• Subjects: cs.LG, cs.GR, cs.MA
• Abstract URL: https://arxiv.org/abs/2504.00010
• Pdf URL: https://arxiv.org/pdf/2504.00010
• Copy Paste: [[2504.00010]] LayerCraft: Enhancing Text-to-Image Generation with CoT Reasoning and Layered Object Integration(https://arxiv.org/abs/2504.00010)
• Keywords: generation
• Abstract: Text-to-image generation (T2I) has become a key area of research with broad applications. However, existing methods often struggle with complex spatial relationships and fine-grained control over multiple concepts. Many existing approaches require significant architectural modifications, extensive training, or expert-level prompt engineering. To address these challenges, we introduce \textbf{LayerCraft}, an automated framework that leverages large language models (LLMs) as autonomous agents for structured procedural generation. LayerCraft enables users to customize objects within an image and supports narrative-driven creation with minimal effort. At its core, the system includes a coordinator agent that directs the process, along with two specialized agents: \textbf{ChainArchitect}, which employs chain-of-thought (CoT) reasoning to generate a dependency-aware 3D layout for precise instance-level control, and the \textbf{Object-Integration Network (OIN)}, which utilizes LoRA fine-tuning on pre-trained T2I models to seamlessly blend objects into specified regions of an image based on textual prompts without requiring architectural changes. Extensive evaluations demonstrate LayerCraft's versatility in applications ranging from multi-concept customization to storytelling. By providing non-experts with intuitive, precise control over T2I generation, our framework democratizes creative image creation. Our code will be released upon acceptance at this http URL
• 摘要：文本到图像生成（T2I）已成为广泛应用的研究的关键领域。但是，现有的方法通常在复杂的空间关系和对多个概念的细粒度控制方面遇到困难。许多现有的方法需要重大的建筑修改，广泛的培训或专家级及时的工程。为了应对这些挑战，我们介绍了\ textbf {layercraft}，这是一种自动化框架，利用大型语言模型（LLMS）作为结构化过程生成的自主剂。 LayerCraft使用户可以自定义图像中的对象，并以最少的精力支持叙事驱动的创建。 At its core, the system includes a coordinator agent that directs the process, along with two specialized agents: \textbf{ChainArchitect}, which employs chain-of-thought (CoT) reasoning to generate a dependency-aware 3D layout for precise instance-level control, and the \textbf{Object-Integration Network (OIN)}, which utilizes LoRA fine-tuning on pre-trained T2I基于文本提示，无需架构更改即可无缝将对象无缝融合到图像的指定区域。广泛的评估表明，Layercraft在从多概念自定义到讲故事的应用中的多功能性。通过为非专家提供直观，精确的对T2i代的控制，我们的框架使创造性的形象创造民主化。我们的代码将在此HTTP URL接受后发布

Title: A Novel Distance-Based Metric for Quality Assessment in Image Segmentation

• Authors: Niklas Rottmayer, Claudia Redenbach
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2504.00023
• Pdf URL: https://arxiv.org/pdf/2504.00023
• Copy Paste: [[2504.00023]] A Novel Distance-Based Metric for Quality Assessment in Image Segmentation(https://arxiv.org/abs/2504.00023)
• Keywords: quality assessment
• Abstract: The assessment of segmentation quality plays a fundamental role in the development, optimization, and comparison of segmentation methods which are used in a wide range of applications. With few exceptions, quality assessment is performed using traditional metrics, which are based on counting the number of erroneous pixels but do not capture the spatial distribution of errors. Established distance-based metrics such as the average Hausdorff distance are difficult to interpret and compare for different methods and datasets. In this paper, we introduce the Surface Consistency Coefficient (SCC), a novel distance-based quality metric that quantifies the spatial distribution of errors based on their proximity to the surface of the structure. Through a rigorous analysis using synthetic data and real segmentation results, we demonstrate the robustness and effectiveness of SCC in distinguishing errors near the surface from those further away. At the same time, SCC is easy to interpret and comparable across different structural contexts.
• 摘要：分割质量的评估在开发，优化和比较分割方法中起着基本作用，这些方法用于广泛应用。除了少数例外，使用传统指标进行质量评估，这些指标基于计算错误像素的数量，但不会捕获错误的空间分布。建立的基于距离的指标，例如平均Hausdorff距离，对于不同的方法和数据集很难解释和比较。在本文中，我们介绍了表面一致性系数（SCC），这是一种基于距离的新型质量度量，该指标可根据误差的空间分布，该指标基于其靠近结构表面的空间分布。通过使用合成数据和实际分割结果进行严格的分析，我们证明了SCC在将表面附近的误差与更远的误差区分开的鲁棒性和有效性。同时，在不同的结构环境中，SCC易于解释和可比。

Title: Few-Shot Generation of Brain Tumors for Secure and Fair Data Sharing

• Authors: Yongyi Shi, Ge Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00150
• Pdf URL: https://arxiv.org/pdf/2504.00150
• Copy Paste: [[2504.00150]] Few-Shot Generation of Brain Tumors for Secure and Fair Data Sharing(https://arxiv.org/abs/2504.00150)
• Keywords: generation, generative
• Abstract: Leveraging multi-center data for medical analytics presents challenges due to privacy concerns and data heterogeneity. While distributed approaches such as federated learning has gained traction, they remain vulnerable to privacy breaches, particularly in sensitive domains like medical imaging. Generative models, such as diffusion models, enhance privacy by synthesizing realistic data. However, they are prone to memorization, especially when trained on small datasets. This study proposes a decentralized few-shot generative model (DFGM) to synthesize brain tumor images while fully preserving privacy. DFGM harmonizes private tumor data with publicly shareable healthy images from multiple medical centers, constructing a new dataset by blending tumor foregrounds with healthy backgrounds. This approach ensures stringent privacy protection and enables controllable, high-quality synthesis by preserving both the healthy backgrounds and tumor foregrounds. We assess DFGM's effectiveness in brain tumor segmentation using a UNet, achieving Dice score improvements of 3.9% for data augmentation and 4.6% for fairness on a separate dataset.
• 摘要：由于隐私问题和数据异质性，利用多中心数据进行医学分析提出了挑战。尽管联合学习之类的分布式方法已引起关注，但它们仍然容易受到隐私漏洞的影响，尤其是在诸如医学成像之类的敏感领域。生成模型（例如扩散模型）通过综合现实数据来增强隐私。但是，它们很容易记忆，尤其是在小型数据集中接受培训时。这项研究提出了一个分散的少数发电模型（DFGM），以合成脑肿瘤图像，同时完全保留隐私。 DFGM将私人肿瘤数据与来自多个医疗中心的公开共享健康图像进行协调，从而通过将肿瘤前景融合具有健康背景的肿瘤前景来构建新数据集。这种方法可确保严格的隐私保护，并通过保留健康的背景和肿瘤前景来实现可控制的高质量综合。我们使用UNET评估DFGM在脑肿瘤分割方面的有效性，以提高数据增强的骰子得分为3.9％，而在单独的数据集中获得了4.6％的公平性。

Title: Self-Evolving Visual Concept Library using Vision-Language Critics

• Authors: Atharva Sehgal, Patrick Yuan, Ziniu Hu, Yisong Yue, Jennifer J. Sun, Swarat Chaudhuri
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.00185
• Pdf URL: https://arxiv.org/pdf/2504.00185
• Copy Paste: [[2504.00185]] Self-Evolving Visual Concept Library using Vision-Language Critics(https://arxiv.org/abs/2504.00185)
• Keywords: generation
• Abstract: We study the problem of building a visual concept library for visual recognition. Building effective visual concept libraries is challenging, as manual definition is labor-intensive, while relying solely on LLMs for concept generation can result in concepts that lack discriminative power or fail to account for the complex interactions between them. Our approach, ESCHER, takes a library learning perspective to iteratively discover and improve visual concepts. ESCHER uses a vision-language model (VLM) as a critic to iteratively refine the concept library, including accounting for interactions between concepts and how they affect downstream classifiers. By leveraging the in-context learning abilities of LLMs and the history of performance using various concepts, ESCHER dynamically improves its concept generation strategy based on the VLM critic's feedback. Finally, ESCHER does not require any human annotations, and is thus an automated plug-and-play framework. We empirically demonstrate the ability of ESCHER to learn a concept library for zero-shot, few-shot, and fine-tuning visual classification tasks. This work represents, to our knowledge, the first application of concept library learning to real-world visual tasks.
• 摘要：我们研究建立视觉概念库以进行视觉识别的问题。构建有效的视觉概念库是具有挑战性的，因为手动定义是劳动密集型的，而仅依靠LLM进行概念生成可能会导致缺乏歧视力或无法解释它们之间的复杂相互作用的概念。我们的方法埃舍尔（Escher）将图书馆的学习视角迭代地发现并改善了视觉概念。 Escher使用视觉模型（VLM）作为批评家迭代地完善概念库，包括考虑概念之间的相互作用以及它们如何影响下游分类器。通过利用LLM的文化学习能力和使用各种概念的绩效历史，Escher基于VLM评论家的反馈，动态改善了其概念生成策略。最后，Escher不需要任何人类注释，因此是自动插入式框架。我们从经验上证明了Escher学习一个概念库的能力，以进行零拍，很少的和微调的视觉分类任务。据我们所知，这项工作代表了概念库学习到现实世界视觉任务的首次应用。

Title: MultiMorph: On-demand Atlas Construction

• Authors: S. Mazdak Abulnaga, Andrew Hoopes, Neel Dey, Malte Hoffmann, Marianne Rakic, Bruce Fischl, John Guttag, Adrian Dalca
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.00247
• Pdf URL: https://arxiv.org/pdf/2504.00247
• Copy Paste: [[2504.00247]] MultiMorph: On-demand Atlas Construction(https://arxiv.org/abs/2504.00247)
• Keywords: generation
• Abstract: We present MultiMorph, a fast and efficient method for constructing anatomical atlases on the fly. Atlases capture the canonical structure of a collection of images and are essential for quantifying anatomical variability across populations. However, current atlas construction methods often require days to weeks of computation, thereby discouraging rapid experimentation. As a result, many scientific studies rely on suboptimal, precomputed atlases from mismatched populations, negatively impacting downstream analyses. MultiMorph addresses these challenges with a feedforward model that rapidly produces high-quality, population-specific atlases in a single forward pass for any 3D brain dataset, without any fine-tuning or optimization. MultiMorph is based on a linear group-interaction layer that aggregates and shares features within the group of input images. Further, by leveraging auxiliary synthetic data, MultiMorph generalizes to new imaging modalities and population groups at test-time. Experimentally, MultiMorph outperforms state-of-the-art optimization-based and learning-based atlas construction methods in both small and large population settings, with a 100-fold reduction in time. This makes MultiMorph an accessible framework for biomedical researchers without machine learning expertise, enabling rapid, high-quality atlas generation for diverse studies.
• 摘要：我们提出了多态，这是一种快速有效的方法，用于即时构建解剖图谱。地图集捕获图像集合的规范结构，对于量化种群的解剖变异性至关重要。但是，当前的Atlas施工方法通常需要数天至数周的计算，从而阻止快速实验。结果，许多科学研究依赖于不匹配种群的次优的，预先计算的地图集，对下游分析产生了负面影响。 Multimphl通过前馈模型解决了这些挑战，该模型在任何3D Brain数据集中迅速在单个正向通道中迅速产生高质量的特定于人群的图书馆，而无需进行任何微调或优化。 MultiMph基于线性组相互作用层，该层在输入图像组中汇总并共享特征。此外，通过利用辅助综合数据，多态将在测试时间中概括为新的成像方式和人群群体。在实验上，多态在小人口和大种群设置中的基于最先进的优化和基于学习的Atlas构建方法，时间降低了100倍。这使得多层层成为无机器人学习专业知识的生物医学研究人员的可访问框架，从而为多种研究提供了快速，高质量的地图集的生成。

Title: Diffusion models for probabilistic precipitation generation from atmospheric variables

• Authors: Michael Aich, Sebastian Bathiany, Philipp Hess, Yu Huang, Niklas Boers
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2504.00307
• Pdf URL: https://arxiv.org/pdf/2504.00307
• Copy Paste: [[2504.00307]] Diffusion models for probabilistic precipitation generation from atmospheric variables(https://arxiv.org/abs/2504.00307)
• Keywords: generation, generative
• Abstract: Improving the representation of precipitation in Earth system models (ESMs) is critical for assessing the impacts of climate change and especially of extreme events like floods and droughts. In existing ESMs, precipitation is not resolved explicitly, but represented by parameterizations. These typically rely on resolving approximated but computationally expensive column-based physics, not accounting for interactions between locations. They struggle to capture fine-scale precipitation processes and introduce significant biases. We present a novel approach, based on generative machine learning, which integrates a conditional diffusion model with a UNet architecture to generate accurate, high-resolution (0.25°) global daily precipitation fields from a small set of prognostic atmospheric variables. Unlike traditional parameterizations, our framework efficiently produces ensemble predictions, capturing uncertainties in precipitation, and does not require fine-tuning by hand. We train our model on the ERA5 reanalysis and present a method that allows us to apply it to arbitrary ESM data, enabling fast generation of probabilistic forecasts and climate scenarios. By leveraging interactions between global prognostic variables, our approach provides an alternative parameterization scheme that mitigates biases present in the ESM precipitation while maintaining consistency with its large-scale (annual) trends. This work demonstrates that complex precipitation patterns can be learned directly from large-scale atmospheric variables, offering a computationally efficient alternative to conventional schemes.
• 摘要：改善地球系统模型（ESM）中降水的表示对于评估气候变化的影响，尤其是洪水和干旱等极端事件的影响至关重要。在现有的ESM中，降水不是明确解决的，而是由参数化表示。这些通常依赖于解决近似但基于计算的基于计算昂贵的物理，而不是考虑位置之间的相互作用。他们努力捕获细小的降水过程并引入重大偏见。我们提出了一种基于生成机器学习的新方法，该方法将条件扩散模型与UNET结构集成在一起，以生成一小部分预后大气变量的准确，高分辨率（0.25°）的全球每日降水场。与传统的参数化不同，我们的框架有效地产生了集合预测，捕获降水中的不确定性，并且不需要手工进行微调。我们在ERA5重新分析上训练模型，并提出一种方法，使我们能够将其应用于任意ESM数据，从而可以快速生成概率的预测和气候场景。通过利用全球预后变量之间的相互作用，我们的方法提供了一种替代参数化方案，可减轻ESM降水中存在的偏差，同时保持与其大规模（年度）趋势的一致性。这项工作表明，可以直接从大规模的大气变量中学到复杂的降水模式，从而提供了传统方案的计算有效替代方案。

Title: Agentic Multimodal AI for Hyperpersonalized B2B and B2C Advertising in Competitive Markets: An AI-Driven Competitive Advertising Framework

• Authors: Sakhinana Sagar Srinivas, Akash Das, Shivam Gupta, Venkataramana Runkana
• Subjects: cs.LG, cs.AI, cs.MA, cs.SI
• Abstract URL: https://arxiv.org/abs/2504.00338
• Pdf URL: https://arxiv.org/pdf/2504.00338
• Copy Paste: [[2504.00338]] Agentic Multimodal AI for Hyperpersonalized B2B and B2C Advertising in Competitive Markets: An AI-Driven Competitive Advertising Framework(https://arxiv.org/abs/2504.00338)
• Keywords: generation
• Abstract: The growing use of foundation models (FMs) in real-world applications demands adaptive, reliable, and efficient strategies for dynamic markets. In the chemical industry, AI-discovered materials drive innovation, but commercial success hinges on market adoption, requiring FM-driven advertising frameworks that operate in-the-wild. We present a multilingual, multimodal AI framework for autonomous, hyper-personalized advertising in B2B and B2C markets. By integrating retrieval-augmented generation (RAG), multimodal reasoning, and adaptive persona-based targeting, our system generates culturally relevant, market-aware ads tailored to shifting consumer behaviors and competition. Validation combines real-world product experiments with a Simulated Humanistic Colony of Agents to model consumer personas, optimize strategies at scale, and ensure privacy compliance. Synthetic experiments mirror real-world scenarios, enabling cost-effective testing of ad strategies without risky A/B tests. Combining structured retrieval-augmented reasoning with in-context learning (ICL), the framework boosts engagement, prevents market cannibalization, and maximizes ROAS. This work bridges AI-driven innovation and market adoption, advancing multimodal FM deployment for high-stakes decision-making in commercial marketing.
• 摘要：在现实世界中，基础模型（FMS）的使用日益增长的使用需要适应性，可靠和有效的动态市场策略。在化学工业中，AI发现的材料推动了创新，但商业上的成功取决于市场采用，需要以FM驱动的广告框架来运作野外。我们提出了一个多语言，多模式的AI框架，用于在B2B和B2C市场中自主，超个性化的广告。通过整合检索型发电（RAG），多模式推理和基于自适应角色的目标，我们的系统生成了针对改变消费者行为和竞争的文化相关的市场感知广告。验证将现实世界中的产品实验与模拟的代理人的人文殖民地结合在一起，以模拟消费者角色，优化大规模的策略并确保隐私合规。合成实验反映了现实世界的情况，可以对AD策略进行具有成本效益的测试，而无需冒险A/B测试。结合结构化检索的推理与内在学习（ICL）（框架）相结合，可以提高参与度，防止市场蚕食并最大化ROAS。这项工作桥接了AI驱动的创新和市场采用，推进了多模式FM部署，以进行商业营销中的高风险决策。

Title: CamoSAM2: Motion-Appearance Induced Auto-Refining Prompts for Video Camouflaged Object Detection

• Authors: Xin Zhang, Keren Fu, Qijun Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00375
• Pdf URL: https://arxiv.org/pdf/2504.00375
• Copy Paste: [[2504.00375]] CamoSAM2: Motion-Appearance Induced Auto-Refining Prompts for Video Camouflaged Object Detection(https://arxiv.org/abs/2504.00375)
• Keywords: generation
• Abstract: The Segment Anything Model 2 (SAM2), a prompt-guided video foundation model, has remarkably performed in video object segmentation, drawing significant attention in the community. Due to the high similarity between camouflaged objects and their surroundings, which makes them difficult to distinguish even by the human eye, the application of SAM2 for automated segmentation in real-world scenarios faces challenges in camouflage perception and reliable prompts generation. To address these issues, we propose CamoSAM2, a motion-appearance prompt inducer (MAPI) and refinement framework to automatically generate and refine prompts for SAM2, enabling high-quality automatic detection and segmentation in VCOD task. Initially, we introduce a prompt inducer that simultaneously integrates motion and appearance cues to detect camouflaged objects, delivering more accurate initial predictions than existing methods. Subsequently, we propose a video-based adaptive multi-prompts refinement (AMPR) strategy tailored for SAM2, aimed at mitigating prompt error in initial coarse masks and further producing good prompts. Specifically, we introduce a novel three-step process to generate reliable prompts by camouflaged object determination, pivotal prompting frame selection, and multi-prompts formation. Extensive experiments conducted on two benchmark datasets demonstrate that our proposed model, CamoSAM2, significantly outperforms existing state-of-the-art methods, achieving increases of 8.0% and 10.1% in mIoU metric. Additionally, our method achieves the fastest inference speed compared to current VCOD models.
• 摘要：任何模型2（SAM2）是一个迅速引入的视频基础模型，在视频对象细分中表现出色，引起了社区的极大关注。由于伪装的物体及其周围环境之间的相似性很高，这使得它们甚至通过人眼而难以区分，因此在现实世界中，SAM2在自动分段中的应用在伪装感知和可靠的提示中会面临挑战。为了解决这些问题，我们提出了CamoSAM2，即运动表现提示诱导器（MAPI）和改进框架，以自动生成和完善SAM2的提示，从而在VCOD任务中实现高质量的自动检测和分段。最初，我们引入了一个迅速诱导器，该诱导剂同时整合运动和外观提示以检测伪装对象，比现有方法提供更准确的初始预测。随后，我们提出了一个基于视频的自适应多项目改进（AMPR）策略，该策略旨在减轻初始粗蒙版的及时错误，并进一步产生良好的提示。具体而言，我们引入了一个新颖的三步过程，以通过伪装的对象确定，关键提示框架选择和多点形成来生成可靠的提示。在两个基准数据集上进行的广泛实验表明，我们提出的模型CamoSAM2极大地胜过现有的最新方法，在MIOU指标中实现了8.0％和10.1％的增加。此外，与当前的VCOD模型相比，我们的方法达到了最快的推理速度。

Title: MPDrive: Improving Spatial Understanding with Marker-Based Prompt Learning for Autonomous Driving

• Authors: Zhiyuan Zhang, Xiaofan Li, Zhihao Xu, Wenjie Peng, Zijian Zhou, Miaojing Shi, Shuangping Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00379
• Pdf URL: https://arxiv.org/pdf/2504.00379
• Copy Paste: [[2504.00379]] MPDrive: Improving Spatial Understanding with Marker-Based Prompt Learning for Autonomous Driving(https://arxiv.org/abs/2504.00379)
• Keywords: generation
• Abstract: Autonomous driving visual question answering (AD-VQA) aims to answer questions related to perception, prediction, and planning based on given driving scene images, heavily relying on the model's spatial understanding capabilities. Prior works typically express spatial information through textual representations of coordinates, resulting in semantic gaps between visual coordinate representations and textual descriptions. This oversight hinders the accurate transmission of spatial information and increases the expressive burden. To address this, we propose a novel Marker-based Prompt learning framework (MPDrive), which represents spatial coordinates by concise visual markers, ensuring linguistic expressive consistency and enhancing the accuracy of both visual perception and spatial expression in AD-VQA. Specifically, we create marker images by employing a detection expert to overlay object regions with numerical labels, converting complex textual coordinate generation into straightforward text-based visual marker predictions. Moreover, we fuse original and marker images as scene-level features and integrate them with detection priors to derive instance-level features. By combining these features, we construct dual-granularity visual prompts that stimulate the LLM's spatial perception capabilities. Extensive experiments on the DriveLM and CODA-LM datasets show that MPDrive achieves state-of-the-art performance, particularly in cases requiring sophisticated spatial understanding.
• 摘要：自动驾驶视觉问题回答（AD-VQA）旨在根据给定的驾驶场景图像来回答与感知，预测和计划有关的问题，从而在很大程度上依赖于模型的空间理解能力。先前的工作通常通过坐标的文本表示表达空间信息，从而导致视觉坐标表示和文本描述之间的语义差距。这种疏忽阻碍了空间信息的准确传输，并增加了表达负担。为了解决这个问题，我们提出了一个新型的基于标记的及时学习框架（MPDRIVE），该框架代表通过简洁的视觉标记物来代表空间坐标，从而确保语言表达性一致性并提高AD-VQA中视觉感知和空间表达的准确性。具体而言，我们通过使用检测专家来覆盖具有数值标签的对象区域，从而创建标记图像，从而将复杂的文本坐标生成转换为基于文本的直观视觉标记预测。此外，我们将原始图像和标记图像融合为场景级特征，并将它们与检测先验集成在一起，以得出实例级别的特征。通过结合这些功能，我们构建了双重粒度视觉提示，以刺激LLM的空间感知能力。在DRIVELM和CODA-LM数据集上进行的广泛实验表明，MPDRIVE实现了最先进的性能，尤其是在需要复杂的空间理解的情况下。

Title: AP-CAP: Advancing High-Quality Data Synthesis for Animal Pose Estimation via a Controllable Image Generation Pipeline

• Authors: Lei Wang, Yujie Zhong, Xiaopeng Sun, Jingchun Cheng, Chengjian Feng, Qiong Cao, Lin Ma, Zhaoxin Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00394
• Pdf URL: https://arxiv.org/pdf/2504.00394
• Copy Paste: [[2504.00394]] AP-CAP: Advancing High-Quality Data Synthesis for Animal Pose Estimation via a Controllable Image Generation Pipeline(https://arxiv.org/abs/2504.00394)
• Keywords: generation
• Abstract: The task of 2D animal pose estimation plays a crucial role in advancing deep learning applications in animal behavior analysis and ecological research. Despite notable progress in some existing approaches, our study reveals that the scarcity of high-quality datasets remains a significant bottleneck, limiting the full potential of current methods. To address this challenge, we propose a novel Controllable Image Generation Pipeline for synthesizing animal pose estimation data, termed AP-CAP. Within this pipeline, we introduce a Multi-Modal Animal Image Generation Model capable of producing images with expected poses. To enhance the quality and diversity of the generated data, we further propose three innovative strategies: (1) Modality-Fusion-Based Animal Image Synthesis Strategy to integrate multi-source appearance representations, (2) Pose-Adjustment-Based Animal Image Synthesis Strategy to dynamically capture diverse pose variations, and (3) Caption-Enhancement-Based Animal Image Synthesis Strategy to enrich visual semantic understanding. Leveraging the proposed model and strategies, we create the MPCH Dataset (Modality-Pose-Caption Hybrid), the first hybrid dataset that innovatively combines synthetic and real data, establishing the largest-scale multi-source heterogeneous benchmark repository for animal pose estimation to date. Extensive experiments demonstrate the superiority of our method in improving both the performance and generalization capability of animal pose estimators.
• 摘要：2D动物姿势估计的任务在推进动物行为分析和生态研究中的深度学习应用中起着至关重要的作用。尽管在某些现有方法中取得了显着进展，但我们的研究表明，高质量数据集的稀缺性仍然是一个重要的瓶颈，这限制了当前方法的全部潜力。为了应对这一挑战，我们提出了一种新型的可控图像生成管道，用于合成动物姿势估计数据，称为AP-CAP。在这条管道中，我们引入了一个多模式动物图像生成模型，能够产生具有预期姿势的图像。为了提高生成数据的质量和多样性，我们进一步提出了三种创新的策略：（1）基于模态融合的动物图像合成策略，以整合多源外观外观表现形式，（2）基于姿势调整的动物图像同步策略，以动态地捕获多样化的姿势变化，以及（3）基于基于Enhancement的动物图像的动物图像策略，以了解原始的动物图像策略，以吸引视觉上的综合图像。利用提出的模型和策略，我们创建了MPCH数据集（模态姿势捕获混合），这是第一个创新地结合合成数据和真实数据的混合数据集，建立了最大的多源多源多源的异构基准基准基准存储库，以估计动物姿势估计到迄今为止。广泛的实验表明，我们方法在提高动物姿势估计剂的性能和概括能力方面的优越性。

Title: NCAP: Scene Text Image Super-Resolution with Non-CAtegorical Prior

• Authors: Dongwoo Park, Suk Pil Ko
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00410
• Pdf URL: https://arxiv.org/pdf/2504.00410
• Copy Paste: [[2504.00410]] NCAP: Scene Text Image Super-Resolution with Non-CAtegorical Prior(https://arxiv.org/abs/2504.00410)
• Keywords: super-resolution
• Abstract: Scene text image super-resolution (STISR) enhances the resolution and quality of low-resolution images. Unlike previous studies that treated scene text images as natural images, recent methods using a text prior (TP), extracted from a pre-trained text recognizer, have shown strong performance. However, two major issues emerge: (1) Explicit categorical priors, like TP, can negatively impact STISR if incorrect. We reveal that these explicit priors are unstable and propose replacing them with Non-CAtegorical Prior (NCAP) using penultimate layer representations. (2) Pre-trained recognizers used to generate TP struggle with low-resolution images. To address this, most studies jointly train the recognizer with the STISR network to bridge the domain gap between low- and high-resolution images, but this can cause an overconfidence phenomenon in the prior modality. We highlight this issue and propose a method to mitigate it by mixing hard and soft labels. Experiments on the TextZoom dataset demonstrate an improvement by 3.5%, while our method significantly enhances generalization performance by 14.8\% across four text recognition datasets. Our method generalizes to all TP-guided STISR networks.
• 摘要：场景文本图像超分辨率（STISR）增强了低分辨率图像的分辨率和质量。与以前将场景文本图像视为自然图像的研究不同，从预先训练的文本识别器中提取的最新方法（TP）的最新方法表现出了很强的性能。但是，出现了两个主要问题：（1）明确的分类先验，例如TP，如果不正确，可能会对Stisr产生负面影响。我们透露，这些明确的先验是不稳定的，并建议使用倒数第二层表示将其替换为非分类先验（NCAP）。 （2）用于产生具有低分辨率图像的TP斗争的预训练识别者。为了解决这个问题，大多数研究都会通过Stisr网络共同训练识别器，以弥合低分辨率图像和高分辨率图像之间的域间隙，但这可能会导致先前模式中的过度自信现象。我们强调了这个问题，并提出了一种通过混合硬标签和软标签来减轻它的方法。 TextZoom数据集上的实验证明了3.5％的提高，而我们的方法在四个文本识别数据集中大大提高了概括性能的14.8 \％。我们的方法概括为所有TP引导的STISR网络。

Title: DecoFuse: Decomposing and Fusing the "What", "Where", and "How" for Brain-Inspired fMRI-to-Video Decoding

• Authors: Chong Li, Jingyang Huo, Weikang Gong, Yanwei Fu, Xiangyang Xue, Jianfeng Feng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00432
• Pdf URL: https://arxiv.org/pdf/2504.00432
• Copy Paste: [[2504.00432]] DecoFuse: Decomposing and Fusing the "What", "Where", and "How" for Brain-Inspired fMRI-to-Video Decoding(https://arxiv.org/abs/2504.00432)
• Keywords: generation
• Abstract: Decoding visual experiences from brain activity is a significant challenge. Existing fMRI-to-video methods often focus on semantic content while overlooking spatial and motion information. However, these aspects are all essential and are processed through distinct pathways in the brain. Motivated by this, we propose DecoFuse, a novel brain-inspired framework for decoding videos from fMRI signals. It first decomposes the video into three components - semantic, spatial, and motion - then decodes each component separately before fusing them to reconstruct the video. This approach not only simplifies the complex task of video decoding by decomposing it into manageable sub-tasks, but also establishes a clearer connection between learned representations and their biological counterpart, as supported by ablation studies. Further, our experiments show significant improvements over previous state-of-the-art methods, achieving 82.4% accuracy for semantic classification, 70.6% accuracy in spatial consistency, a 0.212 cosine similarity for motion prediction, and 21.9% 50-way accuracy for video generation. Additionally, neural encoding analyses for semantic and spatial information align with the two-streams hypothesis, further validating the distinct roles of the ventral and dorsal pathways. Overall, DecoFuse provides a strong and biologically plausible framework for fMRI-to-video decoding. Project page: this https URL.
• 摘要：从大脑活动中解码视觉体验是一个重大挑战。现有的fMRI到视频方法通常集中在语义内容上，同时忽略空间和运动信息。但是，这些方面都是必不可少的，并且通过大脑中的不同途径进行处理。在此激励的情况下，我们提出了Decofuese，这是一种新型的脑灵感框架，用于解码来自fMRI信号的视频。它首先将视频分解为三个组件 - 语义，空间和运动 - 然后将每个组件分别解码，然后再融合它们重建视频。这种方法不仅通过将视频解码分解为可管理的子任务来简化视频解码的复杂任务，而且还可以在消融研究的支持下建立了学到的表示形式与其生物学对应物之间的更明显的联系。此外，我们的实验显示出比以前的最新方法的显着改善，达到语义分类的精度为82.4％，空间一致性的准确性为70.6％，运动预测的0.212余弦相似性和21.9％的50-way速度准确性。另外，对语义和空间信息的神经编码分析与两条假说一致，进一步验证了腹侧和背途径的不同作用。总体而言，Decofuse为fMRI到VIDEO解码提供了一个强大且具有生物学上合理的框架。项目页面：此HTTPS URL。

Title: Distilling Multi-view Diffusion Models into 3D Generators

• Authors: Hao Qin, Luyuan Chen, Ming Kong, Mengxu Lu, Qiang Zhu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.00457
• Pdf URL: https://arxiv.org/pdf/2504.00457
• Copy Paste: [[2504.00457]] Distilling Multi-view Diffusion Models into 3D Generators(https://arxiv.org/abs/2504.00457)
• Keywords: generation
• Abstract: We introduce DD3G, a formulation that Distills a multi-view Diffusion model (MV-DM) into a 3D Generator using gaussian splatting. DD3G compresses and integrates extensive visual and spatial geometric knowledge from the MV-DM by simulating its ordinary differential equation (ODE) trajectory, ensuring the distilled generator generalizes better than those trained solely on 3D data. Unlike previous amortized optimization approaches, we align the MV-DM and 3D generator representation spaces to transfer the teacher's probabilistic flow to the student, thus avoiding inconsistencies in optimization objectives caused by probabilistic sampling. The introduction of probabilistic flow and the coupling of various attributes in 3D Gaussians introduce challenges in the generation process. To tackle this, we propose PEPD, a generator consisting of Pattern Extraction and Progressive Decoding phases, which enables efficient fusion of probabilistic flow and converts a single image into 3D Gaussians within 0.06 seconds. Furthermore, to reduce knowledge loss and overcome sparse-view supervision, we design a joint optimization objective that ensures the quality of generated samples through explicit supervision and implicit verification. Leveraging existing 2D generation models, we compile 120k high-quality RGBA images for distillation. Experiments on synthetic and public datasets demonstrate the effectiveness of our method. Our project is available at: this https URL
• 摘要：我们介绍了DD3G，该公式将多视图扩散模型（MV-DM）提炼到使用高斯裂缝的3D发电机中。 DD3G通过模拟其普通的微分方程（ODE）轨迹来压缩和整合来自MV-DM的广泛的视觉和空间几何知识，从而确保蒸馏的生成器比仅针对3D数据训练的那些更能推广。与以前的摊销优化方法不同，我们对齐MV-DM和3D发生器表示空间，以将教师的概率流传递给学生，从而避免了概率抽样引起的优化目标的不一致。概率流的引入和3D高斯人中各种属性的耦合引入了生成过程中的挑战。为了解决这个问题，我们提出了PEPD，这是一种由模式提取和进行渐进的解码阶段组成的发电机，可以在0.06秒内有效地融合概率流量并将单个图像转换为3D高斯人。此外，为了减少知识损失并克服稀疏视图的监督，我们设计了一个联合优化目标，以通过明确的监督和隐式验证来确保生成样品的质量。利用现有的第二代模型，我们编译了120K高质量的RGBA图像进行蒸馏。关于合成和公共数据集的实验证明了我们方法的有效性。我们的项目可用：此HTTPS URL

Title: MetaLoRA: Tensor-Enhanced Adaptive Low-Rank Fine-tuning

• Authors: Maolin Wang, Xiangyu Zhao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.00460
• Pdf URL: https://arxiv.org/pdf/2504.00460
• Copy Paste: [[2504.00460]] MetaLoRA: Tensor-Enhanced Adaptive Low-Rank Fine-tuning(https://arxiv.org/abs/2504.00460)
• Keywords: generation
• Abstract: There has been a significant increase in the deployment of neural network models, presenting substantial challenges in model adaptation and fine-tuning. Efficient adaptation is crucial in maintaining model performance across diverse tasks and domains. While Low-Rank Adaptation (LoRA) has emerged as a promising parameter-efficient fine-tuning method, its fixed parameter nature limits its ability to handle dynamic task requirements effectively. Adapting models to new tasks can be challenging due to the need for extensive fine-tuning. Current LoRA variants primarily focus on general parameter reduction while overlooking the importance of dynamic parameter adjustment and meta-learning capabilities. Moreover, existing approaches mainly address static adaptations, neglecting the potential benefits of task-aware parameter generation in handling diverse task distributions. To address these limitations, this Ph.D. research proposes a LoRA generation approach to model task relationships and introduces MetaLoRA, a novel parameter-efficient adaptation framework incorporating meta-learning principles. This work develops a comprehensive architecture that integrates meta-parameter generation with adaptive low-rank decomposition, enabling efficient handling of both task-specific and task-agnostic features. MetaLoRA accurately captures task patterns by incorporating meta-learning mechanisms and dynamic parameter adjustment strategies. To our knowledge, this research represents the first attempt to provide a meta-learning enhanced LoRA variant, offering improved adaptation capability while maintaining computational efficiency in model fine-tuning.
• 摘要：神经网络模型的部署已经大大增加，在模型适应和微调方面提出了重大挑战。有效的适应性对于维持各种任务和领域的模型性能至关重要。虽然低排名适应（LORA）已成为一种有希望的参数效率微调量方法，但其固定参数性质限制了其有效处理动态任务需求的能力。由于需要进行广泛的微调，因此将模型适应新任务可能是具有挑战性的。当前的Lora变体主要集中于一般参数降低，同时忽略了动态参数调整和元学习能力的重要性。此外，现有方法主要解决静态适应，忽略了在处理多种任务分布中任务感知参数的潜在好处。为了解决这些限制，该博士学位研究提出了一种洛拉生成的方法来模拟任务关系，并引入了Metalora，Metalora是一种新型参数适应框架，结合了元学习原理。这项工作开发了一种全面的体系结构，该体系结构将元参数的生成与自适应低级分解相结合，从而有效地处理特定于任务和任务的功能。 Metalora通过合并元学习机制和动态参数调整策略来准确捕获任务模式。据我们所知，这项研究代表了提供元学习增强的Lora变体的首次尝试，从而提高了适应能力，同时保持模型微调的计算效率。

Title: High-Quality Pseudo-Label Generation Based on Visual Prompt Assisted Cloud Model Update

• Authors: Xinrun Xu, Qiuhong Zhang, Jianwen Yang, Zhanbiao Lian, Jin Yan, Zhiming Ding, Shan Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.00526
• Pdf URL: https://arxiv.org/pdf/2504.00526
• Copy Paste: [[2504.00526]] High-Quality Pseudo-Label Generation Based on Visual Prompt Assisted Cloud Model Update(https://arxiv.org/abs/2504.00526)
• Keywords: generation
• Abstract: Generating high-quality pseudo-labels on the cloud is crucial for cloud-edge object detection, especially in dynamic traffic monitoring where data distributions evolve. Existing methods often assume reliable cloud models, neglecting potential errors or struggling with complex distribution shifts. This paper proposes Cloud-Adaptive High-Quality Pseudo-label generation (CA-HQP), addressing these limitations by incorporating a learnable Visual Prompt Generator (VPG) and dual feature alignment into cloud model updates. The VPG enables parameter-efficient adaptation by injecting visual prompts, enhancing flexibility without extensive fine-tuning. CA-HQP mitigates domain discrepancies via two feature alignment techniques: global Domain Query Feature Alignment (DQFA) capturing scene-level shifts, and fine-grained Temporal Instance-Aware Feature Embedding Alignment (TIAFA) addressing instance variations. Experiments on the Bellevue traffic dataset demonstrate that CA-HQP significantly improves pseudo-label quality compared to existing methods, leading to notable performance gains for the edge model and showcasing CA-HQP's adaptation effectiveness. Ablation studies validate each component (DQFA, TIAFA, VPG) and the synergistic effect of combined alignment strategies, highlighting the importance of adaptive cloud updates and domain adaptation for robust object detection in evolving scenarios. CA-HQP provides a promising solution for enhancing cloud-edge object detection systems in real-world applications.
• 摘要：在云上生成高质量的伪标记对于云边缘对象检测至关重要，尤其是在数据分布演变的动态流量监视中。现有方法通常假设可靠的云模型，忽略潜在错误或在复杂的分配变化中挣扎。本文提出了云自适应高质量的伪标签生成（CA-HQP），通过将可学习的视觉提示发电机（VPG）和双功能对齐方式纳入云模型更新，以解决这些限制。 VPG通过注入视觉提示，增强灵活性而无需大量微调来启用参数有效的适应。 CA-HQP通过两种特征对齐技术减轻域差异：全局域查询特征对齐（DQFA）捕获场景级别的偏移以及细粒度的时间实例 - 实例感知特征嵌入对齐（TIAFA），以解决实例变体。 Bellevue流量数据集的实验表明，与现有方法相比，CA-HQP显着提高了伪标签的质量，从而导致边缘模型的绩效提高并展示了CA-HQP的适应效果。消融研究验证了每个组件（DQFA，TIAFA，VPG）以及组合对齐策略的协同作用，突出了自适应云更新和域适应性在不断发展的情况下适应强大的对象检测的重要性。 CA-HQP提供了一种有希望的解决方案，用于增强现实世界应用中的云边缘对象检测系统。

Title: Generalization-aware Remote Sensing Change Detection via Domain-agnostic Learning

• Authors: Qi Zang, Shuang Wang, Dong Zhao, Dou Quan, Yang Hu, Licheng Jiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00543
• Pdf URL: https://arxiv.org/pdf/2504.00543
• Copy Paste: [[2504.00543]] Generalization-aware Remote Sensing Change Detection via Domain-agnostic Learning(https://arxiv.org/abs/2504.00543)
• Keywords: generative
• Abstract: Change detection has essential significance for the region's development, in which pseudo-changes between bitemporal images induced by imaging environmental factors are key challenges. Existing transformation-based methods regard pseudo-changes as a kind of style shift and alleviate it by transforming bitemporal images into the same style using generative adversarial networks (GANs). However, their efforts are limited by two drawbacks: 1) Transformed images suffer from distortion that reduces feature discrimination. 2) Alignment hampers the model from learning domain-agnostic representations that degrades performance on scenes with domain shifts from the training data. Therefore, oriented from pseudo-changes caused by style differences, we present a generalizable domain-agnostic difference learning network (DonaNet). For the drawback 1), we argue for local-level statistics as style proxies to assist against domain shifts. For the drawback 2), DonaNet learns domain-agnostic representations by removing domain-specific style of encoded features and highlighting the class characteristics of objects. In the removal, we propose a domain difference removal module to reduce feature variance while preserving discriminative properties and propose its enhanced version to provide possibilities for eliminating more style by decorrelating the correlation between features. In the highlighting, we propose a cross-temporal generalization learning strategy to imitate latent domain shifts, thus enabling the model to extract feature representations more robust to shifts actively. Extensive experiments conducted on three public datasets demonstrate that DonaNet outperforms existing state-of-the-art methods with a smaller model size and is more robust to domain shift.
• 摘要：变更检测对该地区的发展具有重要意义，在该地区的发展中，由成像环境因素引起的偶然图像之间的伪变化是关键挑战。现有的基于转换的方法将伪变化视为一种样式的变化，并通过使用生成对抗网络（GAN）将Bitemal图像转换为相同样式来减轻它。但是，他们的努力受到两个缺点的限制：1）转换的图像遭受减少特征歧视的失真。 2）对齐使模型从学习域 - 不合时宜的表示形式中降低，这些表示会在域中从训练数据转移的场景上降低性能。因此，从风格差异引起的伪变换为导向，我们提出了一个可推广的域差异学习网络（Donanet）。对于缺点1），我们主张本地级别的统计信息作为风格代理，以帮助域转移。对于缺点2），Donanet通过删除特定于域的编码功能样式并突出对象的类特征来学习域形信息表示。在删除中，我们提出了一个域差删除模块，以减少特征差异，同时保留歧视性属性，并提出其增强版本，以通过将特征之间的相关性去授权，以提供消除更多样式的可能性。在突出显示的过程中，我们提出了一个跨时空的概括学习策略，以模仿潜在领域的变化，从而使模型能够更加稳健地积极地提取特征表示。在三个公共数据集上进行的广泛实验表明，Donanet的表现优于现有模型尺寸较小的现有最新方法，并且对域转移更强大。

Title: Geometric Median Matching for Robust k-Subset Selection from Noisy Data

• Authors: Anish Acharya, Sujay Sanghavi, Alexandros G Dimakis, Inderjit S Dhillon
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.00564
• Pdf URL: https://arxiv.org/pdf/2504.00564
• Copy Paste: [[2504.00564]] Geometric Median Matching for Robust k-Subset Selection from Noisy Data(https://arxiv.org/abs/2504.00564)
• Keywords: generation
• Abstract: Data pruning -- the combinatorial task of selecting a small and representative subset from a large dataset, is crucial for mitigating the enormous computational costs associated with training data-hungry modern deep learning models at scale. Since large scale data collections are invariably noisy, developing data pruning strategies that remain robust even in the presence of corruption is critical in practice. However, existing data pruning methods often fail under high corruption rates due to their reliance on empirical mean estimation, which is highly sensitive to outliers. In response, we propose Geometric Median (GM) Matching, a novel k-subset selection strategy that leverages Geometric Median -- a robust estimator with an optimal breakdown point of 1/2; to enhance resilience against noisy data. Our method iteratively selects a k-subset such that the mean of the subset approximates the GM of the (potentially) noisy dataset, ensuring robustness even under arbitrary corruption. We provide theoretical guarantees, showing that GM Matching enjoys an improved O(1/k) convergence rate -- a quadratic improvement over random sampling, even under arbitrary corruption. Extensive experiments across image classification and image generation tasks demonstrate that GM Matching consistently outperforms existing pruning approaches, particularly in high-corruption settings and at high pruning rates; making it a strong baseline for robust data pruning.
• 摘要：数据修剪 - 从大数据集中选择一个小型和代表性子集的组合任务对于减轻与培训数据渴望数据的现代深度学习模型相关的巨大计算成本至关重要。由于大规模的数据收集总是嘈杂，因此即使在腐败存在的情况下，制定数据修剪策略在实践中至关重要。但是，由于对经验平均值的依赖，现有的数据修剪方法通常在高腐败率下失败，这对异常值高度敏感。作为响应，我们提出了几何中位数（GM）匹配，这是一种新型的K-subset选择策略，利用几何中位数 - 一种强大的估计器，最佳分解点为1/2；以增强针对嘈杂数据的弹性。我们的方法迭代选择一个k-subset，以便子集的平均值近似于（可能）嘈杂的数据集的GM，从而确保即使在任意腐败下也可以鲁棒性。我们提供理论保证，表明GM匹配享有改善的O（1/K）融合率 - 即使在任意腐败下，对于随机采样的二次改善。跨图像分类和图像生成任务进行的广泛实验表明，匹配的通用汽车持续优于现有的修剪方法，尤其是在高腐败设置和高修剪率的情况下；使其成为强大的数据修剪的强大基准。

Title: Data Cleansing for GANs

• Authors: Naoyuki Terashita, Hiroki Ohashi, Satoshi Hara
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2504.00603
• Pdf URL: https://arxiv.org/pdf/2504.00603
• Copy Paste: [[2504.00603]] Data Cleansing for GANs(https://arxiv.org/abs/2504.00603)
• Keywords: generative
• Abstract: As the application of generative adversarial networks (GANs) expands, it becomes increasingly critical to develop a unified approach that improves performance across various generative tasks. One effective strategy that applies to any machine learning task is identifying harmful instances, whose removal improves the performance. While previous studies have successfully estimated these harmful training instances in supervised settings, their approaches are not easily applicable to GANs. The challenge lies in two requirements of the previous approaches that do not apply to GANs. First, previous approaches require that the absence of a training instance directly affects the parameters. However, in the training for GANs, the instances do not directly affect the generator's parameters since they are only fed into the discriminator. Second, previous approaches assume that the change in loss directly quantifies the harmfulness of the instance to a model's performance, while common types of GAN losses do not always reflect the generative performance. To overcome the first challenge, we propose influence estimation methods that use the Jacobian of the generator's gradient with respect to the discriminator's parameters (and vice versa). Such a Jacobian represents the indirect effect between two models: how removing an instance from the discriminator's training changes the generator's parameters. Second, we propose an instance evaluation scheme that measures the harmfulness of each training instance based on how a GAN evaluation metric (e.g., Inception score) is expected to change by the instance's removal. Furthermore, we demonstrate that removing the identified harmful instances significantly improves the generative performance on various GAN evaluation metrics.
• 摘要：随着生成对抗网络（GAN）的应用的扩展，开发一种统一方法越来越重要，从而改善各种生成任务的性能。适用于任何机器学习任务的一种有效策略是确定有害实例，其删除可以改善性能。尽管以前的研究已成功估算了有害环境中的这些有害培训实例，但它们的方法并不容易适用于甘斯。挑战在于先前不适用于gan的方法的两个要求。首先，先前的方法要求缺乏训练实例直接影响参数。但是，在对GAN的培训中，这些实例并不直接影响发电机的参数，因为它们仅被送入歧视器中。其次，以前的方法假定损失的变化直接量化了实例对模型性能的有害性，而常见的GAN损失类型并不总是反映生成性能。为了克服第一个挑战，我们提出了使用发电机梯度的Jacobian的影响估计方法，相对于歧视者的参数（反之亦然）。这样的雅各布代表了两个模型之间的间接效果：如何从歧视者的训练中删除实例会改变发生器的参数。其次，我们提出了一个实例评估方案，该方案根据GAN评估指标（例如，启动评分）如何通过实例的去除来改变每个培训实例的有害性。此外，我们证明，消除确定的有害实例可显着改善各种GAN评估指标的生成性能。

Title: ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Experts

• Authors: Yuanchen Wu, Junlong Du, Ke Yan, Shouhong Ding, Xiaoqiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00691
• Pdf URL: https://arxiv.org/pdf/2504.00691
• Copy Paste: [[2504.00691]] ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Experts(https://arxiv.org/abs/2504.00691)
• Keywords: generation
• Abstract: Vision-language (VL) learning requires extensive visual perception capabilities, such as fine-grained object recognition and spatial perception. Recent works typically rely on training huge models on massive datasets to develop these capabilities. As a more efficient alternative, this paper proposes a new framework that Transfers the knowledge from a hub of Vision Experts (ToVE) for efficient VL learning, leveraging pre-trained vision expert models to promote visual perception capability. Specifically, building on a frozen CLIP encoder that provides vision tokens for image-conditioned language generation, ToVE introduces a hub of multiple vision experts and a token-aware gating network that dynamically routes expert knowledge to vision tokens. In the transfer phase, we propose a "residual knowledge transfer" strategy, which not only preserves the generalizability of the vision tokens but also allows detachment of low-contributing experts to improve inference efficiency. Further, we explore to merge these expert knowledge to a single CLIP encoder, creating a knowledge-merged CLIP that produces more informative vision tokens without expert inference during deployment. Experiment results across various VL tasks demonstrate that the proposed ToVE achieves competitive performance with two orders of magnitude fewer training data.
• 摘要：视觉语言（VL）学习需要广泛的视觉感知能力，例如细粒对象识别和空间感知。最近的工作通常依靠在大型数据集上培训大型模型来开发这些功能。作为一种更有效的替代方案，本文提出了一个新框架，该框架从视力专家（TOVE）的枢纽转移知识以进行有效的VL学习，利用预先训练的视觉专家模型来促进视觉感知能力。具体而言，建立在一个冷冻的剪辑编码器上，该编码器为图像条件的语言生成提供了视觉令牌，Tove引入了一个由多个视觉专家组成的枢纽和一个令牌感知的门控网络，该网络将专家知识将专家知识转移到视力令牌。在转移阶段，我们提出了一种“剩余知识转移”策略，该策略不仅保留了视觉令牌的普遍性，而且还允许脱离低限制专家以提高推理效率。此外，我们探索将这些专家知识合并到单个剪辑编码器中，创建一个知识合并的剪辑，该剪辑在部署期间没有专家推断而产生更有信息的愿景令牌。各种VL任务的实验结果表明，提议的TOVE可以通过较少的训练数据来实现竞争性能。

Title: GraphMaster: Automated Graph Synthesis via LLM Agents in Data-Limited Environments

• Authors: Enjun Du, Xunkai Li, Tian Jin, Zhihan Zhang, Rong-Hua Li, Guoren Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.00711
• Pdf URL: https://arxiv.org/pdf/2504.00711
• Copy Paste: [[2504.00711]] GraphMaster: Automated Graph Synthesis via LLM Agents in Data-Limited Environments(https://arxiv.org/abs/2504.00711)
• Keywords: generation
• Abstract: The era of foundation models has revolutionized AI research, yet Graph Foundation Models (GFMs) remain constrained by the scarcity of large-scale graph corpora. Traditional graph data synthesis techniques primarily focus on simplistic structural operations, lacking the capacity to generate semantically rich nodes with meaningful textual attributes: a critical limitation for real-world applications. While large language models (LLMs) demonstrate exceptional text generation capabilities, their direct application to graph synthesis is impeded by context window limitations, hallucination phenomena, and structural consistency challenges. To address these issues, we introduce GraphMaster, the first multi-agent framework specifically designed for graph data synthesis in data-limited environments. GraphMaster orchestrates four specialized LLM agents (Manager, Perception, Enhancement, and Evaluation) that collaboratively optimize the synthesis process through iterative refinement, ensuring both semantic coherence and structural integrity. To rigorously evaluate our approach, we create new data-limited "Sub" variants of six standard graph benchmarks, specifically designed to test synthesis capabilities under realistic constraints. Additionally, we develop a novel interpretability assessment framework that combines human evaluation with a principled Grassmannian manifold-based analysis, providing both qualitative and quantitative measures of semantic coherence. Experimental results demonstrate that GraphMaster significantly outperforms traditional synthesis methods across multiple datasets, establishing a strong foundation for advancing GFMs in data-scarce environments.
• 摘要：基础模型的时代已经彻底改变了AI研究，但是图形基础模型（GFM）仍然受到大规模图表的稀缺性的限制。传统的图形数据综合技术主要集中于简单的结构操作，缺乏具有有意义的文本属性的语义上丰富节点的能力：对现实世界应用的关键限制。尽管大型语言模型（LLMS）表现出非凡的文本生成能力，但它们在图形合成中的直接应用会受到上下文窗口限制，幻觉现象和结构一致性挑战的阻碍。为了解决这些问题，我们介绍了Graphmaster，这是第一个专门为数据限制环境中图形数据综合而设计的多代理框架。 GraphMaster协调了四种专业的LLM代理（经理，感知，增强和评估），它们通过迭代改进协作优化综合过程，以确保语义连贯性和结构完整性。为了严格评估我们的方法，我们创建了六个标准图基准的新数据限制的“子”变体，该变体专门设计用于在现实约束下测试合成能力。此外，我们开发了一种新颖的可解释性评估框架，该框架将人类评估与基于格拉曼尼亚的原则分析相结合，从而提供了语义连贯性的定性和定量衡量标准。实验结果表明，Graphmaster在多个数据集中显着胜过传统的合成方法，为在数据筛选环境中推进GFM的坚实基础。

Title: TAMIS: Tailored Membership Inference Attacks on Synthetic Data

• Authors: Paul Andrey, Batiste Le Bars, Marc Tommasi
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2504.00758
• Pdf URL: https://arxiv.org/pdf/2504.00758
• Copy Paste: [[2504.00758]] TAMIS: Tailored Membership Inference Attacks on Synthetic Data(https://arxiv.org/abs/2504.00758)
• Keywords: generation
• Abstract: Membership Inference Attacks (MIA) enable to empirically assess the privacy of a machine learning algorithm. In this paper, we propose TAMIS, a novel MIA against differentially-private synthetic data generation methods that rely on graphical models. This attack builds upon MAMA-MIA, a recently-published state-of-the-art method. It lowers its computational cost and requires less attacker knowledge. Our attack is the product of a two-fold improvement. First, we recover the graphical model having generated a synthetic dataset by using solely that dataset, rather than shadow-modeling over an auxiliary one. This proves less costly and more performant. Second, we introduce a more mathematically-grounded attack score, that provides a natural threshold for binary predictions. In our experiments, TAMIS achieves better or similar performance as MAMA-MIA on replicas of the SNAKE challenge.
• 摘要：会员推理攻击（MIA）可以从经验上评估机器学习算法的隐私。在本文中，我们提出了塔米斯（Tamis），这是一种针对依赖图形模型的差异性合成数据生成方法的新型MIA。这次攻击是建立在最近出版的最先进方法的妈妈基础上的。它降低了其计算成本，需要更少的攻击者知识。我们的攻击是两倍改进的产物。首先，我们通过仅使用该数据集生成合成数据集的图形模型，而不是在辅助模型上进行阴影模型。事实证明，成本较低，性能较高。其次，我们引入了更具数学上的攻击评分，为二进制预测提供了自然阈值。在我们的实验中，塔米斯（Tamis）在蛇挑战（Snake Challenge）的复制品上取得了更好或类似的表现。

Title: DropGaussian: Structural Regularization for Sparse-view Gaussian Splatting

• Authors: Hyunwoo Park, Gun Ryu, Wonjun Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00773
• Pdf URL: https://arxiv.org/pdf/2504.00773
• Copy Paste: [[2504.00773]] DropGaussian: Structural Regularization for Sparse-view Gaussian Splatting(https://arxiv.org/abs/2504.00773)
• Keywords: generative
• Abstract: Recently, 3D Gaussian splatting (3DGS) has gained considerable attentions in the field of novel view synthesis due to its fast performance while yielding the excellent image quality. However, 3DGS in sparse-view settings (e.g., three-view inputs) often faces with the problem of overfitting to training views, which significantly drops the visual quality of novel view images. Many existing approaches have tackled this issue by using strong priors, such as 2D generative contextual information and external depth signals. In contrast, this paper introduces a prior-free method, so-called DropGaussian, with simple changes in 3D Gaussian splatting. Specifically, we randomly remove Gaussians during the training process in a similar way of dropout, which allows non-excluded Gaussians to have larger gradients while improving their visibility. This makes the remaining Gaussians to contribute more to the optimization process for rendering with sparse input views. Such simple operation effectively alleviates the overfitting problem and enhances the quality of novel view synthesis. By simply applying DropGaussian to the original 3DGS framework, we can achieve the competitive performance with existing prior-based 3DGS methods in sparse-view settings of benchmark datasets without any additional complexity. The code and model are publicly available at: this https URL release.
• 摘要：最近，由于其快速性能，同时产生了出色的图像质量，因此3D高斯脱落（3DGS）在新型视图综合领域引起了相当大的关注。但是，稀疏视图设置中的3DG（例如，三视图输入）通常面临着过度拟合训练视图的问题，从而大大降低了新型视图图像的视觉质量。许多现有的方法通过使用强大的先验（例如2D生成上下文信息和外部深度信号）来解决此问题。相比之下，本文引入了一种先前的方法，即所谓的dropgaussian，并在3D高斯分裂中进行了简单的变化。具体而言，我们在训练过程中以类似的辍学方式随机删除高斯人，这使得非排斥的高斯人可以具有更大的梯度，同时提高其可见度。这使剩下的高斯人更加为呈现稀疏的输入视图渲染的优化过程做出贡献。如此简单的操作有效地减轻了过度拟合的问题，并提高了新型视图合成的质量。通过简单地将Dropgaussian应用于原始的3DGS框架，我们可以在基准数据集的稀疏视图设置中使用现有的基于先前的3DGS方法实现竞争性能，而无需任何其他复杂性。代码和模型可公开可用：此HTTPS URL版本。

Title: Deep Generative Models: Complexity, Dimensionality, and Approximation

• Authors: Kevin Wang, Hongqian Niu, Yixin Wang, Didong Li
• Subjects: cs.LG, math.DG, stat.ML
• Abstract URL: https://arxiv.org/abs/2504.00820
• Pdf URL: https://arxiv.org/pdf/2504.00820
• Copy Paste: [[2504.00820]] Deep Generative Models: Complexity, Dimensionality, and Approximation(https://arxiv.org/abs/2504.00820)
• Keywords: generative
• Abstract: Generative networks have shown remarkable success in learning complex data distributions, particularly in generating high-dimensional data from lower-dimensional inputs. While this capability is well-documented empirically, its theoretical underpinning remains unclear. One common theoretical explanation appeals to the widely accepted manifold hypothesis, which suggests that many real-world datasets, such as images and signals, often possess intrinsic low-dimensional geometric structures. Under this manifold hypothesis, it is widely believed that to approximate a distribution on a $d$-dimensional Riemannian manifold, the latent dimension needs to be at least $d$ or $d+1$. In this work, we show that this requirement on the latent dimension is not necessary by demonstrating that generative networks can approximate distributions on $d$-dimensional Riemannian manifolds from inputs of any arbitrary dimension, even lower than $d$, taking inspiration from the concept of space-filling curves. This approach, in turn, leads to a super-exponential complexity bound of the deep neural networks through expanded neurons. Our findings thus challenge the conventional belief on the relationship between input dimensionality and the ability of generative networks to model data distributions. This novel insight not only corroborates the practical effectiveness of generative networks in handling complex data structures, but also underscores a critical trade-off between approximation error, dimensionality, and model complexity.
• 摘要：生成网络在学习复杂的数据分布方面取得了显着的成功，尤其是从低维输入中生成高维数据。尽管这种能力在经验上是有据可查的，但其理论的基础尚不清楚。一种常见的理论解释吸引了广泛接受的歧管假说，这表明许多现实世界中的数据集（例如图像和信号）通常具有内在的低维几何结构。在此歧管假设下，人们普遍认为，要近似于$ d $维的Riemannian歧管上的分布，潜在维度必须至少为$ d $或$ d+1+1 $。在这项工作中，我们表明，通过证明生成网络可以从任何任意维度的输入中，甚至低于$ d $，从空间填充曲线的概念中获取灵感，就可以在$ d $ d $维的Riemannian歧管上近似分布，这是不需要的。反过来，这种方法通过扩展的神经元导致了深神经网络的超指定复杂性。因此，我们的发现挑战了对输入维度与生成网络对数据分布建模的能力之间关系的传统信念。这种新颖的洞察力不仅证实了生成网络在处理复杂数据结构中的实际有效性，而且还强调了近似误差，维度和模型复杂性之间的关键权衡。

Title: PRISM-0: A Predicate-Rich Scene Graph Generation Framework for Zero-Shot Open-Vocabulary Tasks

• Authors: Abdelrahman Elskhawy, Mengze Li, Nassir Navab, Benjamin Busam
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.00844
• Pdf URL: https://arxiv.org/pdf/2504.00844
• Copy Paste: [[2504.00844]] PRISM-0: A Predicate-Rich Scene Graph Generation Framework for Zero-Shot Open-Vocabulary Tasks(https://arxiv.org/abs/2504.00844)
• Keywords: generation
• Abstract: In Scene Graphs Generation (SGG) one extracts structured representation from visual inputs in the form of objects nodes and predicates connecting them. This facilitates image-based understanding and reasoning for various downstream tasks. Although fully supervised SGG approaches showed steady performance improvements, they suffer from a severe training bias. This is caused by the availability of only small subsets of curated data and exhibits long-tail predicate distribution issues with a lack of predicate diversity adversely affecting downstream tasks. To overcome this, we introduce PRISM-0, a framework for zero-shot open-vocabulary SGG that bootstraps foundation models in a bottom-up approach to capture the whole spectrum of diverse, open-vocabulary predicate prediction. Detected object pairs are filtered and passed to a Vision Language Model (VLM) that generates descriptive captions. These are used to prompt an LLM to generate fine-andcoarse-grained predicates for the pair. The predicates are then validated using a VQA model to provide a final SGG. With the modular and dataset-independent PRISM-0, we can enrich existing SG datasets such as Visual Genome (VG). Experiments illustrate that PRIMS-0 generates semantically meaningful graphs that improve downstream tasks such as Image Captioning and Sentence-to-Graph Retrieval with a performance on par to the best fully supervised methods.
• 摘要：在场景图生成（SGG）中，一个人以对象节点的形式从视觉输入中提取结构化表示形式，并谓词连接它们。这有助于基于图像的各种下游任务的理解和推理。尽管完全有监督的SGG方法显示出稳定的性能提高，但它们患有严重的训练偏见。这是由于仅少量的策划数据子集的可用性引起的，并且表现出长尾谓词分布问题，缺乏谓词多样性会对下游任务产生不利影响。为了克服这一点，我们引入了Prism-0，这是一个零射击开放式摄影SGG的框架，该框架以自下而上的方法启动基础模型，以捕获各种开放式摄影式谓词预测的全部频谱。检测到的对象对被过滤并传递到产生描述性字幕的视觉语言模型（VLM）。这些用于提示LLM为这对生成细透明粒的谓词。然后使用VQA模型验证谓词以提供最终的SGG。使用模块化和数据集独立于Prism-0，我们可以丰富现有的SG数据集，例如视觉基因组（VG）。实验表明，PRIMS-0会生成语义上有意义的图形，这些图表改善了下游任务，例如图像字幕和句子到绘制的检索，并且在最佳完全监督的方法上表现出色。

Title: DBF-UNet: A Two-Stage Framework for Carotid Artery Segmentation with Pseudo-Label Generation

• Authors: Haoxuan Li, Wei Song, Aofan Liu, Peiwu Qin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00908
• Pdf URL: https://arxiv.org/pdf/2504.00908
• Copy Paste: [[2504.00908]] DBF-UNet: A Two-Stage Framework for Carotid Artery Segmentation with Pseudo-Label Generation(https://arxiv.org/abs/2504.00908)
• Keywords: generation
• Abstract: Medical image analysis faces significant challenges due to limited annotation data, particularly in three-dimensional carotid artery segmentation tasks, where existing datasets exhibit spatially discontinuous slice annotations with only a small portion of expert-labeled slices in complete 3D volumetric data. To address this challenge, we propose a two-stage segmentation framework. First, we construct continuous vessel centerlines by interpolating between annotated slice centroids and propagate labels along these centerlines to generate interpolated annotations for unlabeled slices. The slices with expert annotations are used for fine-tuning SAM-Med2D, while the interpolated labels on unlabeled slices serve as prompts to guide segmentation during inference. In the second stage, we propose a novel Dense Bidirectional Feature Fusion UNet (DBF-UNet). This lightweight architecture achieves precise segmentation of complete 3D vascular structures. The network incorporates bidirectional feature fusion in the encoder and integrates multi-scale feature aggregation with dense connectivity for effective feature reuse. Experimental validation on public datasets demonstrates that our proposed method effectively addresses the sparse annotation challenge in carotid artery segmentation while achieving superior performance compared to existing approaches. The source code is available at this https URL.
• 摘要：医疗图像分析由于注释数据有限而面临重大挑战，特别是在三维颈动脉分割任务中，现有数据集在完整的3D体积数据中只有一小部分专家标记的切片显示出空间不连续的切片注释。为了应对这一挑战，我们提出了一个两阶段的分割框架。首先，我们通过在注释的切片中插值并沿这些中心线传播标签来构建连续的容器中心线，以生成未标记的切片的插值注释。带有专家注释的切片用于微调SAM-MED2D，而未标记的切片上的插值标签则是指导推理过程中指导细分的提示。在第二阶段，我们提出了一种新型的致密双向特征融合UNET（DBF-UNET）。这种轻巧的体系结构可实现完整3D血管结构的精确分割。该网络在编码器中结合了双向特征融合，并将多尺度特征聚合与密集连接性集成在一起，以有效重复使用。对公共数据集的实验验证表明，我们提出的方法有效地解决了颈动脉分割中稀疏的注释挑战，同时与现有方法相比实现了卓越的性能。源代码可在此HTTPS URL上找到。

Title: WikiVideo: Article Generation from Multiple Videos

• Authors: Alexander Martin, Reno Kriz, William Gantt Walden, Kate Sanders, Hannah Recknor, Eugene Yang, Francis Ferraro, Benjamin Van Durme
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2504.00939
• Pdf URL: https://arxiv.org/pdf/2504.00939
• Copy Paste: [[2504.00939]] WikiVideo: Article Generation from Multiple Videos(https://arxiv.org/abs/2504.00939)
• Keywords: generation
• Abstract: We present the challenging task of automatically creating a high-level Wikipedia-style article that aggregates information from multiple diverse videos about real-world events, such as natural disasters or political elections. Videos are intuitive sources for retrieval-augmented generation (RAG), but most contemporary RAG workflows focus heavily on text and existing methods for video-based summarization focus on low-level scene understanding rather than high-level event semantics. To close this gap, we introduce WikiVideo, a benchmark consisting of expert-written articles and densely annotated videos that provide evidence for articles' claims, facilitating the integration of video into RAG pipelines and enabling the creation of in-depth content that is grounded in multimodal sources. We further propose Collaborative Article Generation (CAG), a novel interactive method for article creation from multiple videos. CAG leverages an iterative interaction between an r1-style reasoning model and a VideoLLM to draw higher level inferences about the target event than is possible with VideoLLMs alone, which fixate on low-level visual features. We benchmark state-of-the-art VideoLLMs and CAG in both oracle retrieval and RAG settings and find that CAG consistently outperforms alternative methods, while suggesting intriguing avenues for future work.
• 摘要：我们提出了一项艰巨的任务，即自动创建高级Wikipedia风格的文章，该文章汇总了有关真实事件的多种视频，例如自然灾害或政治选举。视频是检索型生成（RAG）的直观来源（RAG），但是大多数当代的RAG工作流程主要集中在文本和现有方法的基于视频的摘要的方法上，集中于低级场景理解，而不是高级事件语义。为了缩小这一差距，我们介绍了Wikivideo，Wikivideo是一种由专家写的文章和密集注释的视频组成的基准，这些视频提供了文章的主张的证据，促进了视频将视频整合到Rag Pipelines中，并促进了在多模态来源中扎根的深入内容。我们进一步提出了协作文章生成（CAG），这是一种用于从多个视频创建文章创建的新型交互式方法。 CAG利用R1风格的推理模型和视频仪之间的迭代相互作用，以比单独使用视频仪的更高级别的推理来绘制有关目标事件的更高级别的推断，这固定在低级视觉功能上。我们在Oracle检索和抹布设置中基准了最新的视频和CAG，并发现CAG始终优于替代方法，同时暗示着吸引未来工作的途径。

Title: GKAN: Explainable Diagnosis of Alzheimer's Disease Using Graph Neural Network with Kolmogorov-Arnold Networks

• Authors: Tianqi Ding, Dawei Xiang, Keith E Schubert, Liang Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00946
• Pdf URL: https://arxiv.org/pdf/2504.00946
• Copy Paste: [[2504.00946]] GKAN: Explainable Diagnosis of Alzheimer's Disease Using Graph Neural Network with Kolmogorov-Arnold Networks(https://arxiv.org/abs/2504.00946)
• Keywords: generative
• Abstract: Alzheimer's Disease (AD) is a progressive neurodegenerative disorder that poses significant diagnostic challenges due to its complex etiology. Graph Convolutional Networks (GCNs) have shown promise in modeling brain connectivity for AD diagnosis, yet their reliance on linear transformations limits their ability to capture intricate nonlinear patterns in neuroimaging data. To address this, we propose GCN-KAN, a novel single-modal framework that integrates Kolmogorov-Arnold Networks (KAN) into GCNs to enhance both diagnostic accuracy and interpretability. Leveraging structural MRI data, our model employs learnable spline-based transformations to better represent brain region interactions. Evaluated on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, GCN-KAN outperforms traditional GCNs by 4-8% in classification accuracy while providing interpretable insights into key brain regions associated with AD. This approach offers a robust and explainable tool for early AD diagnosis.
• 摘要：阿尔茨海默氏病（AD）是一种进行性神经退行性疾病，由于其复杂的病因而带来了重大诊断挑战。图形卷积网络（GCN）在为AD诊断的大脑连接性建模时表现出了希望，但是它们对线性转化的依赖限制了它们在神经影像学数据中捕获复杂的非线性模式的能力。为了解决这个问题，我们提出了GCN-KAN，这是一种新型的单模式框架，将Kolmogorov-Arnold网络（KAN）集成到GCN中，以增强诊断准确性和解释性。利用结构MRI数据，我们的模型采用了可学习的基于样条的转换来更好地代表大脑区域的相互作用。 GCN-KAN对阿尔茨海默氏病神经影像倡议（ADNI）数据集进行了评估，分类精度的分类准确性优于传统GCN，同时向与AD相关的关键大脑区域提供可解释的见解。这种方法为早期AD诊断提供了强大而可解释的工具。

Title: Personalized Federated Training of Diffusion Models with Privacy Guarantees

• Authors: Kumar Kshitij Patel, Weitong Zhang, Lingxiao Wang
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00952
• Pdf URL: https://arxiv.org/pdf/2504.00952
• Copy Paste: [[2504.00952]] Personalized Federated Training of Diffusion Models with Privacy Guarantees(https://arxiv.org/abs/2504.00952)
• Keywords: generative
• Abstract: The scarcity of accessible, compliant, and ethically sourced data presents a considerable challenge to the adoption of artificial intelligence (AI) in sensitive fields like healthcare, finance, and biomedical research. Furthermore, access to unrestricted public datasets is increasingly constrained due to rising concerns over privacy, copyright, and competition. Synthetic data has emerged as a promising alternative, and diffusion models -- a cutting-edge generative AI technology -- provide an effective solution for generating high-quality and diverse synthetic data. In this paper, we introduce a novel federated learning framework for training diffusion models on decentralized private datasets. Our framework leverages personalization and the inherent noise in the forward diffusion process to produce high-quality samples while ensuring robust differential privacy guarantees. Our experiments show that our framework outperforms non-collaborative training methods, particularly in settings with high data heterogeneity, and effectively reduces biases and imbalances in synthetic data, resulting in fairer downstream models.
• 摘要：在医疗保健，金融和生物医学研究等敏感领域中采用人工智能（AI）的稀缺性，对采用人工智能（AI）的稀缺性构成了巨大的挑战。此外，由于对隐私，版权和竞争的关注不断增加，因此访问不受限制的公共数据集受到越来越多的限制。合成数据已成为一种有希望的替代方案，扩散模型（一种尖端的生成AI技术）为生成高质量和多样化的合成数据提供了有效的解决方案。在本文中，我们介绍了一个新颖的联合学习框架，用于在分散的私人数据集上培训扩散模型。我们的框架利用了向前扩散过程中的个性化和固有的噪声，以生成高质量的样本，同时确保可靠的差异隐私保证。我们的实验表明，我们的框架的表现优于非授权培训方法，尤其是在具有高数据异质性的设置中，并有效地减少了合成数据的偏见和失衡，从而导致下游模型更公平。

Title: SuperDec: 3D Scene Decomposition with Superquadric Primitives

• Authors: Elisabetta Fedele, Boyang Sun, Leonidas Guibas, Marc Pollefeys, Francis Engelmann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.00992
• Pdf URL: https://arxiv.org/pdf/2504.00992
• Copy Paste: [[2504.00992]] SuperDec: 3D Scene Decomposition with Superquadric Primitives(https://arxiv.org/abs/2504.00992)
• Keywords: generation
• Abstract: We present SuperDec, an approach for creating compact 3D scene representations via decomposition into superquadric primitives. While most recent works leverage geometric primitives to obtain photorealistic 3D scene representations, we propose to leverage them to obtain a compact yet expressive representation. We propose to solve the problem locally on individual objects and leverage the capabilities of instance segmentation methods to scale our solution to full 3D scenes. In doing that, we design a new architecture which efficiently decompose point clouds of arbitrary objects in a compact set of superquadrics. We train our architecture on ShapeNet and we prove its generalization capabilities on object instances extracted from the ScanNet++ dataset as well as on full Replica scenes. Finally, we show how a compact representation based on superquadrics can be useful for a diverse range of downstream applications, including robotic tasks and controllable visual content generation and editing.
• 摘要：我们提出了SuperDec，这是一种通过分解为超季节来创建紧凑型3D场景表示的方法。尽管最新的作品利用几何原料来获取逼真的3D场景表示形式，但我们建议利用它们来获得紧凑而表达的表示。我们建议在各个对象上本地解决问题，并利用实例分割方法的功能将解决方案扩展到完整的3D场景。在此过程中，我们设计了一种新的体系结构，该体系结构有效地分解了一组紧凑型超Quadrics中任意对象的点云。我们在Shapenet上训练我们的体系结构，并在从扫描仪++数据集以及完整的副本场景中提取的对象实例上证明了它的概括功能。最后，我们展示了基于超级准则的紧凑表示形式如何对各种下游应用程序（包括机器人任务以及可控的视觉内容生成和编辑）有用。

Title: MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization

• Authors: Siyuan Li, Luyuan Zhang, Zedong Wang, Juanxi Tian, Cheng Tan, Zicheng Liu, Chang Yu, Qingsong Xie, Haonan Lu, Haoqian Wang, Zhen Lei
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.00999
• Pdf URL: https://arxiv.org/pdf/2504.00999
• Copy Paste: [[2504.00999]] MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization(https://arxiv.org/abs/2504.00999)
• Keywords: generation, generative
• Abstract: Masked Image Modeling (MIM) with Vector Quantization (VQ) has achieved great success in both self-supervised pre-training and image generation. However, most existing methods struggle to address the trade-off in shared latent space for generation quality vs. representation learning and efficiency. To push the limits of this paradigm, we propose MergeVQ, which incorporates token merging techniques into VQ-based generative models to bridge the gap between image generation and visual representation learning in a unified architecture. During pre-training, MergeVQ decouples top-k semantics from latent space with the token merge module after self-attention blocks in the encoder for subsequent Look-up Free Quantization (LFQ) and global alignment and recovers their fine-grained details through cross-attention in the decoder for reconstruction. As for the second-stage generation, we introduce MergeAR, which performs KV Cache compression for efficient raster-order prediction. Extensive experiments on ImageNet verify that MergeVQ as an AR generative model achieves competitive performance in both visual representation learning and image generation tasks while maintaining favorable token efficiency and inference speed. The code and model will be available at this https URL.
• 摘要：具有矢量量化（VQ）的掩盖图像建模（MIM）在自我监督的预训练和图像生成方面都取得了巨大的成功。但是，大多数现有的方法都难以解决共享的潜在空间中的权衡，以进行发电质量与表示和效率。为了按照此范式的限制，我们提出了MergeVQ，该范式将令牌合并技术纳入了基于VQ的生成模型中，以弥合统一体系结构中图像生成和视觉表示学习之间的差距。在预训练期间，MergeVQ在编码器中自我发项块块后，从潜在空间与代币合并模块的顶级语义解析，以进行随后查找的无查找量化（LFQ）和全球对齐方式，并通过解码器进行再构造来恢复其细粒度的细节。至于第二阶段，我们介绍了Memgear，该Memgear执行KV缓存压缩以进行有效的光栅订单预测。对Imagenet的广泛实验验证了合并为AR生成模型的合并在视觉表示学习和图像生成任务中都能达到竞争性能，同时保持有利的令牌效率和推理速度。代码和模型将在此HTTPS URL上可用。

Title: IntrinsiX: High-Quality PBR Generation using Image Priors

• Authors: Peter Kocsis (1), Lukas Höllein (1), Matthias Nießner (1) ((1) Technical University of Munich)
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.01008
• Pdf URL: https://arxiv.org/pdf/2504.01008
• Copy Paste: [[2504.01008]] IntrinsiX: High-Quality PBR Generation using Image Priors(https://arxiv.org/abs/2504.01008)
• Keywords: generation
• Abstract: We introduce IntrinsiX, a novel method that generates high-quality intrinsic images from text description. In contrast to existing text-to-image models whose outputs contain baked-in scene lighting, our approach predicts physically-based rendering (PBR) maps. This enables the generated outputs to be used for content creation scenarios in core graphics applications that facilitate re-lighting, editing, and texture generation tasks. In order to train our generator, we exploit strong image priors, and pre-train separate models for each PBR material component (albedo, roughness, metallic, normals). We then align these models with a new cross-intrinsic attention formulation that concatenates key and value features in a consistent fashion. This allows us to exchange information between each output modality and to obtain semantically coherent PBR predictions. To ground each intrinsic component, we propose a rendering loss which provides image-space signals to constrain the model, thus facilitating sharp details also in the output BRDF properties. Our results demonstrate detailed intrinsic generation with strong generalization capabilities that outperforms existing intrinsic image decomposition methods used with generated images by a significant margin. Finally, we show a series of applications, including re-lighting, editing, and text-conditioned room-scale PBR texture generation.
• 摘要：我们介绍了Intrinsix，这是一种新颖的方法，该方法从文本描述中生成高质量的内在图像。与现有的文本到图像模型相比，其输出包含烘焙场景照明，我们的方法预测了基于物理的渲染图（PBR）地图。这使生成的输出可用于核心图形应用程序中的内容创建方案，以促进重新亮起，编辑和纹理生成任务。为了训练我们的发电机，我们利用强大的图像先验，并为每个PBR材料成分（反照率，粗糙度，金属，正常）预先培训单独的模型。然后，我们将这些模型与新的跨内在注意力配方对齐，该公式以一致的方式串联关键和价值特征。这使我们能够在每种输出方式之间交换信息并获得语义相干PBR预测。为了接地每个固有组件，我们提出了一个渲染损失，该损失提供图像空间信号以限制模型，从而促进了输出BRDF属性中的尖锐细节。我们的结果表明，具有强大的概括能力的详细固有产生，胜过现有的固有图像分解方法，与生成的图像相当大。最后，我们展示了一系列应用程序，包括重新照明，编辑和文本条件的房间尺度PBR纹理生成。

Title: AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction

• Authors: Junhao Cheng, Yuying Ge, Yixiao Ge, Jing Liao, Ying Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01014
• Pdf URL: https://arxiv.org/pdf/2504.01014
• Copy Paste: [[2504.01014]] AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction(https://arxiv.org/abs/2504.01014)
• Keywords: generation, generative
• Abstract: Recent advancements in image and video synthesis have opened up new promise in generative games. One particularly intriguing application is transforming characters from anime films into interactive, playable entities. This allows players to immerse themselves in the dynamic anime world as their favorite characters for life simulation through language instructions. Such games are defined as infinite game since they eliminate predetermined boundaries and fixed gameplay rules, where players can interact with the game world through open-ended language and experience ever-evolving storylines and environments. Recently, a pioneering approach for infinite anime life simulation employs large language models (LLMs) to translate multi-turn text dialogues into language instructions for image generation. However, it neglects historical visual context, leading to inconsistent gameplay. Furthermore, it only generates static images, failing to incorporate the dynamics necessary for an engaging gaming experience. In this work, we propose AnimeGamer, which is built upon Multimodal Large Language Models (MLLMs) to generate each game state, including dynamic animation shots that depict character movements and updates to character states, as illustrated in Figure 1. We introduce novel action-aware multimodal representations to represent animation shots, which can be decoded into high-quality video clips using a video diffusion model. By taking historical animation shot representations as context and predicting subsequent representations, AnimeGamer can generate games with contextual consistency and satisfactory dynamics. Extensive evaluations using both automated metrics and human evaluations demonstrate that AnimeGamer outperforms existing methods in various aspects of the gaming experience. Codes and checkpoints are available at this https URL.
• 摘要：图像和视频综合方面的最新进展为生成游戏开辟了新的希望。一个特别有趣的应用程序是将角色从动漫电影转变为互动，可玩的实体。这使玩家可以通过语言说明将自己沉浸在动态动漫世界中，作为他们最喜欢的生活模拟角色。这样的游戏被定义为无限游戏，因为它们消除了预定的边界和固定的游戏规则，玩家可以通过开放式语言与游戏世界进行互动，并体验不断发展的故事情节和环境。最近，无限动漫生活模拟的开创性方法采用大型语言模型（LLM）将多转向文本对话转化为图像生成的语言说明。但是，它忽略了历史视觉上下文，导致游戏不一致。此外，它仅生成静态图像，未能结合引人入胜的游戏体验所需的动态。在这项工作中，我们提出了基于多模式的大语模型（MLLMS）来生成每个游戏状态的动画游戏，包括动态动画镜头，描绘了角色运动和对角色状态的更新，如图1所示。我们引入了新颖的动作感动的多模式表示形式，以使用高级视频模型来代表动画，并使用视频模型来解码。通过将历史动画镜头表示作为上下文并预测后续表示形式，AnimeGamer可以生成具有上下文一致性和令人满意的动态的游戏。使用自动指标和人类评估的广泛评估表明，在游戏体验的各个方面，动画游戏的表现优于现有方法。该HTTPS URL可用代码和检查点。

Title: MixerMDM: Learnable Composition of Human Motion Diffusion Models

• Authors: Pablo Ruiz-Ponce, German Barquero, Cristina Palmero, Sergio Escalera, José García-Rodríguez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01019
• Pdf URL: https://arxiv.org/pdf/2504.01019
• Copy Paste: [[2504.01019]] MixerMDM: Learnable Composition of Human Motion Diffusion Models(https://arxiv.org/abs/2504.01019)
• Keywords: generation, generative
• Abstract: Generating human motion guided by conditions such as textual descriptions is challenging due to the need for datasets with pairs of high-quality motion and their corresponding conditions. The difficulty increases when aiming for finer control in the generation. To that end, prior works have proposed to combine several motion diffusion models pre-trained on datasets with different types of conditions, thus allowing control with multiple conditions. However, the proposed merging strategies overlook that the optimal way to combine the generation processes might depend on the particularities of each pre-trained generative model and also the specific textual descriptions. In this context, we introduce MixerMDM, the first learnable model composition technique for combining pre-trained text-conditioned human motion diffusion models. Unlike previous approaches, MixerMDM provides a dynamic mixing strategy that is trained in an adversarial fashion to learn to combine the denoising process of each model depending on the set of conditions driving the generation. By using MixerMDM to combine single- and multi-person motion diffusion models, we achieve fine-grained control on the dynamics of every person individually, and also on the overall interaction. Furthermore, we propose a new evaluation technique that, for the first time in this task, measures the interaction and individual quality by computing the alignment between the mixed generated motions and their conditions as well as the capabilities of MixerMDM to adapt the mixing throughout the denoising process depending on the motions to mix.
• 摘要：由于需要具有一对高质量运动及其相应条件的数据集，因此以文本描述等条件为指导的人类运动具有挑战性。当目标在这一代人中的控制范围内时，难度会增加。为此，先前的工作提出了将在具有不同类型条件的数据集上预先训练的几个运动扩散模型组合，从而可以通过多种条件进行控制。但是，提出的合并策略忽略了结合生成过程的最佳方法可能取决于每个预训练的生成模型的特殊性以及特定的文本描述。在这种情况下，我们介绍了MixerMDM，这是第一个可学习的模型组成技术，用于结合预先训练的文本条件的人体运动扩散模型。与以前的方法不同，MixerMDM提供了一种动态混合策略，该策略以对抗性方式进行了训练，以学会根据驱动一代人的一组条件来结合每个模型的脱索过程。通过使用MixerMDM结合单人运动扩散模型，我们可以单独地对每个人的动力学以及整体相互作用实现精细的控制。此外，我们提出了一种新的评估技术，该技术在此任务中首次通过计算混合产生的动作及其条件之间的对齐方式以及MixerMDM的能力来衡量的相互作用和个人质量，以调整整个DeNoising过程中的混合过程，取决于混合过程。

Title: Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation

• Authors: Junyu Xie, Tengda Han, Max Bain, Arsha Nagrani, Eshika Khandelwal, Gül Varol, Weidi Xie, Andrew Zisserman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.01020
• Pdf URL: https://arxiv.org/pdf/2504.01020
• Copy Paste: [[2504.01020]] Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation(https://arxiv.org/abs/2504.01020)
• Keywords: generation
• Abstract: Our objective is the automatic generation of Audio Descriptions (ADs) for edited video material, such as movies and TV series. To achieve this, we propose a two-stage framework that leverages "shots" as the fundamental units of video understanding. This includes extending temporal context to neighbouring shots and incorporating film grammar devices, such as shot scales and thread structures, to guide AD generation. Our method is compatible with both open-source and proprietary Visual-Language Models (VLMs), integrating expert knowledge from add-on modules without requiring additional training of the VLMs. We achieve state-of-the-art performance among all prior training-free approaches and even surpass fine-tuned methods on several benchmarks. To evaluate the quality of predicted ADs, we introduce a new evaluation measure -- an action score -- specifically targeted to assessing this important aspect of AD. Additionally, we propose a novel evaluation protocol that treats automatic frameworks as AD generation assistants and asks them to generate multiple candidate ADs for selection.
• 摘要：我们的目标是用于编辑的视频材料（例如电影和电视连续剧）的自动生成音频说明（AD）。为了实现这一目标，我们提出了一个两阶段的框架，该框架将“拍摄”作为视频理解的基本单位。这包括将时间上下文扩展到相邻的镜头，并结合薄膜语法设备，例如射击秤和线程结构，以指导广告的生成。我们的方法与开源和专有视觉语言模型（VLMS）兼容，从而在不需要对VLM的额外培训的情况下整合了来自附加模块的专家知识。我们在所有先前的无培训方法中实现最先进的性能，甚至超过几种基准测试的微调方法。为了评估预测广告的质量，我们介绍了一种新的评估措施（一种动作得分），专门针对评估AD的这一重要方面。此外，我们提出了一种新的评估协议，该协议将自动框架视为广告生成助手，并要求他们生成多个候选广告进行选择。