2025-03-26

Title: Generative Data Imputation for Sparse Learner Performance Data Using Generative Adversarial Imputation Networks

• Authors: Liang Zhang, Jionghao Lin, John Sabatini, Diego Zapata-Rivera, Carol Forsyth, Yang Jiang, John Hollander, Xiangen Hu, Arthur C. Graesser
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.18982
• Pdf URL: https://arxiv.org/pdf/2503.18982
• Copy Paste: [[2503.18982]] Generative Data Imputation for Sparse Learner Performance Data Using Generative Adversarial Imputation Networks(https://arxiv.org/abs/2503.18982)
• Keywords: generative
• Abstract: Learner performance data collected by Intelligent Tutoring Systems (ITSs), such as responses to questions, is essential for modeling and predicting learners' knowledge states. However, missing responses due to skips or incomplete attempts create data sparsity, challenging accurate assessment and personalized instruction. To address this, we propose a generative imputation approach using Generative Adversarial Imputation Networks (GAIN). Our method features a three-dimensional (3D) framework (learners, questions, and attempts), flexibly accommodating various sparsity levels. Enhanced by convolutional neural networks and optimized with a least squares loss function, the GAIN-based method aligns input and output dimensions to question-attempt matrices along the learners' dimension. Extensive experiments using datasets from AutoTutor Adult Reading Comprehension (ARC), ASSISTments, and MATHia demonstrate that our approach significantly outperforms tensor factorization and alternative GAN methods in imputation accuracy across different attempt scenarios. Bayesian Knowledge Tracing (BKT) further validates the effectiveness of the imputed data by estimating learning parameters: initial knowledge (P(L0)), learning rate (P(T)), guess rate (P(G)), and slip rate (P(S)). Results indicate the imputed data enhances model fit and closely mirrors original distributions, capturing underlying learning behaviors reliably. Kullback-Leibler (KL) divergence assessments confirm minimal divergence, showing the imputed data preserves essential learning characteristics effectively. These findings underscore GAIN's capability as a robust imputation tool in ITSs, alleviating data sparsity and supporting adaptive, individualized instruction, ultimately leading to more precise and responsive learner assessments and improved educational outcomes.
• 摘要：智能补习系统（ITS）收集的学习者绩效数据，例如对问题的回答，对于对学习者的知识状态进行建模和预测至关重要。但是，由于跳过或不完整的尝试而缺少响应会产生数据稀疏性，挑战准确的评估和个性化的指导。为了解决这个问题，我们提出了一种使用生成的对抗归合网络（增益）的生成插补方法。我们的方法具有三维（3D）框架（学习者，问题和尝试），灵活地适应各种稀疏水平。通过卷积神经网络增强，并通过最小二乘损失函数进行了优化，基于增益的方法将输入和输出维度与学习者尺寸沿问答矩阵保持一致。使用来自自动化的成人阅读理解（ARC），辅助和MATHIA的数据集进行的广泛实验表明，我们的方法在不同尝试方案中的归纳精度极高地超过了张力分解和替代GAN方法。贝叶斯知识追踪（BKT）通过估计学习参数进一步验证了估计数据的有效性：初始知识（P（l0）），学习率（P（t）），猜测率（P（g））和滑移率（P（p（s））。结果表明，估算的数据增强了模型拟合，并密切反映了原始分布，从而可靠地捕获了潜在的学习行为。 Kullback-Leibler（KL）差异评估证实了最小的差异，显示了估算的数据可有效地保留基本学习特征。这些发现强调了收益作为ITS中强大的归纳工具的能力，减轻了数据稀疏性并支持适应性的，个性化的教学，最终导致了更精确，更敏感的学习者评估和改善的教育成果。

Title: SplitFrozen: Split Learning with Device-side Model Frozen for Fine-Tuning LLM on Heterogeneous Resource-Constrained Devices

• Authors: Jian Ma, Xinchen Lyu, Jun Jiang, Qimei Cui, Haipeng Yao, Xiaofeng Tao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.18986
• Pdf URL: https://arxiv.org/pdf/2503.18986
• Copy Paste: [[2503.18986]] SplitFrozen: Split Learning with Device-side Model Frozen for Fine-Tuning LLM on Heterogeneous Resource-Constrained Devices(https://arxiv.org/abs/2503.18986)
• Keywords: generation
• Abstract: Fine-tuning large language models (LLMs) on private, on-device data can empower tailored personalized AI agents. However, fine-tuning LLMs on resource-constrained edge devices faces significant challenges, including excessive computation overhead, device heterogeneity, and data imbalance. This paper proposes SplitFrozen, a split learning framework that enables efficient LLM fine-tuning by strategically freezing device-side model layers while centralizing parameter-efficient fine-tuning on the server. Our framework partitions LLMs into device-side frozen layers and server-side fine-tuning layers, where heterogeneous resource-constrained devices execute only forward propagation. To minimize server-side training costs, we integrate Low-Rank Adaptation (LoRA) into the server-side layers. A pipeline parallelism strategy further optimizes training efficiency by decoupling device-server computations and leveraging decomposed backward propagation. Experiments on GPT-2 with the MRPC, MNLI-matched, and SST-2 datasets demonstrate that SplitFrozen outperforms FedLoRA and SplitLoRA by 69.4\% model accuracy under extremely imbalanced data, while reducing up to 86.8\% device-side computations and 50.2\% total training time. Experiments also validate the scalability of SplitFrozen on content generation task using Llama-3.2 model on GSM8K dataset.
• 摘要：私人，设备数据上的大型语言模型（LLM）可以增强量身定制的个性化AI代理。但是，在资源受限的边缘设备上进行微调LLM面临重大挑战，包括过度的计算开销，设备异质性和数据失衡。本文提出了SplitFrozen，这是一个分裂学习框架，通过战略性地冻结设备端模型层，同时将参数有效的微调在服务器上进行策略性冻结，从而实现有效的LLM微调。我们的框架将LLMS分配到设备侧冷冻层和服务器端微调层中，其中异质资源受限的设备仅执行正向传播。为了最大程度地减少服务器端培训成本，我们将低级适应（LORA）集成到服务器端层中。管道并行性策略通过将设备服务器计算取消并利用分解后的传播来进一步优化训练效率。在GPT-2上使用MRPC进行了MNLI匹配和SST-2数据集的实验表明，在极不平衡的数据下，SplitFrozen的表现优于Fedlora和Splitlora的69.4 \％模型精度，而最多减少了86.8 \％的设备侧计算以及50.2 \％的总培训时间。实验还使用GSM8K数据集上的Llama-3.2模型验证了在内容生成任务上拆分的可扩展性。

Title: SG-Tailor: Inter-Object Commonsense Relationship Reasoning for Scene Graph Manipulation

• Authors: Haoliang Shang, Hanyu Wu, Guangyao Zhai, Boyang Sun, Fangjinhua Wang, Federico Tombari, Marc Pollefeys
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2503.18988
• Pdf URL: https://arxiv.org/pdf/2503.18988
• Copy Paste: [[2503.18988]] SG-Tailor: Inter-Object Commonsense Relationship Reasoning for Scene Graph Manipulation(https://arxiv.org/abs/2503.18988)
• Keywords: generation
• Abstract: Scene graphs capture complex relationships among objects, serving as strong priors for content generation and manipulation. Yet, reasonably manipulating scene graphs -- whether by adding nodes or modifying edges -- remains a challenging and untouched task. Tasks such as adding a node to the graph or reasoning about a node's relationships with all others are computationally intractable, as even a single edge modification can trigger conflicts due to the intricate interdependencies within the graph. To address these challenges, we introduce SG-Tailor, an autoregressive model that predicts the conflict-free relationship between any two nodes. SG-Tailor not only infers inter-object relationships, including generating commonsense edges for newly added nodes but also resolves conflicts arising from edge modifications to produce coherent, manipulated graphs for downstream tasks. For node addition, the model queries the target node and other nodes from the graph to predict the appropriate relationships. For edge modification, SG-Tailor employs a Cut-And-Stitch strategy to solve the conflicts and globally adjust the graph. Extensive experiments demonstrate that SG-Tailor outperforms competing methods by a large margin and can be seamlessly integrated as a plug-in module for scene generation and robotic manipulation tasks.
• 摘要：场景图捕获了对象之间的复杂关系，作为内容生成和操纵的强大先验。然而，无论是添加节点还是修改边缘，合理地操纵场景图 - 仍然是一项具有挑战性且没有触及的任务。诸如将节点添加到图形或有关节点与所有其他关系的关系的推理之类的任务在计算上是棘手的，因为即使是单个边缘修改也可能触发冲突，因为图形中的复杂相互依赖性。为了应对这些挑战，我们介绍了SG-Tailor，这是一种自回归模型，可预测任何两个节点之间无冲突的关系。 SG-Tailor不仅会渗透对象间的关系，包括生成新添加的节点的常识边缘，而且解决了由边缘修改引起的冲突，以产生连贯的，操纵的图形，以实现下游任务。对于节点添加，该模型查询目标节点和图形中的其他节点以预测适当的关系。对于边缘修改，SG-Tailor采用剪裁策略来解决冲突并在全球调整图表。广泛的实验表明，SG-Tailor的表现优于相互竞争的方法，并且可以无缝集成为现场生成和机器人操纵任务的插件模块。

Title: DisentTalk: Cross-lingual Talking Face Generation via Semantic Disentangled Diffusion Model

• Authors: Kangwei Liu, Junwu Liu, Yun Cao, Jinlin Guo, Xiaowei Yi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.19001
• Pdf URL: https://arxiv.org/pdf/2503.19001
• Copy Paste: [[2503.19001]] DisentTalk: Cross-lingual Talking Face Generation via Semantic Disentangled Diffusion Model(https://arxiv.org/abs/2503.19001)
• Keywords: generation
• Abstract: Recent advances in talking face generation have significantly improved facial animation synthesis. However, existing approaches face fundamental limitations: 3DMM-based methods maintain temporal consistency but lack fine-grained regional control, while Stable Diffusion-based methods enable spatial manipulation but suffer from temporal inconsistencies. The integration of these approaches is hindered by incompatible control mechanisms and semantic entanglement of facial representations. This paper presents DisentTalk, introducing a data-driven semantic disentanglement framework that decomposes 3DMM expression parameters into meaningful subspaces for fine-grained facial control. Building upon this disentangled representation, we develop a hierarchical latent diffusion architecture that operates in 3DMM parameter space, integrating region-aware attention mechanisms to ensure both spatial precision and temporal coherence. To address the scarcity of high-quality Chinese training data, we introduce CHDTF, a Chinese high-definition talking face dataset. Extensive experiments show superior performance over existing methods across multiple metrics, including lip synchronization, expression quality, and temporal consistency. Project Page: this https URL.
• 摘要：谈话发电的最新进展显着改善了面部动画的综合。但是，现有方法面临基本局限性：基于3DMM的方法保持时间一致性，但缺乏细粒度的区域控制，而基于稳定的基于扩散的方法可以实现空间操作，但会遇到时间不一致。这些方法的整合受到不兼容的控制机制和面部表征的语义纠缠的阻碍。本文介绍了DisentTalk，引入了数据驱动的语义分离框架，该框架将3DMM表达参数分解为有意义的子空间，以进行细粒度的面部控制。在此分离表示的基础上，我们开发了一个分层的潜扩散体系结构，该结构在3DMM参数空间中运行，集成了区域感知的注意机制，以确保空间精度和时间连贯性。为了解决高质量中国培训数据的稀缺性，我们介绍了中国高清面部数据集CHDTF。广泛的实验表明，超过多个指标的现有方法，包括唇部同步，表达质量和时间一致性。项目页面：此HTTPS URL。

Title: Option Discovery Using LLM-guided Semantic Hierarchical Reinforcement Learning

• Authors: Chak Lam Shek, Pratap Tokekar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.19007
• Pdf URL: https://arxiv.org/pdf/2503.19007
• Copy Paste: [[2503.19007]] Option Discovery Using LLM-guided Semantic Hierarchical Reinforcement Learning(https://arxiv.org/abs/2503.19007)
• Keywords: generation
• Abstract: Large Language Models (LLMs) have shown remarkable promise in reasoning and decision-making, yet their integration with Reinforcement Learning (RL) for complex robotic tasks remains underexplored. In this paper, we propose an LLM-guided hierarchical RL framework, termed LDSC, that leverages LLM-driven subgoal selection and option reuse to enhance sample efficiency, generalization, and multi-task adaptability. Traditional RL methods often suffer from inefficient exploration and high computational cost. Hierarchical RL helps with these challenges, but existing methods often fail to reuse options effectively when faced with new tasks. To address these limitations, we introduce a three-stage framework that uses LLMs for subgoal generation given natural language description of the task, a reusable option learning and selection method, and an action-level policy, enabling more effective decision-making across diverse tasks. By incorporating LLMs for subgoal prediction and policy guidance, our approach improves exploration efficiency and enhances learning performance. On average, LDSC outperforms the baseline by 55.9\% in average reward, demonstrating its effectiveness in complex RL settings. More details and experiment videos could be found in \href{this https URL}{this link\footnote{this https URL}}.
• 摘要：大型语言模型（LLM）在推理和决策方面表现出了巨大的希望，但是它们与加强学习（RL）的集成在一起，用于复杂的机器人任务。在本文中，我们提出了一个称为LDSC的LLM引导的层次RL框架，该框架利用LLM驱动的子目标选择和期权重用来提高样本效率，概括和多任务适应性。传统的RL方法通常遭受效率低下的探索和高计算成本的困扰。分层RL有助于解决这些挑战，但是现有方法在面对新任务时通常无法有效地重复使用选项。为了解决这些限制，我们引入了一个三阶段的框架，鉴于对任务的自然语言描述，可重复使用的选项学习和选择方法以及动作级别的策略，该框架将LLM用于亚目标生成，并实现了跨不同任务的更有效的决策。通过将LLM纳入子目标预测和政策指导，我们的方法提高了勘探效率并提高了学习绩效。平均而言，LDSC在平均奖励中优于基线55.9％，表明其在复杂的RL设置中的有效性。更多详细信息和实验视频可以在\ href {此https url} {this link \ footNote {this https url}}}中找到。

Title: RomanTex: Decoupling 3D-aware Rotary Positional Embedded Multi-Attention Network for Texture Synthesis

• Authors: Yifei Feng, Mingxin Yang, Shuhui Yang, Sheng Zhang, Jiaao Yu, Zibo Zhao, Yuhong Liu, Jie Jiang, Chunchao Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19011
• Pdf URL: https://arxiv.org/pdf/2503.19011
• Copy Paste: [[2503.19011]] RomanTex: Decoupling 3D-aware Rotary Positional Embedded Multi-Attention Network for Texture Synthesis(https://arxiv.org/abs/2503.19011)
• Keywords: generation
• Abstract: Painting textures for existing geometries is a critical yet labor-intensive process in 3D asset generation. Recent advancements in text-to-image (T2I) models have led to significant progress in texture generation. Most existing research approaches this task by first generating images in 2D spaces using image diffusion models, followed by a texture baking process to achieve UV texture. However, these methods often struggle to produce high-quality textures due to inconsistencies among the generated multi-view images, resulting in seams and ghosting artifacts. In contrast, 3D-based texture synthesis methods aim to address these inconsistencies, but they often neglect 2D diffusion model priors, making them challenging to apply to real-world objects To overcome these limitations, we propose RomanTex, a multiview-based texture generation framework that integrates a multi-attention network with an underlying 3D representation, facilitated by our novel 3D-aware Rotary Positional Embedding. Additionally, we incorporate a decoupling characteristic in the multi-attention block to enhance the model's robustness in image-to-texture task, enabling semantically-correct back-view synthesis. Furthermore, we introduce a geometry-related Classifier-Free Guidance (CFG) mechanism to further improve the alignment with both geometries and images. Quantitative and qualitative evaluations, along with comprehensive user studies, demonstrate that our method achieves state-of-the-art results in texture quality and consistency.
• 摘要：现有几何形状的绘画纹理是3D资产生成的关键但劳动密集型的过程。文本对图像（T2I）模型的最新进展导致纹理产生取得了重大进展。大多数现有的研究通过使用图像扩散模型在2D空间中首先生成图像，然后进行纹理烘焙过程以实现UV纹理。但是，由于生成的多视图图像之间的不一致，这些方法通常很难产生高质量的纹理，从而导致接缝和幽灵伪影。相比之下，基于3D的纹理合成方法旨在解决这些不一致，但它们经常忽略2D扩散模型先验，使它们具有挑战性地应用于现实世界中的对象以克服这些局限性，我们建议将基于多的纹理框架与基于多型的3D代表相结合的多型纹理框架，由Indering 3D代表组成，我们的小说3D代表了我们的小说3D。此外，我们将一个脱钩特征纳入多发障碍物中，以增强模型在图像到文本任务中的鲁棒性，从而实现语义上正确的背面视图合成。此外，我们引入了与几何相关的无分类器引导（CFG）机制，以进一步改善几何和图像的比对。定量和定性评估以及全面的用户研究表明，我们的方法实现了最先进的方法，从而带来了质地质量和一致性。

Title: Color Conditional Generation with Sliced Wasserstein Guidance

• Authors: Alexander Lobashev, Maria Larchenko, Dmitry Guskov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19034
• Pdf URL: https://arxiv.org/pdf/2503.19034
• Copy Paste: [[2503.19034]] Color Conditional Generation with Sliced Wasserstein Guidance(https://arxiv.org/abs/2503.19034)
• Keywords: generation
• Abstract: We propose SW-Guidance, a training-free approach for image generation conditioned on the color distribution of a reference image. While it is possible to generate an image with fixed colors by first creating an image from a text prompt and then applying a color style transfer method, this approach often results in semantically meaningless colors in the generated image. Our method solves this problem by modifying the sampling process of a diffusion model to incorporate the differentiable Sliced 1-Wasserstein distance between the color distribution of the generated image and the reference palette. Our method outperforms state-of-the-art techniques for color-conditional generation in terms of color similarity to the reference, producing images that not only match the reference colors but also maintain semantic coherence with the original text prompt. Our source code is available at this https URL.
• 摘要：我们提出了SW-guidance，这是一种以参考图像的颜色分布为条件的图像产生的无训练方法。虽然可以通过首先从文本提示符创建图像，然后应用色彩样式传输方法来生成具有固定颜色的图像，但这种方法通常会在生成的图像中产生毫无意义的颜色。我们的方法通过修改扩散模型的采样过程来解决此问题，以在生成的图像的颜色分布和参考调色板之间结合可区分的切片1-wassestein距离。我们的方法的表现优于颜色相似性的颜色条件生成的最先进技术，产生的图像不仅与参考颜色相匹配，而且还与原始文本提示符保持语义连贯性。我们的源代码可在此HTTPS URL上找到。

Title: WikiAutoGen: Towards Multi-Modal Wikipedia-Style Article Generation

• Authors: Zhongyu Yang, Jun Chen, Dannong Xu, Junjie Fei, Xiaoqian Shen, Liangbing Zhao, Chun-Mei Feng, Mohamed Elhoseiny
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19065
• Pdf URL: https://arxiv.org/pdf/2503.19065
• Copy Paste: [[2503.19065]] WikiAutoGen: Towards Multi-Modal Wikipedia-Style Article Generation(https://arxiv.org/abs/2503.19065)
• Keywords: generation
• Abstract: Knowledge discovery and collection are intelligence-intensive tasks that traditionally require significant human effort to ensure high-quality outputs. Recent research has explored multi-agent frameworks for automating Wikipedia-style article generation by retrieving and synthesizing information from the internet. However, these methods primarily focus on text-only generation, overlooking the importance of multimodal content in enhancing informativeness and engagement. In this work, we introduce WikiAutoGen, a novel system for automated multimodal Wikipedia-style article generation. Unlike prior approaches, WikiAutoGen retrieves and integrates relevant images alongside text, enriching both the depth and visual appeal of generated content. To further improve factual accuracy and comprehensiveness, we propose a multi-perspective self-reflection mechanism, which critically assesses retrieved content from diverse viewpoints to enhance reliability, breadth, and coherence, etc. Additionally, we introduce WikiSeek, a benchmark comprising Wikipedia articles with topics paired with both textual and image-based representations, designed to evaluate multimodal knowledge generation on more challenging topics. Experimental results show that WikiAutoGen outperforms previous methods by 8%-29% on our WikiSeek benchmark, producing more accurate, coherent, and visually enriched Wikipedia-style articles. We show some of our generated examples in this https URL .
• 摘要：知识发现和收集是智力密集型任务，传统上需要大量的人类努力来确保高质量的产出。最近的研究探索了通过从Internet检索和合成信息来自动化Wikipedia风格文章生成的多代理框架。但是，这些方法主要集中于仅文本生成，忽略了多模式内容在增强信息性和参与度中的重要性。在这项工作中，我们介绍了Wikiautogen，这是一种新型自动化多模式Wikipedia风格的文章一代的系统。与先前的方法不同，Wikiautogen检索并将相关图像与文本一起集成在一起，从而丰富了生成的内容的深度和视觉吸引力。为了进一步提高事实的准确性和全面性，我们提出了一种多人自我反思机制，该机制从不同的角度评估了内容，以提高可靠性，宽度和相干性等。此外，我们介绍了Wikiseek，我们介绍了构成Wwikipedia文章的基准和图像构造的基于五个挑战的基准，该基准是一种基于互补的构图。实验结果表明，Wikiautogen在我们的Wikiseek基准上的表现优于先前的方法，而Wikiautogen的表现更准确，相干和视觉上富集的Wikipedia风格的文章。我们在此HTTPS URL中显示了一些生成的示例。

Title: HingeRLC-GAN: Combating Mode Collapse with Hinge Loss and RLC Regularization

• Authors: Osman Goni, Himadri Saha Arka, Mithun Halder, Mir Moynuddin Ahmed Shibly, Swakkhar Shatabda
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.19074
• Pdf URL: https://arxiv.org/pdf/2503.19074
• Copy Paste: [[2503.19074]] HingeRLC-GAN: Combating Mode Collapse with Hinge Loss and RLC Regularization(https://arxiv.org/abs/2503.19074)
• Keywords: generative
• Abstract: Recent advances in Generative Adversarial Networks (GANs) have demonstrated their capability for producing high-quality images. However, a significant challenge remains mode collapse, which occurs when the generator produces a limited number of data patterns that do not reflect the diversity of the training dataset. This study addresses this issue by proposing a number of architectural changes aimed at increasing the diversity and stability of GAN models. We start by improving the loss function with Wasserstein loss and Gradient Penalty to better capture the full range of data variations. We also investigate various network architectures and conclude that ResNet significantly contributes to increased diversity. Building on these findings, we introduce HingeRLC-GAN, a novel approach that combines RLC Regularization and the Hinge loss function. With a FID Score of 18 and a KID Score of 0.001, our approach outperforms existing methods by effectively balancing training stability and increased diversity.
• 摘要：生成对抗网络（GAN）的最新进展证明了它们产生高质量图像的能力。但是，一个重大的挑战仍然是模式崩溃，这发生在发电机产生有限数量的数据模式时发生，这些数据模式不反映培训数据集的多样性。这项研究通过提出许多旨在提高GAN模型的多样性和稳定性的建筑变化来解决此问题。我们首先要通过Wasserstein损失和梯度惩罚来改善损失功能，以更好地捕获数据变化的全部范围。我们还研究了各种网络架构，并得出结论，重新NET会大大导致多样性的增加。在这些发现的基础上，我们引入了Hingerlc-Gan，这是一种结合RLC正则化和铰链损耗函数的新方法。我们的方法得分为18，孩子得分为0.001，我们的方法通过有效平衡训练稳定性和增加的多样性来优于现有方法。

Title: Risk-Based Thresholding for Reliable Anomaly Detection in Concentrated Solar Power Plants

• Authors: Yorick Estievenart, Sukanya Patra, Souhaib Ben Taieb
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19146
• Pdf URL: https://arxiv.org/pdf/2503.19146
• Copy Paste: [[2503.19146]] Risk-Based Thresholding for Reliable Anomaly Detection in Concentrated Solar Power Plants(https://arxiv.org/abs/2503.19146)
• Keywords: generative
• Abstract: Efficient and reliable operation of Concentrated Solar Power (CSP) plants is essential for meeting the growing demand for sustainable energy. However, high-temperature solar receivers face severe operational risks, such as freezing, deformation, and corrosion, resulting in costly downtime and maintenance. To monitor CSP plants, cameras mounted on solar receivers record infrared images at irregular intervals ranging from one to five minutes throughout the day. Anomalous images can be detected by thresholding an anomaly score, where the threshold is chosen to optimize metrics such as the F1-score on a validation set. This work proposes a framework for generating more reliable decision thresholds with finite-sample coverage guarantees on any chosen risk function. Our framework also incorporates an abstention mechanism, allowing high-risk predictions to be deferred to domain experts. Second, we propose a density forecasting method to estimate the likelihood of an observed image given a sequence of previously observed images, using this likelihood as its anomaly score. Third, we analyze the deployment results of our framework across multiple training scenarios over several months for two CSP plants. This analysis provides valuable insights to our industry partner for optimizing maintenance operations. Finally, given the confidential nature of our dataset, we provide an extended simulated dataset, leveraging recent advancements in generative modeling to create diverse thermal images that simulate multiple CSP plants. Our code is publicly available.
• 摘要：浓缩太阳能（CSP）工厂的高效和可靠运行对于满足对可持续能源不断增长的需求至关重要。但是，高温太阳接收机面临着严重的操作风险，例如冻结，变形和腐蚀，导致昂贵的停机时间和维护。为了监视CSP植物，安装在太阳接收机上的摄像机以不规则的间隔为纪录红外图像，范围从一整天的一到五分钟。可以通过阈值对异常分数检测到异常图像，其中选择阈值以优化诸如验证集中的F1分数之类的指标。这项工作提出了一个框架，以生成更可靠的决策阈值，并在任何选择的风险功能上保证有限样本覆盖范围。我们的框架还结合了弃权机制，可以将高风险预测推迟到领域专家。其次，我们提出了一种密度预测方法，以使用这种可能性作为其异常得分来估计观察到的图像的可能性。第三，我们分析了两种CSP工厂的几个月内多种培训场景的框架的部署结果。该分析为我们的行业合作伙伴提供了宝贵的见解，以优化维护操作。最后，鉴于我们数据集的机密性质，我们提供了一个扩展的模拟数据集，利用生成建模的最新进步来创建模拟多个CSP工厂的多种热图像。我们的代码公开可用。

Title: HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Models

• Authors: Mingzhen Huang, Fu-Jen Chu, Bugra Tekin, Kevin J Liang, Haoyu Ma, Weiyao Wang, Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, Kris Kitani, Matt Feiszli, Hao Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19157
• Pdf URL: https://arxiv.org/pdf/2503.19157
• Copy Paste: [[2503.19157]] HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Models(https://arxiv.org/abs/2503.19157)
• Keywords: generation, generative
• Abstract: We introduce HOIGPT, a token-based generative method that unifies 3D hand-object interactions (HOI) perception and generation, offering the first comprehensive solution for captioning and generating high-quality 3D HOI sequences from a diverse range of conditional signals (\eg text, objects, partial sequences). At its core, HOIGPT utilizes a large language model to predict the bidrectional transformation between HOI sequences and natural language descriptions. Given text inputs, HOIGPT generates a sequence of hand and object meshes; given (partial) HOI sequences, HOIGPT generates text descriptions and completes the sequences. To facilitate HOI understanding with a large language model, this paper introduces two key innovations: (1) a novel physically grounded HOI tokenizer, the hand-object decomposed VQ-VAE, for discretizing HOI sequences, and (2) a motion-aware language model trained to process and generate both text and HOI tokens. Extensive experiments demonstrate that HOIGPT sets new state-of-the-art performance on both text generation (+2.01% R Precision) and HOI generation (-2.56 FID) across multiple tasks and benchmarks.
• 摘要：我们介绍了Hoigpt，这是一种基于令牌的生成方法，该方法统一了3D手对象相互作用（HOI）感知和生成，为从各种有条件信号（\ EG EG文本，对象，部分序列）中提供了首个字幕和生成高质量的3D HOI序列的全面解决方案。 Hoigpt以核心利用大型语言模型来预测HOI序列和自然语言描述之间的双向转换。给定文本输入，Hoigpt生成了一系列手和对象网格。给定的（部分）HOI序列，Hoigpt生成文本描述并完成序列。为了通过大型语言模型来促进HOI理解，本文介绍了两个关键的创新：（1）一种新颖的物理扎根的HOI令牌，手工对象挖掘的VQ-VAE，用于离散的HOI序列，以及（2）一种运动意识的语言模型，以训练流程和生成文本和Hoi dokens。广泛的实验表明，Hoigpt在文本生成（+2.01％r Precision）和HOI生成（-2.56 FID）（-2.56 FID）（-2.56 FID）（-2.56 FID）（-2.56 FID）上设定了新的最先进的性能。

Title: FRESA:Feedforward Reconstruction of Personalized Skinned Avatars from Few Images

• Authors: Rong Wang, Fabian Prada, Ziyan Wang, Zhongshi Jiang, Chengxiang Yin, Junxuan Li, Shunsuke Saito, Igor Santesteban, Javier Romero, Rohan Joshi, Hongdong Li, Jason Saragih, Yaser Sheikh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19207
• Pdf URL: https://arxiv.org/pdf/2503.19207
• Copy Paste: [[2503.19207]] FRESA:Feedforward Reconstruction of Personalized Skinned Avatars from Few Images(https://arxiv.org/abs/2503.19207)
• Keywords: generation
• Abstract: We present a novel method for reconstructing personalized 3D human avatars with realistic animation from only a few images. Due to the large variations in body shapes, poses, and cloth types, existing methods mostly require hours of per-subject optimization during inference, which limits their practical applications. In contrast, we learn a universal prior from over a thousand clothed humans to achieve instant feedforward generation and zero-shot generalization. Specifically, instead of rigging the avatar with shared skinning weights, we jointly infer personalized avatar shape, skinning weights, and pose-dependent deformations, which effectively improves overall geometric fidelity and reduces deformation artifacts. Moreover, to normalize pose variations and resolve coupled ambiguity between canonical shapes and skinning weights, we design a 3D canonicalization process to produce pixel-aligned initial conditions, which helps to reconstruct fine-grained geometric details. We then propose a multi-frame feature aggregation to robustly reduce artifacts introduced in canonicalization and fuse a plausible avatar preserving person-specific identities. Finally, we train the model in an end-to-end framework on a large-scale capture dataset, which contains diverse human subjects paired with high-quality 3D scans. Extensive experiments show that our method generates more authentic reconstruction and animation than state-of-the-arts, and can be directly generalized to inputs from casually taken phone photos. Project page and code is available at this https URL.
• 摘要：我们提出了一种新颖的方法，用于重建个性化的3D人体化身，其中只有几张图像逼真的动画。由于身体形状，姿势和布类型的差异很大，现有方法通常需要在推断过程中进行数小时的每项受试者优化，这限制了其实际应用。相比之下，我们从一千多人的人类中学到了一个普遍的先前，以实现即时的前馈产生和零拍的概括。具体而言，我们共同推断出个性化的头像形状，皮肤重量和姿势依赖性变形，而不是用共同的皮肤重量索具，这有效地改善了整体几何富裕性并减少了变形文物。此外，为了使姿势变化归一化并解决规范形状和皮肤重量之间的歧义性歧义，我们设计了一个3D规范化过程，以产生与像素对齐的初始条件，这有助于重建细粒的几何细节。然后，我们提出了一个多帧特征聚合，以稳健地减少规范化中引入的伪影并融合合理的化身保留特定于人的身份。最后，我们在大规模捕获数据集上的端到端框架中训练该模型，该数据集包含不同的人类受试者，并配对高质量的3D扫描。广泛的实验表明，我们的方法比最新的方法生成更真实的重建和动画，并且可以直接概括为随意拍摄的电话照片的输入。项目页面和代码可在此HTTPS URL上找到。

Title: Continual Reinforcement Learning for HVAC Systems Control: Integrating Hypernetworks and Transfer Learning

• Authors: Gautham Udayakumar Bekal, Ahmed Ghareeb, Ashish Pujari
• Subjects: cs.LG, cs.AI, eess.SY
• Abstract URL: https://arxiv.org/abs/2503.19212
• Pdf URL: https://arxiv.org/pdf/2503.19212
• Copy Paste: [[2503.19212]] Continual Reinforcement Learning for HVAC Systems Control: Integrating Hypernetworks and Transfer Learning(https://arxiv.org/abs/2503.19212)
• Keywords: generation
• Abstract: Buildings with Heating, Ventilation, and Air Conditioning (HVAC) systems play a crucial role in ensuring indoor comfort and efficiency. While traditionally governed by physics-based models, the emergence of big data has enabled data-driven methods like Deep Reinforcement Learning (DRL). However, Reinforcement Learning (RL)-based techniques often suffer from sample inefficiency and limited generalization, especially across varying HVAC systems. We introduce a model-based reinforcement learning framework that uses a Hypernetwork to continuously learn environment dynamics across tasks with different action spaces. This enables efficient synthetic rollout generation and improved sample usage. Our approach demonstrates strong backward transfer in a continual learning setting after training on a second task, minimal fine-tuning on the first task allows rapid convergence within just 5 episodes and thus outperforming Model Free Reinforcement Learning (MFRL) and effectively mitigating catastrophic forgetting. These findings have significant implications for reducing energy consumption and operational costs in building management, thus supporting global sustainability goals. Keywords: Deep Reinforcement Learning, HVAC Systems Control, Hypernetworks, Transfer and Continual Learning, Catastrophic Forgetting
• 摘要：带有供暖，通风和空调（HVAC）系统的建筑物在确保室内舒适性和效率方面起着至关重要的作用。尽管传统上由基于物理的模型支配，但大数据的出现启用了数据驱动的方法，例如深入增强学习（DRL）。但是，加强学习（RL）的技术通常会遭受样本效率低下和有限的概括，尤其是在不同的HVAC系统中。我们介绍了一个基于模型的增强学习框架，该框架使用超网络来跨不同动作空间的任务不断学习环境动态。这使得有效的合成推出生成并改进了样品使用情况。我们的方法表明，在训练第二项任务后，在连续学习的环境中表现出强大的向后转移，对第一个任务的微调最少可以在仅5集中快速融合，从而超过了模型的自由加固学习（MFRL），并有效地减轻了灾难性的遗忘。这些发现对减少建筑管理中的能耗和运营成本具有重要意义，从而支持全球可持续性目标。关键字：深度强化学习，HVAC系统控制，超网络，转移和持续学习，灾难性遗忘

Title: Learning Hazing to Dehazing: Towards Realistic Haze Generation for Real-World Image Dehazing

• Authors: Ruiyi Wang, Yushuo Zheng, Zicheng Zhang, Chunyi Li, Shuaicheng Liu, Guangtao Zhai, Xiaohong Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19262
• Pdf URL: https://arxiv.org/pdf/2503.19262
• Copy Paste: [[2503.19262]] Learning Hazing to Dehazing: Towards Realistic Haze Generation for Real-World Image Dehazing(https://arxiv.org/abs/2503.19262)
• Keywords: generation, generative
• Abstract: Existing real-world image dehazing methods primarily attempt to fine-tune pre-trained models or adapt their inference procedures, thus heavily relying on the pre-trained models and associated training data. Moreover, restoring heavily distorted information under dense haze requires generative diffusion models, whose potential in dehazing remains underutilized partly due to their lengthy sampling processes. To address these limitations, we introduce a novel hazing-dehazing pipeline consisting of a Realistic Hazy Image Generation framework (HazeGen) and a Diffusion-based Dehazing framework (DiffDehaze). Specifically, HazeGen harnesses robust generative diffusion priors of real-world hazy images embedded in a pre-trained text-to-image diffusion model. By employing specialized hybrid training and blended sampling strategies, HazeGen produces realistic and diverse hazy images as high-quality training data for DiffDehaze. To alleviate the inefficiency and fidelity concerns associated with diffusion-based methods, DiffDehaze adopts an Accelerated Fidelity-Preserving Sampling process (AccSamp). The core of AccSamp is the Tiled Statistical Alignment Operation (AlignOp), which can provide a clean and faithful dehazing estimate within a small fraction of sampling steps to reduce complexity and enable effective fidelity guidance. Extensive experiments demonstrate the superior dehazing performance and visual quality of our approach over existing methods. The code is available at this https URL.
• 摘要：现有的现实世界图像脱掩的方法主要尝试微调预训练的模型或调整其推理程序，从而在很大程度上依赖于预先训练的模型和相关的培训数据。此外，在密集的雾期下恢复严重扭曲的信息需要生成的扩散模型，由于其漫长的采样过程，其潜在的去悬空的潜力部分仍未得到充分利用。为了解决这些局限性，我们引入了一条新颖的危险管道，该管道由现实的朦胧图像生成框架（Hazegen）和基于扩散的Dhazing框架（Diffdehaze）组成。具体而言，Hazegen利用现实世界中的朦胧图像的鲁棒生成扩散先验，该图像嵌入了预训练的文本对图像扩散模型中。通过采用专门的混合培训并混合了采样策略，Hazegen可以作为Diffdehaze的高质量培训数据产生现实和多样化的朦胧图像。为了减轻与基于扩散的方法相关的效率低下和忠诚问题，Diffdehaze采用了加速富裕性的采样过程（ACCSAMP）。 ACCSAMP的核心是瓷砖统计对准操作（Alignop），它可以在采样步骤的一小部分中提供干净而忠实的除尘估算，以降低复杂性并实现有效的忠实指导。广泛的实验证明了我们的方法优于现有方法的卓越表现和视觉质量。该代码可在此HTTPS URL上找到。

Title: DWIM: Towards Tool-aware Visual Reasoning via Discrepancy-aware Workflow Generation & Instruct-Masking Tuning

• Authors: Fucai Ke, Vijay Kumar B G, Xingjian Leng, Zhixi Cai, Zaid Khan, Weiqing Wang, Pari Delir Haghighi, Hamid Rezatofighi, Manmohan Chandraker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19263
• Pdf URL: https://arxiv.org/pdf/2503.19263
• Copy Paste: [[2503.19263]] DWIM: Towards Tool-aware Visual Reasoning via Discrepancy-aware Workflow Generation & Instruct-Masking Tuning(https://arxiv.org/abs/2503.19263)
• Keywords: generation
• Abstract: Visual reasoning (VR), which is crucial in many fields for enabling human-like visual understanding, remains highly challenging. Recently, compositional visual reasoning approaches, which leverage the reasoning abilities of large language models (LLMs) with integrated tools to solve problems, have shown promise as more effective strategies than end-to-end VR methods. However, these approaches face limitations, as frozen LLMs lack tool awareness in VR, leading to performance bottlenecks. While leveraging LLMs for reasoning is widely used in other domains, they are not directly applicable to VR due to limited training data, imperfect tools that introduce errors and reduce data collection efficiency in VR, and challenging in fine-tuning on noisy workflows. To address these challenges, we propose DWIM: i) Discrepancy-aware training Workflow generation, which assesses tool usage and extracts more viable workflows for training; and ii) Instruct-Masking fine-tuning, which guides the model to only clone effective actions, enabling the generation of more practical solutions. Our experiments demonstrate that DWIM achieves state-of-the-art performance across various VR tasks, exhibiting strong generalization on multiple widely-used datasets.
• 摘要：视觉推理（VR）在许多领域至关重要的是具有人类般的视觉理解，这仍然是高度挑战。最近，使用集成工具来解决问题的大型语言模型（LLMS）的推理能力（LLMS）的构图视觉推理方法比端到端VR方法更有效地表现出了有效的策略。但是，这些方法面临局限性，因为冷冻的LLM缺乏VR中的工具意识，从而导致性能瓶颈。尽管利用LLM进行推理广泛用于其他域，但由于培训数据有限，引入错误和降低VR的数据收集效率以及对嘈杂的工作流进行微调挑战，它们并不直接适用于VR。为了应对这些挑战，我们提出了DWIM：i）差异感知的培训工作流程，该培训的生成，它评估了工具的使用并提取了更多可行的培训工作流程； ii）指导掩盖微调，该调整指导模型仅克隆有效的动作，从而能够生成更实际的解决方案。我们的实验表明，DWIM在各种VR任务中实现了最先进的性能，在多个广泛使用的数据集上表现出强烈的概括。

Title: ISPDiffuser: Learning RAW-to-sRGB Mappings with Texture-Aware Diffusion Models and Histogram-Guided Color Consistency

• Authors: Yang Ren, Hai Jiang, Menglong Yang, Wei Li, Shuaicheng Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19283
• Pdf URL: https://arxiv.org/pdf/2503.19283
• Copy Paste: [[2503.19283]] ISPDiffuser: Learning RAW-to-sRGB Mappings with Texture-Aware Diffusion Models and Histogram-Guided Color Consistency(https://arxiv.org/abs/2503.19283)
• Keywords: generative
• Abstract: RAW-to-sRGB mapping, or the simulation of the traditional camera image signal processor (ISP), aims to generate DSLR-quality sRGB images from raw data captured by smartphone sensors. Despite achieving comparable results to sophisticated handcrafted camera ISP solutions, existing learning-based methods still struggle with detail disparity and color distortion. In this paper, we present ISPDiffuser, a diffusion-based decoupled framework that separates the RAW-to-sRGB mapping into detail reconstruction in grayscale space and color consistency mapping from grayscale to sRGB. Specifically, we propose a texture-aware diffusion model that leverages the generative ability of diffusion models to focus on local detail recovery, in which a texture enrichment loss is further proposed to prompt the diffusion model to generate more intricate texture details. Subsequently, we introduce a histogram-guided color consistency module that utilizes color histogram as guidance to learn precise color information for grayscale to sRGB color consistency mapping, with a color consistency loss designed to constrain the learned color information. Extensive experimental results show that the proposed ISPDiffuser outperforms state-of-the-art competitors both quantitatively and visually. The code is available at this https URL.
• 摘要：原始映射或传统摄像机图像信号处理器（ISP）的仿真旨在从智能手机传感器捕获的原始数据中生成DSLR质量的SRGB图像。尽管取得了与复杂的手工ISP解决方案相当的结果，但现有的基于学习的方法仍然在细节差异和颜色扭曲方面挣扎。在本文中，我们提出了ISPDiffuser，这是一个基于扩散的解耦框架，将原始到SRGB映射分为灰度空间中的细节重建，并将其从灰度到SRGB到SRGB。具体而言，我们提出了一个纹理感知的扩散模型，该模型利用扩散模型专注于局部细节恢复的生成能力，其中进一步提出了纹理富集损失，以促使扩散模型生成更复杂的纹理细节。随后，我们引入了一个直方图引导的颜色一致性模块，该模块利用颜色直方图作为指导，以学习灰度的精确颜色信息，以使灰度到SRGB颜色一致性映射，其颜色一致性损失旨在限制学习的颜色信息。广泛的实验结果表明，拟议的ISPDIFFUSER在定量和视觉上都优于最先进的竞争者。该代码可在此HTTPS URL上找到。

Title: Exploring Semantic Feature Discrimination for Perceptual Image Super-Resolution and Opinion-Unaware No-Reference Image Quality Assessment

• Authors: Guanglu Dong, Xiangyu Liao, Mingyang Li, Guihuan Guo, Chao Ren
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2503.19295
• Pdf URL: https://arxiv.org/pdf/2503.19295
• Copy Paste: [[2503.19295]] Exploring Semantic Feature Discrimination for Perceptual Image Super-Resolution and Opinion-Unaware No-Reference Image Quality Assessment(https://arxiv.org/abs/2503.19295)
• Keywords: super-resolution, generative, quality assessment
• Abstract: Generative Adversarial Networks (GANs) have been widely applied to image super-resolution (SR) to enhance the perceptual quality. However, most existing GAN-based SR methods typically perform coarse-grained discrimination directly on images and ignore the semantic information of images, making it challenging for the super resolution networks (SRN) to learn fine-grained and semantic-related texture details. To alleviate this issue, we propose a semantic feature discrimination method, SFD, for perceptual SR. Specifically, we first design a feature discriminator (Feat-D), to discriminate the pixel-wise middle semantic features from CLIP, aligning the feature distributions of SR images with that of high-quality images. Additionally, we propose a text-guided discrimination method (TG-D) by introducing learnable prompt pairs (LPP) in an adversarial manner to perform discrimination on the more abstract output feature of CLIP, further enhancing the discriminative ability of our method. With both Feat-D and TG-D, our SFD can effectively distinguish between the semantic feature distributions of low-quality and high-quality images, encouraging SRN to generate more realistic and semantic-relevant textures. Furthermore, based on the trained Feat-D and LPP, we propose a novel opinion-unaware no-reference image quality assessment (OU NR-IQA) method, SFD-IQA, greatly improving OU NR-IQA performance without any additional targeted training. Extensive experiments on classical SISR, real-world SISR, and OU NR-IQA tasks demonstrate the effectiveness of our proposed methods.
• 摘要：生成对抗网络（GAN）已被广泛应用于图像超分辨率（SR），以提高感知质量。但是，大多数现有的基于GAN的SR方法通常直接在图像上执行粗粒歧视，而忽略了图像的语义信息，这使得超级分辨率网络（SRN）具有挑战性，以学习与细粒度和语义相关的纹理细节。为了减轻此问题，我们提出了一种语义特征歧视方法SFD，以进行感知SR。具体来说，我们首先设计一个功能歧视器（feat-d），以歧视剪辑中的像素中间语义特征，将SR图像的特征分布与高质量图像的特征分布对齐。此外，我们通过以对抗性方式引入可学习的提示对（LPP）来提出一种文本引导的歧视方法（TG-D），以对剪辑的更抽象输出功能进行歧视，从而进一步增强我们方法的歧视能力。借助Feat-D和TG-D，我们的SFD可以有效地区分低质量和高质量图像的语义特征分布，从而鼓励SRN产生更逼真和与语义相关的纹理。此外，根据训练有素的壮举D和LPP，我们提出了一种新颖的意见 - 统一的参考图像质量评估（OU NR-IQA）方法，SFD-IQA，大大改善了OU NR-IQA性能，而无需任何其他有针对性的培训。关于经典SISR，现实世界SISR和OU NR-IQA任务的广泛实验证明了我们提出的方法的有效性。

Title: UniMoMo: Unified Generative Modeling of 3D Molecules for De Novo Binder Design

• Authors: Xiangzhe Kong, Zishen Zhang, Ziting Zhang, Rui Jiao, Jianzhu Ma, Kai Liu, Wenbing Huang, Yang Liu
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2503.19300
• Pdf URL: https://arxiv.org/pdf/2503.19300
• Copy Paste: [[2503.19300]] UniMoMo: Unified Generative Modeling of 3D Molecules for De Novo Binder Design(https://arxiv.org/abs/2503.19300)
• Keywords: generation, generative
• Abstract: The design of target-specific molecules such as small molecules, peptides, and antibodies is vital for biological research and drug discovery. Existing generative methods are restricted to single-domain molecules, failing to address versatile therapeutic needs or utilize cross-domain transferability to enhance model performance. In this paper, we introduce Unified generative Modeling of 3D Molecules (UniMoMo), the first framework capable of designing binders of multiple molecular domains using a single model. In particular, UniMoMo unifies the representations of different molecules as graphs of blocks, where each block corresponds to either a standard amino acid or a molecular fragment. Based on these unified representations, UniMoMo utilizes a geometric latent diffusion model for 3D molecular generation, featuring an iterative full-atom autoencoder to compress blocks into latent space points, followed by an E(3)-equivariant diffusion process. Extensive benchmarks across peptides, antibodies, and small molecules demonstrate the superiority of our unified framework over existing domain-specific models, highlighting the benefits of multi-domain training.
• 摘要：靶标特异性分子（例如小分子，肽和抗体）的设计对于生物学研究和药物发现至关重要。现有的生成方法仅限于单域分子，无法满足多功能治疗需求或利用跨域转移性来增强模型性能。在本文中，我们介绍了3D分子（Unimomo）的统一生成建模，这是一个能够使用单个模型设计多个分子域的粘合剂的第一个框架。特别是，Unimomo将不同分子的表示形式统一为块的图，其中每个块对应于标准氨基酸或分子片段。基于这些统一的表示，Unimomo利用了3D分子生成的几何潜在扩散模型，具有迭代全原子自动编码器来将块压缩到潜在的空间点中，然后进行E（3） - 等级式扩散过程。跨肽，抗体和小分子之间的广泛基准表明了我们统一框架比现有域特异性模型的优越性，突显了多域训练的好处。

Title: ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoning

• Authors: Jiaqi Liao, Zhengyuan Yang, Linjie Li, Dianqi Li, Kevin Lin, Yu Cheng, Lijuan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19312
• Pdf URL: https://arxiv.org/pdf/2503.19312
• Copy Paste: [[2503.19312]] ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoning(https://arxiv.org/abs/2503.19312)
• Keywords: generation
• Abstract: In this work, we study the problem of Text-to-Image In-Context Learning (T2I-ICL). While Unified Multimodal LLMs (MLLMs) have advanced rapidly in recent years, they struggle with contextual reasoning in T2I-ICL scenarios. To address this limitation, we propose a novel framework that incorporates a thought process called ImageGen-CoT prior to image generation. To avoid generating unstructured ineffective reasoning steps, we develop an automatic pipeline to curate a high-quality ImageGen-CoT dataset. We then fine-tune MLLMs using this dataset to enhance their contextual reasoning capabilities. To further enhance performance, we explore test-time scale-up strategies and propose a novel hybrid scaling approach. This approach first generates multiple ImageGen-CoT chains and then produces multiple images for each chain via sampling. Extensive experiments demonstrate the effectiveness of our proposed method. Notably, fine-tuning with the ImageGen-CoT dataset leads to a substantial 80\% performance gain for SEED-X on T2I-ICL tasks. See our project page at this https URL. Code and model weights will be open-sourced.
• 摘要：在这项工作中，我们研究了文本对图像中的文本内部学习（T2-ICL）的问题。尽管近年来统一的多模式LLM（MLLM）迅速发展，但它们在T2I-ICL方案中的上下文推理挣扎。为了解决这一限制，我们提出了一个新颖的框架，该框架在图像生成之前包含了一个称为ImageGen-Cot的思维过程。为了避免生成无效的推理步骤，我们开发了自动管道来策划高质量的ImageGen-COT数据集。然后，我们使用此数据集微调MLLM，以增强其上下文推理功能。为了进一步提高性能，我们探讨了测试时间扩展策略，并提出了一种新型的混合规模方法。该方法首先生成多个ImageGen-Cot链，然后通过采样为每个链生成多个图像。广泛的实验证明了我们提出的方法的有效性。值得注意的是，对ImageGen-COT数据集进行微调导致T2I-ICL任务上的Seed-X的80 \％性能增长。在此HTTPS URL上查看我们的项目页面。代码和模型权重将是开源的。

Title: Long-Context Autoregressive Video Modeling with Next-Frame Prediction

• Authors: Yuchao Gu, Weijia Mao, Mike Zheng Shou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19325
• Pdf URL: https://arxiv.org/pdf/2503.19325
• Copy Paste: [[2503.19325]] Long-Context Autoregressive Video Modeling with Next-Frame Prediction(https://arxiv.org/abs/2503.19325)
• Keywords: generation
• Abstract: Long-context autoregressive modeling has significantly advanced language generation, but video generation still struggles to fully utilize extended temporal contexts. To investigate long-context video modeling, we introduce Frame AutoRegressive (FAR), a strong baseline for video autoregressive modeling. Just as language models learn causal dependencies between tokens (i.e., Token AR), FAR models temporal causal dependencies between continuous frames, achieving better convergence than Token AR and video diffusion transformers. Building on FAR, we observe that long-context vision modeling faces challenges due to visual redundancy. Existing RoPE lacks effective temporal decay for remote context and fails to extrapolate well to long video sequences. Additionally, training on long videos is computationally expensive, as vision tokens grow much faster than language tokens. To tackle these issues, we propose balancing locality and long-range dependency. We introduce FlexRoPE, an test-time technique that adds flexible temporal decay to RoPE, enabling extrapolation to 16x longer vision contexts. Furthermore, we propose long short-term context modeling, where a high-resolution short-term context window ensures fine-grained temporal consistency, while an unlimited long-term context window encodes long-range information using fewer tokens. With this approach, we can train on long video sequences with a manageable token context length. We demonstrate that FAR achieves state-of-the-art performance in both short- and long-video generation, providing a simple yet effective baseline for video autoregressive modeling.
• 摘要：长篇小说自回旋建模具有显着高级的语言生成，但是视频生成仍在努力充分利用扩展的时间上下文。为了调查长篇小说视频建模，我们介绍了框架自回归（FAR），这是视频自回归建模的强大基线。就像语言模型学习令牌（即令牌AR）之间的因果关系一样，远处的临时因果关系依赖性在连续框架之间的依赖性，从而获得了比令牌AR和视频扩散变压器更好的收敛性。在远处，我们观察到，由于视觉冗余而导致的长篇文化视觉建模面临挑战。现有绳索缺乏远程上下文的有效时间衰减，并且无法很好地推断出长时间的视频序列。此外，随着视觉令牌的增长速度比语言令牌快得多，对长视频的培训在计算上是昂贵的。为了解决这些问题，我们建议平衡当地和远程依赖性。我们介绍了一种测试时间技术，它为绳索增加了灵活的时间衰减，从而使外推到16倍更长的视力上下文。此外，我们提出了长期的短期上下文建模，其中高分辨率的短期上下文窗口确保了细粒度的时间一致性，而无限的长期上下文窗口则使用较少的令牌来编码长距离信息。通过这种方法，我们可以使用可管理的令牌上下文长度进行长时间的视频序列训练。我们证明，远距离短期和长时间发电的远面表现，为视频自动回归建模提供了简单而有效的基线。

Title: BADGR: Bundle Adjustment Diffusion Conditioned by GRadients for Wide-Baseline Floor Plan Reconstruction

• Authors: Yuguang Li, Ivaylo Boyadzhiev, Zixuan Liu, Linda Shapiro, Alex Colburn
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19340
• Pdf URL: https://arxiv.org/pdf/2503.19340
• Copy Paste: [[2503.19340]] BADGR: Bundle Adjustment Diffusion Conditioned by GRadients for Wide-Baseline Floor Plan Reconstruction(https://arxiv.org/abs/2503.19340)
• Keywords: generation
• Abstract: Reconstructing precise camera poses and floor plan layouts from wide-baseline RGB panoramas is a difficult and unsolved problem. We introduce BADGR, a novel diffusion model that jointly performs reconstruction and bundle adjustment (BA) to refine poses and layouts from a coarse state, using 1D floor boundary predictions from dozens of images of varying input densities. Unlike a guided diffusion model, BADGR is conditioned on dense per-entity outputs from a single-step Levenberg Marquardt (LM) optimizer and is trained to predict camera and wall positions while minimizing reprojection errors for view-consistency. The objective of layout generation from denoising diffusion process complements BA optimization by providing additional learned layout-structural constraints on top of the co-visible features across images. These constraints help BADGR to make plausible guesses on spatial relations which help constrain pose graph, such as wall adjacency, collinearity, and learn to mitigate errors from dense boundary observations with global contexts. BADGR trains exclusively on 2D floor plans, simplifying data acquisition, enabling robust augmentation, and supporting variety of input densities. Our experiments and analysis validate our method, which significantly outperforms the state-of-the-art pose and floor plan layout reconstruction with different input densities.
• 摘要：从宽基线RGB全景中重建精确的相机姿势和平面图布局是一个困难且未解决的问题。我们介绍了Badgr，这是一种新型扩散模型，共同执行重建和束调整（BA），以使用数十个不同输入密度的图像中的1D层边界预测来完善粗糙状态的姿势和布局。与引导扩散模型不同，Badgr以单步Levenberg Marquardt（LM）优化器的致密性输出为条件，并经过训练以预测相机和墙壁位置，同时最大程度地减少了视图抗性的再估算错误。从denoising扩散过程中布局生成的目的通过在图像跨图像的可共同特征上提供其他学习的布局结构约束来补充BA优化。这些约束有助于Badgr对空间关系做出合理的猜测，这有助于限制姿势图，例如墙壁邻接，截然性，并学会从具有全球环境的密集边界观察中减轻错误。 Badgr专门针对2D楼平面图进行训练，简化数据获取，实现强大的增强和支持各种输入密度。我们的实验和分析验证了我们的方法，这显着超过了具有不同输入密度的最先进的姿势和平面图布局重建。

Title: VGAT: A Cancer Survival Analysis Framework Transitioning from Generative Visual Question Answering to Genomic Reconstruction

• Authors: Zizhi Chen, Minghao Han, Xukun Zhang, Shuwei Ma, Tao Liu, Xing Wei, Lihua Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19367
• Pdf URL: https://arxiv.org/pdf/2503.19367
• Copy Paste: [[2503.19367]] VGAT: A Cancer Survival Analysis Framework Transitioning from Generative Visual Question Answering to Genomic Reconstruction(https://arxiv.org/abs/2503.19367)
• Keywords: generative
• Abstract: Multimodal learning combining pathology images and genomic sequences enhances cancer survival analysis but faces clinical implementation barriers due to limited access to genomic sequencing in under-resourced regions. To enable survival prediction using only whole-slide images (WSI), we propose the Visual-Genomic Answering-Guided Transformer (VGAT), a framework integrating Visual Question Answering (VQA) techniques for genomic modality reconstruction. By adapting VQA's text feature extraction approach, we derive stable genomic representations that circumvent dimensionality challenges in raw genomic data. Simultaneously, a cluster-based visual prompt module selectively enhances discriminative WSI patches, addressing noise from unfiltered image regions. Evaluated across five TCGA datasets, VGAT outperforms existing WSI-only methods, demonstrating the viability of genomic-informed inference without sequencing. This approach bridges multimodal research and clinical feasibility in resource-constrained settings. The code link is this https URL.
• 摘要：将病理图像和基因组序列结合的多模式学习增强了癌症的生存分析，但由于资源不足的区域中获得基因组测序的机会有限，因此面临临床实施障碍。为了仅使用全扫描图像（WSI）启用生存预测，我们提出了视觉基因组答案引导的变压器（VGAT），这是一个集成了视觉问题答录（VQA）技术的框架，以进行基因组模态重建技术。通过调整VQA的文本特征提取方法，我们得出了稳定的基因组表示，从而规避原始基因组数据中的维度挑战。同时，基于群集的视觉提示模块选择性地增强了判别WSI贴片，从而解决了未经过滤的图像区域的噪声。 VGAT在五个TCGA数据集中进行了评估，优于现有的仅WSI-Forly方法，证明了基因组信息的推理无需测序的可行性。这种方法桥接了在资源约束设置中的多模式研究和临床可行性。代码链接是此HTTPS URL。

Title: EfficientMT: Efficient Temporal Adaptation for Motion Transfer in Text-to-Video Diffusion Models

• Authors: Yufei Cai, Hu Han, Yuxiang Wei, Shiguang Shan, Xilin Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19369
• Pdf URL: https://arxiv.org/pdf/2503.19369
• Copy Paste: [[2503.19369]] EfficientMT: Efficient Temporal Adaptation for Motion Transfer in Text-to-Video Diffusion Models(https://arxiv.org/abs/2503.19369)
• Keywords: generation, generative
• Abstract: The progress on generative models has led to significant advances on text-to-video (T2V) generation, yet the motion controllability of generated videos remains limited. Existing motion transfer methods explored the motion representations of reference videos to guide generation. Nevertheless, these methods typically rely on sample-specific optimization strategy, resulting in high computational burdens. In this paper, we propose \textbf{EfficientMT}, a novel and efficient end-to-end framework for video motion transfer. By leveraging a small set of synthetic paired motion transfer samples, EfficientMT effectively adapts a pretrained T2V model into a general motion transfer framework that can accurately capture and reproduce diverse motion patterns. Specifically, we repurpose the backbone of the T2V model to extract temporal information from reference videos, and further propose a scaler module to distill motion-related information. Subsequently, we introduce a temporal integration mechanism that seamlessly incorporates reference motion features into the video generation process. After training on our self-collected synthetic paired samples, EfficientMT enables general video motion transfer without requiring test-time optimization. Extensive experiments demonstrate that our EfficientMT outperforms existing methods in efficiency while maintaining flexible motion controllability. Our code will be available this https URL.
• 摘要：生成模型的进展导致了文本到视频（T2V）生成的重大进展，但生成的视频的运动可控性仍然有限。现有的运动转移方法探索了参考视频的运动表示，以指导生成。然而，这些方法通常依赖于特定于样本的优化策略，从而导致较高的计算负担。在本文中，我们建议\ textbf {效率mt}，这是一个新颖而有效的视频运动传输端到端框架。通过利用一小组合成的配对运动转移样品，有效地将预告片的T2V模型有效地调整为一般运动转移框架，该框架可以准确捕获和再现各种运动模式。具体而言，我们将T2V模型的主干重新实现以从参考视频中提取时间信息，并进一步提出一个缩放器模块以提炼与运动相关的信息。随后，我们引入了一种时间整合机制，该机制将参考运动特征无缝纳入视频生成过程。在对我们自我收集的合成配对样品进行了训练之后，有效MT可以实现一般的视频运动传输，而无需测试时间优化。广泛的实验表明，我们的有效MT在效率方面优于现有方法，同时保持灵活的运动可控性。我们的代码将提供此HTTPS URL。

Title: Interpretable Generative Models through Post-hoc Concept Bottlenecks

• Authors: Akshay Kulkarni, Ge Yan, Chung-En Sun, Tuomas Oikarinen, Tsui-Wei Weng
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.19377
• Pdf URL: https://arxiv.org/pdf/2503.19377
• Copy Paste: [[2503.19377]] Interpretable Generative Models through Post-hoc Concept Bottlenecks(https://arxiv.org/abs/2503.19377)
• Keywords: generative
• Abstract: Concept bottleneck models (CBM) aim to produce inherently interpretable models that rely on human-understandable concepts for their predictions. However, existing approaches to design interpretable generative models based on CBMs are not yet efficient and scalable, as they require expensive generative model training from scratch as well as real images with labor-intensive concept supervision. To address these challenges, we present two novel and low-cost methods to build interpretable generative models through post-hoc techniques and we name our approaches: concept-bottleneck autoencoder (CB-AE) and concept controller (CC). Our proposed approaches enable efficient and scalable training without the need of real data and require only minimal to no concept supervision. Additionally, our methods generalize across modern generative model families including generative adversarial networks and diffusion models. We demonstrate the superior interpretability and steerability of our methods on numerous standard datasets like CelebA, CelebA-HQ, and CUB with large improvements (average ~25%) over the prior work, while being 4-15x faster to train. Finally, a large-scale user study is performed to validate the interpretability and steerability of our methods.
• 摘要：概念瓶颈模型（CBM）旨在产生固有的可解释模型，这些模型依赖于人类可行的概念来进行预测。但是，现有的设计基于CBM的可解释生成模型的方法尚不高效且可扩展，因为它们需要从头开始昂贵的生成模型培训以及具有劳动密集型概念监督的真实图像。为了应对这些挑战，我们提出了两种新颖和低成本的方法，可以通过事后技术来构建可解释的生成模型，并将我们的方法命名：概念 - 底层自动编码器（CB-AE）和概念控制器（CC）。我们提出的方法可以实现高效且可扩展的培训，而无需真正的数据，只需要最少或不需要概念监督即可。此外，我们的方法在包括生成对抗网络和扩散模型在内的现代生成模型家族中概括了。我们在许多标准数据集（如Celeba，Celeba-HQ和Cub）上证明了我们方法的卓越可解释性和可转让性，并且在先前的工作中具有较大的改进（平均约25％），同时训练的速度更快4-15倍。最后，进行了大规模的用户研究，以验证我们方法的可解释性和可取性。

Title: MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation

• Authors: Yukang Lin, Hokit Fung, Jianjin Xu, Zeping Ren, Adela S.M. Lau, Guosheng Yin, Xiu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19383
• Pdf URL: https://arxiv.org/pdf/2503.19383
• Copy Paste: [[2503.19383]] MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation(https://arxiv.org/abs/2503.19383)
• Keywords: generation
• Abstract: Recent portrait animation methods have made significant strides in generating realistic lip synchronization. However, they often lack explicit control over head movements and facial expressions, and cannot produce videos from multiple viewpoints, resulting in less controllable and expressive animations. Moreover, text-guided portrait animation remains underexplored, despite its user-friendly nature. We present a novel two-stage text-guided framework, MVPortrait (Multi-view Vivid Portrait), to generate expressive multi-view portrait animations that faithfully capture the described motion and emotion. MVPortrait is the first to introduce FLAME as an intermediate representation, effectively embedding facial movements, expressions, and view transformations within its parameter space. In the first stage, we separately train the FLAME motion and emotion diffusion models based on text input. In the second stage, we train a multi-view video generation model conditioned on a reference portrait image and multi-view FLAME rendering sequences from the first stage. Experimental results exhibit that MVPortrait outperforms existing methods in terms of motion and emotion control, as well as view consistency. Furthermore, by leveraging FLAME as a bridge, MVPortrait becomes the first controllable portrait animation framework that is compatible with text, speech, and video as driving signals.
• 摘要：最近的肖像画方法在产生逼真的唇部同步方面取得了重大进步。但是，他们通常缺乏对头部动作和面部表情的明确控制，并且无法从多个观点产生视频，从而导致了不太可控和表现力的动画。此外，尽管具有用户友好的性质，但文字引导的肖像画仍然没有被忽略。我们提出了一个新颖的两阶段文本引导框架Mvportrait（Multi-View Vivid Portrait），以生成表现力的多视图肖像画动画，该动画忠实地捕捉了所描述的动作和情感。 Mvportrait是第一个将火焰作为中间表示形式引入的，有效地嵌入了面部运动，表达式和视图转换在其参数空间内。在第一阶段，我们根据文本输入分别训练火焰运动和情感扩散模型。在第二阶段，我们训练一个以参考肖像图像和第一阶段的多视图火焰渲染序列为条件的多视频视频生成模型。实验结果表明，Mvportrait在运动和情绪控制方面优于现有方法，以及视图一致性。此外，通过利用火焰作为桥梁，Mvportrait成为第一个可控的肖像画框架，与文本，语音和视频作为驾驶信号兼容。

Title: Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcing

• Authors: Jaihoon Kim, Taehoon Yoon, Jisung Hwang, Minhyuk Sung
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.19385
• Pdf URL: https://arxiv.org/pdf/2503.19385
• Copy Paste: [[2503.19385]] Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcing(https://arxiv.org/abs/2503.19385)
• Keywords: generation, generative
• Abstract: We propose an inference-time scaling approach for pretrained flow models. Recently, inference-time scaling has gained significant attention in LLMs and diffusion models, improving sample quality or better aligning outputs with user preferences by leveraging additional computation. For diffusion models, particle sampling has allowed more efficient scaling due to the stochasticity at intermediate denoising steps. On the contrary, while flow models have gained popularity as an alternative to diffusion models--offering faster generation and high-quality outputs in state-of-the-art image and video generative models--efficient inference-time scaling methods used for diffusion models cannot be directly applied due to their deterministic generative process. To enable efficient inference-time scaling for flow models, we propose three key ideas: 1) SDE-based generation, enabling particle sampling in flow models, 2) Interpolant conversion, broadening the search space and enhancing sample diversity, and 3) Rollover Budget Forcing (RBF), an adaptive allocation of computational resources across timesteps to maximize budget utilization. Our experiments show that SDE-based generation, particularly variance-preserving (VP) interpolant-based generation, improves the performance of particle sampling methods for inference-time scaling in flow models. Additionally, we demonstrate that RBF with VP-SDE achieves the best performance, outperforming all previous inference-time scaling approaches.
• 摘要：我们建议用于预审流的流动模型的推理时间缩放方法。最近，推理时间缩放在LLM和扩散模型中引起了极大的关注，通过利用其他计算来改善样品质量或更好地对准输出，以用户偏好对齐。对于扩散模型，由于中间去胶的步骤的随机性，粒子采样允许更有效的缩放。相反，尽管流动模型已成为扩散模型的替代品（可在最新的图像和视频生成模型中提供更快的生成和高质量输出）的替代性，因此无法直接应用用于扩散模型的推理时间缩放方法，因为它们的确定性生成过程无法直接应用。为了实现流量模型的有效推理时间缩放，我们提出了三个关键思想：1）基于SDE的生成，在流模型中启用粒子采样，2）插值转换，扩大搜索空间和增强样本多样性，以及3）推销预算强迫（RBF），一种适应性的计算资源分配，以跨时间浏览到跨时间跨度的预算UPIDIATION。我们的实验表明，基于SDE的生成，尤其是基于方差的interpolant interpolant生成，可提高流量模型中推理时间缩放的粒子采样方法的性能。此外，我们证明了带有VP-SDE的RBF实现最佳性能，表现优于所有以前的推理时间缩放方法。

Title: EmoHead: Emotional Talking Head via Manipulating Semantic Expression Parameters

• Authors: Xuli Shen, Hua Cai, Dingding Yu, Weilin Shen, Qing Xu, Xiangyang Xue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19416
• Pdf URL: https://arxiv.org/pdf/2503.19416
• Copy Paste: [[2503.19416]] EmoHead: Emotional Talking Head via Manipulating Semantic Expression Parameters(https://arxiv.org/abs/2503.19416)
• Keywords: generation
• Abstract: Generating emotion-specific talking head videos from audio input is an important and complex challenge for human-machine interaction. However, emotion is highly abstract concept with ambiguous boundaries, and it necessitates disentangled expression parameters to generate emotionally expressive talking head videos. In this work, we present EmoHead to synthesize talking head videos via semantic expression parameters. To predict expression parameter for arbitrary audio input, we apply an audio-expression module that can be specified by an emotion tag. This module aims to enhance correlation from audio input across various emotions. Furthermore, we leverage pre-trained hyperplane to refine facial movements by probing along the vertical direction. Finally, the refined expression parameters regularize neural radiance fields and facilitate the emotion-consistent generation of talking head videos. Experimental results demonstrate that semantic expression parameters lead to better reconstruction quality and controllability.
• 摘要：从音频输入中产生特定于情绪的会说话的头视频是人机互动的重要挑战。然而，情感是具有模棱两可的界限的高度抽象的概念，因此需要散布的表达参数才能产生情感表达的说话头视频。在这项工作中，我们提出情报台，以通过语义表达参数综合说话的头视频。为了预测任意音频输入的表达参数，我们应用一个可以通过情感标签指定的音频表达模块。该模块旨在增强各种情绪中音频输入的相关性。此外，我们通过沿垂直方向进行探测来利用预训练的超平面来完善面部运动。最后，精致的表达参数使神经辐射场定向，并促进了情感上的会说话的头部视频。实验结果表明，语义表达参数会导致更好的重建质量和可控性。

Title: SparseGS-W: Sparse-View 3D Gaussian Splatting in the Wild with Generative Priors

• Authors: Yiqing Li, Xuan Wang, Jiawei Wu, Yikun Ma, Zhi Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19452
• Pdf URL: https://arxiv.org/pdf/2503.19452
• Copy Paste: [[2503.19452]] SparseGS-W: Sparse-View 3D Gaussian Splatting in the Wild with Generative Priors(https://arxiv.org/abs/2503.19452)
• Keywords: generative
• Abstract: Synthesizing novel views of large-scale scenes from unconstrained in-the-wild images is an important but challenging task in computer vision. Existing methods, which optimize per-image appearance and transient occlusion through implicit neural networks from dense training views (approximately 1000 images), struggle to perform effectively under sparse input conditions, resulting in noticeable artifacts. To this end, we propose SparseGS-W, a novel framework based on 3D Gaussian Splatting that enables the reconstruction of complex outdoor scenes and handles occlusions and appearance changes with as few as five training images. We leverage geometric priors and constrained diffusion priors to compensate for the lack of multi-view information from extremely sparse input. Specifically, we propose a plug-and-play Constrained Novel-View Enhancement module to iteratively improve the quality of rendered novel views during the Gaussian optimization process. Furthermore, we propose an Occlusion Handling module, which flexibly removes occlusions utilizing the inherent high-quality inpainting capability of constrained diffusion priors. Both modules are capable of extracting appearance features from any user-provided reference image, enabling flexible modeling of illumination-consistent scenes. Extensive experiments on the PhotoTourism and Tanks and Temples datasets demonstrate that SparseGS-W achieves state-of-the-art performance not only in full-reference metrics, but also in commonly used non-reference metrics such as FID, ClipIQA, and MUSIQ.
• 摘要：从不受限制的野外图像中综合大规模场景的新型观点是计算机视觉中的重要但具有挑战性的任务。现有方法，通过从密集的训练视图（约1000张图像）中的隐式神经网络优化人均外观和瞬态闭塞，难以在稀疏输入条件下有效执行，从而导致明显的人工制品。为此，我们提出了基于3D高斯碎片的新型框架，这是一个新型框架，可以重建复杂的室外场景和手柄的遮挡和外观，而外观变化只有五个训练图像。我们利用几何先验和受限的扩散先验来补偿来自极稀疏输入的多视图信息。具体而言，我们建议在高斯优化过程中迭代地提高渲染的新型视图质量，提出一个插件限制的新型视图增强模块。此外，我们提出了一个遮挡处理模块，该模块通过受约束扩散先验的固有高质量的高质量填充能力灵活地消除了阻塞。这两个模块均能够从任何用户提供的参考图像中提取外观特征，从而可以灵活地建模照明场景。关于光载和储罐和寺庙数据集的广泛实验表明，Sparsegs-W不仅在全参考指标中实现了最先进的性能，而且在常用的非参考指标（例如FID，Clipiqa和Musiq）中也实现了最先进的性能。

Title: Data-centric Federated Graph Learning with Large Language Models

• Authors: Bo Yan, Zhongjian Zhang, Huabin Sun, Mengmei Zhang, Yang Cao, Chuan Shi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.19455
• Pdf URL: https://arxiv.org/pdf/2503.19455
• Copy Paste: [[2503.19455]] Data-centric Federated Graph Learning with Large Language Models(https://arxiv.org/abs/2503.19455)
• Keywords: generation
• Abstract: In federated graph learning (FGL), a complete graph is divided into multiple subgraphs stored in each client due to privacy concerns, and all clients jointly train a global graph model by only transmitting model parameters. A pain point of FGL is the heterogeneity problem, where nodes or structures present non-IID properties among clients (e.g., different node label distributions), dramatically undermining the convergence and performance of FGL. To address this, existing efforts focus on design strategies at the model level, i.e., they design models to extract common knowledge to mitigate heterogeneity. However, these model-level strategies fail to fundamentally address the heterogeneity problem as the model needs to be designed from scratch when transferring to other tasks. Motivated by large language models (LLMs) having achieved remarkable success, we aim to utilize LLMs to fully understand and augment local text-attributed graphs, to address data heterogeneity at the data level. In this paper, we propose a general framework LLM4FGL that innovatively decomposes the task of LLM for FGL into two sub-tasks theoretically. Specifically, for each client, it first utilizes the LLM to generate missing neighbors and then infers connections between generated nodes and raw nodes. To improve the quality of generated nodes, we design a novel federated generation-and-reflection mechanism for LLMs, without the need to modify the parameters of the LLM but relying solely on the collective feedback from all clients. After neighbor generation, all the clients utilize a pre-trained edge predictor to infer the missing edges. Furthermore, our framework can seamlessly integrate as a plug-in with existing FGL methods. Experiments on three real-world datasets demonstrate the superiority of our method compared to advanced baselines.
• 摘要：在联合图学习（FGL）中，由于隐私问题，完整的图被分为每个客户端存储的多个子图，所有客户都仅通过传输模型参数来共同训练全局图模型。 FGL的疼痛点是异质性问题，其中节点或结构在客户之间存在非IID特性（例如不同的节点标签分布），从而极大地破坏了FGL的收敛性和性能。为了解决这一问题，现有的努力集中在模型级别上的设计策略上，即他们设计模型以提取常识以减轻异质性。但是，这些模型级策略无法从根本上解决异质性问题，因为在转移到其他任务时需要从头开始设计模型。由大型语言模型（LLMS）取得了显着成功的动机，我们旨在利用LLM充分理解和增强本地文本属性图，以解决数据级别的数据异质性。在本文中，我们提出了一个通用框架LLM4FGL，该框架在理论上创新地分解了FGL的LLM任务为两个子任务。具体来说，对于每个客户，它首先使用LLM生成缺失的邻居，然后进一步在生成的节点和原始节点之间进行连接。为了提高生成节点的质量，我们为LLM设计了一种新颖的联合生成和反思机制，而无需修改LLM的参数，而仅依靠所有客户的集体反馈。邻居生成后，所有客户端都使用预训练的边缘预测变量来推断缺失的边缘。此外，我们的框架可以与现有的FGL方法无缝集成为插件。在三个现实世界数据集上进行的实验证明了我们方法的优势与高级基准相比。

Title: G-DexGrasp: Generalizable Dexterous Grasping Synthesis Via Part-Aware Prior Retrieval and Prior-Assisted Generation

• Authors: Juntao Jian, Xiuping Liu, Zixuan Chen, Manyi Li, Jian Liu, Ruizhen Hu
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2503.19457
• Pdf URL: https://arxiv.org/pdf/2503.19457
• Copy Paste: [[2503.19457]] G-DexGrasp: Generalizable Dexterous Grasping Synthesis Via Part-Aware Prior Retrieval and Prior-Assisted Generation(https://arxiv.org/abs/2503.19457)
• Keywords: generation, generative
• Abstract: Recent advances in dexterous grasping synthesis have demonstrated significant progress in producing reasonable and plausible grasps for many task purposes. But it remains challenging to generalize to unseen object categories and diverse task instructions. In this paper, we propose G-DexGrasp, a retrieval-augmented generation approach that can produce high-quality dexterous hand configurations for unseen object categories and language-based task instructions. The key is to retrieve generalizable grasping priors, including the fine-grained contact part and the affordance-related distribution of relevant grasping instances, for the following synthesis pipeline. Specifically, the fine-grained contact part and affordance act as generalizable guidance to infer reasonable grasping configurations for unseen objects with a generative model, while the relevant grasping distribution plays as regularization to guarantee the plausibility of synthesized grasps during the subsequent refinement optimization. Our comparison experiments validate the effectiveness of our key designs for generalization and demonstrate the remarkable performance against the existing approaches. Project page: this https URL
• 摘要：敏感抓地合成的最新进展表明，出于许多任务目的，在产生合理和合理的掌握方面取得了重大进展。但是，概括到看不见的对象类别和多样化的任务说明仍然是一项挑战。在本文中，我们提出了G-DexGrasp，这是一种检索效果的生成方法，可以为看不见的对象类别和基于语言的任务说明产生高质量的灵活手动配置。关键是要在以下合成管道中检索可概括的先验，包括细粒度的接触部件和相关握把实例的与负担相关的分布。具体而言，细颗粒的接触部分和负担能力作为可推广的指导，以通过生成模型来推断对看不见的对象的合理握把配置，而相关的握把分布则作为正则化，以确保在随后的精炼优化过程中合成的GRASPS的合理性。我们的比较实验验证了我们的关键设计对概括的有效性，并证明了针对现有方法的显着性能。项目页面：此HTTPS URL

Title: AccVideo: Accelerating Video Diffusion Model with Synthetic Dataset

• Authors: Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, Yu Qiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19462
• Pdf URL: https://arxiv.org/pdf/2503.19462
• Copy Paste: [[2503.19462]] AccVideo: Accelerating Video Diffusion Model with Synthetic Dataset(https://arxiv.org/abs/2503.19462)
• Keywords: generation
• Abstract: Diffusion models have achieved remarkable progress in the field of video generation. However, their iterative denoising nature requires a large number of inference steps to generate a video, which is slow and computationally expensive. In this paper, we begin with a detailed analysis of the challenges present in existing diffusion distillation methods and propose a novel efficient method, namely AccVideo, to reduce the inference steps for accelerating video diffusion models with synthetic dataset. We leverage the pretrained video diffusion model to generate multiple valid denoising trajectories as our synthetic dataset, which eliminates the use of useless data points during distillation. Based on the synthetic dataset, we design a trajectory-based few-step guidance that utilizes key data points from the denoising trajectories to learn the noise-to-video mapping, enabling video generation in fewer steps. Furthermore, since the synthetic dataset captures the data distribution at each diffusion timestep, we introduce an adversarial training strategy to align the output distribution of the student model with that of our synthetic dataset, thereby enhancing the video quality. Extensive experiments demonstrate that our model achieves 8.5x improvements in generation speed compared to the teacher model while maintaining comparable performance. Compared to previous accelerating methods, our approach is capable of generating videos with higher quality and resolution, i.e., 5-seconds, 720x1280, 24fps.
• 摘要：扩散模型在视频生成领域取得了显着进步。但是，它们的迭代性化性质需要大量的推理步骤来生成一个较慢且计算昂贵的视频。在本文中，我们从对现有扩散蒸馏方法中存在的挑战的详细分析开始，并提出了一种新颖的有效方法，即AccVideo，以减少使用合成数据集加速视频扩散模型的推理步骤。我们利用验证的视频扩散模型作为合成数据集生成多个有效的脱氧轨迹，从而消除了蒸馏过程中无用的数据点的使用。基于合成数据集，我们设计了一个基于轨迹的几步指导，该指南利用来自DeNoising轨迹的关键数据点来学习噪声到视频映射，从而使视频生成更少。此外，由于合成数据集捕获了每个扩散时间末的数据分布，因此我们引入了一种对抗性培训策略，以使学生模型的输出分布与我们的合成数据集的输出分布相结合，从而提高了视频质量。广泛的实验表明，与教师模型相比，我们的模型在保持可比的性能的同时，生成速度的增长8.5倍。与以前的加速方法相比，我们的方法能够生成具有更高质量和分辨率的视频，即5秒，720x1280，24fps。

Title: Noisier2Inverse: Self-Supervised Learning for Image Reconstruction with Correlated Noise

• Authors: Nadja Gruber, Johannes Schwab, Markus Haltmeier, Ander Biguri, Clemens Dlaska, Gyeongha Hwang
• Subjects: cs.CV, eess.IV, math.OC
• Abstract URL: https://arxiv.org/abs/2503.19468
• Pdf URL: https://arxiv.org/pdf/2503.19468
• Copy Paste: [[2503.19468]] Noisier2Inverse: Self-Supervised Learning for Image Reconstruction with Correlated Noise(https://arxiv.org/abs/2503.19468)
• Keywords: generation
• Abstract: We propose Noisier2Inverse, a correction-free self-supervised deep learning approach for general inverse prob- lems. The proposed method learns a reconstruction function without the need for ground truth samples and is ap- plicable in cases where measurement noise is statistically correlated. This includes computed tomography, where detector imperfections or photon scattering create correlated noise patterns, as well as microscopy and seismic imaging, where physical interactions during measurement introduce dependencies in the noise structure. Similar to Noisier2Noise, a key step in our approach is the generation of noisier data from which the reconstruction net- work learns. However, unlike Noisier2Noise, the proposed loss function operates in measurement space and is trained to recover an extrapolated image instead of the original noisy one. This eliminates the need for an extrap- olation step during inference, which would otherwise suffer from ill-posedness. We numerically demonstrate that our method clearly outperforms previous self-supervised approaches that account for correlated noise.
• 摘要：我们提出了Noisier2inverse，这是一种无校正的自我监督的深度学习方法，用于一般反向概率。所提出的方法在不需要地面真相样本的情况下学习重建函数，并且在统计上相关的测量噪声相关的情况下是可观的。这包括计算机断层扫描，其中检测器缺陷或光子散射会产生相关的噪声模式，以及显微镜和地震成像，其中测量过程中的物理相互作用会在噪声结构中引入依赖性。与noisier2noise类似，我们方法的关键步骤是生成嘈杂的数据，重建网络从中学习。但是，与Noisier2noise不同，所提出的损失函数在测量空间中运行，并经过训练以恢复外推图像而不是原始噪声。这消除了推断期间对推断步骤的需求，否则这将遭受不良状态。我们从数值上证明，我们的方法清楚地表现出了以前的自我监督的方法，这些方法解释了相关的噪声。

Title: GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancers

• Authors: Shijie Ma, Yuying Ge, Teng Wang, Yuxin Guo, Yixiao Ge, Ying Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19480
• Pdf URL: https://arxiv.org/pdf/2503.19480
• Copy Paste: [[2503.19480]] GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancers(https://arxiv.org/abs/2503.19480)
• Keywords: generation, generative
• Abstract: The synergy between generative and discriminative models receives growing attention. While discriminative Contrastive Language-Image Pre-Training (CLIP) excels in high-level semantics, it struggles with perceiving fine-grained visual details. Generally, to enhance representations, generative models take CLIP's visual features as conditions for reconstruction. However, the underlying principle remains underexplored. In this work, we empirically found that visually perfect generations are not always optimal for representation enhancement. The essence lies in effectively extracting fine-grained knowledge from generative models while mitigating irrelevant information. To explore critical factors, we delve into three aspects: (1) Conditioning mechanisms: We found that even a small number of local tokens can drastically reduce the difficulty of reconstruction, leading to collapsed training. We thus conclude that utilizing only global visual tokens as conditions is the most effective strategy. (2) Denoising configurations: We observed that end-to-end training introduces extraneous information. To address this, we propose a two-stage training strategy to prioritize learning useful visual knowledge. Additionally, we demonstrate that lightweight denoisers can yield remarkable improvements. (3) Generation paradigms: We explore both continuous and discrete denoisers with desirable outcomes, validating the versatility of our method. Through our in-depth explorations, we have finally arrived at an effective method, namely GenHancer, which consistently outperforms prior arts on the MMVP-VLM benchmark, e.g., 6.0% on OpenAICLIP. The enhanced CLIP can be further plugged into multimodal large language models for better vision-centric performance. All the models and codes are made publicly available.
• 摘要：生成和判别模型之间的协同作用受到了越来越多的关注。虽然歧视性的对比语言图像预训练（剪辑）在高级语义上表现出色，但它在感知细粒度的视觉细节方面挣扎。通常，为了增强表示形式，生成模型将Clip的视觉特征作为重建条件。但是，基本原理仍然没有被忽略。在这项工作中，我们从经验上发现，视觉上完美的一代并不总是最佳的表示表示。本质在于有效地从生成模型中提取细颗粒知识，同时减轻无关信息。为了探索关键因素，我们深入研究了三个方面：（1）调节机制：我们发现，即使是少数局部令牌也可以大大减少重建的困难，从而导致训练崩溃。因此，我们得出的结论是，仅利用全球视觉令牌作为条件是最有效的策略。 （2）降级配置：我们观察到端到端培训引入了无关的信息。为了解决这个问题，我们提出了一个两阶段的培训策略，以优先学习有用的视觉知识。此外，我们证明了轻巧的Denoisers可以产生显着的改进。 （3）生成范式：我们探索具有理想结果的连续和离散的Denoisers，从而验证了我们方法的多功能性。通过我们的深入探索，我们终于获得了一种有效的方法，即Genhancer，该方法始终超过MMVP-VLM基准上的先前艺术，例如OpenAiClip的6.0％。增强的剪辑可以进一步插入多模式的大语言模型，以更好地以视力为中心。所有模型和代码均可公开使用。

Title: Exploring Disentangled and Controllable Human Image Synthesis: From End-to-End to Stage-by-Stage

• Authors: Zhengwentai Sun, Heyuan Li, Xihe Yang, Keru Zheng, Shuliang Ning, Yihao Zhi, Hongjie Liao, Chenghong Li, Shuguang Cui, Xiaoguang Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19486
• Pdf URL: https://arxiv.org/pdf/2503.19486
• Copy Paste: [[2503.19486]] Exploring Disentangled and Controllable Human Image Synthesis: From End-to-End to Stage-by-Stage(https://arxiv.org/abs/2503.19486)
• Keywords: generation, generative
• Abstract: Achieving fine-grained controllability in human image synthesis is a long-standing challenge in computer vision. Existing methods primarily focus on either facial synthesis or near-frontal body generation, with limited ability to simultaneously control key factors such as viewpoint, pose, clothing, and identity in a disentangled manner. In this paper, we introduce a new disentangled and controllable human synthesis task, which explicitly separates and manipulates these four factors within a unified framework. We first develop an end-to-end generative model trained on MVHumanNet for factor disentanglement. However, the domain gap between MVHumanNet and in-the-wild data produce unsatisfacotry results, motivating the exploration of virtual try-on (VTON) dataset as a potential solution. Through experiments, we observe that simply incorporating the VTON dataset as additional data to train the end-to-end model degrades performance, primarily due to the inconsistency in data forms between the two datasets, which disrupts the disentanglement process. To better leverage both datasets, we propose a stage-by-stage framework that decomposes human image generation into three sequential steps: clothed A-pose generation, back-view synthesis, and pose and view control. This structured pipeline enables better dataset utilization at different stages, significantly improving controllability and generalization, especially for in-the-wild scenarios. Extensive experiments demonstrate that our stage-by-stage approach outperforms end-to-end models in both visual fidelity and disentanglement quality, offering a scalable solution for real-world tasks. Additional demos are available on the project page: this https URL.
• 摘要：在人类图像合成中实现细粒度的可控性是计算机视觉中的长期挑战。现有的方法主要集中于面部合成或近额外的身体产生，其能力有限，无法以分离的方式控制关键因素，例如观点，姿势，衣服和身份。在本文中，我们介绍了一个新的分离和可控的人类综合任务，该任务明确分开并操纵了统一框架内的这四个因素。我们首先开发了一种在MVHumannet培训的端到端生成模型，以进行因素分解。但是，MVHumannet和野外数据之间的域间隙产生了不抗浓度的结果，激发了将虚拟试验（VTON）数据集作为潜在解决方案的探索。通过实验，我们观察到，仅将VTON数据集合并为训练端到端模型的附加数据，这主要是由于两个数据集之间的数据形式不一致，这破坏了分解过程。为了更好地利用两个数据集，我们提出了一个阶段框架，将人类图像生成分解为三个顺序的步骤：穿衣服的A置式生成，背景合成以及姿势和视图控制。该结构化管道可以在不同阶段进行更好的数据集利用，从而显着改善了可控性和概括，尤其是在野外场景中。广泛的实验表明，我们的阶段方法在视觉保真度和分离质量上都优于端到端模型，为现实世界任务提供了可扩展的解决方案。在项目页面上提供其他演示：此HTTPS URL。

Title: VectorFit : Adaptive Singular & Bias Vector Fine-Tuning of Pre-trained Foundation Models

• Authors: Suhas G Hegde, Shilpy Kaur, Aruna Tiwari
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.19530
• Pdf URL: https://arxiv.org/pdf/2503.19530
• Copy Paste: [[2503.19530]] VectorFit : Adaptive Singular & Bias Vector Fine-Tuning of Pre-trained Foundation Models(https://arxiv.org/abs/2503.19530)
• Keywords: generation
• Abstract: Popular PEFT methods achieve parameter efficiency by assuming that incremental weight updates are inherently low-rank, which often leads to a performance gap compared to full fine-tuning. While recent methods have attempted to address this limitation, they typically lack sufficient parameter and memory efficiency. We propose VectorFit, an effective and easily deployable approach that adaptively trains the singular vectors and biases of pre-trained weight matrices. We demonstrate that the utilization of structural and transformational characteristics of pre-trained weights enables high-rank updates comparable to those of full fine-tuning. As a result, VectorFit achieves superior performance with 9X less trainable parameters compared to state-of-the-art PEFT methods. Through extensive experiments over 17 datasets spanning diverse language and vision tasks such as natural language understanding and generation, question answering, image classification, and image generation, we exhibit that VectorFit consistently outperforms baselines, even in extremely low-budget scenarios.
• 摘要：流行的PEFT方法通过假设增量重量更新本质上是低级别来达到参数效率，这通常会导致性能差距与完整的微调相比。尽管最近的方法试图解决此限制，但它们通常缺乏足够的参数和记忆效率。我们提出了一种矢量法，这是一种有效且易于部署的方法，可适应训练预训练的重量矩阵的奇异向量和偏见。我们证明，预训练重量的结构和变革性特征的利用使高级更新可与完整的微调相媲美。结果，与最先进的PEFT方法相比，VectorFit的训练参数低9倍。通过17个数据集的广泛实验，涵盖了各种语言和视觉任务，例如自然语言理解和产生，问题答案，图像分类和图像产生，我们展示了矢量fit的表现始终超过基本线，即使在极低的低估场景中也是如此。

Title: Dance Like a Chicken: Low-Rank Stylization for Human Motion Diffusion

• Authors: Haim Sawdayee, Chuan Guo, Guy Tevet, Bing Zhou, Jian Wang, Amit H. Bermano
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19557
• Pdf URL: https://arxiv.org/pdf/2503.19557
• Copy Paste: [[2503.19557]] Dance Like a Chicken: Low-Rank Stylization for Human Motion Diffusion(https://arxiv.org/abs/2503.19557)
• Keywords: generation, generative
• Abstract: Text-to-motion generative models span a wide range of 3D human actions but struggle with nuanced stylistic attributes such as a "Chicken" style. Due to the scarcity of style-specific data, existing approaches pull the generative prior towards a reference style, which often results in out-of-distribution low quality generations. In this work, we introduce LoRA-MDM, a lightweight framework for motion stylization that generalizes to complex actions while maintaining editability. Our key insight is that adapting the generative prior to include the style, while preserving its overall distribution, is more effective than modifying each individual motion during generation. Building on this idea, LoRA-MDM learns to adapt the prior to include the reference style using only a few samples. The style can then be used in the context of different textual prompts for generation. The low-rank adaptation shifts the motion manifold in a semantically meaningful way, enabling realistic style infusion even for actions not present in the reference samples. Moreover, preserving the distribution structure enables advanced operations such as style blending and motion editing. We compare LoRA-MDM to state-of-the-art stylized motion generation methods and demonstrate a favorable balance between text fidelity and style consistency.
• 摘要：文本到动作生成模型涵盖了3D人类动作的广泛范围，但与“鸡”风格等细微的风格属性斗争。由于特定于样式的数据的稀缺性，现有方法将生成剂带入参考样式，这通常会导致分布量的低质量一代。在这项工作中，我们介绍了Lora-MDM，这是一个轻巧的运动风格框架，在维护编辑性的同时将其推广到复杂的动作。我们的关键见解是，在包含样式之前调整生成剂，同时保留其整体分布，比在发电过程中修改每个单独运动更有效。在这个想法的基础上，Lora-MDM学会了在仅使用几个样本中包含参考样式之前调整。然后，该样式可以在不同的文本提示的上下文中使用。低级别的适应性以语义上有意义的方式改变了运动歧管，即使参考样本中不存在的动作，也可以实现逼真的风格输液。此外，保留分配结构可以实现高级操作，例如样式混合和运动编辑。我们将Lora-MDM与最先进的程式化运动生成方法进行了比较，并在文本保真度和样式一致性之间表现出了良好的平衡。

Title: Improved tissue sodium concentration quantification in breast cancer by reducing partial volume effects: a preliminary study

• Authors: Olgica Zaric, Carmen Leser, Vladimir Juras, Alex Farr, Malina Gologan, Stanislas Rapacchi, Laura Villazan Garcia, Christian Singer, Siegfried Trattnig, Christian Licht, Ramona Woitek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19570
• Pdf URL: https://arxiv.org/pdf/2503.19570
• Copy Paste: [[2503.19570]] Improved tissue sodium concentration quantification in breast cancer by reducing partial volume effects: a preliminary study(https://arxiv.org/abs/2503.19570)
• Keywords: quality assessment
• Abstract: Introduction: In sodium (23Na) MRI, partial volume effects (PVE) are one of the most common causes of errors in the quantification of tissue sodium concentration (TSC) in vivo. Advanced image reconstruction algorithms, such as compressed sensing (CS), have been shown to potentially reduce PVE. Therefore, we investigated the feasibility of CS-based methods for image quality and TSC quantification accuracy improvement in patients with breast cancer (BC). Subjects and Methods: Three healthy participants and 12 female participants with BC were examined on a 7T MRI scanner in this study. We reconstructed 23Na-MRI images using the weighted total variation (wTV) and directional total variation (dTV), anatomically guided total variation (AG-TV), and adaptive combine (ADC) reconstruction and performed image quality assessment. We evaluated agreement in tumor volumes delineated on sodium data using the Dice score and performed TSC quantification for different image reconstruction approaches. Results: All methods provided sodium images of the breast with good quality. The mean Dice scores for wTV, dTV, and AG-TV were 65%, 72%, and 75%, respectively. In the breast tumors, average TSC values were 83.0, 72.0, 80.0, and 84.0 mmol/L, respectively. There was a significant difference between dTV and wTV (p<0.001), as well as between dTV and AG-TV (p<0.001) and dTV and ADC algorithm (p<0.001). Conclusion: The results of this study showed that there are differences in tumor appearance and TSC estimations that might be depending on the type of image reconstruction and parameters used, most likely due to differences in their robustness in reducing PVE.
• 摘要：简介：在钠（23NA）MRI中，部分体积效应（PVE）是体内钠浓度（TSC）定量误差的最常见原因之一。已经显示出高级图像重建算法（例如压缩传感（CS））有可能减少PVE。因此，我们研究了基于CS的方法对乳腺癌患者（BC）的图像质量和TSC定量精度提高的可行性。受试者和方法：在本研究中，在7T MRI扫描仪上检查了三名健康参与者和12名BC女性参与者。我们使用加权总变异（WTV）和定向总变异（DTV），解剖学上引导的总变异（AG-TV）以及自适应联合收割机（ADC）重建和执行图像质量评估来重建23NA-MRI图像。我们评估了使用骰子分数在钠数据上描述的肿瘤体积的一致性，并对不同的图像重建方法进行了TSC定量。结果：所有方法提供了质量良好的乳房钠图像。 WTV，DTV和AG-TV的平均骰子得分分别为65％，72％和75％。在乳腺肿瘤中，平均TSC值分别为83.0、72.0、80.0和84.0 mmol/l。 DTV和WTV之间存在显着差异（P <0.001），DTV和AG-TV（P <0.001）和DTV和ADC算法之间存在显着差异（P <0.001）。结论：这项研究的结果表明，肿瘤外观和TSC估计存在差异，这些估计可能取决于图像重建和所使用的参数的类型，这很可能是由于它们在减少PVE的鲁棒性差异所致。

Title: Optimizing Language Models for Inference Time Objectives using Reinforcement Learning

• Authors: Yunhao Tang, Kunhao Zheng, Gabriel Synnaeve, Rémi Munos
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2503.19595
• Pdf URL: https://arxiv.org/pdf/2503.19595
• Copy Paste: [[2503.19595]] Optimizing Language Models for Inference Time Objectives using Reinforcement Learning(https://arxiv.org/abs/2503.19595)
• Keywords: generation
• Abstract: In this work, we investigate the merits of explicitly optimizing for inference time algorithmic performance during model training. We show how optimizing for inference time performance can improve overall model efficacy. We consider generic inference time objectives with $k$ samples, with a focus on pass@$k$ and majority voting as two main applications. With language model training on reasoning datasets, we showcase the performance trade-off enabled by training with such objectives. When training on code generation tasks, we show that the approach significantly improves pass@$k$ objectives compared to the baseline method.
• 摘要：在这项工作中，我们研究了模型培训期间明确优化推理时间算法性能的优点。我们展示了推理时间性能的优化如何提高整体模型功效。我们认为使用$ K $样本的通用推理时间目标，重点关注Pass@$ K $，多数投票作为两个主要应用程序。通过有关推理数据集的语言模型培训，我们通过此类目标来展示通过培训来启用的性能权衡。在代码生成任务进行培训时，我们表明该方法与基线方法相比显着改善了PASS@$ K $目标。

Title: RL-finetuning LLMs from on- and off-policy data with a single algorithm

• Authors: Yunhao Tang, Taco Cohen, David W. Zhang, Michal Valko, Rémi Munos
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2503.19612
• Pdf URL: https://arxiv.org/pdf/2503.19612
• Copy Paste: [[2503.19612]] RL-finetuning LLMs from on- and off-policy data with a single algorithm(https://arxiv.org/abs/2503.19612)
• Keywords: generation
• Abstract: We introduce a novel reinforcement learning algorithm (AGRO, for Any-Generation Reward Optimization) for fine-tuning large-language models. AGRO leverages the concept of generation consistency, which states that the optimal policy satisfies the notion of consistency across any possible generation of the model. We derive algorithms that find optimal solutions via the sample-based policy gradient and provide theoretical guarantees on their convergence. Our experiments demonstrate the effectiveness of AGRO in both on-policy and off-policy settings, showing improved performance on the mathematical reasoning dataset over baseline algorithms.
• 摘要：我们引入了一种新颖的增强学习算法（用于微调大型模型的农业，用于任何产生奖励优化）。 Agro利用了发电一致性的概念，该概念指出，最佳政策满足了任何可能产生的模型的一致性概念。我们得出通过基于样本的策略梯度找到最佳解决方案的算法，并为其收敛提供理论保证。我们的实验证明了Agro在跨政策和非政策环境中的有效性，显示了基线算法上数学推理数据集的性能的提高。

Title: Burst Image Super-Resolution with Mamba

• Authors: Ozan Unal, Steven Marty, Dengxin Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19634
• Pdf URL: https://arxiv.org/pdf/2503.19634
• Copy Paste: [[2503.19634]] Burst Image Super-Resolution with Mamba(https://arxiv.org/abs/2503.19634)
• Keywords: super-resolution
• Abstract: Burst image super-resolution (BISR) aims to enhance the resolution of a keyframe by leveraging information from multiple low-resolution images captured in quick succession. In the deep learning era, BISR methods have evolved from fully convolutional networks to transformer-based architectures, which, despite their effectiveness, suffer from the quadratic complexity of self-attention. We see Mamba as the next natural step in the evolution of this field, offering a comparable global receptive field and selective information routing with only linear time complexity. In this work, we introduce BurstMamba, a Mamba-based architecture for BISR. Our approach decouples the task into two specialized branches: a spatial module for keyframe super-resolution and a temporal module for subpixel prior extraction, striking a balance between computational efficiency and burst information integration. To further enhance burst processing with Mamba, we propose two novel strategies: (i) optical flow-based serialization, which aligns burst sequences only during state updates to preserve subpixel details, and (ii) a wavelet-based reparameterization of the state-space update rules, prioritizing high-frequency features for improved burst-to-keyframe information passing. Our framework achieves SOTA performance on public benchmarks of SyntheticSR, RealBSR-RGB, and RealBSR-RAW.
• 摘要：爆发图像超分辨率（BISR）旨在通过利用快速连续捕获的多个低分辨率图像的信息来增强密钥帧的分辨率。在深度学习时代，BISR方法已经从完全卷积的网络演变为基于变压器的架构，尽管它们有效，但它们却遭受了自我注意的二次复杂性。我们将Mamba视为该领域演变的下一个自然步骤，仅具有线性时间复杂性提供了可比的全局接受场和选择性信息路由。在这项工作中，我们介绍了BISR的基于Mamba的建筑Bustmamba。我们的方法将任务分解为两个专用分支：一个用于钥匙帧超分辨率的空间模块和一个用于子像素的时间模块，以提取之前提取，在计算效率和爆发信息集成之间达到平衡。为了进一步增强与Mamba的突发处理，我们提出了两种新型策略：（i）基于光流的序列化，仅在状态更新期间与爆发序列保持一致，以保留子像素细节，以及（ii）基于小波的更新规则的基于小波的重新聚集规则，优先考虑爆发到爆发的高频特征，以改进爆发到爆发的爆发信息。我们的框架在Syntheticsr，RealBSR-RGB和RealBSR-RAW的公共基准上实现了SOTA性能。

Title: CoSimGen: Controllable Diffusion Model for Simultaneous Image and Mask Generation

• Authors: Rupak Bose, Chinedu Innocent Nwoye, Aditya Bhat, Nicolas Padoy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19661
• Pdf URL: https://arxiv.org/pdf/2503.19661
• Copy Paste: [[2503.19661]] CoSimGen: Controllable Diffusion Model for Simultaneous Image and Mask Generation(https://arxiv.org/abs/2503.19661)
• Keywords: generation, generative
• Abstract: The acquisition of annotated datasets with paired images and segmentation masks is a critical challenge in domains such as medical imaging, remote sensing, and computer vision. Manual annotation demands significant resources, faces ethical constraints, and depends heavily on domain expertise. Existing generative models often target single-modality outputs, either images or segmentation masks, failing to address the need for high-quality, simultaneous image-mask generation. Additionally, these models frequently lack adaptable conditioning mechanisms, restricting control over the generated outputs and limiting their applicability for dataset augmentation and rare scenario simulation. We propose CoSimGen, a diffusion-based framework for controllable simultaneous image and mask generation. Conditioning is intuitively achieved through (1) text prompts grounded in class semantics, (2) spatial embedding of context prompts to provide spatial coherence, and (3) spectral embedding of timestep information to model noise levels during diffusion. To enhance controllability and training efficiency, the framework incorporates contrastive triplet loss between text and class embeddings, alongside diffusion and adversarial losses. Initial low-resolution outputs 128 x 128 are super-resolved to 512 x 512, producing high-fidelity images and masks with strict adherence to conditions. We evaluate CoSimGen on metrics such as FID, KID, LPIPS, Class FID, Positive predicted value for image fidelity and semantic alignment of generated samples over 4 diverse datasets. CoSimGen achieves state-of-the-art performance across all datasets, achieving the lowest KID of 0.11 and LPIPS of 0.53 across datasets.
• 摘要：在医学成像，遥感和计算机视觉等领域中，使用配对图像和分割面罩的注释数据集获取是一个至关重要的挑战。手动注释需要大量资源，面临道德限制，并在很大程度上取决于领域的专业知识。现有的生成模型通常针对单模式输出，无论是图像还是分割掩码，都无法满足对高质量，同时的图像掩码生成的需求。此外，这些模型经常缺乏适应性的调节机制，限制了对生成的输出的控制，并限制了其对数据集扩展和稀有场景模拟的适用性。我们提出了Cosimgen，这是一个基于扩散的框架，用于可控的同时图像和掩模生成。条件是通过（1）基于类语义基础的文本提示，（2）上下文提示提示提供空间连贯性的空间嵌入，以及（3）在扩散期间，时间到时的光谱嵌入到模型噪声水平。为了提高可控性和训练效率，该框架结合了文本和类嵌入之间的对比三胞胎损失，以及扩散和对抗性损失。最初的低分辨率输出128 x 128被超级分辨至512 x 512，从而产生了严格遵守条件的高保真图像和掩模。我们评估COSIMGEN，例如FID，KID，LPIPS，类FID，图像保真度的积极预测值以及在4个不同数据集上产生的样品的语义比对。 Cosimgen在所有数据集中都达到了最先进的性能，在数据集中达到了0.11的最低孩子，LPIPS为0.53。

Title: High-Quality Spatial Reconstruction and Orthoimage Generation Using Efficient 2D Gaussian Splatting

• Authors: Qian Wang, Zhihao Zhan, Jialei He, Zhituo Tu, Xiang Zhu, Jie Yuan
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2503.19703
• Pdf URL: https://arxiv.org/pdf/2503.19703
• Copy Paste: [[2503.19703]] High-Quality Spatial Reconstruction and Orthoimage Generation Using Efficient 2D Gaussian Splatting(https://arxiv.org/abs/2503.19703)
• Keywords: generation
• Abstract: Highly accurate geometric precision and dense image features characterize True Digital Orthophoto Maps (TDOMs), which are in great demand for applications such as urban planning, infrastructure management, and environmental monitoring. Traditional TDOM generation methods need sophisticated processes, such as Digital Surface Models (DSM) and occlusion detection, which are computationally expensive and prone to errors. This work presents an alternative technique rooted in 2D Gaussian Splatting (2DGS), free of explicit DSM and occlusion detection. With depth map generation, spatial information for every pixel within the TDOM is retrieved and can reconstruct the scene with high precision. Divide-and-conquer strategy achieves excellent GS training and rendering with high-resolution TDOMs at a lower resource cost, which preserves higher quality of rendering on complex terrain and thin structure without a decrease in efficiency. Experimental results demonstrate the efficiency of large-scale scene reconstruction and high-precision terrain modeling. This approach provides accurate spatial data, which assists users in better planning and decision-making based on maps.
• 摘要：高度准确的几何精度和密集的图像特征表征了真正的数字矫正图（TDORS），这些图（TDORS）对应用程序，例如城市规划，基础架构管理和环境监控的需求非常巨大。传统的TDOR生成方法需要复杂的过程，例如数字表面模型（DSM）和遮挡检测，这些过程在计算上昂贵且容易出现错误。这项工作提出了一种植根于2D高斯裂（2DG）的替代技术，没有明确的DSM和遮挡检测。通过深度图生成，检索了TDOR中每个像素的空间信息，并可以高精度地重建场景。分裂策略以较低的资源成本以高分辨率的TDOD进行了出色的GS培训和渲染，从而在复杂的地形和薄结构上保留了较高的渲染质量，而效率降低了。实验结果证明了大型场景重建和高精度地形建模的效率。这种方法提供了准确的空间数据，该数据可帮助用户根据地图进行更好的计划和决策制定。

Title: CamSAM2: Segment Anything Accurately in Camouflaged Videos

• Authors: Yuli Zhou, Guolei Sun, Yawei Li, Yuqian Fu, Luca Benini, Ender Konukoglu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.19730
• Pdf URL: https://arxiv.org/pdf/2503.19730
• Copy Paste: [[2503.19730]] CamSAM2: Segment Anything Accurately in Camouflaged Videos(https://arxiv.org/abs/2503.19730)
• Keywords: generation
• Abstract: Video camouflaged object segmentation (VCOS), aiming at segmenting camouflaged objects that seamlessly blend into their environment, is a fundamental vision task with various real-world applications. With the release of SAM2, video segmentation has witnessed significant progress. However, SAM2's capability of segmenting camouflaged videos is suboptimal, especially when given simple prompts such as point and box. To address the problem, we propose Camouflaged SAM2 (CamSAM2), which enhances SAM2's ability to handle camouflaged scenes without modifying SAM2's parameters. Specifically, we introduce a decamouflaged token to provide the flexibility of feature adjustment for VCOS. To make full use of fine-grained and high-resolution features from the current frame and previous frames, we propose implicit object-aware fusion (IOF) and explicit object-aware fusion (EOF) modules, respectively. Object prototype generation (OPG) is introduced to abstract and memorize object prototypes with informative details using high-quality features from previous frames. Extensive experiments are conducted to validate the effectiveness of our approach. While CamSAM2 only adds negligible learnable parameters to SAM2, it substantially outperforms SAM2 on three VCOS datasets, especially achieving 12.2 mDice gains with click prompt on MoCA-Mask and 19.6 mDice gains with mask prompt on SUN-SEG-Hard, with Hiera-T as the backbone. The code will be available at \href{this https URL}{this http URL}.
• 摘要：视频伪装的对象细分（VCO）旨在将无缝融合到其环境中的伪装对象进行分割，这是具有各种真实世界应用程序的基本视觉任务。随着SAM2的发布，视频细分目睹了重大进展。但是，SAM2分割伪装视频的能力是次优的，尤其是当给出的简单提示（例如点和框）时。为了解决该问题，我们提出了伪装的SAM2（CAMSAM2），从而增强了SAM2在不修改SAM2参数的情况下处理伪装场景的能力。具体来说，我们引入了一个decamofflef的令牌，以提供VCO的功能调整的灵活性。为了完全利用当前帧和先前帧的细粒度和高分辨率特征，我们分别提出了隐式对象感知融合（IOF）和显式对象感知融合（EOF）模块。将对象原型生成（OPG）引入抽象和记住对象原型，并使用先前框架的高质量功能提供信息的详细信息。进行了广泛的实验以验证我们方法的有效性。虽然CAMSAM2仅在SAM2中添加可忽略不计的可学习参数，但它在三个VCOS数据集上的表现基本上都优于SAM2，尤其是在MOCA蒙版上单击即可获得12.2 MDICE的增长，并在Moca Mask上获得了19.6 MDICE的增长，并在Sun-Seg-Hard上使用蒙版提示，并带有Hiera-T，并带有Hiera-t，如hiera-t一样。该代码将在\ href {此https url} {this HTTP url}上可用。

Title: PCM : Picard Consistency Model for Fast Parallel Sampling of Diffusion Models

• Authors: Junhyuk So, Jiwoong Shin, Chaeyeon Jang, Eunhyeok Park
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19731
• Pdf URL: https://arxiv.org/pdf/2503.19731
• Copy Paste: [[2503.19731]] PCM : Picard Consistency Model for Fast Parallel Sampling of Diffusion Models(https://arxiv.org/abs/2503.19731)
• Keywords: generation
• Abstract: Recently, diffusion models have achieved significant advances in vision, text, and robotics. However, they still face slow generation speeds due to sequential denoising processes. To address this, a parallel sampling method based on Picard iteration was introduced, effectively reducing sequential steps while ensuring exact convergence to the original output. Nonetheless, Picard iteration does not guarantee faster convergence, which can still result in slow generation in practice. In this work, we propose a new parallelization scheme, the Picard Consistency Model (PCM), which significantly reduces the number of generation steps in Picard iteration. Inspired by the consistency model, PCM is directly trained to predict the fixed-point solution, or the final output, at any stage of the convergence trajectory. Additionally, we introduce a new concept called model switching, which addresses PCM's limitations and ensures exact convergence. Extensive experiments demonstrate that PCM achieves up to a 2.71x speedup over sequential sampling and a 1.77x speedup over Picard iteration across various tasks, including image generation and robotic control.
• 摘要：最近，扩散模型在视觉，文本和机器人技术方面取得了重大进步。但是，由于顺序降解过程，它们仍然面临缓慢的生成速度。为了解决这个问题，引入了基于PICARD迭代的并行抽样方法，有效地降低了顺序步骤，同时确保与原始输出的确切收敛。尽管如此，Picard迭代并不能保证更快的收敛性，这在实践中仍然会导致缓慢的产生。在这项工作中，我们提出了一种新的并行化方案，即PICARD一致性模型（PCM），该模型大大减少了PICARD迭代中的生成步骤数量。受一致性模型的启发，PCM经过直接训练以预测收敛轨迹的任何阶段的定点解决方案或最终输出。此外，我们引入了一个称为模型切换的新概念，该概念解决了PCM的局限性并确保确切的收敛。广泛的实验表明，PCM在顺序采样上达到了2.71倍的速度，并且在各种任务（包括图像产生和机器人控制）上对PICARD迭代进行了1.77倍的速度。

Title: ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation

• Authors: Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, Xiang Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19755
• Pdf URL: https://arxiv.org/pdf/2503.19755
• Copy Paste: [[2503.19755]] ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation(https://arxiv.org/abs/2503.19755)
• Keywords: generation, generative
• Abstract: End-to-end (E2E) autonomous driving methods still struggle to make correct decisions in interactive closed-loop evaluation due to limited causal reasoning capability. Current methods attempt to leverage the powerful understanding and reasoning abilities of Vision-Language Models (VLMs) to resolve this dilemma. However, the problem is still open that few VLMs for E2E methods perform well in the closed-loop evaluation due to the gap between the semantic reasoning space and the purely numerical trajectory output in the action space. To tackle this issue, we propose ORION, a holistic E2E autonomous driving framework by vision-language instructed action generation. ORION uniquely combines a QT-Former to aggregate long-term history context, a Large Language Model (LLM) for driving scenario reasoning, and a generative planner for precision trajectory prediction. ORION further aligns the reasoning space and the action space to implement a unified E2E optimization for both visual question-answering (VQA) and planning tasks. Our method achieves an impressive closed-loop performance of 77.74 Driving Score (DS) and 54.62% Success Rate (SR) on the challenge Bench2Drive datasets, which outperforms state-of-the-art (SOTA) methods by a large margin of 14.28 DS and 19.61% SR.
• 摘要：由于因果推理能力有限，端到端（E2E）自主驾驶方法仍在努力在交互式闭环评估中做出正确的决定。当前的方法试图利用视觉模型（VLM）的强大理解和推理能力来解决这一困境。但是，由于语义推理空间和动作空间中纯粹的数值轨迹输出之间的差距，因此，在闭环评估中，几乎没有E2E方法的VLM在闭环评估中表现良好。为了解决这个问题，我们提出了Orion，这是通过视觉语言指示的动作生成的整体E2E自动驾驶框架。 Orion独特地结合了QT形式，以汇总长期历史上下文，用于驱动方案推理的大语言模型（LLM）以及用于精确轨迹预测的生成计划者。 Orion进一步使推理空间和行动空间保持一致，以实现为视觉提问（VQA）和计划任务实施统一的E2E优化。我们的方法在Challenge Bench2Drive数据集上实现了令人印象深刻的闭环性能（DS）和54.62％的成功率（SR），该数据集以14.28 DS和19.61％的SR优于最先进的方法（SOTA）方法。

Title: Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models

• Authors: Kartik Thakral, Tamar Glaser, Tal Hassner, Mayank Vatsa, Richa Singh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19783
• Pdf URL: https://arxiv.org/pdf/2503.19783
• Copy Paste: [[2503.19783]] Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models(https://arxiv.org/abs/2503.19783)
• Keywords: generative
• Abstract: Existing unlearning algorithms in text-to-image generative models often fail to preserve the knowledge of semantically related concepts when removing specific target concepts: a challenge known as adjacency. To address this, we propose FADE (Fine grained Attenuation for Diffusion Erasure), introducing adjacency aware unlearning in diffusion models. FADE comprises two components: (1) the Concept Neighborhood, which identifies an adjacency set of related concepts, and (2) Mesh Modules, employing a structured combination of Expungement, Adjacency, and Guidance loss components. These enable precise erasure of target concepts while preserving fidelity across related and unrelated concepts. Evaluated on datasets like Stanford Dogs, Oxford Flowers, CUB, I2P, Imagenette, and ImageNet1k, FADE effectively removes target concepts with minimal impact on correlated concepts, achieving atleast a 12% improvement in retention performance over state-of-the-art methods.
• 摘要：在删除特定目标概念时，文本到图像生成模型中现有的未学习算法通常无法保留语义相关概念的知识：一种称为邻接的挑战。为了解决这个问题，我们提出了淡出的淡出（扩散擦除的细粒度衰减），在扩散模型中引入了邻接的意识到的学习。淡出包括两个组成部分：（1）概念邻域，该概念邻域识别相关概念的邻接集，以及（2）网格模块，采用了结构化摄入，邻接和指导损失成分的结构组合。这些可以精确擦除目标概念，同时保留跨相关和无关概念的忠诚度。在斯坦福犬，牛津花，幼崽，I2p，Imagenette和Imagenet1k等数据集上进行了评估，有效地消除了目标概念，对相关概念的影响最小，从而使保留效果至少提高了12％的概念。

Title: SITA: Structurally Imperceptible and Transferable Adversarial Attacks for Stylized Image Generation

• Authors: Jingdan Kang, Haoxin Yang, Yan Cai, Huaidong Zhang, Xuemiao Xu, Yong Du, Shengfeng He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19791
• Pdf URL: https://arxiv.org/pdf/2503.19791
• Copy Paste: [[2503.19791]] SITA: Structurally Imperceptible and Transferable Adversarial Attacks for Stylized Image Generation(https://arxiv.org/abs/2503.19791)
• Keywords: generation
• Abstract: Image generation technology has brought significant advancements across various fields but has also raised concerns about data misuse and potential rights infringements, particularly with respect to creating visual artworks. Current methods aimed at safeguarding artworks often employ adversarial attacks. However, these methods face challenges such as poor transferability, high computational costs, and the introduction of noticeable noise, which compromises the aesthetic quality of the original artwork. To address these limitations, we propose a Structurally Imperceptible and Transferable Adversarial (SITA) attacks. SITA leverages a CLIP-based destylization loss, which decouples and disrupts the robust style representation of the image. This disruption hinders style extraction during stylized image generation, thereby impairing the overall stylization process. Importantly, SITA eliminates the need for a surrogate diffusion model, leading to significantly reduced computational overhead. The method's robust style feature disruption ensures high transferability across diverse models. Moreover, SITA introduces perturbations by embedding noise within the imperceptible structural details of the image. This approach effectively protects against style extraction without compromising the visual quality of the artwork. Extensive experiments demonstrate that SITA offers superior protection for artworks against unauthorized use in stylized generation. It significantly outperforms existing methods in terms of transferability, computational efficiency, and noise imperceptibility. Code is available at this https URL.
• 摘要：图像生成技术在各个领域都带来了重大进步，但也引起了人们对数据滥用和潜在权利侵犯的关注，尤其是在创建视觉艺术品方面。旨在保护艺术品的当前方法通常采用对抗性攻击。但是，这些方法面临着挑战，例如可传递性差，高计算成本以及引入明显的噪音，这损害了原始艺术品的美学质量。为了解决这些局限性，我们提出了一个在结构上不可察觉和可转移的对抗（SITA）攻击。 SITA利用基于夹子的命运损失，将图像的稳健样式表示分解并破坏了图像的强大样式表示。这种破坏会阻碍样式化图像生成期间的样式提取，从而损害了整体风格化过程。重要的是，SITA消除了对替代扩散模型的需求，从而大大减少了计算开销。该方法的强大风格功能破坏可确保各种模型之间的高传递性。此外，SITA通过将噪声嵌入图像的不可感知的结构细节中，从而引入扰动。这种方法有效地保护了风格提取，而不会损害艺术品的视觉质量。广泛的实验表明，SITA为艺术品提供了优越的保护，以防止未经授权使用在程式化的一代中。就可传递性，计算效率和噪声不可识别而言，它极大地优于现有方法。代码可在此HTTPS URL上找到。

Title: In the Blink of an Eye: Instant Game Map Editing using a Generative-AI Smart Brush

• Authors: Vitaly Gnatyuk, Valeriia Koriukina Ilya Levoshevich, Pavel Nurminskiy, Guenter Wallner
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19793
• Pdf URL: https://arxiv.org/pdf/2503.19793
• Copy Paste: [[2503.19793]] In the Blink of an Eye: Instant Game Map Editing using a Generative-AI Smart Brush(https://arxiv.org/abs/2503.19793)
• Keywords: generation, generative
• Abstract: With video games steadily increasing in complexity, automated generation of game content has found widespread interest. However, the task of 3D gaming map art creation remains underexplored to date due to its unique complexity and domain-specific challenges. While recent works have addressed related topics such as retro-style level generation and procedural terrain creation, these works primarily focus on simpler data distributions. To the best of our knowledge, we are the first to demonstrate the application of modern AI techniques for high-resolution texture manipulation in complex, highly detailed AAA 3D game environments. We introduce a novel Smart Brush for map editing, designed to assist artists in seamlessly modifying selected areas of a game map with minimal effort. By leveraging generative adversarial networks and diffusion models we propose two variants of the brush that enable efficient and context-aware generation. Our hybrid workflow aims to enhance both artistic flexibility and production efficiency, enabling the refinement of environments without manually reworking every detail, thus helping to bridge the gap between automation and creative control in game development. A comparative evaluation of our two methods with adapted versions of several state-of-the art models shows that our GAN-based brush produces the sharpest and most detailed outputs while preserving image context while the evaluated state-of-the-art models tend towards blurrier results and exhibit difficulties in maintaining contextual consistency.
• 摘要：随着视频游戏的复杂性稳步增长，自动生成游戏内容引起了广泛的兴趣。但是，由于其独特的复杂性和特定于领域的挑战，3D游戏地图艺术创作的任务仍然尚未得到充实。尽管最近的作品已经解决了相关的主题，例如复古级别的生成和程序地形创建，但这些作品主要集中于更简单的数据分布。据我们所知，我们是第一个证明现代AI技术在复杂，高度详细的AAA 3D游戏环境中应用高分辨率纹理操作的应用。我们介绍了一种用于地图编辑的新颖智能刷，旨在帮助艺术家以最小的努力无缝修改游戏地图的选定区域。通过利用生成的对抗网络和扩散模型，我们提出了刷子的两个变体，以实现有效和上下文感知的生成。我们的混合工作流程旨在提高艺术灵活性和生产效率，从而在不手动重新加工所有细节的情况下对环境进行完善，从而有助于弥合游戏开发中自动化和创造性控制之间的差距。对几种最先进模型的适应版本的两种方法的比较评估表明，我们的基于GAN的刷子会产生最清晰，最详细的输出，同时保留图像上下文，而评估的最先进模型则倾向于更明显，并且在维持上下文一致性方面遇到困难。

Title: Unpaired Object-Level SAR-to-Optical Image Translation for Aircraft with Keypoints-Guided Diffusion Models

• Authors: Ruixi You, Hecheng Jia, Feng Xu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2503.19798
• Pdf URL: https://arxiv.org/pdf/2503.19798
• Copy Paste: [[2503.19798]] Unpaired Object-Level SAR-to-Optical Image Translation for Aircraft with Keypoints-Guided Diffusion Models(https://arxiv.org/abs/2503.19798)
• Keywords: generation
• Abstract: Synthetic Aperture Radar (SAR) imagery provides all-weather, all-day, and high-resolution imaging capabilities but its unique imaging mechanism makes interpretation heavily reliant on expert knowledge, limiting interpretability, especially in complex target tasks. Translating SAR images into optical images is a promising solution to enhance interpretation and support downstream tasks. Most existing research focuses on scene-level translation, with limited work on object-level translation due to the scarcity of paired data and the challenge of accurately preserving contour and texture details. To address these issues, this study proposes a keypoint-guided diffusion model (KeypointDiff) for SAR-to-optical image translation of unpaired aircraft targets. This framework introduces supervision on target class and azimuth angle via keypoints, along with a training strategy for unpaired data. Based on the classifier-free guidance diffusion architecture, a class-angle guidance module (CAGM) is designed to integrate class and angle information into the diffusion generation process. Furthermore, adversarial loss and consistency loss are employed to improve image fidelity and detail quality, tailored for aircraft targets. During sampling, aided by a pre-trained keypoint detector, the model eliminates the requirement for manually labeled class and azimuth information, enabling automated SAR-to-optical translation. Experimental results demonstrate that the proposed method outperforms existing approaches across multiple metrics, providing an efficient and effective solution for object-level SAR-to-optical translation and downstream tasks. Moreover, the method exhibits strong zero-shot generalization to untrained aircraft types with the assistance of the keypoint detector.
• 摘要：合成孔径雷达（SAR）成像提供全天候，全天和高分辨率成像功能，但其独特的成像机制使解释非常依赖专家知识，限制了解释性，尤其是在复杂的目标任务中。将SAR图像转换为光学图像是增强解释和支持下游任务的有前途解决方案。大多数现有的研究都集中在场景级翻译上，由于配对数据的稀缺性以及准确保留轮廓和纹理细节的挑战，对对象级翻译的工作有限。为了解决这些问题，本研究提出了一个关键引导的扩散模型（KeyPointDiff），以用于未配对飞机目标的SAR到光学图像翻译。该框架通过关键点对目标类别和方位角进行监督，以及针对未配对数据的培训策略。基于无分类器引导扩散体系结构，类角度指导模块（CAGM）旨在将类和角度信息整合到扩散生成过程中。此外，针对飞机目标量身定制的对抗性损失和一致性损失是为了提高图像保真度和细节质量。在采样期间，在预先训练的关键点检测器的帮助下，该模型消除了对手动标记的类和方位角信息的要求，从而实现了自动化的SAR到光学翻译。实验结果表明，所提出的方法的表现优于多个指标的现有方法，为对象级别的SAR到光学翻译和下游任务提供了有效的解决方案。此外，该方法在按键检测器的帮助下对未经训练的飞机类型表现出强烈的零弹性概括。

Title: Domain-incremental White Blood Cell Classification with Privacy-aware Continual Learning

• Authors: Pratibha Kumari, Afshin Bozorgpour, Daniel Reisenbüchler, Edgar Jost, Martina Crysandt, Christian Matek, Dorit Merhof
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19819
• Pdf URL: https://arxiv.org/pdf/2503.19819
• Copy Paste: [[2503.19819]] Domain-incremental White Blood Cell Classification with Privacy-aware Continual Learning(https://arxiv.org/abs/2503.19819)
• Keywords: generative
• Abstract: White blood cell (WBC) classification plays a vital role in hematology for diagnosing various medical conditions. However, it faces significant challenges due to domain shifts caused by variations in sample sources (e.g., blood or bone marrow) and differing imaging conditions across hospitals. Traditional deep learning models often suffer from catastrophic forgetting in such dynamic environments, while foundation models, though generally robust, experience performance degradation when the distribution of inference data differs from that of the training data. To address these challenges, we propose a generative replay-based Continual Learning (CL) strategy designed to prevent forgetting in foundation models for WBC classification. Our method employs lightweight generators to mimic past data with a synthetic latent representation to enable privacy-preserving replay. To showcase the effectiveness, we carry out extensive experiments with a total of four datasets with different task ordering and four backbone models including ResNet50, RetCCL, CTransPath, and UNI. Experimental results demonstrate that conventional fine-tuning methods degrade performance on previously learned tasks and struggle with domain shifts. In contrast, our continual learning strategy effectively mitigates catastrophic forgetting, preserving model performance across varying domains. This work presents a practical solution for maintaining reliable WBC classification in real-world clinical settings, where data distributions frequently evolve.
• 摘要：白血细胞（WBC）分类在诊断各种疾病的血液学中起着至关重要的作用。但是，由于样本源（例如，血液或骨髓）的变化以及医院之间不同的成像条件而导致的域变化，它面临着重大挑战。传统的深度学习模型通常会遭受这种动态环境中灾难性遗忘的困扰，而当推断数据的分布与培训数据的分布不同时，基础模型虽然通常强大，但经验强劲，但经历了性能下降。为了应对这些挑战，我们提出了一种基于生成重播的持续学习（CL）策略，旨在防止在WBC分类基础模型中忘记。我们的方法采用轻量级发电机来模仿过去的数据，并具有合成的潜在表示，以实现隐私权重播。为了展示有效性，我们进行了广泛的实验，共有四个具有不同任务订购的数据集和四个骨干模型，包括Resnet50，RetCCL，Ctranspath和Uni。实验结果表明，常规的微调方法会在先前学习的任务上降低性能并与域转移斗争。相比之下，我们的持续学习策略有效地减轻了灾难性的遗忘，从而保留了各个领域的模型性能。这项工作提出了一种实用解决方案，用于在现实世界中维持可靠的WBC分类，其中数据分布经常发展。

Title: Mask$^2$DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long Video Generation

• Authors: Tianhao Qi, Jianlong Yuan, Wanquan Feng, Shancheng Fang, Jiawei Liu, SiYu Zhou, Qian He, Hongtao Xie, Yongdong Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19881
• Pdf URL: https://arxiv.org/pdf/2503.19881
• Copy Paste: [[2503.19881]] Mask$^2$DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long Video Generation(https://arxiv.org/abs/2503.19881)
• Keywords: generation
• Abstract: Sora has unveiled the immense potential of the Diffusion Transformer (DiT) architecture in single-scene video generation. However, the more challenging task of multi-scene video generation, which offers broader applications, remains relatively underexplored. To bridge this gap, we propose Mask$^2$DiT, a novel approach that establishes fine-grained, one-to-one alignment between video segments and their corresponding text annotations. Specifically, we introduce a symmetric binary mask at each attention layer within the DiT architecture, ensuring that each text annotation applies exclusively to its respective video segment while preserving temporal coherence across visual tokens. This attention mechanism enables precise segment-level textual-to-visual alignment, allowing the DiT architecture to effectively handle video generation tasks with a fixed number of scenes. To further equip the DiT architecture with the ability to generate additional scenes based on existing ones, we incorporate a segment-level conditional mask, which conditions each newly generated segment on the preceding video segments, thereby enabling auto-regressive scene extension. Both qualitative and quantitative experiments confirm that Mask$^2$DiT excels in maintaining visual consistency across segments while ensuring semantic alignment between each segment and its corresponding text description. Our project page is this https URL.
• 摘要：索拉（Sora）在单场景视频生成中推出了扩散变压器（DIT）架构的巨大潜力。但是，提供更广泛应用程序的多场景视频生成的更具挑战性的任务仍然相对不受影响。为了弥合这一差距，我们提出了面具$^2 $ DIT，这是一种新颖的方法，可以在视频片段及其相应的文本注释之间建立细粒度，一对一的对齐。具体来说，我们在DIT体系结构中的每个注意力层上引入了对称二进制掩码，以确保每个文本注释仅适用于其各自的视频段，同时保留跨视觉令牌的时间连贯性。这种注意机制可以使精确的细分级文本到视觉对齐，从而使DIT体系结构能够有效地处理具有固定场景的视频生成任务。为了进一步为DIT体系结构提供基于现有场景的其他场景的能力，我们结合了一个细分级别的条件掩码，该掩模条件在前一个视频片段上的每个新生成的细分市场都有条件，从而实现自动回归场景扩展。定性和定量实验都证实，蒙版$^2 $ dit在保持各个段的视觉一致性方面表现出色，同时确保每个段之间的语义对齐及其相应的文本描述。我们的项目页面是此HTTPS URL。

Title: Scaling Down Text Encoders of Text-to-Image Diffusion Models

• Authors: Lifu Wang, Daqing Liu, Xinchen Liu, Xiaodong He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19897
• Pdf URL: https://arxiv.org/pdf/2503.19897
• Copy Paste: [[2503.19897]] Scaling Down Text Encoders of Text-to-Image Diffusion Models(https://arxiv.org/abs/2503.19897)
• Keywords: generation
• Abstract: Text encoders in diffusion models have rapidly evolved, transitioning from CLIP to T5-XXL. Although this evolution has significantly enhanced the models' ability to understand complex prompts and generate text, it also leads to a substantial increase in the number of parameters. Despite T5 series encoders being trained on the C4 natural language corpus, which includes a significant amount of non-visual data, diffusion models with T5 encoder do not respond to those non-visual prompts, indicating redundancy in representational power. Therefore, it raises an important question: "Do we really need such a large text encoder?" In pursuit of an answer, we employ vision-based knowledge distillation to train a series of T5 encoder models. To fully inherit its capabilities, we constructed our dataset based on three criteria: image quality, semantic understanding, and text-rendering. Our results demonstrate the scaling down pattern that the distilled T5-base model can generate images of comparable quality to those produced by T5-XXL, while being 50 times smaller in size. This reduction in model size significantly lowers the GPU requirements for running state-of-the-art models such as FLUX and SD3, making high-quality text-to-image generation more accessible.
• 摘要：扩散模型中的文本编码迅速发展，从夹子到T5-XXL过渡。尽管这种演变显着增强了模型理解复杂提示和生成文本的能力，但它也导致参数数量大幅增加。尽管T5系列编码器接受了C4自然语言语料库的培训，其中包括大量非视觉数据，但带有T5编码器的扩散模型并未响应那些非视觉提示，表明代表权的冗余。因此，它提出了一个重要的问题：“我们真的需要如此大的文本编码器吗？”为了追求答案，我们采用基于视觉的知识蒸馏来培训一系列T5编码模型。为了完全继承其功能，我们根据三个标准构建了数据集：图像质量，语义理解和文本渲染。我们的结果表明，蒸馏T5碱模型可以生成与T5-XXL生成的图像相当的图像，而大小小的50倍，蒸馏型T5基碱模型可以生成可比质量的图像。型号的降低显着降低了运行最先进模型（例如Flux和SD3）的GPU要求，从而使高质量的文本对图像生成更容易访问。

Title: CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning

• Authors: Hao Yu, Zhuokai Zhao, Shen Yan, Lukasz Korycki, Jianyu Wang, Baosheng He, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hanchao Yu
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2503.19900
• Pdf URL: https://arxiv.org/pdf/2503.19900
• Copy Paste: [[2503.19900]] CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning(https://arxiv.org/abs/2503.19900)
• Keywords: generation, generative
• Abstract: The rapid advancement of large vision-language models (LVLMs) has driven significant progress in multimodal tasks, enabling models to interpret, reason, and generate outputs across both visual and textual domains. While excelling in generative tasks, existing LVLMs often face limitations in tasks requiring high-fidelity representation learning, such as generating image or text embeddings for retrieval. Recent work has proposed finetuning LVLMs for representational learning, but the fine-tuned model often loses its generative capabilities due to the representational learning training paradigm. To address this trade-off, we introduce CAFe, a contrastive-autoregressive fine-tuning framework that enhances LVLMs for both representation and generative tasks. By integrating a contrastive objective with autoregressive language modeling, our approach unifies these traditionally separate tasks, achieving state-of-the-art results in both multimodal retrieval and multimodal generative benchmarks, including object hallucination (OH) mitigation. CAFe establishes a novel framework that synergizes embedding and generative functionalities in a single model, setting a foundation for future multimodal models that excel in both retrieval precision and coherent output generation.
• 摘要：大型视觉模型（LVLM）的快速发展已驱动了多模式任务的重大进展，使模型能够解释，推理和在视觉和文本域中生成输出。尽管在生成任务方面出色，但现有的LVLM经常面临需要高保真表示学习的任务的局限性，例如生成图像或文本嵌入以进行检索。最近的工作提出了用于代表性学习的Finetuntuntuntuntuntuntun，但是由于代表性学习培训范式，微调模型经常失去其生成能力。为了解决这一权衡，我们引入了CAFE，这是一个对比的自我回调的微调框架，可增强代表和生成任务的LVLM。通过将对比度目标与自回归语言建模相结合，我们的方法统一了这些传统上单独的任务，从而实现最先进的任务会导致多模式检索和多模式生成基准测试，包括对象幻觉（OH）缓解。咖啡馆建立了一个新颖的框架，该框架在单个模型中协同嵌入和生成功能，为未来的多模型模型奠定了基础，这些模型在检索精度和相干输出生成中都表现出色。

Title: ICE: Intrinsic Concept Extraction from a Single Image via Diffusion Models

• Authors: Fernando Julio Cendra, Kai Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19902
• Pdf URL: https://arxiv.org/pdf/2503.19902
• Copy Paste: [[2503.19902]] ICE: Intrinsic Concept Extraction from a Single Image via Diffusion Models(https://arxiv.org/abs/2503.19902)
• Keywords: generative
• Abstract: The inherent ambiguity in defining visual concepts poses significant challenges for modern generative models, such as the diffusion-based Text-to-Image (T2I) models, in accurately learning concepts from a single image. Existing methods lack a systematic way to reliably extract the interpretable underlying intrinsic concepts. To address this challenge, we present ICE, short for Intrinsic Concept Extraction, a novel framework that exclusively utilizes a T2I model to automatically and systematically extract intrinsic concepts from a single image. ICE consists of two pivotal stages. In the first stage, ICE devises an automatic concept localization module to pinpoint relevant text-based concepts and their corresponding masks within the image. This critical stage streamlines concept initialization and provides precise guidance for subsequent analysis. The second stage delves deeper into each identified mask, decomposing the object-level concepts into intrinsic concepts and general concepts. This decomposition allows for a more granular and interpretable breakdown of visual elements. Our framework demonstrates superior performance on intrinsic concept extraction from a single image in an unsupervised manner. Project page: this https URL
• 摘要：定义视觉概念的固有歧义为现代生成模型带来了重大挑战，例如基于扩散的文本对图像（T2I）模型，可以从单个图像中准确地学习概念。现有方法缺乏一种系统的方法来可靠地提取可解释的基本固有概念。为了应对这一挑战，我们提出了冰，这是内在概念提取的缩写，这是一个新颖的框架，该框架专门利用T2i模型自动并系统地从单个图像中提取固有的概念。冰由两个关键阶段组成。在第一阶段，ICE设计了一个自动概念本地化模块，以查明相关的基于文本的概念及其在图像中的相应掩码。这个关键阶段简化了概念初始化，并为后续分析提供了精确的指导。第二阶段深入研究了每个已确定的面具，将对象级概念分解为内在的概念和一般概念。这种分解允许对视觉元素的更详细和可解释的分解。我们的框架以无监督的方式从单个图像中提取了固有概念上的卓越性能。项目页面：此HTTPS URL

Title: AvatarArtist: Open-Domain 4D Avatarization

• Authors: Hongyu Liu, Xuan Wang, Ziyu Wan, Yue Ma, Jingye Chen, Yanbo Fan, Yujun Shen, Yibing Song, Qifeng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19906
• Pdf URL: https://arxiv.org/pdf/2503.19906
• Copy Paste: [[2503.19906]] AvatarArtist: Open-Domain 4D Avatarization(https://arxiv.org/abs/2503.19906)
• Keywords: generative
• Abstract: This work focuses on open-domain 4D avatarization, with the purpose of creating a 4D avatar from a portrait image in an arbitrary style. We select parametric triplanes as the intermediate 4D representation and propose a practical training paradigm that takes advantage of both generative adversarial networks (GANs) and diffusion models. Our design stems from the observation that 4D GANs excel at bridging images and triplanes without supervision yet usually face challenges in handling diverse data distributions. A robust 2D diffusion prior emerges as the solution, assisting the GAN in transferring its expertise across various domains. The synergy between these experts permits the construction of a multi-domain image-triplane dataset, which drives the development of a general 4D avatar creator. Extensive experiments suggest that our model, AvatarArtist, is capable of producing high-quality 4D avatars with strong robustness to various source image domains. The code, the data, and the models will be made publicly available to facilitate future studies..
• 摘要：这项工作着重于开放域4D Avatarization，目的是用任意风格的肖像图像创建4D化身。我们选择参数三链型作为中间4D表示，并提出了一种实用的训练范式，该范式既利用生成的对抗网络（GAN）和扩散模型。我们的设计源于这样的观察，即4D GAN在没有监督的情况下毫无疑问地桥接图像和三型，但通常在处理各种数据分布方面面临挑战。强大的2D扩散先验作为解决方案出现，协助GAN将其专业知识转移到各个领域。这些专家之间的协同作用允许构建多域图像triplane数据集，该数据集驱动了一般的4D阿凡达创作者的开发。广泛的实验表明，我们的模型AVATARARTIST能够生成具有强大鲁棒性的高质量4D化身。代码，数据和模型将公开使用以促进未来的研究。

Title: FullDiT: Multi-Task Video Generative Foundation Model with Full Attention

• Authors: Xuan Ju, Weicai Ye, Quande Liu, Qiulin Wang, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, Qiang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.19907
• Pdf URL: https://arxiv.org/pdf/2503.19907
• Copy Paste: [[2503.19907]] FullDiT: Multi-Task Video Generative Foundation Model with Full Attention(https://arxiv.org/abs/2503.19907)
• Keywords: generation, generative
• Abstract: Current video generative foundation models primarily focus on text-to-video tasks, providing limited control for fine-grained video content creation. Although adapter-based approaches (e.g., ControlNet) enable additional controls with minimal fine-tuning, they encounter challenges when integrating multiple conditions, including: branch conflicts between independently trained adapters, parameter redundancy leading to increased computational cost, and suboptimal performance compared to full fine-tuning. To address these challenges, we introduce FullDiT, a unified foundation model for video generation that seamlessly integrates multiple conditions via unified full-attention mechanisms. By fusing multi-task conditions into a unified sequence representation and leveraging the long-context learning ability of full self-attention to capture condition dynamics, FullDiT reduces parameter overhead, avoids conditions conflict, and shows scalability and emergent ability. We further introduce FullBench for multi-task video generation evaluation. Experiments demonstrate that FullDiT achieves state-of-the-art results, highlighting the efficacy of full-attention in complex multi-task video generation.
• 摘要：当前的视频生成基础模型主要集中于文本到视频任务，为创建细粒度的视频内容提供了有限的控制。尽管基于适配器的方法（例如ControlNET）能够以最少的微调进行其他控制，但在整合多种条件时，它们会遇到挑战，包括：独立训练的适配器之间的分支冲突，参数冗余，导致计算成本增加，与完整的微调相比，计算成本增加和次优绩效。为了应对这些挑战，我们介绍了Fulldit，这是视频生成的统一基础模型，该模型通过统一的全注意机制无缝整合了多种条件。通过将多任务条件融合到统一的序列表示中，并利用完全自我注意力捕获条件动态的长篇文化学习能力，fulldit降低了参数开销，避免了条件冲突，并显示出可扩展性和出现的能力。我们进一步介绍了全任务视频生成评估。实验表明，Fulldit取得了最新的结果，突出了复杂的多任务视频生成中全注意的功效。