2025-02-13

Title: Movie Weaver: Tuning-Free Multi-Concept Video Personalization with Anchored Prompts

• Authors: Feng Liang, Haoyu Ma, Zecheng He, Tingbo Hou, Ji Hou, Kunpeng Li, Xiaoliang Dai, Felix Juefei-Xu, Samaneh Azadi, Animesh Sinha, Peizhao Zhang, Peter Vajda, Diana Marculescu
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.07802
• Pdf URL: https://arxiv.org/pdf/2502.07802
• Copy Paste: [[2502.07802]] Movie Weaver: Tuning-Free Multi-Concept Video Personalization with Anchored Prompts(https://arxiv.org/abs/2502.07802)
• Keywords: generation
• Abstract: Video personalization, which generates customized videos using reference images, has gained significant attention. However, prior methods typically focus on single-concept personalization, limiting broader applications that require multi-concept integration. Attempts to extend these models to multiple concepts often lead to identity blending, which results in composite characters with fused attributes from multiple sources. This challenge arises due to the lack of a mechanism to link each concept with its specific reference image. We address this with anchored prompts, which embed image anchors as unique tokens within text prompts, guiding accurate referencing during generation. Additionally, we introduce concept embeddings to encode the order of reference images. Our approach, Movie Weaver, seamlessly weaves multiple concepts-including face, body, and animal images-into one video, allowing flexible combinations in a single model. The evaluation shows that Movie Weaver outperforms existing methods for multi-concept video personalization in identity preservation and overall quality.
• 摘要：视频个性化使用参考图像生成定制视频，引起了广泛关注。然而，之前的方法通常侧重于单概念个性化，限制了需要多概念集成的更广泛应用。尝试将这些模型扩展到多个概念通常会导致身份混合，从而产生具有来自多个来源的融合属性的复合角色。这一挑战的出现是因为缺乏将每个概念与其特定参考图像链接起来的机制。我们使用锚定提示来解决这个问题，它将图像锚点作为唯一标记嵌入文本提示中，在生成过程中指导准确引用。此外，我们引入了概念嵌入来编码参考图像的顺序。我们的方法 Movie Weaver 将多个概念（包括面部、身体和动物图像）无缝编织到一个视频中，允许在单个模型中进行灵活组合。评估表明，Movie Weaver 在身份保存和整体质量方面优于现有的多概念视频个性化方法。

Title: Pre-Trained Video Generative Models as World Simulators

• Authors: Haoran He, Yang Zhang, Liang Lin, Zhongwen Xu, Ling Pan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.07825
• Pdf URL: https://arxiv.org/pdf/2502.07825
• Copy Paste: [[2502.07825]] Pre-Trained Video Generative Models as World Simulators(https://arxiv.org/abs/2502.07825)
• Keywords: generative
• Abstract: Video generative models pre-trained on large-scale internet datasets have achieved remarkable success, excelling at producing realistic synthetic videos. However, they often generate clips based on static prompts (e.g., text or images), limiting their ability to model interactive and dynamic scenarios. In this paper, we propose Dynamic World Simulation (DWS), a novel approach to transform pre-trained video generative models into controllable world simulators capable of executing specified action trajectories. To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module that seamlessly integrates into any existing model. Instead of focusing on complex visual details, we demonstrate that consistent dynamic transition modeling is the key to building powerful world simulators. Building upon this insight, we further introduce a motion-reinforced loss that enhances action controllability by compelling the model to capture dynamic changes more effectively. Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models, achieving significant improvements in generating action-controllable, dynamically consistent videos across games and robotics domains. Moreover, to facilitate the applications of the learned world simulator in downstream tasks such as model-based reinforcement learning, we propose prioritized imagination to improve sample efficiency, demonstrating competitive performance compared with state-of-the-art methods.
• 摘要：在大型互联网数据集上预训练的视频生成模型取得了显著的成功，擅长制作逼真的合成视频。然而，它们通常基于静态提示（例如文本或图像）生成剪辑，这限制了它们对交互式和动态场景进行建模的能力。在本文中，我们提出了动态世界模拟 (DWS)，这是一种将预训练的视频生成模型转换为可控制的世界模拟器的新方法，能够执行指定的动作轨迹。为了实现条件动作和生成的视觉变化之间的精确对齐，我们引入了一个轻量级的通用动作条件模块，可以无缝集成到任何现有模型中。我们证明了一致的动态转换建模是构建强大的世界模拟器的关键，而不是专注于复杂的视觉细节。基于这一见解，我们进一步引入了运动强化损失，通过强制模型更有效地捕捉动态变化来增强动作的可控性。实验表明，DWS 可以灵活地应用于扩散和自回归变换器模型，在游戏和机器人领域生成动作可控、动态一致的视频方面取得了显着的改进。此外，为了促进学习世界模拟器在基于模型的强化学习等下游任务中的应用，我们提出了优先想象来提高样本效率，与最先进的方法相比表现出竞争力。

Title: Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing

• Authors: Sihao Wu, Xiaonan Si, Chi Xing, Jianhong Wang, Gaojie Jin, Guangliang Cheng, Lijun Zhang, Xiaowei Huang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.07829
• Pdf URL: https://arxiv.org/pdf/2502.07829
• Copy Paste: [[2502.07829]] Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing(https://arxiv.org/abs/2502.07829)
• Keywords: generation
• Abstract: The integration of preference alignment with diffusion models (DMs) has emerged as a transformative approach to enhance image generation and editing capabilities. Although integrating diffusion models with preference alignment strategies poses significant challenges for novices at this intersection, comprehensive and systematic reviews of this subject are still notably lacking. To bridge this gap, this paper extensively surveys preference alignment with diffusion models in image generation and editing. First, we systematically review cutting-edge optimization techniques such as reinforcement learning with human feedback (RLHF), direct preference optimization (DPO), and others, highlighting their pivotal role in aligning preferences with DMs. Then, we thoroughly explore the applications of aligning preferences with DMs in autonomous driving, medical imaging, robotics, and more. Finally, we comprehensively discuss the challenges of preference alignment with DMs. To our knowledge, this is the first survey centered on preference alignment with DMs, providing insights to drive future innovation in this dynamic area.
• 摘要：偏好对齐与扩散模型 (DM) 的集成已成为增强图像生成和编辑能力的一种变革性方法。尽管将扩散模型与偏好对齐策略相结合对新手来说提出了重大挑战，但对这一主题的全面系统的评论仍然明显缺乏。为了弥补这一差距，本文广泛调查了图像生成和编辑中偏好与扩散模型的对齐。首先，我们系统地回顾了强化学习与人类反馈 (RLHF)、直接偏好优化 (DPO) 等前沿优化技术，强调了它们在将偏好与 DM 对齐方面的关键作用。然后，我们彻底探讨了将偏好与 DM 对齐在自动驾驶、医学成像、机器人技术等领域的应用。最后，我们全面讨论了偏好与 DM 对齐的挑战。据我们所知，这是第一项以偏好与 DM 对齐为中心的调查，为推动这一充满活力的领域的未来创新提供了见解。

Title: Spread them Apart: Towards Robust Watermarking of Generated Content

• Authors: Mikhail Pautov, Danil Ivanov, Andrey V. Galichin, Oleg Rogov, Ivan Oseledets
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.07845
• Pdf URL: https://arxiv.org/pdf/2502.07845
• Copy Paste: [[2502.07845]] Spread them Apart: Towards Robust Watermarking of Generated Content(https://arxiv.org/abs/2502.07845)
• Keywords: generative
• Abstract: Generative models that can produce realistic images have improved significantly in recent years. The quality of the generated content has increased drastically, so sometimes it is very difficult to distinguish between the real images and the generated ones. Such an improvement comes at a price of ethical concerns about the usage of the generative models: the users of generative models can improperly claim ownership of the generated content protected by a license. In this paper, we propose an approach to embed watermarks into the generated content to allow future detection of the generated content and identification of the user who generated it. The watermark is embedded during the inference of the model, so the proposed approach does not require the retraining of the latter. We prove that watermarks embedded are guaranteed to be robust against additive perturbations of a bounded magnitude. We apply our method to watermark diffusion models and show that it matches state-of-the-art watermarking schemes in terms of robustness to different types of synthetic watermark removal attacks.
• 摘要：近年来，能够生成逼真图像的生成模型得到了显著改进。生成内容的质量大幅提高，因此有时很难区分真实图像和生成图像。这种改进的代价是人们对生成模型的使用存在道德担忧：生成模型的用户可能会不正当地声称拥有受许可证保护的生成内容。在本文中，我们提出了一种将水印嵌入生成内容的方法，以便将来检测生成的内容并识别生成该内容的用户。水印是在模型推理过程中嵌入的，因此所提出的方法不需要对后者进行重新训练。我们证明嵌入的水印保证对有限幅度的加性扰动具有鲁棒性。我们将我们的方法应用于水印扩散模型，并表明它在对不同类型的合成水印去除攻击的鲁棒性方面与最先进的水印方案相匹配。

Title: Understanding Classifier-Free Guidance: High-Dimensional Theory and Non-Linear Generalizations

• Authors: Krunoslav Lehman Pavasovic, Jakob Verbeek, Giulio Biroli, Marc Mezard
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.07849
• Pdf URL: https://arxiv.org/pdf/2502.07849
• Copy Paste: [[2502.07849]] Understanding Classifier-Free Guidance: High-Dimensional Theory and Non-Linear Generalizations(https://arxiv.org/abs/2502.07849)
• Keywords: generation
• Abstract: Recent studies have raised concerns about the effectiveness of Classifier-Free Guidance (CFG), indicating that in low-dimensional settings, it can lead to overshooting the target distribution and reducing sample diversity. In this work, we demonstrate that in infinite and sufficiently high-dimensional contexts CFG effectively reproduces the target distribution, revealing a blessing-of-dimensionality result. Additionally, we explore finite-dimensional effects, precisely characterizing overshoot and variance reduction. Based on our analysis, we introduce non-linear generalizations of CFG. Through numerical simulations on Gaussian mixtures and experiments on class-conditional and text-to-image diffusion models, we validate our analysis and show that our non-linear CFG offers improved flexibility and generation quality without additional computation cost.
• 摘要：最近的研究引起了人们对无分类器指导 (CFG) 有效性的担忧，表明在低维设置中，它可能导致目标分布超调并降低样本多样性。在这项工作中，我们证明在无限且足够高维的环境中，CFG 有效地重现了目标分布，揭示了维度增益的结果。此外，我们还探索了有限维效应，精确地表征了超调和方差减少。基于我们的分析，我们引入了 CFG 的非线性泛化。通过对高斯混合进行数值模拟以及对类条件和文本到图像扩散模型进行实验，我们验证了我们的分析，并表明我们的非线性 CFG 提供了更高的灵活性和生成质量，而无需额外的计算成本。

Title: MRS: A Fast Sampler for Mean Reverting Diffusion based on ODE and SDE Solvers

• Authors: Ao Li, Wei Fang, Hongbo Zhao, Le Lu, Ge Yang, Minfeng Xu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.07856
• Pdf URL: https://arxiv.org/pdf/2502.07856
• Copy Paste: [[2502.07856]] MRS: A Fast Sampler for Mean Reverting Diffusion based on ODE and SDE Solvers(https://arxiv.org/abs/2502.07856)
• Keywords: restoration, generation
• Abstract: In applications of diffusion models, controllable generation is of practical significance, but is also challenging. Current methods for controllable generation primarily focus on modifying the score function of diffusion models, while Mean Reverting (MR) Diffusion directly modifies the structure of the stochastic differential equation (SDE), making the incorporation of image conditions simpler and more natural. However, current training-free fast samplers are not directly applicable to MR Diffusion. And thus MR Diffusion requires hundreds of NFEs (number of function evaluations) to obtain high-quality samples. In this paper, we propose a new algorithm named MRS (MR Sampler) to reduce the sampling NFEs of MR Diffusion. We solve the reverse-time SDE and the probability flow ordinary differential equation (PF-ODE) associated with MR Diffusion, and derive semi-analytical solutions. The solutions consist of an analytical function and an integral parameterized by a neural network. Based on this solution, we can generate high-quality samples in fewer steps. Our approach does not require training and supports all mainstream parameterizations, including noise prediction, data prediction and velocity prediction. Extensive experiments demonstrate that MR Sampler maintains high sampling quality with a speedup of 10 to 20 times across ten different image restoration tasks. Our algorithm accelerates the sampling procedure of MR Diffusion, making it more practical in controllable generation.
• 摘要：在扩散模型的应用中，可控生成具有现实意义，但同时也具有挑战性。当前的可控生成方法主要侧重于修改扩散模型的得分函数，而均值回复（MR）扩散则直接修改随机微分方程（SDE）的结构，使图像条件的融入更加简单自然。然而，目前的免训练快速采样器并不直接适用于MR扩散，而MR扩散需要数百次NFE（函数求值次数）才能获得高质量样本。在本文中，我们提出了一种名为MRS（MR采样器）的新算法来减少MR扩散的采样NFE。我们求解与MR扩散相关的逆时间SDE和概率流常微分方程（PF-ODE），并推导出半解析解。该解由一个解析函数和一个由神经网络参数化的积分组成。基于该解，我们可以用更少的步骤生成高质量的样本。我们的方法不需要训练，并且支持所有主流的参数化，包括噪声预测、数据预测和速度预测。大量实验表明，MR Sampler 在十种不同的图像恢复任务中保持了较高的采样质量，速度提高了 10 到 20 倍。我们的算法加速了 MR Diffusion 的采样过程，使其在可控生成中更加实用。

Title: BalanceKV: KV Cache Compression through Discrepancy Theory

• Authors: Insu Han, Michael Kapralov, Ekaterina Kochetkova, Kshiteej Sheth, Amir Zandieh
• Subjects: cs.LG, cs.AI, cs.DS
• Abstract URL: https://arxiv.org/abs/2502.07861
• Pdf URL: https://arxiv.org/pdf/2502.07861
• Copy Paste: [[2502.07861]] BalanceKV: KV Cache Compression through Discrepancy Theory(https://arxiv.org/abs/2502.07861)
• Keywords: generation
• Abstract: Large language models (LLMs) have achieved impressive success, but their high memory requirements present challenges for long-context token generation. The memory complexity of long-context LLMs is primarily due to the need to store Key-Value (KV) embeddings in their KV cache. We present BalanceKV, a KV cache compression method based on geometric sampling process stemming from Banaszczyk's vector balancing theory, which introduces dependencies informed by the geometry of keys and value tokens, and improves precision. BalanceKV offers both theoretically proven and empirically validated performance improvements over existing methods.
• 摘要：大型语言模型 (LLM) 取得了令人瞩目的成功，但它们的高内存要求给长上下文标记生成带来了挑战。长上下文 LLM 的内存复杂性主要是由于需要在其 KV 缓存中存储键值 (KV) 嵌入。我们提出了 BalanceKV，这是一种基于几何采样过程的 KV 缓存压缩方法，源自 Banaszczyk 的向量平衡理论，它引入了由键和值标记的几何形状决定的依赖关系，并提高了精度。与现有方法相比，BalanceKV 提供了理论证明和经验验证的性能改进。

Title: TextAtlas5M: A Large-scale Dataset for Dense Text Image Generation

• Authors: Alex Jinpeng Wang, Dongxing Mao, Jiawei Zhang, Weiming Han, Zhuobai Dong, Linjie Li, Yiqi Lin, Zhengyuan Yang, Libo Qin, Fuwei Zhang, Lijuan Wang, Min Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.07870
• Pdf URL: https://arxiv.org/pdf/2502.07870
• Copy Paste: [[2502.07870]] TextAtlas5M: A Large-scale Dataset for Dense Text Image Generation(https://arxiv.org/abs/2502.07870)
• Keywords: generation, generative
• Abstract: Text-conditioned image generation has gained significant attention in recent years and are processing increasingly longer and comprehensive text prompt. In everyday life, dense and intricate text appears in contexts like advertisements, infographics, and signage, where the integration of both text and visuals is essential for conveying complex information. However, despite these advances, the generation of images containing long-form text remains a persistent challenge, largely due to the limitations of existing datasets, which often focus on shorter and simpler text. To address this gap, we introduce TextAtlas5M, a novel dataset specifically designed to evaluate long-text rendering in text-conditioned image generation. Our dataset consists of 5 million long-text generated and collected images across diverse data types, enabling comprehensive evaluation of large-scale generative models on long-text image generation. We further curate 3000 human-improved test set TextAtlasEval across 3 data domains, establishing one of the most extensive benchmarks for text-conditioned generation. Evaluations suggest that the TextAtlasEval benchmarks present significant challenges even for the most advanced proprietary models (e.g. GPT4o with DallE-3), while their open-source counterparts show an even larger performance gap. These evidences position TextAtlas5M as a valuable dataset for training and evaluating future-generation text-conditioned image generation models.
• 摘要：近年来，文本条件图像生成引起了广泛关注，并且正在处理越来越长、越来越全面的文本提示。在日常生活中，密集而复杂的文本出现在广告、信息图表和标牌等环境中，其中文本和视觉效果的整合对于传达复杂信息至关重要。然而，尽管取得了这些进展，但包含长文本的图像生成仍然是一个持续的挑战，这主要是由于现有数据集的局限性，这些数据集通常侧重于较短和较简单的文本。为了解决这一差距，我们引入了 TextAtlas5M，这是一个专门用于评估文本条件图像生成中长文本渲染的新数据集。我们的数据集由 500 万张跨不同数据类型生成和收集的长文本图像组成，可以全面评估长文本图像生成的大规模生成模型。我们进一步在 3 个数据域中整理了 3000 个人工改进的测试集 TextAtlasEval，为文本条件生成建立了最广泛的基准之一。评估表明，即使对于最先进的专有模型（例如 GPT4o 和 DallE-3），TextAtlasEval 基准也带来了重大挑战，而其开源模型则显示出更大的性能差距。这些证据表明 TextAtlas5M 是训练和评估未来一代文本条件图像生成模型的宝贵数据集。

Title: Generative Risk Minimization for Out-of-Distribution Generalization on Graphs

• Authors: Song Wang, Zhen Tan, Yaochen Zhu, Chuxu Zhang, Jundong Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.07968
• Pdf URL: https://arxiv.org/pdf/2502.07968
• Copy Paste: [[2502.07968]] Generative Risk Minimization for Out-of-Distribution Generalization on Graphs(https://arxiv.org/abs/2502.07968)
• Keywords: generative
• Abstract: Out-of-distribution (OOD) generalization on graphs aims at dealing with scenarios where the test graph distribution differs from the training graph distributions. Compared to i.i.d. data like images, the OOD generalization problem on graph-structured data remains challenging due to the non-i.i.d. property and complex structural information on graphs. Recently, several works on graph OOD generalization have explored extracting invariant subgraphs that share crucial classification information across different distributions. Nevertheless, such a strategy could be suboptimal for entirely capturing the invariant information, as the extraction of discrete structures could potentially lead to the loss of invariant information or the involvement of spurious information. In this paper, we propose an innovative framework, named Generative Risk Minimization (GRM), designed to generate an invariant subgraph for each input graph to be classified, instead of extraction. To address the challenge of optimization in the absence of optimal invariant subgraphs (i.e., ground truths), we derive a tractable form of the proposed GRM objective by introducing a latent causal variable, and its effectiveness is validated by our theoretical analysis. We further conduct extensive experiments across a variety of real-world graph datasets for both node-level and graph-level OOD generalization, and the results demonstrate the superiority of our framework GRM.
• 摘要：图上的分布外 (OOD) 泛化旨在处理测试图分布与训练图分布不同的情况。与图像等独立同分布数据相比，由于图上的非独立同分布属性和复杂的结构信息，图结构数据的 OOD 泛化问题仍然具有挑战性。最近，一些关于图 OOD 泛化的研究探索了提取在不同分布之间共享关键分类信息的不变子图。然而，这种策略可能不是完全捕获不变信息的最优方法，因为提取离散结构可能会导致不变信息的丢失或包含虚假信息。在本文中，我们提出了一个创新框架，称为生成风险最小化 (GRM)，旨在为每个要分类的输入图生成一个不变子图，而不是提取。为了解决在缺乏最优不变子图（即基本事实）的情况下进行优化的难题，我们通过引入潜在因果变量推导出所提出的 GRM 目标的可处理形式，并通过理论分析验证了其有效性。我们进一步在各种现实世界图数据集上进行了广泛的实验，以进行节点级和图级 OOD 泛化，结果证明了我们的框架 GRM 的优越性。

Title: Towards Training One-Step Diffusion Models Without Distillation

• Authors: Mingtian Zhang, Jiajun He, Wenlin Chen, Zijing Ou, José Miguel Hernández-Lobato, Bernhard Schölkopf, David Barber
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08005
• Pdf URL: https://arxiv.org/pdf/2502.08005
• Copy Paste: [[2502.08005]] Towards Training One-Step Diffusion Models Without Distillation(https://arxiv.org/abs/2502.08005)
• Keywords: generative
• Abstract: Recent advances in one-step generative models typically follow a two-stage process: first training a teacher diffusion model and then distilling it into a one-step student model. This distillation process traditionally relies on both the teacher model's score function to compute the distillation loss and its weights for student initialization. In this paper, we explore whether one-step generative models can be trained directly without this distillation process. First, we show that the teacher's score function is not essential and propose a family of distillation methods that achieve competitive results without relying on score estimation. Next, we demonstrate that initialization from teacher weights is indispensable in successful training. Surprisingly, we find that this benefit is not due to improved ``input-output" mapping but rather the learned feature representations, which dominate distillation quality. Our findings provide a better understanding of the role of initialization in one-step model training and its impact on distillation quality.
• 摘要：一步生成模型的最新进展通常遵循两阶段过程：首先训练教师扩散模型，然后将其提炼为一步学生模型。这种提炼过程传统上依赖于教师模型的得分函数来计算提炼损失，并依赖于其权重来初始化学生。在本文中，我们探讨是否可以在没有这个提炼过程的情况下直接训练一步生成模型。首先，我们表明教师的得分函数并不是必不可少的，并提出了一系列提炼方法，这些方法可以在不依赖分数估计的情况下实现有竞争力的结果。接下来，我们证明从教师权重进行初始化对于成功的训练是必不可少的。令人惊讶的是，我们发现这种好处不是由于改进的“输入-输出”映射，而是学习到的特征表示，这决定了提炼质量。我们的研究结果更好地理解了初始化在一步模型训练中的作用及其对提炼质量的影响。

Title: Greed is Good: Guided Generation from a Greedy Perspective

• Authors: Zander W. Blasingame, Chen Liu
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.08006
• Pdf URL: https://arxiv.org/pdf/2502.08006
• Copy Paste: [[2502.08006]] Greed is Good: Guided Generation from a Greedy Perspective(https://arxiv.org/abs/2502.08006)
• Keywords: generation, generative
• Abstract: Training-free guided generation is a widely used and powerful technique that allows the end user to exert further control over the generative process of diffusion models. In this work, we explore the guided generation from the perspective of optimizing the solution trajectory of a neural differential equation in a greedy manner. We present such a strategy as a unifying view on training-free guidance by showing that the greedy strategy is a first-order discretization of end-to-end optimization techniques. We show that a greedy guidance strategy makes good decisions and compare it to a guidance strategy using the ideal gradients found via the continuous adjoint equations. We then show how other popular training-free guidance strategies can be viewed in a unified manner from this perspective.
• 摘要：无需训练的引导生成是一种广泛使用且功能强大的技术，它允许最终用户进一步控制扩散模型的生成过程。在这项工作中，我们从贪婪方式优化神经微分方程解轨迹的角度探索引导生成。我们通过展示贪婪策略是端到端优化技术的一阶离散化，将这种策略作为对无需训练引导的统一观点。我们表明贪婪引导策略可以做出正确的决策，并将其与使用通过连续伴随方程找到的理想梯度的引导策略进行比较。然后，我们展示了如何从这个角度以统一的方式看待其他流行的无需训练的引导策略。

Title: Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol

• Authors: Pai Liu, Lingfeng Zhao, Shivangi Agarwal, Jinghan Liu, Audrey Huang, Philip Amortila, Nan Jiang
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.08021
• Pdf URL: https://arxiv.org/pdf/2502.08021
• Copy Paste: [[2502.08021]] Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol(https://arxiv.org/abs/2502.08021)
• Keywords: generation
• Abstract: Holdout validation and hyperparameter tuning from data is a long-standing problem in offline reinforcement learning (RL). A standard framework is to use off-policy evaluation (OPE) methods to evaluate and select the policies, but OPE either incurs exponential variance (e.g., importance sampling) or has hyperparameters on their own (e.g., FQE and model-based). In this work we focus on hyperparameter tuning for OPE itself, which is even more under-investigated. Concretely, we select among candidate value functions ("model-free") or dynamics ("model-based") to best assess the performance of a target policy. Our contributions are two fold. We develop: (1) new model-free and model-based selectors with theoretical guarantees, and (2) a new experimental protocol for empirically evaluating them. Compared to the model-free protocol in prior works, our new protocol allows for more stable generation of candidate value functions, better control of misspecification, and evaluation of model-free and model-based methods alike. We exemplify the protocol on a Gym environment, and find that our new model-free selector, LSTD-Tournament, demonstrates promising empirical performance.
• 摘要：从数据中进行保留验证和超参数调整是离线强化学习 (RL) 中长期存在的问题。标准框架是使用离策略评估 (OPE) 方法来评估和选择策略，但 OPE 要么产生指数方差（例如重要性采样），要么有自己的超参数（例如 FQE 和基于模型的）。在这项工作中，我们专注于 OPE 本身的超参数调整，这方面的研究还不够充分。具体来说，我们在候选值函数（“无模型”）或动态（“基于模型”）中进行选择，以最好地评估目标策略的性能。我们的贡献有两个方面。我们开发了：(1) 具有理论保证的新型无模型和基于模型的选择器，以及 (2) 用于实证评估它们的新实验协议。与之前工作中的无模型协议相比，我们的新协议可以更稳定地生成候选值函数，更好地控制错误指定，并评估无模型和基于模型的方法。我们在 Gym 环境中举例说明了该协议，并发现我们的新无模型选择器 LSTD-Tournament 表现出良好的经验性能。

Title: Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning

• Authors: Zijian He, Reyna Abhyankar, Vikranth Srivatsa, Yiying Zhang
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2502.08056
• Pdf URL: https://arxiv.org/pdf/2502.08056
• Copy Paste: [[2502.08056]] Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning(https://arxiv.org/abs/2502.08056)
• Keywords: generation
• Abstract: Today's gen-AI workflows that involve multiple ML model calls, tool/API calls, data retrieval, or generic code execution are often tuned manually in an ad-hoc way that is both time-consuming and error-prone. In this paper, we propose a systematic approach for automatically tuning gen-AI workflows. Our key insight is that gen-AI workflows can benefit from structure, operator, and prompt changes, but unique properties of gen-AI workflows require new optimization techniques. We propose AdaSeek, an adaptive hierarchical search algorithm for autotuning gen-AI workflows. AdaSeek organizes workflow tuning methods into different layers based on the user-specified total search budget and distributes the budget across different layers based on the complexity of each layer. During its hierarchical search, AdaSeek redistributes the search budget from less useful to more promising tuning configurations based on workflow-level evaluation results. We implement AdaSeek in a workflow autotuning framework called Cognify and evaluate Cognify using six types of workflows such as RAG-based QA and text-to-SQL transformation. Overall, Cognify improves these workflows' generation quality by up to 2.8x, reduces execution monetary cost by up to 10x, and reduces end-to-end latency by 2.7x.
• 摘要：当今的 gen-AI 工作流涉及多个 ML 模型调用、工具/API 调用、数据检索或通用代码执行，通常以临时方式手动调整，既耗时又容易出错。在本文中，我们提出了一种自动调整 gen-AI 工作流的系统方法。我们的主要见解是，gen-AI 工作流可以从结构、运算符和提示更改中受益，但 gen-AI 工作流的独特属性需要新的优化技术。我们提出了 AdaSeek，一种用于自动调整 gen-AI 工作流的自适应分层搜索算法。AdaSeek 根据用户指定的总搜索预算将工作流调整方法组织到不同的层中，并根据每层的复杂性将预算分配到不同的层中。在分层搜索过程中，AdaSeek 根据工作流级评估结果将搜索预算从不太有用的调整配置重新分配到更有希望的调整配置。我们在名为 Cognify 的工作流自动调整框架中实现了 AdaSeek，并使用六种类型的工作流（例如基于 RAG 的 QA 和文本到 SQL 转换）对 Cognify 进行了评估。总体而言，Cognify 将这些工作流的生成质量提高了 2.8 倍，将执行成本降低了 10 倍，并将端到端延迟降低了 2.7 倍。

Title: ID-Cloak: Crafting Identity-Specific Cloaks Against Personalized Text-to-Image Generation

• Authors: Qianrui Teng, Xing Cui, Xuannan Liu, Peipei Li, Zekun Li, Huaibo Huang, Ran He
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2502.08097
• Pdf URL: https://arxiv.org/pdf/2502.08097
• Copy Paste: [[2502.08097]] ID-Cloak: Crafting Identity-Specific Cloaks Against Personalized Text-to-Image Generation(https://arxiv.org/abs/2502.08097)
• Keywords: generation
• Abstract: Personalized text-to-image models allow users to generate images of new concepts from several reference photos, thereby leading to critical concerns regarding civil privacy. Although several anti-personalization techniques have been developed, these methods typically assume that defenders can afford to design a privacy cloak corresponding to each specific image. However, due to extensive personal images shared online, image-specific methods are limited by real-world practical applications. To address this issue, we are the first to investigate the creation of identity-specific cloaks (ID-Cloak) that safeguard all images belong to a specific identity. Specifically, we first model an identity subspace that preserves personal commonalities and learns diverse contexts to capture the image distribution to be protected. Then, we craft identity-specific cloaks with the proposed novel objective that encourages the cloak to guide the model away from its normal output within the subspace. Extensive experiments show that the generated universal cloak can effectively protect the images. We believe our method, along with the proposed identity-specific cloak setting, marks a notable advance in realistic privacy protection.
• 摘要：个性化的文本到图像模型允许用户从几张参考照片中生成新概念的图像，从而引发了对公民隐私的严重担忧。尽管已经开发了几种反个性化技术，但这些方法通常假设防御者有能力设计与每张特定图像相对应的隐私斗篷。然而，由于网上共享了大量的个人图像，特定于图像的方法受到现实世界实际应用的限制。为了解决这个问题，我们率先研究了创建身份特定斗篷 (ID-Cloak)，以保护所有属于特定身份的图像。具体来说，我们首先对一个身份子空间进行建模，该子空间保留了个人共性并学习了不同的上下文以捕获要保护的图像分布。然后，我们根据提出的新目标制作了身份特定斗篷，鼓励斗篷引导模型远离子空间内的正常输出。大量实验表明，生成的通用斗篷可以有效地保护图像。我们相信，我们的方法以及提出的身份特定斗篷设置标志着现实隐私保护的显著进步。

Title: Rethinking Tokenized Graph Transformers for Node Classification

• Authors: Jinsong Chen, Chenyang Li, GaiChao Li, John E. Hopcroft, Kun He
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08101
• Pdf URL: https://arxiv.org/pdf/2502.08101
• Copy Paste: [[2502.08101]] Rethinking Tokenized Graph Transformers for Node Classification(https://arxiv.org/abs/2502.08101)
• Keywords: generation
• Abstract: Node tokenized graph Transformers (GTs) have shown promising performance in node classification. The generation of token sequences is the key module in existing tokenized GTs which transforms the input graph into token sequences, facilitating the node representation learning via Transformer. In this paper, we observe that the generations of token sequences in existing GTs only focus on the first-order neighbors on the constructed similarity graphs, which leads to the limited usage of nodes to generate diverse token sequences, further restricting the potential of tokenized GTs for node classification. To this end, we propose a new method termed SwapGT. SwapGT first introduces a novel token swapping operation based on the characteristics of token sequences that fully leverages the semantic relevance of nodes to generate more informative token sequences. Then, SwapGT leverages a Transformer-based backbone to learn node representations from the generated token sequences. Moreover, SwapGT develops a center alignment loss to constrain the representation learning from multiple token sequences, further enhancing the model performance. Extensive empirical results on various datasets showcase the superiority of SwapGT for node classification.
• 摘要：节点标记图 Transformer (GT) 在节点分类中表现出色。标记序列的生成是现有标记图 GT 中的关键模块，它将输入图转换为标记序列，从而促进通过 Transformer 进行节点表示学习。在本文中，我们观察到现有 GT 中标记序列的生成仅关注构建的相似性图上的一阶邻居，这导致节点在生成多样化标记序列方面的使用有限，进一步限制了标记图 GT 用于节点分类的潜力。为此，我们提出了一种称为 SwapGT 的新方法。SwapGT 首先根据标记序列的特征引入一种新颖的标记交换操作，充分利用节点的语义相关性来生成更具信息量的标记序列。然后，SwapGT 利用基于 Transformer 的主干从生成的标记序列中学习节点表示。此外，SwapGT 开发了一个中心对齐损失来约束从多个标记序列中进行表示学习，进一步提高了模型性能。在各种数据集上进行的大量实证结果证明了 SwapGT 在节点分类方面的优越性。

Title: PoGDiff: Product-of-Gaussians Diffusion Models for Imbalanced Text-to-Image Generation

• Authors: Ziyan Wang, Sizhe Wei, Xiaoming Huo, Hao Wang
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.08106
• Pdf URL: https://arxiv.org/pdf/2502.08106
• Copy Paste: [[2502.08106]] PoGDiff: Product-of-Gaussians Diffusion Models for Imbalanced Text-to-Image Generation(https://arxiv.org/abs/2502.08106)
• Keywords: generation
• Abstract: Diffusion models have made significant advancements in recent years. However, their performance often deteriorates when trained or fine-tuned on imbalanced datasets. This degradation is largely due to the disproportionate representation of majority and minority data in image-text pairs. In this paper, we propose a general fine-tuning approach, dubbed PoGDiff, to address this challenge. Rather than directly minimizing the KL divergence between the predicted and ground-truth distributions, PoGDiff replaces the ground-truth distribution with a Product of Gaussians (PoG), which is constructed by combining the original ground-truth targets with the predicted distribution conditioned on a neighboring text embedding. Experiments on real-world datasets demonstrate that our method effectively addresses the imbalance problem in diffusion models, improving both generation accuracy and quality.
• 摘要：近年来，扩散模型取得了重大进展。然而，在对不平衡数据集进行训练或微调时，它们的性能往往会下降。这种下降主要是由于图像-文本对中多数数据和少数数据的表示不成比例。在本文中，我们提出了一种称为 PoGDiff 的通用微调方法来应对这一挑战。PoGDiff 不是直接最小化预测分布和真实分布之间的 KL 散度，而是用高斯积 (PoG) 替换真实分布，后者是通过将原始真实目标与以相邻文本嵌入为条件的预测分布相结合而构建的。在真实数据集上的实验表明，我们的方法有效地解决了扩散模型中的不平衡问题，提高了生成的准确性和质量。

Title: Force Matching with Relativistic Constraints: A Physics-Inspired Approach to Stable and Efficient Generative Modeling

• Authors: Yang Cao, Bo Chen, Xiaoyu Li, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song, Mingda Wan
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08150
• Pdf URL: https://arxiv.org/pdf/2502.08150
• Copy Paste: [[2502.08150]] Force Matching with Relativistic Constraints: A Physics-Inspired Approach to Stable and Efficient Generative Modeling(https://arxiv.org/abs/2502.08150)
• Keywords: generative
• Abstract: This paper introduces Force Matching (ForM), a novel framework for generative modeling that represents an initial exploration into leveraging special relativistic mechanics to enhance the stability of the sampling process. By incorporating the Lorentz factor, ForM imposes a velocity constraint, ensuring that sample velocities remain bounded within a constant limit. This constraint serves as a fundamental mechanism for stabilizing the generative dynamics, leading to a more robust and controlled sampling process. We provide a rigorous theoretical analysis demonstrating that the velocity constraint is preserved throughout the sampling procedure within the ForM framework. To validate the effectiveness of our approach, we conduct extensive empirical evaluations. On the \textit{half-moons} dataset, ForM significantly outperforms baseline methods, achieving the lowest Euclidean distance loss of \textbf{0.714}, in contrast to vanilla first-order flow matching (5.853) and first- and second-order flow matching (5.793). Additionally, we perform an ablation study to further investigate the impact of our velocity constraint, reaffirming the superiority of ForM in stabilizing the generative process. The theoretical guarantees and empirical results underscore the potential of integrating special relativity principles into generative modeling. Our findings suggest that ForM provides a promising pathway toward achieving stable, efficient, and flexible generative processes. This work lays the foundation for future advancements in high-dimensional generative modeling, opening new avenues for the application of physical principles in machine learning.
• 摘要：本文介绍了力匹配 (ForM)，这是一种用于生成建模的新型框架，代表了利用特殊相对论力学来增强采样过程稳定性的初步探索。通过结合洛伦兹因子，ForM 施加了速度约束，确保样本速度保持在恒定的范围内。此约束是稳定生成动力学的基本机制，可实现更稳健、更可控的采样过程。我们提供了严格的理论分析，证明在 ForM 框架内，速度约束在整个采样过程中都得以保留。为了验证我们方法的有效性，我们进行了广泛的实证评估。在 \textit{half-moons} 数据集上，ForM 的表现明显优于基线方法，与普通的一阶流匹配 (5.853) 和一阶和二阶流匹配 (5.793) 相比，实现了最低的欧几里得距离损失 \textbf{0.714}。此外，我们进行了一项消融研究，以进一步研究速度约束的影响，重申了 ForM 在稳定生成过程方面的优势。理论保证和实证结果强调了将狭义相对论原理融入生成模型的潜力。我们的研究结果表明，ForM 为实现稳定、高效和灵活的生成过程提供了一条有希望的途径。这项工作为高维生成模型的未来发展奠定了基础，为物理原理在机器学习中的应用开辟了新途径。

Title: AnyCharV: Bootstrap Controllable Character Video Generation with Fine-to-Coarse Guidance

• Authors: Zhao Wang, Hao Wen, Lingting Zhu, Chenming Shang, Yujiu Yang, Qi Dou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08189
• Pdf URL: https://arxiv.org/pdf/2502.08189
• Copy Paste: [[2502.08189]] AnyCharV: Bootstrap Controllable Character Video Generation with Fine-to-Coarse Guidance(https://arxiv.org/abs/2502.08189)
• Keywords: generation
• Abstract: Character video generation is a significant real-world application focused on producing high-quality videos featuring specific characters. Recent advancements have introduced various control signals to animate static characters, successfully enhancing control over the generation process. However, these methods often lack flexibility, limiting their applicability and making it challenging for users to synthesize a source character into a desired target scene. To address this issue, we propose a novel framework, AnyCharV, that flexibly generates character videos using arbitrary source characters and target scenes, guided by pose information. Our approach involves a two-stage training process. In the first stage, we develop a base model capable of integrating the source character with the target scene using pose guidance. The second stage further bootstraps controllable generation through a self-boosting mechanism, where we use the generated video in the first stage and replace the fine mask with the coarse one, enabling training outcomes with better preservation of character details. Experimental results demonstrate the effectiveness and robustness of our proposed method. Our project page is this https URL.
• 摘要：角色视频生成是一项重要的现实世界应用，专注于制作具有特定角色的高质量视频。最近的进展引入了各种控制信号来为静态角色制作动画，成功地增强了对生成过程的控制。然而，这些方法往往缺乏灵活性，限制了它们的适用性，并使用户难以将源角色合成到所需的目标场景中。为了解决这个问题，我们提出了一个新颖的框架 AnyCharV，它可以灵活地使用任意源角色和目标场景生成角色视频，并由姿势信息引导。我们的方法涉及一个两阶段的训练过程。在第一阶段，我们开发了一个基础模型，该模型能够使用姿势引导将源角色与目标场景结合起来。第二阶段通过自增强机制进一步引导可控生成，其中我们使用第一阶段生成的视频并将精细蒙版替换为粗蒙版，从而使训练结果能够更好地保留角色细节。实验结果证明了我们提出的方法的有效性和鲁棒性。我们的项目页面是这个 https URL。

Title: Take What You Need: Flexible Multi-Task Semantic Communications with Channel Adaptation

• Authors: Xiang Chen, Shuying Gan, Chenyuan Feng, Xijun Wang, Tony Q. S. Quek
• Subjects: cs.CV, cs.IT, cs.NI
• Abstract URL: https://arxiv.org/abs/2502.08221
• Pdf URL: https://arxiv.org/pdf/2502.08221
• Copy Paste: [[2502.08221]] Take What You Need: Flexible Multi-Task Semantic Communications with Channel Adaptation(https://arxiv.org/abs/2502.08221)
• Keywords: generation
• Abstract: The growing demand for efficient semantic communication systems capable of managing diverse tasks and adapting to fluctuating channel conditions has driven the development of robust, resource-efficient frameworks. This article introduces a novel channel-adaptive and multi-task-aware semantic communication framework based on a masked auto-encoder architecture. Our framework optimizes the transmission of meaningful information by incorporating a multi-task-aware scoring mechanism that identifies and prioritizes semantically significant data across multiple concurrent tasks. A channel-aware extractor is employed to dynamically select relevant information in response to real-time channel conditions. By jointly optimizing semantic relevance and transmission efficiency, the framework ensures minimal performance degradation under resource constraints. Experimental results demonstrate the superior performance of our framework compared to conventional methods in tasks such as image reconstruction and object detection. These results underscore the framework's adaptability to heterogeneous channel environments and its scalability for multi-task applications, positioning it as a promising solution for next-generation semantic communication networks.
• 摘要：人们对能够管理各种任务并适应波动的信道条件的高效语义通信系统的需求日益增长，这推动了稳健、资源高效的框架的发展。本文介绍了一种基于掩蔽自动编码器架构的新型信道自适应和多任务感知语义通信框架。我们的框架通过结合多任务感知评分机制来优化有意义信息的传输，该机制可识别并优先处理多个并发任务中语义上重要的数据。采用信道感知提取器来动态选择相关信息以响应实时信道条件。通过联合优化语义相关性和传输效率，该框架可确保在资源受限的情况下将性能下降降至最低。实验结果表明，与传统方法相比，我们的框架在图像重建和对象检测等任务中具有卓越的性能。这些结果强调了该框架对异构信道环境的适应性及其对多任务应用的可扩展性，使其成为下一代语义通信网络的有前途的解决方案。

Title: Learning Human Skill Generators at Key-Step Levels

• Authors: Yilu Wu, Chenhui Zhu, Shuai Wang, Hanlin Wang, Jing Wang, Zhaoxiang Zhang, Limin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08234
• Pdf URL: https://arxiv.org/pdf/2502.08234
• Copy Paste: [[2502.08234]] Learning Human Skill Generators at Key-Step Levels(https://arxiv.org/abs/2502.08234)
• Keywords: generation
• Abstract: We are committed to learning human skill generators at key-step levels. The generation of skills is a challenging endeavor, but its successful implementation could greatly facilitate human skill learning and provide more experience for embodied intelligence. Although current video generation models can synthesis simple and atomic human operations, they struggle with human skills due to their complex procedure process. Human skills involve multi-step, long-duration actions and complex scene transitions, so the existing naive auto-regressive methods for synthesizing long videos cannot generate human skills. To address this, we propose a novel task, the Key-step Skill Generation (KS-Gen), aimed at reducing the complexity of generating human skill videos. Given the initial state and a skill description, the task is to generate video clips of key steps to complete the skill, rather than a full-length video. To support this task, we introduce a carefully curated dataset and define multiple evaluation metrics to assess performance. Considering the complexity of KS-Gen, we propose a new framework for this task. First, a multimodal large language model (MLLM) generates descriptions for key steps using retrieval argument. Subsequently, we use a Key-step Image Generator (KIG) to address the discontinuity between key steps in skill videos. Finally, a video generation model uses these descriptions and key-step images to generate video clips of the key steps with high temporal consistency. We offer a detailed analysis of the results, hoping to provide more insights on human skill generation. All models and data are available at this https URL.
• 摘要：我们致力于在关键步骤层面学习人类技能生成器。技能的生成是一项具有挑战性的任务，但成功实施可以极大地促进人类技能的学习，并为具身智能提供更多经验。尽管当前的视频生成模型可以合成简单和原子的人类操作，但由于其复杂的程序过程，它们在处理人类技能方面遇到了困难。人类技能涉及多步骤、长时间的动作和复杂的场景转换，因此现有的用于合成长视频的简单自回归方法无法生成人类技能。为了解决这个问题，我们提出了一项新任务，即关键步骤技能生成 (KS-Gen)，旨在降低生成人类技能视频的复杂性。给定初始状态和技能描述，任务是生成完成技能的关键步骤的视频片段，而不是全长视频。为了支持这项任务，我们引入了一个精心策划的数据集并定义了多个评估指标来评估性能。考虑到 KS-Gen 的复杂性，我们为这项任务提出了一个新的框架。首先，多模态大语言模型 (MLLM) 使用检索参数生成关键步骤的描述。随后，我们使用关键步骤图像生成器 (KIG) 来解决技能视频中关键步骤之间的不连续性问题。最后，视频生成模型使用这些描述和关键步骤图像来生成具有高度时间一致性的关键步骤视频片段。我们对结果进行了详细分析，希望为人类技能生成提供更多见解。所有模型和数据均可在此 https URL 上找到。

Title: FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis

• Authors: Wonjoon Jin, Qi Dai, Chong Luo, Seung-Hwan Baek, Sunghyun Cho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08244
• Pdf URL: https://arxiv.org/pdf/2502.08244
• Copy Paste: [[2502.08244]] FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis(https://arxiv.org/abs/2502.08244)
• Keywords: generation
• Abstract: This paper presents FloVD, a novel optical-flow-based video diffusion model for camera-controllable video generation. FloVD leverages optical flow maps to represent motions of the camera and moving objects. This approach offers two key benefits. Since optical flow can be directly estimated from videos, our approach allows for the use of arbitrary training videos without ground-truth camera parameters. Moreover, as background optical flow encodes 3D correlation across different viewpoints, our method enables detailed camera control by leveraging the background motion. To synthesize natural object motion while supporting detailed camera control, our framework adopts a two-stage video synthesis pipeline consisting of optical flow generation and flow-conditioned video synthesis. Extensive experiments demonstrate the superiority of our method over previous approaches in terms of accurate camera control and natural object motion synthesis.
• 摘要：本文介绍了一种基于光流的新型视频扩散模型 FloVD，用于生成可由摄像机控制的视频。FloVD 利用光流图来表示摄像机和移动物体的运动。这种方法有两个主要优点。由于可以直接从视频中估计光流，因此我们的方法允许使用任意训练视频，而无需真实的摄像机参数。此外，由于背景光流编码了不同视点之间的 3D 相关性，因此我们的方法可以通过利用背景运动来实现详细的摄像机控制。为了在支持详细的摄像机控制的同时合成自然物体运动，我们的框架采用了由光流生成和流条件视频合成组成的两阶段视频合成流水线。大量实验证明了我们的方法在精确的摄像机控制和自然物体运动合成方面优于以前的方法。

Title: UniCoRN: Unified Commented Retrieval Network with LMMs

• Authors: Maximilian Jaritz, Matthieu Guillaumin, Sabine Sternig, Loris Bazzani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08254
• Pdf URL: https://arxiv.org/pdf/2502.08254
• Copy Paste: [[2502.08254]] UniCoRN: Unified Commented Retrieval Network with LMMs(https://arxiv.org/abs/2502.08254)
• Keywords: generation, generative
• Abstract: Multimodal retrieval methods have limitations in handling complex, compositional queries that require reasoning about the visual content of both the query and the retrieved entities. On the other hand, Large Multimodal Models (LMMs) can answer with language to more complex visual questions, but without the inherent ability to retrieve relevant entities to support their answers. We aim to address these limitations with UniCoRN, a Unified Commented Retrieval Network that combines the strengths of composed multimodal retrieval methods and generative language approaches, going beyond Retrieval-Augmented Generation (RAG). We introduce an entity adapter module to inject the retrieved multimodal entities back into the LMM, so it can attend to them while generating answers and comments. By keeping the base LMM frozen, UniCoRN preserves its original capabilities while being able to perform both retrieval and text generation tasks under a single integrated framework. To assess these new abilities, we introduce the Commented Retrieval task (CoR) and a corresponding dataset, with the goal of retrieving an image that accurately answers a given question and generate an additional textual response that provides further clarification and details about the visual information. We demonstrate the effectiveness of UniCoRN on several datasets showing improvements of +4.5% recall over the state of the art for composed multimodal retrieval and of +14.9% METEOR / +18.4% BEM over RAG for commenting in CoR.
• 摘要：多模态检索方法在处理复杂的组合查询方面存在局限性，因为这些查询需要对查询和检索到的实体的视觉内容进行推理。另一方面，大型多模态模型 (LMM) 可以用语言回答更复杂的视觉问题，但没有检索相关实体来支持其答案的固有能力。我们旨在通过 UniCoRN 解决这些限制，UniCoRN 是一个统一注释检索网络，它结合了组合多模态检索方法和生成语言方法的优势，超越了检索增强生成 (RAG)。我们引入了一个实体适配器模块，将检索到的多模态实体注入回 LMM，以便它可以在生成答案和评论的同时处理它们。通过保持基础 LMM 冻结，UniCoRN 保留了其原有的功能，同时能够在单个集成框架下执行检索和文本生成任务。为了评估这些新功能，我们引入了注释检索任务 (CoR) 和相应的数据集，目标是检索准确回答给定问题的图像，并生成额外的文本响应，以进一步阐明和详细说明视觉信息。我们在多个数据集上展示了 UniCoRN 的有效性，结果显示，在组合多模态检索方面，召回率比现有技术提高了 4.5%，在 CoR 注释方面，METEOR / BEM 比 RAG 提高了 14.9% / 18.4%。

Title: GenIAS: Generator for Instantiating Anomalies in time Series

• Authors: Zahra Zamanzadeh Darban, Qizhou Wang, Geoffrey I. Webb, Shirui Pan, Charu C. Aggarwal, Mahsa Salehi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08262
• Pdf URL: https://arxiv.org/pdf/2502.08262
• Copy Paste: [[2502.08262]] GenIAS: Generator for Instantiating Anomalies in time Series(https://arxiv.org/abs/2502.08262)
• Keywords: generation, generative
• Abstract: A recent and promising approach for building time series anomaly detection (TSAD) models is to inject synthetic samples of anomalies within real data sets. The existing injection mechanisms have significant limitations - most of them rely on ad hoc, hand-crafted strategies which fail to capture the natural diversity of anomalous patterns, or are restricted to univariate time series settings. To address these challenges, we design a generative model for TSAD using a variational autoencoder, which is referred to as a Generator for Instantiating Anomalies in Time Series (GenIAS). GenIAS is designed to produce diverse and realistic synthetic anomalies for TSAD tasks. By employing a novel learned perturbation mechanism in the latent space and injecting the perturbed patterns in different segments of time series, GenIAS can generate anomalies with greater diversity and varying scales. Further, guided by a new triplet loss function, which uses a min-max margin and a new variance-scaling approach to further enforce the learning of compact normal patterns, GenIAS ensures that anomalies are distinct from normal samples while remaining realistic. The approach is effective for both univariate and multivariate time series. We demonstrate the diversity and realism of the generated anomalies. Our extensive experiments demonstrate that GenIAS - when integrated into a TSAD task - consistently outperforms seventeen traditional and deep anomaly detection models, thereby highlighting the potential of generative models for time series anomaly generation.
• 摘要：构建时间序列异常检测 (TSAD) 模型的一种最新且有前景的方法是将合成的异常样本注入真实数据集中。现有的注入机制存在很大的局限性 - 其中大多数依赖于临时的、手工制定的策略，这些策略无法捕捉异常模式的自然多样性，或者仅限于单变量时间序列设置。为了应对这些挑战，我们使用变分自动编码器为 TSAD 设计了一个生成模型，该模型称为时间序列异常实例生成器 (GenIAS)。GenIAS 旨在为 TSAD 任务生成多样化且逼真的合成异常。通过在潜在空间中采用一种新颖的学习扰动机制，并将扰动模式注入时间序列的不同部分，GenIAS 可以生成具有更大多样性和不同规模的异常。此外，在新的三重态损失函数（使用最小-最大边际和新的方差缩放方法来进一步强化紧凑正常模式的学习）的指导下，GenIAS 确保异常与正常样本不同，同时保持真实性。该方法对单变量和多变量时间序列都有效。我们展示了生成的异常的多样性和真实性。我们进行了广泛的实验，表明 GenIAS 在集成到 TSAD 任务中时，其表现始终优于 17 种传统和深度异常检测模型，从而凸显了生成模型在时间序列异常生成方面的潜力。

Title: HDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting

• Authors: Shibo Feng, Peilin Zhao, Liu Liu, Pengcheng Wu, Zhiqi Shen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08302
• Pdf URL: https://arxiv.org/pdf/2502.08302
• Copy Paste: [[2502.08302]] HDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting(https://arxiv.org/abs/2502.08302)
• Keywords: generation, generative
• Abstract: Generative models have gained significant attention in multivariate time series forecasting (MTS), particularly due to their ability to generate high-fidelity samples. Forecasting the probability distribution of multivariate time series is a challenging yet practical task. Although some recent attempts have been made to handle this task, two major challenges persist: 1) some existing generative methods underperform in high-dimensional multivariate time series forecasting, which is hard to scale to higher dimensions; 2) the inherent high-dimensional multivariate attributes constrain the forecasting lengths of existing generative models. In this paper, we point out that discrete token representations can model high-dimensional MTS with faster inference time, and forecasting the target with long-term trends of itself can extend the forecasting length with high accuracy. Motivated by this, we propose a vector quantized framework called Hierarchical Discrete Transformer (HDT) that models time series into discrete token representations with l2 normalization enhanced vector quantized strategy, in which we transform the MTS forecasting into discrete tokens generation. To address the limitations of generative models in long-term forecasting, we propose a hierarchical discrete Transformer. This model captures the discrete long-term trend of the target at the low level and leverages this trend as a condition to generate the discrete representation of the target at the high level that introduces the features of the target itself to extend the forecasting length in high-dimensional MTS. Extensive experiments on five popular MTS datasets verify the effectiveness of our proposed method.
• 摘要：生成模型在多元时间序列预测 (MTS) 中获得了极大的关注，特别是因为它们能够生成高保真样本。预测多元时间序列的概率分布是一项具有挑战性但又很实用的任务。尽管最近已经进行了一些尝试来处理这项任务，但仍然存在两个主要挑战：1) 一些现有的生成方法在高维多元时间序列预测中表现不佳，难以扩展到更高维度；2) 固有的高维多元属性限制了现有生成模型的预测长度。在本文中，我们指出离散标记表示可以以更快的推理时间对高维 MTS 进行建模，并且预测具有长期趋势的目标本身可以以高精度延长预测长度。受此启发，我们提出了一个称为分层离散变换器 (HDT) 的矢量量化框架，该框架使用 l2 规范化增强矢量量化策略将时间序列建模为离散标记表示，其中我们将 MTS 预测转换为离散标记生成。为了解决生成模型在长期预测中的局限性，我们提出了一种分层离散 Transformer。该模型在低层捕获目标的离散长期趋势，并利用此趋势作为条件在高层生成目标的离散表示，该表示引入目标本身的特征以延长高维 MTS 中​​的预测长度。在五个流行的 MTS 数据集上进行的大量实验验证了我们提出的方法的有效性。

Title: Foundation Models in Computational Pathology: A Review of Challenges, Opportunities, and Impact

• Authors: Mohsin Bilal, Aadam, Manahil Raza, Youssef Altherwy, Anas Alsuhaibani, Abdulrahman Abduljabbar, Fahdah Almarshad, Paul Golding, Nasir Rajpoot
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08333
• Pdf URL: https://arxiv.org/pdf/2502.08333
• Copy Paste: [[2502.08333]] Foundation Models in Computational Pathology: A Review of Challenges, Opportunities, and Impact(https://arxiv.org/abs/2502.08333)
• Keywords: generative
• Abstract: From self-supervised, vision-only models to contrastive visual-language frameworks, computational pathology has rapidly evolved in recent years. Generative AI "co-pilots" now demonstrate the ability to mine subtle, sub-visual tissue cues across the cellular-to-pathology spectrum, generate comprehensive reports, and respond to complex user queries. The scale of data has surged dramatically, growing from tens to millions of multi-gigapixel tissue images, while the number of trainable parameters in these models has risen to several billion. The critical question remains: how will this new wave of generative and multi-purpose AI transform clinical diagnostics? In this article, we explore the true potential of these innovations and their integration into clinical practice. We review the rapid progress of foundation models in pathology, clarify their applications and significance. More precisely, we examine the very definition of foundational models, identifying what makes them foundational, general, or multipurpose, and assess their impact on computational pathology. Additionally, we address the unique challenges associated with their development and evaluation. These models have demonstrated exceptional predictive and generative capabilities, but establishing global benchmarks is crucial to enhancing evaluation standards and fostering their widespread clinical adoption. In computational pathology, the broader impact of frontier AI ultimately depends on widespread adoption and societal acceptance. While direct public exposure is not strictly necessary, it remains a powerful tool for dispelling misconceptions, building trust, and securing regulatory support.
• 摘要：近年来，计算病理学从自监督的纯视觉模型发展到对比性视觉语言框架，发展迅速。生成式人工智能“副驾驶”现在展示了从细胞到病理学范围内挖掘细微的、视觉以下组织线索、生成综合报告和响应复杂用户查询的能力。数据规模急剧增加，从数十张到数百万张数千兆像素的组织图像，而这些模型中可训练参数的数量已上升到数十亿。关键问题仍然存在：这一波新的生成式和多用途人工智能将如何改变临床诊断？在本文中，我们探讨了这些创新的真正潜力及其与临床实践的整合。我们回顾了病理学基础模型的快速进展，阐明了它们的应用和意义。更准确地说，我们研究了基础模型的定义，确定了是什么使它们成为基础的、通用的或多用途的，并评估了它们对计算病理学的影响。此外，我们还解决了与它们的开发和评估相关的独特挑战。这些模型已经展现出卓越的预测和生成能力，但建立全球基准对于提高评估标准和促进其在临床上的广泛应用至关重要。在计算病理学中，前沿人工智能的更广泛影响最终取决于广泛应用和社会认可。虽然直接向公众曝光并非绝对必要，但它仍然是消除误解、建立信任和获得监管支持的有力工具。

Title: Trustworthy GNNs with LLMs: A Systematic Review and Taxonomy

• Authors: Ruizhan Xue, Huimin Deng, Fang He, Maojun Wang, Zeyu Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08353
• Pdf URL: https://arxiv.org/pdf/2502.08353
• Copy Paste: [[2502.08353]] Trustworthy GNNs with LLMs: A Systematic Review and Taxonomy(https://arxiv.org/abs/2502.08353)
• Keywords: generation
• Abstract: With the extensive application of Graph Neural Networks (GNNs) across various domains, their trustworthiness has emerged as a focal point of research. Some existing studies have shown that the integration of large language models (LLMs) can improve the semantic understanding and generation capabilities of GNNs, which in turn improves the trustworthiness of GNNs from various aspects. Our review introduces a taxonomy that offers researchers a clear framework for comprehending the principles and applications of different methods and helps clarify the connections and differences among various approaches. Then we systematically survey representative approaches along the four categories of our taxonomy. Through our taxonomy, researchers can understand the applicable scenarios, potential advantages, and limitations of each approach for the the trusted integration of GNNs with LLMs. Finally, we present some promising directions of work and future trends for the integration of LLMs and GNNs to improve model trustworthiness.
• 摘要：随着图神经网络 (GNN) 在各个领域的广泛应用，其可信度已成为研究的焦点。现有的一些研究表明，大型语言模型 (LLM) 的集成可以提高 GNN 的语义理解和生成能力，从而从各个方面提高 GNN 的可信度。我们的综述介绍了一种分类法，它为研究人员提供了一个清晰的框架来理解不同方法的原理和应用，并有助于阐明各种方法之间的联系和区别。然后，我们系统地调查了我们分类法的四个类别中的代表性方法。通过我们的分类法，研究人员可以了解每种方法在 GNN 与 LLM 的可信集成中的适用场景、潜在优势和局限性。最后，我们提出了一些有希望的工作方向和未来趋势，以整合 LLM 和 GNN 来提高模型的可信度。

Title: A Survey on Pre-Trained Diffusion Model Distillations

• Authors: Xuhui Fan, Zhangkai Wu, Hongyu Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08364
• Pdf URL: https://arxiv.org/pdf/2502.08364
• Copy Paste: [[2502.08364]] A Survey on Pre-Trained Diffusion Model Distillations(https://arxiv.org/abs/2502.08364)
• Keywords: generation, generative
• Abstract: Diffusion Models~(DMs) have emerged as the dominant approach in Generative Artificial Intelligence (GenAI), owing to their remarkable performance in tasks such as text-to-image synthesis. However, practical DMs, such as stable diffusion, are typically trained on massive datasets and thus usually require large storage. At the same time, many steps may be required, i.e., recursively evaluating the trained neural network, to generate a high-quality image, which results in significant computational costs during sample generation. As a result, distillation methods on pre-trained DM have become widely adopted practices to develop smaller, more efficient models capable of rapid, few-step generation in low-resource environment. When these distillation methods are developed from different perspectives, there is an urgent need for a systematic survey, particularly from a methodological perspective. In this survey, we review distillation methods through three aspects: output loss distillation, trajectory distillation and adversarial distillation. We also discuss current challenges and outline future research directions in the conclusion.
• 摘要：扩散模型 (DM) 已成为生成式人工智能 (GenAI) 中的主导方法，因为它们在文本到图像合成等任务中表现出色。然而，实际的 DM（例如稳定扩散）通常在海量数据集上进行训练，因此通常需要大量存储空间。同时，可能需要许多步骤（即递归评估训练过的神经网络）才能生成高质量的图像，这会导致样本生成过程中的大量计算成本。因此，在预训练的 DM 上进行蒸馏方法已成为广泛采用的做法，以开发更小、更高效的模型，这些模型能够在低资源环境中快速、仅需几步即可生成。当从不同角度开发这些蒸馏方法时，迫切需要进行系统的调查，特别是从方法论的角度。在本调查中，我们从三个方面回顾了蒸馏方法：输出损失蒸馏、轨迹蒸馏和对抗性蒸馏。我们还在结论中讨论了当前的挑战并概述了未来的研究方向。

Title: Not All Frame Features Are Equal: Video-to-4D Generation via Decoupling Dynamic-Static Features

• Authors: Liying Yang, Chen Liu, Zhenwei Zhu, Ajian Liu, Hui Ma, Jian Nong, Yanyan Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08377
• Pdf URL: https://arxiv.org/pdf/2502.08377
• Copy Paste: [[2502.08377]] Not All Frame Features Are Equal: Video-to-4D Generation via Decoupling Dynamic-Static Features(https://arxiv.org/abs/2502.08377)
• Keywords: generation
• Abstract: Recently, the generation of dynamic 3D objects from a video has shown impressive results. Existing methods directly optimize Gaussians using whole information in frames. However, when dynamic regions are interwoven with static regions within frames, particularly if the static regions account for a large proportion, existing methods often overlook information in dynamic regions and are prone to overfitting on static regions. This leads to producing results with blurry textures. We consider that decoupling dynamic-static features to enhance dynamic representations can alleviate this issue. Thus, we propose a dynamic-static feature decoupling module (DSFD). Along temporal axes, it regards the portions of current frame features that possess significant differences relative to reference frame features as dynamic features. Conversely, the remaining parts are the static features. Then, we acquire decoupled features driven by dynamic features and current frame features. Moreover, to further enhance the dynamic representation of decoupled features from different viewpoints and ensure accurate motion prediction, we design a temporal-spatial similarity fusion module (TSSF). Along spatial axes, it adaptively selects a similar information of dynamic regions. Hinging on the above, we construct a novel approach, DS4D. Experimental results verify our method achieves state-of-the-art (SOTA) results in video-to-4D. In addition, the experiments on a real-world scenario dataset demonstrate its effectiveness on the 4D scene. Our code will be publicly available.
• 摘要：最近，从视频中生成动态 3D 对象取得了令人印象深刻的效果。现有方法直接使用帧中的整个信息来优化高斯分布。然而，当帧内的动态区域与静态区域交织在一起时，特别是当静态区域占很大比例时，现有方法往往会忽略动态区域中的信息，并且容易在静态区域上过度拟合。这会导致产生纹理模糊的结果。我们认为将动态-静态特征解耦以增强动态表示可以缓解这一问题。因此，我们提出了一个动态-静态特征解耦模块 (DSFD)。沿时间轴，它将当前帧特征中与参考帧特征具有显著差异的部分视为动态特征。相反，其余部分是静态特征。然后，我们获得由动态特征和当前帧特征驱动的解耦特征。此外，为了进一步从不同角度增强解耦特征的动态表示并确保准确的运动预测，我们设计了一个时空相似性融合模块 (TSSF)。沿空间轴，它自适应地选择动态区域的相似信息。基于上述内容，我们构建了一种新方法 DS4D。实验结果验证了我们的方法在视频转 4D 方面取得了最佳 (SOTA) 效果。此外，在真实场景数据集上进行的实验证明了其在 4D 场景中的有效性。我们的代码将公开发布。

Title: Training-Free Restoration of Pruned Neural Networks

• Authors: Keonho Lee, Minsoo Kim, Dong-Wan Choi
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08474
• Pdf URL: https://arxiv.org/pdf/2502.08474
• Copy Paste: [[2502.08474]] Training-Free Restoration of Pruned Neural Networks(https://arxiv.org/abs/2502.08474)
• Keywords: restoration
• Abstract: Although network pruning has been highly popularized to compress deep neural networks, its resulting accuracy heavily depends on a fine-tuning process that is often computationally expensive and requires the original data. However, this may not be the case in real-world scenarios, and hence a few recent works attempt to restore pruned networks without any expensive retraining process. Their strong assumption is that every neuron being pruned can be replaced with another one quite similar to it, but unfortunately this does not hold in many neural networks, where the similarity between neurons is extremely low in some layers. In this article, we propose a more rigorous and robust method of restoring pruned networks in a fine-tuning free and data-free manner, called LBYL (Leave Before You Leave). LBYL significantly relaxes the aforementioned assumption in a way that each pruned neuron leaves its pieces of information to as many preserved neurons as possible and thereby multiple neurons together obtain a more robust approximation to the original output of the neuron who just left. Our method is based on a theoretical analysis on how to formulate the reconstruction error between the original network and its approximation, which nicely leads to a closed form solution for our derived loss function. Through the extensive experiments, LBYL is confirmed to be indeed more effective to approximate the original network and consequently able to achieve higher accuracy for restored networks, compared to the recent approaches exploiting the similarity between two neurons. The very first version of this work, which contains major technical and theoretical components, was submitted to NeurIPS 2021 and ICML 2022.
• 摘要：尽管网络修剪已广泛用于压缩深度神经网络，但其最终准确性在很大程度上取决于微调过程，而微调过程通常计算量大且需要原始数据。然而，在现实世界中情况可能并非如此，因此最近有几篇论文试图在不进行任何昂贵的再训练过程的情况下恢复修剪后的网络。他们强有力的假设是，每个被修剪的神经元都可以用另一个非常相似的神经元替换，但不幸的是，这在许多神经网络中并不成立，因为在某些层中，神经元之间的相似性极低。在本文中，我们提出了一种更严格、更稳健的方法，以无需微调和无需数据的方式恢复修剪后的网络，称为 LBYL（离开前留下）。LBYL 大大放宽了上述假设，每个修剪后的神经元将其信息留给尽可能多的保留神经元，从而多个神经元共同获得对刚刚离开的神经元的原始输出的更稳健的近似值。我们的方法基于理论分析，即如何计算原始网络与其近似值之间的重建误差，从而很好地为我们导出的损失函数得出闭式解。通过大量实验，与最近利用两个神经元之间相似性的方法相比，LBYL 确实更有效地近似原始网络，从而能够为恢复的网络实现更高的精度。这项工作的第一个版本包含主要的技术和理论内容，已提交给 NeurIPS 2021 和 ICML 2022。

Title: One-Shot Federated Learning with Classifier-Free Diffusion Models

• Authors: Obaidullah Zaland, Shutong Jin, Florian T. Pokorny, Monowar Bhuyan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08488
• Pdf URL: https://arxiv.org/pdf/2502.08488
• Copy Paste: [[2502.08488]] One-Shot Federated Learning with Classifier-Free Diffusion Models(https://arxiv.org/abs/2502.08488)
• Keywords: generation
• Abstract: Federated learning (FL) enables collaborative learning without data centralization but introduces significant communication costs due to multiple communication rounds between clients and the server. One-shot federated learning (OSFL) addresses this by forming a global model with a single communication round, often relying on the server's model distillation or auxiliary dataset generation - often through pre-trained diffusion models (DMs). Existing DM-assisted OSFL methods, however, typically employ classifier-guided DMs, which require training auxiliary classifier models at each client, introducing additional computation overhead. This work introduces OSCAR (One-Shot Federated Learning with Classifier-Free Diffusion Models), a novel OSFL approach that eliminates the need for auxiliary models. OSCAR uses foundation models to devise category-specific data representations at each client, seamlessly integrated into a classifier-free diffusion model pipeline for server-side data generation. OSCAR is a simple yet cost-effective OSFL approach that outperforms the state-of-the-art on four benchmarking datasets while reducing the communication load by at least 99%.
• 摘要：联邦学习 (FL) 无需数据集中化即可实现协作学习，但由于客户端和服务器之间需要进行多轮通信，因此会产生大量通信成本。一次性联邦学习 (OSFL) 通过单轮通信形成全局模型来解决此问题，通常依赖于服务器的模型提炼或辅助数据集生成 - 通常通过预先训练的扩散模型 (DM)。然而，现有的 DM 辅助 OSFL 方法通常采用分类器引导的 DM，这需要在每个客户端上训练辅助分类器模型，从而引入额外的计算开销。这项工作引入了 OSCAR（具有无分类器扩散模型的一次性联邦学习），这是一种新颖的 OSFL 方法，无需辅助模型。OSCAR 使用基础模型在每个客户端上设计特定于类别的数据表示，并无缝集成到无分类器扩散模型管道中以生成服务器端数据。OSCAR 是一种简单但经济高效的 OSFL 方法，在四个基准数据集上的表现优于最先进的方法，同时将通信负载减少了至少 99%。

Title: The Paradox of Stochasticity: Limited Creativity and Computational Decoupling in Temperature-Varied LLM Outputs of Structured Fictional Data

• Authors: Evgenii Evstafev
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08515
• Pdf URL: https://arxiv.org/pdf/2502.08515
• Copy Paste: [[2502.08515]] The Paradox of Stochasticity: Limited Creativity and Computational Decoupling in Temperature-Varied LLM Outputs of Structured Fictional Data(https://arxiv.org/abs/2502.08515)
• Keywords: generation
• Abstract: This study examines how temperature settings and model architectures affect the generation of structured fictional data (names, birthdates) across three large language models (LLMs): llama3.1:8b, deepseek-r1:8b, and mistral:latest. By systematically testing temperature values from 0.0 to 1.0 in increments of 0.1, we conducted 330 trials yielding 889 structured entities, validated for syntactic consistency. Key findings reveal that model architecture significantly influences computational efficiency, with mistral:latest and llama3.1:8b processing data 8x faster than deepseek-r1:8b. Contrary to expectations, temperature showed no correlation with processing time, challenging assumptions about stochastic sampling costs. Output diversity remained limited, as models consistently defaulted to common name archetypes (e.g., 'John Doe' and 'Jane Smith') across all temperatures, though rare names clustered at intermediate values (0.3-0.7). These results demonstrate that architectural optimizations, rather than temperature adjustments, dominate performance in structured generation tasks. The findings emphasize prioritizing model selection over hyperparameter tuning for efficiency and suggest explicit diversity constraints are necessary to mitigate default output biases in synthetic data pipelines.
• 摘要：本研究考察了温度设置和模型架构如何影响三种大型语言模型 (LLM) 中结构化虚构数据（姓名、出生日期）的生成：llama3.1:8b、deepseek-r1:8b 和 mistral:latest。通过系统地测试从 0.0 到 1.0 的温度值（增量为 0.1），我们进行了 330 次试验，产生了 889 个结构化实体，并验证了句法一致性。主要发现表明，模型架构显著影响计算效率，mistral:latest 和 llama3.1:8b 处理数据的速度比 deepseek-r1:8b 快 8 倍。与预期相反，温度与处理时间没有相关性，这对随机采样成本的假设提出了挑战。输出多样性仍然有限，因为模型在所有温度下始终默认为常见名称原型（例如“John Doe”和“Jane Smith”），尽管罕见名称聚集在中间值（0.3-0.7）。这些结果表明，在结构化生成任务中，架构优化而非温度调整才是决定性能的主要因素。研究结果强调，为了提高效率，模型选择应优先于超参数调整，并表明明确的多样性约束对于缓解合成数据管道中的默认输出偏差是必要的。

Title: FedMHO: Heterogeneous One-Shot Federated Learning Towards Resource-Constrained Edge Devices

• Authors: Dezhong Yao, Yuexin Shi, Tongtong Liu, Zhiqiang Xu
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2502.08518
• Pdf URL: https://arxiv.org/pdf/2502.08518
• Copy Paste: [[2502.08518]] FedMHO: Heterogeneous One-Shot Federated Learning Towards Resource-Constrained Edge Devices(https://arxiv.org/abs/2502.08518)
• Keywords: generation, generative
• Abstract: Federated Learning (FL) is increasingly adopted in edge computing scenarios, where a large number of heterogeneous clients operate under constrained or sufficient resources. The iterative training process in conventional FL introduces significant computation and communication overhead, which is unfriendly for resource-constrained edge devices. One-shot FL has emerged as a promising approach to mitigate communication overhead, and model-heterogeneous FL solves the problem of diverse computing resources across clients. However, existing methods face challenges in effectively managing model-heterogeneous one-shot FL, often leading to unsatisfactory global model performance or reliance on auxiliary datasets. To address these challenges, we propose a novel FL framework named FedMHO, which leverages deep classification models on resource-sufficient clients and lightweight generative models on resource-constrained devices. On the server side, FedMHO involves a two-stage process that includes data generation and knowledge fusion. Furthermore, we introduce FedMHO-MD and FedMHO-SD to mitigate the knowledge-forgetting problem during the knowledge fusion stage, and an unsupervised data optimization solution to improve the quality of synthetic samples. Comprehensive experiments demonstrate the effectiveness of our methods, as they outperform state-of-the-art baselines in various experimental setups.
• 摘要：联邦学习 (FL) 在边缘计算场景中被越来越多地采用，在这种场景中，大量异构客户端在资源受限或充足的情况下运行。传统 FL 中的迭代训练过程引入了大量的计算和通信开销，这对于资源受限的边缘设备来说是不友好的。一次性 FL 已经成为一种有前途的减轻通信开销的方法，而模型异构 FL 解决了跨客户端计算资源多样化的问题。然而，现有方法在有效管理模型异构的一次性 FL 方面面临挑战，通常导致全局模型性能不令人满意或对辅助数据集的依赖。为了应对这些挑战，我们提出了一个名为 FedMHO 的新型 FL 框架，它利用资源充足的客户端上的深度分类模型和资源受限设备上的轻量级生成模型。在服务器端，FedMHO 涉及一个包括数据生成和知识融合的两阶段过程。此外，我们引入了 FedMHO-MD 和 FedMHO-SD 来缓解知识融合阶段的知识遗忘问题，并引入了无监督数据优化解决方案来提高合成样本的质量。全面的实验证明了我们方法的有效性，因为它们在各种实验设置中都优于最先进的基线。

Title: A Survey on Image Quality Assessment: Insights, Analysis, and Future Outlook

• Authors: Chengqian Ma, Zhengyi Shi, Zhiqiang Lu, Shenghao Xie, Fei Chao, Yao Sui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08540
• Pdf URL: https://arxiv.org/pdf/2502.08540
• Copy Paste: [[2502.08540]] A Survey on Image Quality Assessment: Insights, Analysis, and Future Outlook(https://arxiv.org/abs/2502.08540)
• Keywords: quality assessment
• Abstract: Image quality assessment (IQA) represents a pivotal challenge in image-focused technologies, significantly influencing the advancement trajectory of image processing and computer vision. Recently, IQA has witnessed a notable surge in innovative research efforts, driven by the emergence of novel architectural paradigms and sophisticated computational techniques. This survey delivers an extensive analysis of contemporary IQA methodologies, organized according to their application scenarios, serving as a beneficial reference for both beginners and experienced researchers. We analyze the advantages and limitations of current approaches and suggest potential future research pathways. The survey encompasses both general and specific IQA methodologies, including conventional statistical measures, machine learning techniques, and cutting-edge deep learning models such as convolutional neural networks (CNNs) and Transformer models. The analysis within this survey highlights the necessity for distortion-specific IQA methods tailored to various application scenarios, emphasizing the significance of practicality, interpretability, and ease of implementation in future developments.
• 摘要：图像质量评估 (IQA) 是图像技术中的关键挑战，对图像处理和计算机视觉的发展轨迹有重大影响。最近，在新型架构范式和复杂计算技术的推动下，IQA 的创新研究工作出现了显著增长。本综述对当代 IQA 方法进行了广泛的分析，并根据其应用场景进行了组织，为初学者和经验丰富的研究人员提供了有益的参考。我们分析了当前方法的优点和局限性，并提出了未来的潜在研究途径。本综述涵盖了一般和特定的 IQA 方法，包括传统的统计方法、机器学习技术和前沿的深度学习模型，如卷积神经网络 (CNN) 和 Transformer 模型。本综述中的分析强调了针对各种应用场景定制针对失真的 IQA 方法的必要性，强调了实用性、可解释性和易于实施在未来发展中的重要性。

Title: Human-Centric Foundation Models: Perception, Generation and Agentic Modeling

• Authors: Shixiang Tang, Yizhou Wang, Lu Chen, Yuan Wang, Sida Peng, Dan Xu, Wanli Ouyang
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2502.08556
• Pdf URL: https://arxiv.org/pdf/2502.08556
• Copy Paste: [[2502.08556]] Human-Centric Foundation Models: Perception, Generation and Agentic Modeling(https://arxiv.org/abs/2502.08556)
• Keywords: generation
• Abstract: Human understanding and generation are critical for modeling digital humans and humanoid embodiments. Recently, Human-centric Foundation Models (HcFMs) inspired by the success of generalist models, such as large language and vision models, have emerged to unify diverse human-centric tasks into a single framework, surpassing traditional task-specific approaches. In this survey, we present a comprehensive overview of HcFMs by proposing a taxonomy that categorizes current approaches into four groups: (1) Human-centric Perception Foundation Models that capture fine-grained features for multi-modal 2D and 3D understanding. (2) Human-centric AIGC Foundation Models that generate high-fidelity, diverse human-related content. (3) Unified Perception and Generation Models that integrate these capabilities to enhance both human understanding and synthesis. (4) Human-centric Agentic Foundation Models that extend beyond perception and generation to learn human-like intelligence and interactive behaviors for humanoid embodied tasks. We review state-of-the-art techniques, discuss emerging challenges and future research directions. This survey aims to serve as a roadmap for researchers and practitioners working towards more robust, versatile, and intelligent digital human and embodiments modeling.
• 摘要：人类理解和生成对于数字人类和人形化身建模至关重要。最近，受通用模型（例如大型语言和视觉模型）成功的启发，以人为中心的基础模型 (HcFM) 应运而生，将各种以人为中心的任务统一到一个框架中，超越了传统的特定于任务的方法。在本调查中，我们通过提出一种分类法将当前方法分为四类，全面概述了 HcFM：(1) 以人为中心的感知基础模型，可捕获细粒度特征以实现多模态 2D 和 3D 理解。(2) 以人为中心的 AIGC 基础模型，可生成高保真、多样化的人类相关内容。(3) 统一感知和生成模型，集成这些功能以增强人类的理解和综合。(4) 以人为中心的代理基础模型，超越感知和生成，学习类似人类的智能和人形化身任务的交互行为。我们回顾了最先进的技术，讨论了新出现的挑战和未来的研究方向。此项调查旨在为致力于更为稳健、多功能和智能的数字人类和具体化建模的研究人员和从业人员提供路线图。

Title: Ultrasound Image Generation using Latent Diffusion Models

• Authors: Benoit Freiche, Anthony El-Khoury, Ali Nasiri-Sarvi, Mahdi S. Hosseini, Damien Garcia, Adrian Basarab, Mathieu Boily, Hassan Rivaz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08580
• Pdf URL: https://arxiv.org/pdf/2502.08580
• Copy Paste: [[2502.08580]] Ultrasound Image Generation using Latent Diffusion Models(https://arxiv.org/abs/2502.08580)
• Keywords: generation
• Abstract: Diffusion models for image generation have been a subject of increasing interest due to their ability to generate diverse, high-quality images. Image generation has immense potential in medical imaging because open-source medical images are difficult to obtain compared to natural images, especially for rare conditions. The generated images can be used later to train classification and segmentation models. In this paper, we propose simulating realistic ultrasound (US) images by successive fine-tuning of large diffusion models on different publicly available databases. To do so, we fine-tuned Stable Diffusion, a state-of-the-art latent diffusion model, on BUSI (Breast US Images) an ultrasound breast image dataset. We successfully generated high-quality US images of the breast using simple prompts that specify the organ and pathology, which appeared realistic to three experienced US scientists and a US radiologist. Additionally, we provided user control by conditioning the model with segmentations through ControlNet. We will release the source code at this http URL to allow fast US image generation to the scientific community.
• 摘要：用于图像生成的扩散模型因其能够生成多样化、高质量的图像而受到越来越多的关注。图像生成在医学成像中具有巨大的潜力，因为与自然图像相比，开源医学图像很难获得，尤其是在罕见情况下。生成的图像稍后可用于训练分类和分割模型。在本文中，我们建议通过在不同的公开数据库上连续微调大型扩散模型来模拟逼真的超声 (US) 图像。为此，我们在超声乳房图像数据集 BUSI (Breast US Images) 上微调了最先进的潜在扩散模型稳定扩散。我们使用指定器官和病理的简单提示成功生成了高质量的乳房超声图像，这些图像在三位经验丰富的美国科学家和一位美国放射科医生看来很逼真。此外，我们通过 ControlNet 使用分割调节模型来提供用户控制。我们将在此 http URL 发布源代码，以便向科学界快速生成 US 图像。

Title: Light-A-Video: Training-free Video Relighting via Progressive Light Fusion

• Authors: Yujie Zhou, Jiazi Bu, Pengyang Ling, Pan Zhang, Tong Wu, Qidong Huang, Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Anyi Rao, Jiaqi Wang, Li Niu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08590
• Pdf URL: https://arxiv.org/pdf/2502.08590
• Copy Paste: [[2502.08590]] Light-A-Video: Training-free Video Relighting via Progressive Light Fusion(https://arxiv.org/abs/2502.08590)
• Keywords: generation
• Abstract: Recent advancements in image relighting models, driven by large-scale datasets and pre-trained diffusion models, have enabled the imposition of consistent lighting. However, video relighting still lags, primarily due to the excessive training costs and the scarcity of diverse, high-quality video relighting datasets. A simple application of image relighting models on a frame-by-frame basis leads to several issues: lighting source inconsistency and relighted appearance inconsistency, resulting in flickers in the generated videos. In this work, we propose Light-A-Video, a training-free approach to achieve temporally smooth video relighting. Adapted from image relighting models, Light-A-Video introduces two key techniques to enhance lighting consistency. First, we design a Consistent Light Attention (CLA) module, which enhances cross-frame interactions within the self-attention layers to stabilize the generation of the background lighting source. Second, leveraging the physical principle of light transport independence, we apply linear blending between the source video's appearance and the relighted appearance, using a Progressive Light Fusion (PLF) strategy to ensure smooth temporal transitions in illumination. Experiments show that Light-A-Video improves the temporal consistency of relighted video while maintaining the image quality, ensuring coherent lighting transitions across frames. Project page: this https URL.
• 摘要：在大型数据集和预训练扩散模型的推动下，图像重新照明模型的最新进展使得能够实施一致的照明。然而，视频重新照明仍然滞后，主要是因为训练成本过高，以及缺乏多样化、高质量的视频重新照明数据集。逐帧简单地应用图像重新照明模型会导致几个问题：光源不一致和重新照明外观不一致，从而导致生成的视频闪烁。在这项工作中，我们提出了 Light-A-Video，这是一种实现时间平滑的视频重新照明的无需训练的方法。Light-A-Video 改编自图像重新照明模型，引入了两种增强照明一致性的关键技术。首先，我们设计了一个一致的光注意 (CLA) 模块，它增强了自注意层内的跨帧交互，以稳定背景光源的生成。其次，利用光传输独立性的物理原理，我们在源视频的外观和重新点亮的外观之间应用线性混合，使用渐进式光融合 (PLF) 策略来确保照明的平滑时间过渡。实验表明，Light-A-Video 提高了重新点亮的视频的时间一致性，同时保持了图像质量，确保了帧间一致的照明过渡。项目页面：此 https URL。

Title: Enhancing Diffusion Models Efficiency by Disentangling Total-Variance and Signal-to-Noise Ratio

• Authors: Khaled Kahouli, Winfried Ripken, Stefan Gugler, Oliver T. Unke, Klaus-Robert Müller, Shinichi Nakajima
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.08598
• Pdf URL: https://arxiv.org/pdf/2502.08598
• Copy Paste: [[2502.08598]] Enhancing Diffusion Models Efficiency by Disentangling Total-Variance and Signal-to-Noise Ratio(https://arxiv.org/abs/2502.08598)
• Keywords: generation
• Abstract: The long sampling time of diffusion models remains a significant bottleneck, which can be mitigated by reducing the number of diffusion time steps. However, the quality of samples with fewer steps is highly dependent on the noise schedule, i.e., the specific manner in which noise is introduced and the signal is reduced at each step. Although prior work has improved upon the original variance-preserving and variance-exploding schedules, these approaches $\textit{passively}$ adjust the total variance, without direct control over it. In this work, we propose a novel total-variance/signal-to-noise-ratio disentangled (TV/SNR) framework, where TV and SNR can be controlled independently. Our approach reveals that different existing schedules, where the TV explodes exponentially, can be $\textit{improved}$ by setting a constant TV schedule while preserving the same SNR schedule. Furthermore, generalizing the SNR schedule of the optimal transport flow matching significantly improves the performance in molecular structure generation, achieving few step generation of stable molecules. A similar tendency is observed in image generation, where our approach with a uniform diffusion time grid performs comparably to the highly tailored EDM sampler.
• 摘要：扩散模型的长采样时间仍然是一个重大瓶颈，可以通过减少扩散时间步骤的数量来缓解。然而，步骤较少的样本的质量在很大程度上取决于噪声时间表，即在每个步骤中引入噪声和减少信号的具体方式。虽然先前的工作已经改进了原始的方差保持和方差爆炸时间表，但这些方法只是被动地调整总方差，而不是直接控制它。在这项工作中，我们提出了一种新颖的总方差/信噪比解缠结 (TV/SNR) 框架，其中 TV 和 SNR 可以独立控制。我们的方法表明，可以通过设置恒定的 TV 时间表同时保留相同的 SNR 时间表来改进不同的现有时间表，其中 TV 呈指数爆炸。此外，推广最佳传输流匹配的 SNR 时间表可显著提高分子结构生成的性能，实现稳定分子的少步生成。在图像生成中也观察到了类似的趋势，其中我们采用均匀扩散时间网格的方法的性能与高度定制的 EDM 采样器相当。

Title: CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation

• Authors: Qinghe Wang, Yawen Luo, Xiaoyu Shi, Xu Jia, Huchuan Lu, Tianfan Xue, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08639
• Pdf URL: https://arxiv.org/pdf/2502.08639
• Copy Paste: [[2502.08639]] CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation(https://arxiv.org/abs/2502.08639)
• Keywords: generation
• Abstract: In this work, we present CineMaster, a novel framework for 3D-aware and controllable text-to-video generation. Our goal is to empower users with comparable controllability as professional film directors: precise placement of objects within the scene, flexible manipulation of both objects and camera in 3D space, and intuitive layout control over the rendered frames. To achieve this, CineMaster operates in two stages. In the first stage, we design an interactive workflow that allows users to intuitively construct 3D-aware conditional signals by positioning object bounding boxes and defining camera movements within the 3D space. In the second stage, these control signals--comprising rendered depth maps, camera trajectories and object class labels--serve as the guidance for a text-to-video diffusion model, ensuring to generate the user-intended video content. Furthermore, to overcome the scarcity of in-the-wild datasets with 3D object motion and camera pose annotations, we carefully establish an automated data annotation pipeline that extracts 3D bounding boxes and camera trajectories from large-scale video data. Extensive qualitative and quantitative experiments demonstrate that CineMaster significantly outperforms existing methods and implements prominent 3D-aware text-to-video generation. Project page: this https URL.
• 摘要：在这项工作中，我们提出了 CineMaster，一种用于 3D 感知和可控文本到视频生成的新框架。我们的目标是让用户拥有与专业电影导演相当的可控性：在场景中精确放置物体、在 3D 空间中灵活操纵物体和相机，以及对渲染帧进行直观的布局控制。为了实现这一目标，CineMaster 分两个阶段运行。在第一阶段，我们设计了一个交互式工作流程，允许用户通过定位对象边界框和定义 3D 空间内的相机运动来直观地构建 3D 感知条件信号。在第二阶段，这些控制信号（包括渲染的深度图、相机轨迹和对象类别标签）作为文本到视频扩散模型的指导，确保生成用户想要的视频内容。此外，为了克服具有 3D 对象运动和相机姿势注释的野外数据集的稀缺性，我们精心建立了一个自动数据注释管道，从大规模视频数据中提取 3D 边界框和相机轨迹。大量定性和定量实验表明，CineMaster 明显优于现有方法，并实现了卓越的 3D 感知文本到视频生成。项目页面：此 https URL。

Title: SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation

• Authors: Ellie Arar, Yarden Frenkel, Daniel Cohen-Or, Ariel Shamir, Yael Vinker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08642
• Pdf URL: https://arxiv.org/pdf/2502.08642
• Copy Paste: [[2502.08642]] SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation(https://arxiv.org/abs/2502.08642)
• Keywords: generation
• Abstract: Recent advancements in large vision-language models have enabled highly expressive and diverse vector sketch generation. However, state-of-the-art methods rely on a time-consuming optimization process involving repeated feedback from a pretrained model to determine stroke placement. Consequently, despite producing impressive sketches, these methods are limited in practical applications. In this work, we introduce SwiftSketch, a diffusion model for image-conditioned vector sketch generation that can produce high-quality sketches in less than a second. SwiftSketch operates by progressively denoising stroke control points sampled from a Gaussian distribution. Its transformer-decoder architecture is designed to effectively handle the discrete nature of vector representation and capture the inherent global dependencies between strokes. To train SwiftSketch, we construct a synthetic dataset of image-sketch pairs, addressing the limitations of existing sketch datasets, which are often created by non-artists and lack professional quality. For generating these synthetic sketches, we introduce ControlSketch, a method that enhances SDS-based techniques by incorporating precise spatial control through a depth-aware ControlNet. We demonstrate that SwiftSketch generalizes across diverse concepts, efficiently producing sketches that combine high fidelity with a natural and visually appealing style.
• 摘要：大型视觉语言模型的最新进展使得矢量草图生成具有高度表现力和多样性。然而，最先进的方法依赖于耗时的优化过程，该过程涉及来自预训练模型的重复反馈以确定笔划位置。因此，尽管这些方法可以生成令人印象深刻的草图，但它们在实际应用中受到限制。在这项工作中，我们引入了 SwiftSketch，这是一种用于图像条件矢量草图生成的扩散模型，可以在不到一秒的时间内生成高质量的草图。SwiftSketch 通过逐步去噪从高斯分布中采样的笔划控制点来运行。其转换器-解码器架构旨在有效处理矢量表示的离散性质并捕获笔划之间固有的全局依赖关系。为了训练 SwiftSketch，我们构建了一个由图像-草图对组成的合成数据集，解决了现有草图数据集的局限性，这些数据集通常由非艺术家创建并且缺乏专业质量。为了生成这些合成草图，我们引入了 ControlSketch，这是一种通过深度感知 ControlNet 结合精确的空间控制来增强基于 SDS 的技术的方法。我们证明 SwiftSketch 可以跨不同概念进行概括，高效地生成兼具高保真度和自然、视觉吸引力风格的草图。