2025-02-10

Title: On-device Sora: Enabling Diffusion-Based Text-to-Video Generation for Mobile Devices

• Authors: Bosung Kim, Kyuhwan Lee, Isu Jeong, Jungmin Cheon, Yeojin Lee, Seulki Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04363
• Pdf URL: https://arxiv.org/pdf/2502.04363
• Copy Paste: [[2502.04363]] On-device Sora: Enabling Diffusion-Based Text-to-Video Generation for Mobile Devices(https://arxiv.org/abs/2502.04363)
• Keywords: generation, generative
• Abstract: We present On-device Sora, a first pioneering solution for diffusion-based on-device text-to-video generation that operates efficiently on smartphone-grade devices. Building on Open-Sora, On-device Sora applies three novel techniques to address the challenges of diffusion-based text-to-video generation on computation- and memory-limited mobile devices. First, Linear Proportional Leap (LPL) reduces the excessive denoising steps required in video diffusion through an efficient leap-based approach. Second, Temporal Dimension Token Merging (TDTM) minimizes intensive token-processing computation in attention layers by merging consecutive tokens along the temporal dimension. Third, Concurrent Inference with Dynamic Loading (CI-DL) dynamically partitions large models into smaller blocks and loads them into memory for concurrent model inference, effectively addressing the challenges of limited device memory. We implement On-device Sora on the iPhone 15 Pro, and the experimental evaluations demonstrate that it is capable of generating high-quality videos on the device, comparable to those produced by Open-Sora running on high-end GPUs. These results show that On-device Sora enables efficient and high-quality video generation on resource-constrained mobile devices, expanding accessibility, ensuring user privacy, reducing dependence on cloud infrastructure, and lowering associated costs. We envision the proposed On-device Sora as a significant first step toward democratizing state-of-the-art generative technologies, enabling video generation capabilities on commodity mobile and embedded devices. The code implementation is publicly available at an GitHub repository: this https URL.
• 摘要：我们提出了Device Sora，这是第一个开创性的解决方案，用于基于扩散的基于设备的文本到视频生成，可在智能手机级设备上有效运行。在开放式的基础上，开发项目SORA应用了三种新型技术，以应对基于扩散的文本到视频生成在计算和内存限制的移动设备上的挑战。首先，线性比例LEAP（LPL）通过有效的基于LEAP的方法来减少视频扩散所需的过度降解步骤。其次，时间维令牌合并（TDTM）通过沿时间维度合并连续令牌来最大程度地减少注意力层中的密集令牌处理。第三，与动态加载（CI-DL）的同时推断将大型模型动态分配到较小的块中，并将它们加载到内存中以进行并发模型推理，从而有效地解决了有限的设备内存的挑战。我们在iPhone 15 Pro上实现了device Sora，实验评估表明，它能够在设备上生成高质量的视频，与在高端GPU上运行的开放式SORA生产的视频相媲美。这些结果表明，在资源约束的移动设备上，启用了device Sora可以在资源受限的移动设备上产生高效，高质量的视频生成，确保用户隐私，降低对云基础架构的依赖以及降低相关成本。我们将拟议的智商索拉（Sora）视为使最先进的生成技术民主化的重要第一步，从而在商品移动设备和嵌入式设备上实现了视频生成功能。代码实现可在GITHUB存储库中公开可用：此HTTPS URL。

Title: Lost in Edits? A $\lambda$-Compass for AIGC Provenance

• Authors: Wenhao You, Bryan Hooi, Yiwei Wang, Euijin Choo, Ming-Hsuan Yang, Junsong Yuan, Zi Huang, Yujun Cai
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.04364
• Pdf URL: https://arxiv.org/pdf/2502.04364
• Copy Paste: [[2502.04364]] Lost in Edits? A $\lambda$-Compass for AIGC Provenance(https://arxiv.org/abs/2502.04364)
• Keywords: generative
• Abstract: Recent advancements in diffusion models have driven the growth of text-guided image editing tools, enabling precise and iterative modifications of synthesized content. However, as these tools become increasingly accessible, they also introduce significant risks of misuse, emphasizing the critical need for robust attribution methods to ensure content authenticity and traceability. Despite the creative potential of such tools, they pose significant challenges for attribution, particularly in adversarial settings where edits can be layered to obscure an image's origins. We propose LambdaTracer, a novel latent-space attribution method that robustly identifies and differentiates authentic outputs from manipulated ones without requiring any modifications to generative or editing pipelines. By adaptively calibrating reconstruction losses, LambdaTracer remains effective across diverse iterative editing processes, whether automated through text-guided editing tools such as InstructPix2Pix and ControlNet or performed manually with editing software such as Adobe Photoshop. Extensive experiments reveal that our method consistently outperforms baseline approaches in distinguishing maliciously edited images, providing a practical solution to safeguard ownership, creativity, and credibility in the open, fast-evolving AI ecosystems.
• 摘要：扩散模型的最新进展推动了文本引导的图像编辑工具的增长，从而实现了合成内容的精确和迭代修改。但是，随着这些工具变得越来越易于​​访问，它们还引入了滥用的重大风险，强调了对可靠归因方法的关键需求，以确保内容真实性和可追溯性。尽管这种工具具有创造力，但它们对归因构成了重大挑战，尤其是在可以分层以模糊图像起源的对抗环境中。我们提出了LambDatracer，这是一种新型潜在空间归因方法，可牢固地识别和区分真实的输出与受操纵的输出，而无需对生成或编辑管道进行任何修改。通过自适应校准重建损失，LambDatracer在各种迭代编辑过程中仍然有效，无论是通过文本指导的编辑工具（例如ConscessPix2Pix和ControlNet）自动化，还是使用Adobe Photoshop等编辑软件手动执行。广泛的实验表明，我们的方法在区分恶意编辑的图像方面始终优于基线方法，从而提供了一种实用的解决方案来保护公开，快速发展的AI生态系统中所有权，创造力和信誉。

Title: DILLEMA: Diffusion and Large Language Models for Multi-Modal Augmentation

• Authors: Luciano Baresi, Davide Yi Xian Hu, Muhammad Irfan Mas'udi, Giovanni Quattrocchi
• Subjects: cs.CV, cs.GR, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2502.04378
• Pdf URL: https://arxiv.org/pdf/2502.04378
• Copy Paste: [[2502.04378]] DILLEMA: Diffusion and Large Language Models for Multi-Modal Augmentation(https://arxiv.org/abs/2502.04378)
• Keywords: generative
• Abstract: Ensuring the robustness of deep learning models requires comprehensive and diverse testing. Existing approaches, often based on simple data augmentation techniques or generative adversarial networks, are limited in producing realistic and varied test cases. To address these limitations, we present a novel framework for testing vision neural networks that leverages Large Language Models and control-conditioned Diffusion Models to generate synthetic, high-fidelity test cases. Our approach begins by translating images into detailed textual descriptions using a captioning model, allowing the language model to identify modifiable aspects of the image and generate counterfactual descriptions. These descriptions are then used to produce new test images through a text-to-image diffusion process that preserves spatial consistency and maintains the critical elements of the scene. We demonstrate the effectiveness of our method using two datasets: ImageNet1K for image classification and SHIFT for semantic segmentation in autonomous driving. The results show that our approach can generate significant test cases that reveal weaknesses and improve the robustness of the model through targeted retraining. We conducted a human assessment using Mechanical Turk to validate the generated images. The responses from the participants confirmed, with high agreement among the voters, that our approach produces valid and realistic images.
• 摘要：确保深度学习模型的鲁棒性需要全面和多样化的测试。现有的方法通常基于简单的数据增强技术或生成对抗网络，在产生现实和多样化的测试用例方面受到限制。为了解决这些局限性，我们提出了一个新的框架，用于测试视觉神经网络，该框架利用大型语言模型和控制条件的扩散模型生成合成的高保真测试案例。我们的方法首先使用字幕模型将图像转换为详细的文本描述，从而允许语言模型识别图像的可修改方面并生成反事实描述。然后，这些描述用于通过文本对图像扩散过程产生新的测试图像，该过程保留空间一致性并保持场景的关键要素。我们使用两个数据集证明了我们方法的有效性：Imagenet1k用于图像分类和自主驾驶中语义分割的变化。结果表明，我们的方法可以产生重要的测试案例，以揭示弱点并通过靶向重新培训提高模型的鲁棒性。我们使用机械Turk进行了人类评估以验证生成的图像。参与者的回应证实，在选民之间达成了高度同意，我们的方法会产生有效和现实的图像。

Title: Towards Fair and Robust Face Parsing for Generative AI: A Multi-Objective Approach

• Authors: Sophia J. Abraham, Jonathan D. Hauenstein, Walter J. Scheirer
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04391
• Pdf URL: https://arxiv.org/pdf/2502.04391
• Copy Paste: [[2502.04391]] Towards Fair and Robust Face Parsing for Generative AI: A Multi-Objective Approach(https://arxiv.org/abs/2502.04391)
• Keywords: generation, generative
• Abstract: Face parsing is a fundamental task in computer vision, enabling applications such as identity verification, facial editing, and controllable image synthesis. However, existing face parsing models often lack fairness and robustness, leading to biased segmentation across demographic groups and errors under occlusions, noise, and domain shifts. These limitations affect downstream face synthesis, where segmentation biases can degrade generative model outputs. We propose a multi-objective learning framework that optimizes accuracy, fairness, and robustness in face parsing. Our approach introduces a homotopy-based loss function that dynamically adjusts the importance of these objectives during training. To evaluate its impact, we compare multi-objective and single-objective U-Net models in a GAN-based face synthesis pipeline (Pix2PixHD). Our results show that fairness-aware and robust segmentation improves photorealism and consistency in face generation. Additionally, we conduct preliminary experiments using ControlNet, a structured conditioning model for diffusion-based synthesis, to explore how segmentation quality influences guided image generation. Our findings demonstrate that multi-objective face parsing improves demographic consistency and robustness, leading to higher-quality GAN-based synthesis.
• 摘要：面部解析是计算机视觉中的一项基本任务，可以实现诸如身份验证，面部编辑和可控图像合成之类的应用程序。但是，现有的面部解析模型通常缺乏公平和鲁棒性，从而导致人口组群体之间的分割和障碍，噪声和域移动下的错误。这些局限性影响下游面部合成，其中分割偏差会降低生成模型输出。我们提出了一个多目标学习框架，该框架优化了面部解析的准确性，公平性和鲁棒性。我们的方法引入了基于同质损失的功能，该功能会在训练过程中动态调整这些目标的重要性。为了评估其影响，我们比较了基于GAN的面部合成管道（Pix2PixHD）中的多目标和单目标U-NET模型。我们的结果表明，公平意识和稳健的分割可改善面部产生的光真相和一致性。此外，我们使用ControlNet（用于基于扩散的合成的结构化条件模型）进行初步实验，以探索分割质量如何影响引导图像产生。我们的发现表明，多目标面对解析提高了人口统计学的一致性和鲁棒性，从而导致了高质量的基于GAN的合成。

Title: UniCP: A Unified Caching and Pruning Framework for Efficient Video Generation

• Authors: Wenzhang Sun, Qirui Hou, Donglin Di, Jiahui Yang, Yongjia Ma, Jianxun Cui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04393
• Pdf URL: https://arxiv.org/pdf/2502.04393
• Copy Paste: [[2502.04393]] UniCP: A Unified Caching and Pruning Framework for Efficient Video Generation(https://arxiv.org/abs/2502.04393)
• Keywords: generation
• Abstract: Diffusion Transformers (DiT) excel in video generation but encounter significant computational challenges due to the quadratic complexity of attention. Notably, attention differences between adjacent diffusion steps follow a U-shaped pattern. Current methods leverage this property by caching attention blocks, however, they still struggle with sudden error spikes and large discrepancies. To address these issues, we propose UniCP a unified caching and pruning framework for efficient video generation. UniCP optimizes both temporal and spatial dimensions through. Error Aware Dynamic Cache Window (EDCW): Dynamically adjusts cache window sizes for different blocks at various timesteps, adapting to abrupt error changes. PCA based Slicing (PCAS) and Dynamic Weight Shift (DWS): PCAS prunes redundant attention components, and DWS integrates caching and pruning by enabling dynamic switching between pruned and cached outputs. By adjusting cache windows and pruning redundant components, UniCP enhances computational efficiency and maintains video detail fidelity. Experimental results show that UniCP outperforms existing methods in both performance and efficiency.
• 摘要：扩散变压器（DIT）在视频生成中出色，但由于注意力的二次复杂性而遇到了重大的计算挑战。值得注意的是，相邻扩散步骤之间的注意力差异遵循U形模式。当前的方法通过缓存注意力障碍来利用这一属性，但是，它们仍然在突然的错误峰值和较大的差异中挣扎。为了解决这些问题，我们建议UNICP一个统一的缓存和修剪框架，以进行有效的视频生成。 UNICP通过通过时间和空间维度来优化。错误了解动态缓存窗口（EDCW）：在各种时间步中，动态调整不同块的缓存窗口大小，以适应突然的错误更改。基于PCA的切片（PCA）和动态重量移位（DWS）：PCAS修剪冗余注意力组件，而DWS通过在修剪和cached输出之间启用动态切换来集成缓存和修剪。通过调整缓存窗口并修剪冗余组件，UNICP提高了计算效率并保持视频详细信息的保真度。实验结果表明，UNICP在性能和效率方面均优于现有方法。

Title: Illuminating Spaces: Deep Reinforcement Learning and Laser-Wall Partitioning for Architectural Layout Generation

• Authors: Reza Kakooee, Benjamin Dillenburger
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04407
• Pdf URL: https://arxiv.org/pdf/2502.04407
• Copy Paste: [[2502.04407]] Illuminating Spaces: Deep Reinforcement Learning and Laser-Wall Partitioning for Architectural Layout Generation(https://arxiv.org/abs/2502.04407)
• Keywords: generation, generative
• Abstract: Space layout design (SLD), occurring in the early stages of the design process, nonetheless influences both the functionality and aesthetics of the ultimate architectural outcome. The complexity of SLD necessitates innovative approaches to efficiently explore vast solution spaces. While image-based generative AI has emerged as a potential solution, they often rely on pixel-based space composition methods that lack intuitive representation of architectural processes. This paper leverages deep Reinforcement Learning (RL), as it offers a procedural approach that intuitively mimics the process of human designers. Effectively using RL for SLD requires an explorative space composing method to generate desirable design solutions. We introduce "laser-wall", a novel space partitioning method that conceptualizes walls as emitters of imaginary light beams to partition spaces. This approach bridges vector-based and pixel-based partitioning methods, offering both flexibility and exploratory power in generating diverse layouts. We present two planning strategies: one-shot planning, which generates entire layouts in a single pass, and dynamic planning, which allows for adaptive refinement by continuously transforming laser-walls. Additionally, we introduce on-light and off-light wall transformations for smooth and fast layout refinement, as well as identity-less and identity-full walls for versatile room assignment. We developed SpaceLayoutGym, an open-source OpenAI Gym compatible simulator for generating and evaluating space layouts. The RL agent processes the input design scenarios and generates solutions following a reward function that balances geometrical and topological requirements. Our results demonstrate that the RL-based laser-wall approach can generate diverse and functional space layouts that satisfy both geometric constraints and topological requirements and is architecturally intuitive.
• 摘要：空间布局设计（SLD）发生在设计过程的早期阶段，但仍影响最终建筑结果的功能和美学。 SLD的复杂性需要创新的方法来有效探索庞大的解决方案空间。尽管基于图像的生成AI已成为潜在的解决方案，但它们通常依赖于缺乏建筑过程直观表示的基于像素的空间组成方法。本文提供了深入的增强学习（RL），因为它提供了一种程序性方法，可以直观地模仿人类设计师的过程。有效地将RL用于SLD需要一种探索空间构成方法来生成理想的设计解决方案。我们介绍了“激光墙”，这是一种新型的空间分配方法，将墙壁概念化为虚构光束的发射器，以分配空间。这种方法桥接了基于向量的基于矢量和基于像素的分区方法，在产生各种布局方面具有灵活性和探索性。我们提出了两种计划策略：单发计划，该计划在单个通行证中生成了整个布局，并且动态计划，可以通过不断改变激光墙来进行自适应的改进。此外，我们介绍了光滑和浅色的墙面转换，以进行光滑，快速的布局改进，以及无身份和身份的墙壁，用于多功能房间分配。我们开发了Spacelayoutgym，这是一种开源OpenAI健身房兼容模拟器，用于生成和评估太空布局。 RL代理处理输入设计方案并遵循平衡几何和拓扑要求的奖励功能，生成解决方案。我们的结果表明，基于RL的激光壁方法可以生成各种和功能性的空间布局，从而满足几何约束和拓扑要求，并且在架构上是直观的。

Title: Decoder-Only LLMs are Better Controllers for Diffusion Models

• Authors: Ziyi Dong, Yao Xiao, Pengxu Wei, Liang Lin
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04412
• Pdf URL: https://arxiv.org/pdf/2502.04412
• Copy Paste: [[2502.04412]] Decoder-Only LLMs are Better Controllers for Diffusion Models(https://arxiv.org/abs/2502.04412)
• Keywords: generation
• Abstract: Groundbreaking advancements in text-to-image generation have recently been achieved with the emergence of diffusion models. These models exhibit a remarkable ability to generate highly artistic and intricately detailed images based on textual prompts. However, obtaining desired generation outcomes often necessitates repetitive trials of manipulating text prompts just like casting spells on a magic mirror, and the reason behind that is the limited capability of semantic understanding inherent in current image generation models. Specifically, existing diffusion models encode the text prompt input with a pre-trained encoder structure, which is usually trained on a limited number of image-caption pairs. The state-of-the-art large language models (LLMs) based on the decoder-only structure have shown a powerful semantic understanding capability as their architectures are more suitable for training on very large-scale unlabeled data. In this work, we propose to enhance text-to-image diffusion models by borrowing the strength of semantic understanding from large language models, and devise a simple yet effective adapter to allow the diffusion models to be compatible with the decoder-only structure. Meanwhile, we also provide a supporting theoretical analysis with various architectures (e.g., encoder-only, encoder-decoder, and decoder-only), and conduct extensive empirical evaluations to verify its effectiveness. The experimental results show that the enhanced models with our adapter module are superior to the stat-of-the-art models in terms of text-to-image generation quality and reliability.
• 摘要：由于扩散模型的出现，最近已经实现了文本到图像生成的突破性进步。这些模型具有非凡的能力，可以根据文本提示产生高度艺术和错综复杂的图像。但是，获得所需的生成成果通常需要重复的操纵文本提示的试验，就像在魔术镜上铸造法术一样，其原因是当前图像生成模型中固有的语义理解能力有限。具体而言，现有的扩散模型用预先训练的编码器结构编码文本提示输入，该结构通常在有限数量的图像符号对上进行训练。基于仅解码器结构的最先进的大语言模型（LLM）显示出强大的语义理解能力，因为它们的体系结构更适合在非常大规模的未标记数据上培训。在这项工作中，我们建议通过从大型语言模型中借用语义理解的强度来增强文本形象扩散模型，并设计一个简单而有效的适配器，以允许扩散模型与仅解码器的结构兼容。同时，我们还提供了支持各种体系结构（例如，仅编码，编码器编码器和仅解码器）的支持理论分析，并进行了广泛的经验评估以验证其有效性。实验结果表明，在文本到图像生成的质量和可靠性方面，使用适配器模块的增强模型优于最终模型。

Title: Understanding and Mitigating the Bias Inheritance in LLM-based Data Augmentation on Downstream Tasks

• Authors: Miaomiao Li, Hao Chen, Yang Wang, Tingyuan Zhu, Weijia Zhang, Kaijie Zhu, Kam-Fai Wong, Jindong Wang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04419
• Pdf URL: https://arxiv.org/pdf/2502.04419
• Copy Paste: [[2502.04419]] Understanding and Mitigating the Bias Inheritance in LLM-based Data Augmentation on Downstream Tasks(https://arxiv.org/abs/2502.04419)
• Keywords: generation
• Abstract: Generating synthetic datasets via large language models (LLMs) themselves has emerged as a promising approach to improve LLM performance. However, LLMs inherently reflect biases present in their training data, leading to a critical challenge: when these models generate synthetic data for training, they may propagate and amplify their inherent biases that can significantly impact model fairness and robustness on downstream tasks--a phenomenon we term bias inheritance. This work presents the first systematic investigation in understanding, analyzing, and mitigating bias inheritance. We study this problem by fine-tuning LLMs with a combined dataset consisting of original and LLM-augmented data, where bias ratio represents the proportion of augmented data. Through systematic experiments across 10 classification and generation tasks, we analyze how 6 different types of biases manifest at varying bias ratios. Our results reveal that bias inheritance has nuanced effects on downstream tasks, influencing both classification tasks and generation tasks differently. Then, our analysis identifies three key misalignment factors: misalignment of values, group data, and data distributions. Based on these insights, we propose three mitigation strategies: token-based, mask-based, and loss-based approaches. Experiments demonstrate that these strategies also work differently on various tasks and bias, indicating the substantial challenges to fully mitigate bias inheritance. We hope this work can provide valuable insights to the research of LLM data augmentation.
• 摘要：通过大语言模型（LLM）本身生成合成数据集已成为提高LLM性能的有前途的方法。但是，LLM固有地反映了其培训数据中存在的偏见，从而导致了一个关键的挑战：当这些模型生成培训的合成数据时，它们可能会传播并扩大其固有的偏见，从而对下游任务产生严重影响模型的公平性和稳健性 - 这一现象 - 这一现象我们称偏见的继承。这项工作介绍了在理解，分析和减轻偏差遗传方面进行的首次系统调查。我们通过通过由原始和LLM增强数据组成的合并数据集进行微调LLM来研究这个问题，其中偏差比表示增强数据的比例。通过在10个分类和生成任务的系统实验中，我们分析了6种不同类型的偏见如何以不同的偏差比表现出来。我们的结果表明，偏见继承对下游任务有细微的影响，对分类任务和生成任务的影响有所不同。然后，我们的分析确定了三个关键的未对准因素：值，组数据和数据分布的未对准。基于这些见解，我们提出了三种缓解策略：基于令牌的，基于面具和基于损失的方法。实验表明，这些策略在各种任务和偏见上也有所不同，表明完全减轻偏见的遗传面临的重大挑战。我们希望这项工作可以为LLM数据增强研究提供宝贵的见解。

Title: Training Language Models to Reason Efficiently

• Authors: Daman Arora, Andrea Zanette
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04463
• Pdf URL: https://arxiv.org/pdf/2502.04463
• Copy Paste: [[2502.04463]] Training Language Models to Reason Efficiently(https://arxiv.org/abs/2502.04463)
• Keywords: generation
• Abstract: Scaling model size and training data has led to great advances in the performance of Large Language Models (LLMs). However, the diminishing returns of this approach necessitate alternative methods to improve model capabilities, particularly in tasks requiring advanced reasoning. Large reasoning models, which leverage long chain-of-thoughts, bring unprecedented breakthroughs in problem-solving capabilities but at a substantial deployment cost associated to longer generations. Reducing inference costs is crucial for the economic feasibility, user experience, and environmental sustainability of these models. In this work, we propose to train large reasoning models to reason efficiently. More precisely, we use reinforcement learning (RL) to train reasoning models to dynamically allocate inference-time compute based on task complexity. Our method incentivizes models to minimize unnecessary computational overhead while maintaining accuracy, thereby achieving substantial efficiency gains. It enables the derivation of a family of reasoning models with varying efficiency levels, controlled via a single hyperparameter. Experiments on two open-weight large reasoning models demonstrate significant reductions in inference cost while preserving most of the accuracy.
• 摘要：缩放模型的规模和培训数据已导致大语言模型（LLM）的性能取得了长足进步。但是，这种方法的回报减少需要替代方法来提高模型功能，尤其是在需要高级推理的任务中。利用长长的思想链的大型推理模型为解决问题的能力带来了前所未有的突破，但与长代相关的大量部署成本。降低推理成本对于这些模型的经济可行性，用户体验和环境可持续性至关重要。在这项工作中，我们建议培训大型推理模型以有效地推理。更确切地说，我们使用加固学习（RL）来训练推理模型，以根据任务复杂性动态分配推理时间计算。我们的方法激励模型，以最大程度地减少不必要的计算开销，同时保持准确性，从而实现可观的效率提高。它可以通过单个高参数控制具有不同效率水平的推理模型家族。在两个开放重量大推理模型上进行的实验表明，推理成本显着降低，同时保留了大多数精度。

Title: FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks

• Authors: Luca Della Libera, Francesco Paissan, Cem Subakan, Mirco Ravanelli
• Subjects: cs.LG, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2502.04465
• Pdf URL: https://arxiv.org/pdf/2502.04465
• Copy Paste: [[2502.04465]] FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks(https://arxiv.org/abs/2502.04465)
• Keywords: generative
• Abstract: Large language models have revolutionized natural language processing through self-supervised pretraining on massive datasets. Inspired by this success, researchers have explored adapting these methods to speech by discretizing continuous audio into tokens using neural audio codecs. However, existing approaches face limitations, including high bitrates, the loss of either semantic or acoustic information, and the reliance on multi-codebook designs when trying to capture both, which increases architectural complexity for downstream tasks. To address these challenges, we introduce FocalCodec, an efficient low-bitrate codec based on focal modulation that utilizes a single binary codebook to compress speech between 0.16 and 0.65 kbps. FocalCodec delivers competitive performance in speech resynthesis and voice conversion at lower bitrates than the current state-of-the-art, while effectively handling multilingual speech and noisy environments. Evaluation on downstream tasks shows that FocalCodec successfully preserves sufficient semantic and acoustic information, while also being well-suited for generative modeling. Demo samples, code and checkpoints are available at this https URL.
• 摘要：大型语言模型通过在大规模数据集上的自我监督预处理进行了彻底改变自然语言处理。受到这一成功的启发，研究人员探索了通过将连续音频离散到使用神经音频编解码器中的令牌来调整这些方法来进行语音。但是，现有方法面临局限性，包括高比特率，语义或声学信息的丢失以及尝试捕获两者时对多编码书设计的依赖，这增加了下游任务的体系结构复杂性。为了应对这些挑战，我们介绍了基于焦点调制的有效的低焦油编解码器焦距，该焦点是利用单个二进制代码手册来压缩语音在0.16至0.65 kbps之间的。焦距在低比特率的语音重新合成和语音转换方面的竞争性能比当前的最新速度，同时有效地处理多语言语音和嘈杂的环境。对下游任务的评估表明，焦距成功保留了足够的语义和声学信息，同时也非常适合生成建模。此HTTPS URL可用演示样本，代码和检查点。

Title: Iterative Importance Fine-tuning of Diffusion Models

• Authors: Alexander Denker, Shreyas Padhy, Francisco Vargas, Johannes Hertrich
• Subjects: cs.LG, eess.IV, math.PR
• Abstract URL: https://arxiv.org/abs/2502.04468
• Pdf URL: https://arxiv.org/pdf/2502.04468
• Copy Paste: [[2502.04468]] Iterative Importance Fine-tuning of Diffusion Models(https://arxiv.org/abs/2502.04468)
• Keywords: generative
• Abstract: Diffusion models are an important tool for generative modelling, serving as effective priors in applications such as imaging and protein design. A key challenge in applying diffusion models for downstream tasks is efficiently sampling from resulting posterior distributions, which can be addressed using the $h$-transform. This work introduces a self-supervised algorithm for fine-tuning diffusion models by estimating the $h$-transform, enabling amortised conditional sampling. Our method iteratively refines the $h$-transform using a synthetic dataset resampled with path-based importance weights. We demonstrate the effectiveness of this framework on class-conditional sampling and reward fine-tuning for text-to-image diffusion models.
• 摘要：扩散模型是生成建模的重要工具，在成像和蛋白质设计等应用中充当有效的先验。将扩散模型应用于下游任务的一个关键挑战是有效地从结果后分布中进行采样，可以使用$ h $ - 转换来解决。这项工作通过估计$ h $转换，启用摊销条件采样，引入了一种用于微调扩散模型的自制算法。我们的方法使用基于路径的重要性权重重新采样的合成数据集来完善$ h $ - 转换。我们证明了该框架对文本到图像扩散模型的班级条件采样和奖励微调的有效性。

Title: Augmented Conditioning Is Enough For Effective Training Image Generation

• Authors: Jiahui Chen, Amy Zhang, Adriana Romero-Soriano
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.04475
• Pdf URL: https://arxiv.org/pdf/2502.04475
• Copy Paste: [[2502.04475]] Augmented Conditioning Is Enough For Effective Training Image Generation(https://arxiv.org/abs/2502.04475)
• Keywords: generation
• Abstract: Image generation abilities of text-to-image diffusion models have significantly advanced, yielding highly photo-realistic images from descriptive text and increasing the viability of leveraging synthetic images to train computer vision models. To serve as effective training data, generated images must be highly realistic while also sufficiently diverse within the support of the target data distribution. Yet, state-of-the-art conditional image generation models have been primarily optimized for creative applications, prioritizing image realism and prompt adherence over conditional diversity. In this paper, we investigate how to improve the diversity of generated images with the goal of increasing their effectiveness to train downstream image classification models, without fine-tuning the image generation model. We find that conditioning the generation process on an augmented real image and text prompt produces generations that serve as effective synthetic datasets for downstream training. Conditioning on real training images contextualizes the generation process to produce images that are in-domain with the real image distribution, while data augmentations introduce visual diversity that improves the performance of the downstream classifier. We validate augmentation-conditioning on a total of five established long-tail and few-shot image classification benchmarks and show that leveraging augmentations to condition the generation process results in consistent improvements over the state-of-the-art on the long-tailed benchmark and remarkable gains in extreme few-shot regimes of the remaining four benchmarks. These results constitute an important step towards effectively leveraging synthetic data for downstream training.
• 摘要：文本到图像扩散模型的图像生成能力具有显着提高，从描述性文本产生了高度的光真实图像，并提高了利用合成图像的可行性来训练计算机视觉模型。为了充当有效的培训数据，生成的图像必须高度现实，同时在支持目标数据分布的支持下也足够多样化。然而，最新的条件图像生成模型主要针对创意应用进行了优化，优先考虑图像现实主义并迅速遵守条件多样性。在本文中，我们研究了如何改善生成的图像的多样性，目的是提高其训练下游图像分类模型的有效性，而无需微调图像生成模型。我们发现，以增强的真实图像和文本提示为生成生成的生成后代，将生成过程调整为有效的合成数据集用于下游培训。在真实训练图像上进行条件的条件将生成过程与生成具有真实图像分布的图像的生成过程相关，而数据增强则引入了视觉多样性，从而改善了下游分类器的性能。我们验证了对五个已建立的长尾和少量图像分类基准的增强条件并在其余四个基准的极端枪击方面获得了显着的收益。这些结果构成了有效利用合成数据进行下游训练的重要一步。

Title: Fast Video Generation with Sliding Tile Attention

• Authors: Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhenghong Liu, Hao Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04507
• Pdf URL: https://arxiv.org/pdf/2502.04507
• Copy Paste: [[2502.04507]] Fast Video Generation with Sliding Tile Attention(https://arxiv.org/abs/2502.04507)
• Keywords: generation
• Abstract: Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art video generation, but suffer from prohibitive compute cost -- when generating just a 5-second 720P video, attention alone takes 800 out of 945 seconds of total inference time. This paper introduces sliding tile attention (STA) to address this challenge. STA leverages the observation that attention scores in pretrained video diffusion models predominantly concentrate within localized 3D windows. By sliding and attending over the local spatial-temporal region, STA eliminates redundancy from full attention. Unlike traditional token-wise sliding window attention (SWA), STA operates tile-by-tile with a novel hardware-aware sliding window design, preserving expressiveness while being hardware-efficient. With careful kernel-level optimizations, STA offers the first efficient 2D/3D sliding-window-like attention implementation, achieving 58.79% MFU. Precisely, STA accelerates attention by 2.8-17x over FlashAttention-2 (FA2) and 1.6-10x over FlashAttention-3 (FA3). On the leading video DiT, HunyuanVideo, STA reduces end-to-end latency from 945s (FA3) to 685s without quality degradation, requiring no training. Enabling finetuning further lowers latency to 268s with only a 0.09% drop on VBench.
• 摘要：带有3D全部注意力的扩散变压器（DITS），最先进的视频发电，但遭受了高度的计算成本 - 仅生成5秒的720p视频时，仅注意力含量为945秒，总推断时间就需要800 。本文介绍了滑动瓷砖的注意（STA），以应对这一挑战。 STA利用了这样的观察结果，即预处理的视频扩散模型中的注意力分数主要集中在局部3D窗口中。通过在局部时空区域滑动和参加，STA消除了全部关注的冗余。与传统的令牌滑动窗户的关注（SWA）不同，STA通过新颖的硬件滑动窗户设计逐个瓷砖，在硬件有效的同时保持表现力。通过仔细的内核级优化，STA提供了第一个有效的2D/3D滑动窗口般的注意力实现，可实现58.79％的MFU。确切地说，在Flashattention-2（FA2）（FA2）上，STA在Flashattention-3（FA3）（FA3）上加速了2.8-17x的注意力。在领先的视频DIT（Hunyuanvideo）上，STA将端到端的潜伏期从945（FA3）降低到685秒，而无需质量退化，不需要培训。实现Finetun的进一步降低了268S的潜伏期，Vbench下降了0.09％。

Title: Towards Cost-Effective Reward Guided Text Generation

• Authors: Ahmad Rashid, Ruotian Wu, Rongqi Fan, Hongliang Li, Agustinus Kristiadi, Pascal Poupart
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04517
• Pdf URL: https://arxiv.org/pdf/2502.04517
• Copy Paste: [[2502.04517]] Towards Cost-Effective Reward Guided Text Generation(https://arxiv.org/abs/2502.04517)
• Keywords: generation
• Abstract: Reward-guided text generation (RGTG) has emerged as a viable alternative to offline reinforcement learning from human feedback (RLHF). RGTG methods can align baseline language models to human preferences without further training like in standard RLHF methods. However, they rely on a reward model to score each candidate token generated by the language model at inference, incurring significant test-time overhead. Additionally, the reward model is usually only trained to score full sequences, which can lead to sub-optimal choices for partial sequences. In this work, we present a novel reward model architecture that is trained, using a Bradley-Terry loss, to prefer the optimal expansion of a sequence with just a \emph{single call} to the reward model at each step of the generation process. That is, a score for all possible candidate tokens is generated simultaneously, leading to efficient inference. We theoretically analyze various RGTG reward models and demonstrate that prior techniques prefer sub-optimal sequences compared to our method during inference. Empirically, our reward model leads to significantly faster inference than other RGTG methods. It requires fewer calls to the reward model and performs competitively compared to previous RGTG and offline RLHF methods.
• 摘要：奖励指导的文本生成（RGTG）已成为从人类反馈（RLHF）中学习的可行替代方案。 RGTG方法可以将基线语言模型与人类偏好保持一致，而无需像标准RLHF方法中的进一步培训。但是，他们依靠奖励模型来推断语言模型产生的每个候选代币，从而产生大量的测试时间开销。此外，奖励模型通常仅受过训练以得分完整序列，这可能会导致部分序列的亚最佳选择。在这项工作中，我们提出了一种新颖的奖励模型架构，该架构使用Bradley-Terry损失进行了训练，以更喜欢在生成过程的每个步骤中使用\ Emph {single call}的序列最佳扩展到奖励模型。也就是说，所有可能的候选令牌的分数是同时生成的，从而导致有效的推断。我们从理论上分析了各种RGTG奖励模型，并证明与推断期间的方法相比，先前的技术更喜欢亚最佳序列。从经验上讲，与其他RGTG方法相比，我们的奖励模型会导致明显更快的推理。与以前的RGTG和离线RLHF方法相比，它需要更少的奖励模型呼叫，并具有竞争力。

Title: Speeding up Speculative Decoding via Approximate Verification

• Authors: Meiyu Zhong, Noel Teku, Ravi Tandon
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2502.04557
• Pdf URL: https://arxiv.org/pdf/2502.04557
• Copy Paste: [[2502.04557]] Speeding up Speculative Decoding via Approximate Verification(https://arxiv.org/abs/2502.04557)
• Keywords: generation
• Abstract: Speculative Decoding (SD) is a recently proposed technique for faster inference using Large Language Models (LLMs). SD operates by using a smaller draft LLM for autoregressively generating a sequence of tokens and a larger target LLM for parallel verification to ensure statistical consistency. However, periodic parallel calls to the target LLM for verification prevent SD from achieving even lower latencies. We propose SPRINTER, which utilizes a low-complexity verifier trained to predict if tokens generated from a draft LLM would be accepted by the target LLM. By performing approximate sequential verification, SPRINTER does not require verification by the target LLM and is only invoked when a token is deemed unacceptable. This leads to reducing the number of calls to the larger LLM and can achieve further speedups. We present a theoretical analysis of SPRINTER, examining the statistical properties of the generated tokens, as well as the expected reduction in latency as a function of the verifier. We evaluate SPRINTER on several datasets and model pairs, demonstrating that approximate verification can still maintain high quality generation while further reducing latency. For instance, on Wiki-Summaries dataset, SPRINTER achieves a 1.7x latency speedup and requires 8.3x fewer flops relative to SD, while still generating high-quality responses when using GPT2-Small and GPT2-XL as draft/target models.
• 摘要：投机解码（SD）是最近提出的使用大语言模型（LLM）更快推断的技术。 SD通过使用较小的llm草案进行自动加工来生成一系列令牌和较大的目标LLM进行并行验证以确保统计一致性。但是，定期并行呼吁目标LLM进行验证，以防止SD达到较低的潜伏期。我们提出了Sprinter，它利用训练有素的低复杂性验证仪来预测目标LLM从LLM草案中产生的令牌。通过执行近似的顺序验证，Sprinter不需要目标LLM验证，并且仅当令牌被认为是不可接受的时才调用的。这导致减少对较大LLM的呼叫数量，并可以实现进一步的加速。我们提出了对短跑运动员的理论分析，研究了生成的令牌的统计特性，以及预期的延迟降低随验证者的函数。我们在几个数据集和模型对上评估了短跑运动员，表明近似验证仍然可以保持高质量的生成，同时进一步降低延迟。例如，在Wiki-Summaries数据集上，Sprinter实现了1.7倍的延迟速度，相对于SD而言，SPRINTER需要减少8.3倍，同时在使用GPT2-SMALL和GPT2-XL作为草稿/目标模型时仍会生成高质量的响应。

Title: Position-aware Automatic Circuit Discovery

• Authors: Tal Haklay, Hadas Orgad, David Bau, Aaron Mueller, Yonatan Belinkov
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04577
• Pdf URL: https://arxiv.org/pdf/2502.04577
• Copy Paste: [[2502.04577]] Position-aware Automatic Circuit Discovery(https://arxiv.org/abs/2502.04577)
• Keywords: generation
• Abstract: A widely used strategy to discover and understand language model mechanisms is circuit analysis. A circuit is a minimal subgraph of a model's computation graph that executes a specific task. We identify a gap in existing circuit discovery methods: they assume circuits are position-invariant, treating model components as equally relevant across input positions. This limits their ability to capture cross-positional interactions or mechanisms that vary across positions. To address this gap, we propose two improvements to incorporate positionality into circuits, even on tasks containing variable-length examples. First, we extend edge attribution patching, a gradient-based method for circuit discovery, to differentiate between token positions. Second, we introduce the concept of a dataset schema, which defines token spans with similar semantics across examples, enabling position-aware circuit discovery in datasets with variable length examples. We additionally develop an automated pipeline for schema generation and application using large language models. Our approach enables fully automated discovery of position-sensitive circuits, yielding better trade-offs between circuit size and faithfulness compared to prior work.
• 摘要：一种广泛的发现和理解语言模型机制的策略是电路分析。电路是执行特定任务的模型计算图的最小子图。我们在现有电路发现方法中确定差距：他们假设电路是位置不变的，将模型组件视为在输入位置之间同样相关的。这限制了他们捕获跨位置相互作用或各个位置变化的机制的能力。为了解决这一差距，我们提出了两个改进，即使在包含可变长度示例的任务上，将位置纳入电路。首先，我们扩展了Edge归因贴片，这是一种基于梯度的电路发现方法，以区分令牌位置。其次，我们介绍了数据集架构的概念，该概念定义了在示例中具有相似语义的令牌跨度，从而在具有可变长度示例的数据集中启用了位置感知的电路发现。我们还开发了使用大语言模型来生成架构和应用的自动管道。我们的方法可以使对位置敏感的电路完全自动化发现，与先前的工作相比，在电路尺寸和忠诚之间产生了更好的权衡。

Title: Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context

• Authors: Taejong Joo, Diego Klabjan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04580
• Pdf URL: https://arxiv.org/pdf/2502.04580
• Copy Paste: [[2502.04580]] Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context(https://arxiv.org/abs/2502.04580)
• Keywords: generation
• Abstract: Transformers have demonstrated remarkable in-context learning (ICL) capabilities, adapting to new tasks by simply conditioning on demonstrations without parameter updates. Compelling empirical and theoretical evidence suggests that ICL, as a general-purpose learner, could outperform task-specific models. However, it remains unclear to what extent the transformers optimally learn in-context compared to principled learning algorithms. To bridge this gap, we introduce a new framework for quantifying optimality of ICL as a learning algorithm in stylized settings. Our findings reveal a striking dichotomy: while ICL initially matches the efficiency of a Bayes optimal estimator, its efficiency significantly deteriorates in long context. Through an information-theoretic analysis, we show that the diminishing efficiency is inherent to ICL. These results clarify the trade-offs in adopting ICL as a universal problem solver, motivating a new generation of on-the-fly adaptive methods without the diminishing efficiency.
• 摘要：变形金刚已经证明了出色的内在学习（ICL）功能，可以通过简单地调节没有参数更新的演示来适应新任务。引人入胜的经验和理论证据表明，作为通用学习者，ICL可以胜过特定于任务的模型。但是，与原则性学习算法相比，变形金刚在多大程度上最佳地学习的程度尚不清楚。为了弥合这一差距，我们引入了一个新的框架，以量化ICL作为风格化设置中的学习算法的最佳性。我们的发现表明了引人注目的二分法：虽然ICL最初与贝叶斯最佳估计量的效率相匹配，但其效率在长期以来会显着恶化。通过信息理论分析，我们表明降低的效率是ICL固有的。这些结果阐明了采用ICL作为通用问题解决者的权衡，激发了新一代的自适应方法，而无需降低效率。

Title: The $\alpha$-Alternator: Dynamic Adaptation To Varying Noise Levels In Sequences Using The Vendi Score For Improved Robustness and Performance

• Authors: Mohammad Reza Rezaei, Adji Bousso Dieng
• Subjects: cs.LG, cs.AI, cs.NE, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.04593
• Pdf URL: https://arxiv.org/pdf/2502.04593
• Copy Paste: [[2502.04593]] The $\alpha$-Alternator: Dynamic Adaptation To Varying Noise Levels In Sequences Using The Vendi Score For Improved Robustness and Performance(https://arxiv.org/abs/2502.04593)
• Keywords: generative
• Abstract: Current state-of-the-art dynamical models, such as Mamba, assume the same level of noisiness for all elements of a given sequence, which limits their performance on noisy temporal data. In this paper, we introduce the $\alpha$-Alternator, a novel generative model for time-dependent data that dynamically adapts to the complexity introduced by varying noise levels in sequences. The $\alpha$-Alternator leverages the Vendi Score (VS), a flexible similarity-based diversity metric, to adjust, at each time step $t$, the influence of the sequence element at time $t$ and the latent representation of the dynamics up to that time step on the predicted future dynamics. This influence is captured by a parameter that is learned and shared across all sequences in a given dataset. The sign of this parameter determines the direction of influence. A negative value indicates a noisy dataset, where a sequence element that increases the VS is considered noisy, and the model relies more on the latent history when processing that element. Conversely, when the parameter is positive, a sequence element that increases the VS is considered informative, and the $\alpha$-Alternator relies more on this new input than on the latent history when updating its predicted latent dynamics. The $\alpha$-Alternator is trained using a combination of observation masking and Alternator loss minimization. Masking simulates varying noise levels in sequences, enabling the model to be more robust to these fluctuations and improving its performance in trajectory prediction, imputation, and forecasting. Our experimental results demonstrate that the $\alpha$-Alternator outperforms both Alternators and state-of-the-art state-space models across neural decoding and time-series forecasting benchmarks.
• 摘要：当前的最新动力学模型（例如Mamba）对给定序列的所有元素都采用相同的噪声，这限制了它们在嘈杂的时间数据上的性能。在本文中，我们介绍了$ \ alpha $ alternator，这是一种新颖的生成模型，用于时间依赖性数据，该模型动态地适应了序列中不同噪声水平引入的复杂性。 $ \ alpha $ -Anterthator利用基于灵活的相似性多样性度量的Vendi Score（VS）来调整，每次步骤$ t $，在时间$ t $上的序列元素的影响和潜在的表示直到那个时间步骤的动力学步骤，就可以预测的未来动态。这种影响是由在给定数据集中所有序列中学习和共享的参数捕获的。该参数的符号决定了影响方向。负值表示一个嘈杂的数据集，其中增加VS的序列元素被认为是嘈杂的，并且该模型在处理该元素时更多地依赖于潜在历史记录。相反，当参数为正时，增加VS的序列元素被认为是有益的，并且$ \ alpha $ - alternator在更新其预测的潜在动力学时更依赖于此新输入。 $ \ alpha $ -Anternator通过观察掩蔽和交流发电机损耗最小化的组合对其进行培训。掩盖序列模拟了序列的变化噪声水平，从而使模型能够对这些波动更加健壮，并提高了其在轨迹预测，插补和预测中的性能。我们的实验结果表明，$ \ alpha $ alternator在神经解码和预测基准的时间序列上优于交流发电机和最先进的状态空间模型。

Title: Multiscale style transfer based on a Laplacian pyramid for traditional Chinese painting

• Authors: Kunxiao Liu, Guowu Yuan, Hongyu Liu, Hao Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04597
• Pdf URL: https://arxiv.org/pdf/2502.04597
• Copy Paste: [[2502.04597]] Multiscale style transfer based on a Laplacian pyramid for traditional Chinese painting(https://arxiv.org/abs/2502.04597)
• Keywords: generation
• Abstract: Style transfer is adopted to synthesize appealing stylized images that preserve the structure of a content image but carry the pattern of a style image. Many recently proposed style transfer methods use only western oil paintings as style images to achieve image stylization. As a result, unnatural messy artistic effects are produced in stylized images when using these methods to directly transfer the patterns of traditional Chinese paintings, which are composed of plain colors and abstract objects. Moreover, most of them work only at the original image scale and thus ignore multiscale image information during training. In this paper, we present a novel effective multiscale style transfer method based on Laplacian pyramid decomposition and reconstruction, which can transfer unique patterns of Chinese paintings by learning different image features at different scales. In the first stage, the holistic patterns are transferred at low resolution by adopting a Style Transfer Base Network. Then, the details of the content and style are gradually enhanced at higher resolutions by a Detail Enhancement Network with an edge information selection (EIS) module in the second stage. The effectiveness of our method is demonstrated through the generation of appealing high-quality stylization results and a comparison with some state-of-the-art style transfer methods. Datasets and codes are available at this https URL.
• 摘要：采用样式传输来合成具有吸引力的风格化图像，以保留内容图像的结构，但具有样式图像的模式。许多最近提出的样式转移方法仅使用西方油画作为样式图像来实现图像风格化。结果，当使用这些方法直接传递由纯色和抽象对象组成的传统中国绘画的图案时，在风格化的图像中产生了不自然的凌乱艺术效果。此外，它们中的大多数仅在原始图像量表上工作，因此在训练过程中忽略了多尺度图像信息。在本文中，我们提出了一种基于拉普拉斯金字塔分解和重建的新颖有效的多尺度风格转移方法，该方法可以通过在不同尺度上学习不同的图像特征来传递中国绘画的独特模式。在第一阶段，整体模式通过采用样式转移基础网络以低分辨率转移。然后，在第二阶段，通过具有边缘信息选择（EIS）模块的详细信息增强网络在更高分辨率的情况下逐渐增强了内容和样式的细节。通过产生具有吸引力的高质量风格化结果以及与某些最先进的样式转移方法的比较来证明我们方法的有效性。数据集和代码可在此HTTPS URL上找到。

Title: Neural Clustering for Prefractured Mesh Generation in Real-time Object Destruction

• Authors: Seunghwan Kim, Sunha Park, Seungkyu Lee
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2502.04615
• Pdf URL: https://arxiv.org/pdf/2502.04615
• Copy Paste: [[2502.04615]] Neural Clustering for Prefractured Mesh Generation in Real-time Object Destruction(https://arxiv.org/abs/2502.04615)
• Keywords: generation
• Abstract: Prefracture method is a practical implementation for real-time object destruction that is hardly achievable within performance constraints, but can produce unrealistic results due to its heuristic nature. To mitigate it, we approach the clustering of prefractured mesh generation as an unordered segmentation on point cloud data, and propose leveraging the deep neural network trained on a physics-based dataset. Our novel paradigm successfully predicts the structural weakness of object that have been limited, exhibiting ready-to-use results with remarkable quality.
• 摘要：预锻炼方法是实时对象破坏的实际实施，在绩效限制中几乎无法实现，但由于其启发式性质会产生不切实际的结果。为了减轻它，我们将预裂网格生成的聚类作为对点云数据的无序分割，并提出利用基于物理数据集的深度神经网络。我们的新型范式成功地预测了物体的结构弱点，这些范式受到限制，表现出具有出色质量的现成结果。

Title: HetSSNet: Spatial-Spectral Heterogeneous Graph Learning Network for Panchromatic and Multispectral Images Fusion

• Authors: Mengting Ma, Yizhen Jiang, Mengjiao Zhao, Jiaxin Li, Wei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04623
• Pdf URL: https://arxiv.org/pdf/2502.04623
• Copy Paste: [[2502.04623]] HetSSNet: Spatial-Spectral Heterogeneous Graph Learning Network for Panchromatic and Multispectral Images Fusion(https://arxiv.org/abs/2502.04623)
• Keywords: generation
• Abstract: Remote sensing pansharpening aims to reconstruct spatial-spectral properties during the fusion of panchromatic (PAN) images and low-resolution multi-spectral (LR-MS) images, finally generating the high-resolution multi-spectral (HR-MS) images. In the mainstream modeling strategies, i.e., CNN and Transformer, the input images are treated as the equal-sized grid of pixels in the Euclidean space. They have limitations in facing remote sensing images with irregular ground objects. Graph is the more flexible structure, however, there are two major challenges when modeling spatial-spectral properties with graph: \emph{1) constructing the customized graph structure for spatial-spectral relationship priors}; \emph{2) learning the unified spatial-spectral representation through the graph}. To address these challenges, we propose the spatial-spectral heterogeneous graph learning network, named \textbf{HetSSNet}. Specifically, HetSSNet initially constructs the heterogeneous graph structure for pansharpening, which explicitly describes pansharpening-specific relationships. Subsequently, the basic relationship pattern generation module is designed to extract the multiple relationship patterns from the heterogeneous graph. Finally, relationship pattern aggregation module is exploited to collaboratively learn unified spatial-spectral representation across different relationships among nodes with adaptive importance learning from local and global perspectives. Extensive experiments demonstrate the significant superiority and generalization of HetSSNet.
• 摘要：遥感Pansharpening的目的是重建全球图像（PAN）图像和低分辨率多光谱（LR-MS）图像的空间光谱性能，最终生成高分辨率的多光谱（HR-MS）图像。在主流建模策略（即CNN和变压器）中，输入图像被视为欧几里得空间中像素的等级网格。它们在面对具有不规则地面物体的遥感图像时有局限性。但是，图形是更灵活的结构，但是，用图形建模空间 - 光谱属性时有两个主要挑战：\ emph {1）为空间 - 光谱关系priors}构建自定义的图形结构}; \ emph {2）通过图表学习统一的空间光谱表示}。为了应对这些挑战，我们提出了名为\ textbf {hetssnet}的空间光谱异质图学习网络。具体而言，HETSSNET最初构建了Pansharpening的异质图结构，该图明确描述了Pansharpenting特定关系。随后，基本的关系模式生成模块旨在从异质图中提取多个关系模式。最后，利用关系模式聚合模块，以协作学习从本地和全球角度学习具有自适应重要性学习的节点之间不同关系之间的统一空间 - 光谱表示。广泛的实验证明了HETSSNET的显着优势和概括。

Title: Importance Sampling via Score-based Generative Models

• Authors: Heasung Kim, Taekyun Lee, Hyeji Kim, Gustavo de Veciana
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04646
• Pdf URL: https://arxiv.org/pdf/2502.04646
• Copy Paste: [[2502.04646]] Importance Sampling via Score-based Generative Models(https://arxiv.org/abs/2502.04646)
• Keywords: generative
• Abstract: Importance sampling, which involves sampling from a probability density function (PDF) proportional to the product of an importance weight function and a base PDF, is a powerful technique with applications in variance reduction, biased or customized sampling, data augmentation, and beyond. Inspired by the growing availability of score-based generative models (SGMs), we propose an entirely training-free Importance sampling framework that relies solely on an SGM for the base PDF. Our key innovation is realizing the importance sampling process as a backward diffusion process, expressed in terms of the score function of the base PDF and the specified importance weight function--both readily available--eliminating the need for any additional training. We conduct a thorough analysis demonstrating the method's scalability and effectiveness across diverse datasets and tasks, including importance sampling for industrial and natural images with neural importance weight functions. The training-free aspect of our method is particularly compelling in real-world scenarios where a single base distribution underlies multiple biased sampling tasks, each requiring a different importance weight function. To the best of our knowledge our approach is the first importance sampling framework to achieve this.
• 摘要：重要性采样涉及从与重要性权重函数和基本PDF成正比的概率密度函数（PDF）进行采样，是一种强大的技术，具有降低，偏见或自定义的采样，数据扩展等方差的应用。受到基于分数的生成模型（SGM）的可用性的启发，我们提出了一个完全无训练的重要性采样框架，仅依赖于基本PDF的SGM。我们的关键创新是意识到重要的采样过程是一个向后扩散过程，它根据基本PDF的分数功能和指定的重要性权重函数表示 - 很容易获得 - 逐步提供了对任何其他培训的需求。我们进行了彻底的分析，证明了该方法在不同数据集和任务中的可扩展性和有效性，包括对具有神经重要性重量功能的工业和自然图像的重要性采样。我们方法的无训练方面在现实情况下尤其引人注目，在现实世界中，单个基本分布是多个有偏见的采样任务的基础，每项都需要不同的重要性权重函数。据我们所知，我们的方法是实现这一目标的第一个重要性抽样框架。

Title: A Comprehensive Review on Noise Control of Diffusion Model

• Authors: Zhehao Guo, Jiedong Lang, Shuyu Huang, Yunfei Gao, Xintong Ding
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04669
• Pdf URL: https://arxiv.org/pdf/2502.04669
• Copy Paste: [[2502.04669]] A Comprehensive Review on Noise Control of Diffusion Model(https://arxiv.org/abs/2502.04669)
• Keywords: generative
• Abstract: Diffusion models have recently emerged as powerful generative frameworks for producing high-quality images. A pivotal component of these models is the noise schedule, which governs the rate of noise injection during the diffusion process. Since the noise schedule substantially influences sampling quality and training quality, understanding its design and implications is crucial. In this discussion, various noise schedules are examined, and their distinguishing features and performance characteristics are highlighted.
• 摘要：扩散模型最近已成为用于产生高质量图像的强大生成框架。这些模型的关键成分是噪声表，该噪声表控制了扩散过程中噪声注入速率。由于噪声时间表基本影响了采样质量和训练质量，因此了解其设计和含义至关重要。在此讨论中，检查了各种噪声时间表，并突出显示了它们的显着特征和性能特征。

Title: CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation

• Authors: Bowen Song, Zecheng Zhang, Zhaoxu Luo, Jason Hu, Wei Yuan, Jing Jia, Zhengxu Tang, Guanyang Wang, Liyue Shen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04670
• Pdf URL: https://arxiv.org/pdf/2502.04670
• Copy Paste: [[2502.04670]] CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation(https://arxiv.org/abs/2502.04670)
• Keywords: generation, generative
• Abstract: Diffusion models have emerged as powerful tools for generative tasks, producing high-quality outputs across diverse domains. However, how the generated data responds to the initial noise perturbation in diffusion models remains under-explored, which hinders understanding the controllability of the sampling process. In this work, we first observe an interesting phenomenon: the relationship between the change of generation outputs and the scale of initial noise perturbation is highly linear through the diffusion ODE sampling. Then we provide both theoretical and empirical study to justify this linearity property of this input-output (noise-generation data) relationship. Inspired by these new insights, we propose a novel Controllable and Constrained Sampling method (CCS) together with a new controller algorithm for diffusion models to sample with desired statistical properties while preserving good sample quality. We perform extensive experiments to compare our proposed sampling approach with other methods on both sampling controllability and sampled data quality. Results show that our CCS method achieves more precisely controlled sampling while maintaining superior sample quality and diversity.
• 摘要：扩散模型已成为生成任务的强大工具，从而产生了跨不同领域的高质量输出。但是，生成的数据如何响应扩散模型中的初始噪声扰动仍然不足，这阻碍了理解采样过程的可控性。在这项工作中，我们首先观察到一个有趣的现象：通过扩散ODE采样，发电输出变化与初始噪声扰动的尺度之间的关系是高度线性的。然后，我们提供理论和经验研究，以证明此输入输出（噪声生成数据）关系的这种线性性能合理。受这些新见解的启发，我们提出了一种新颖的可控和约束采样方法（CCS），以及用于扩散模型的新控制器算法，以采样具有所需的统计属性的，同时保留良好的样品质量。我们进行了广泛的实验，以将我们提出的抽样方法与采样可控性和采样数据质量的其他方法进行比较。结果表明，我们的CCS方法可实现更精确的采样，同时保持较高的样本质量和多样性。

Title: G2PDiffusion: Genotype-to-Phenotype Prediction with Diffusion Models

• Authors: Mengdi Liu, Zhangyang Gao, Hong Chang, Stan Z. Li, Shiguang Shan, Xinlin Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04684
• Pdf URL: https://arxiv.org/pdf/2502.04684
• Copy Paste: [[2502.04684]] G2PDiffusion: Genotype-to-Phenotype Prediction with Diffusion Models(https://arxiv.org/abs/2502.04684)
• Keywords: generation
• Abstract: Discovering the genotype-phenotype relationship is crucial for genetic engineering, which will facilitate advances in fields such as crop breeding, conservation biology, and personalized medicine. Current research usually focuses on single species and small datasets due to limitations in phenotypic data collection, especially for traits that require visual assessments or physical measurements. Deciphering complex and composite phenotypes, such as morphology, from genetic data at scale remains an open question. To break through traditional generic models that rely on simplified assumptions, this paper introduces G2PDiffusion, the first-of-its-kind diffusion model designed for genotype-to-phenotype generation across multiple species. Specifically, we use images to represent morphological phenotypes across species and redefine phenotype prediction as conditional image generation. To this end, this paper introduces an environment-enhanced DNA sequence conditioner and trains a stable diffusion model with a novel alignment method to improve genotype-to-phenotype consistency. Extensive experiments demonstrate that our approach enhances phenotype prediction accuracy across species, capturing subtle genetic variations that contribute to observable traits.
• 摘要：发现基因型 - 表型关系对于基因工程至关重要，这将有助于诸如农作物育种，保护生物学和个性化医学等领域的进步。由于表型数据收集的局限性，当前的研究通常集中在单个物种和小数据集上，尤其是对于需要视觉评估或物理测量的特征。从大规模遗传数据中解密的复合和复合表型，例如形态学，仍然是一个悬而未决的问题。为了突破依赖简化假设的传统通用模型，本文引入了G2Pdiffusion，这是为多种物种设计的基因型至苯型型生成的首个型扩散模型。具体而言，我们使用图像来表示物种跨物种的形态表型，并重新定义表型预测为条件图像产生。为此，本文介绍了环境增强的DNA序列调节剂，并使用一种新型的比对方法训练稳定的扩散模型，以改善基因型到表型的一致性。广泛的实验表明，我们的方法增强了跨物种的表型预测准确性，从而捕获了有助于观察性状的细微遗传变异。

Title: Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?

• Authors: Yujin Han, Andi Han, Wei Huang, Chaochao Lu, Difan Zou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04725
• Pdf URL: https://arxiv.org/pdf/2502.04725
• Copy Paste: [[2502.04725]] Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?(https://arxiv.org/abs/2502.04725)
• Keywords: generation
• Abstract: Despite the remarkable success of diffusion models (DMs) in data generation, they exhibit specific failure cases with unsatisfactory outputs. We focus on one such limitation: the ability of DMs to learn hidden rules between image features. Specifically, for image data with dependent features ($\mathbf{x}$) and ($\mathbf{y}$) (e.g., the height of the sun ($\mathbf{x}$) and the length of the shadow ($\mathbf{y}$)), we investigate whether DMs can accurately capture the inter-feature rule ($p(\mathbf{y}|\mathbf{x})$). Empirical evaluations on mainstream DMs (e.g., Stable Diffusion 3.5) reveal consistent failures, such as inconsistent lighting-shadow relationships and mismatched object-mirror reflections. Inspired by these findings, we design four synthetic tasks with strongly correlated features to assess DMs' rule-learning abilities. Extensive experiments show that while DMs can identify coarse-grained rules, they struggle with fine-grained ones. Our theoretical analysis demonstrates that DMs trained via denoising score matching (DSM) exhibit constant errors in learning hidden rules, as the DSM objective is not compatible with rule conformity. To mitigate this, we introduce a common technique - incorporating additional classifier guidance during sampling, which achieves (limited) improvements. Our analysis reveals that the subtle signals of fine-grained rules are challenging for the classifier to capture, providing insights for future exploration.
• 摘要：尽管扩散模型（DMS）在数据生成中取得了显着的成功，但它们表现出特定的失败情况，输出不令人满意。我们专注于这样一个限制：DMS在图像特征之间学习隐藏规则的能力。具体而言，对于具有因功能（$ \ mathbf {x} $）和（$ \ mathbf {y} $）的图像数据（例如，太阳的高度（$ \ mathbf {x} $）和阴影的长度（$ \ mathbf {y} $）），我们研究了DMS是否可以准确捕获功能间规则（$ P（\ Mathbf {y} | \ Mathbf {x}）$）。对主流DM的经验评估（例如稳定的扩散3.5）揭示了一致的失败，例如不一致的照明阴影关系和不匹配的对象摩尔反射。受这些发现的启发，我们设计了四个具有密切相关特征的合成任务，以评估DMS的规则学习能力。广泛的实验表明，尽管DMS可以识别粗粒规则，但它们与细粒度的规则斗争。我们的理论分析表明，通过denoising得分匹配（DSM）训练的DM在学习隐藏规则时表现出恒定的错误，因为DSM目标与规则的一致性不兼容。为了减轻这种情况，我们引入了一种通用技术 - 在抽样过程中结合了其他分类器指导，从而实现（有限）的改进。我们的分析表明，细粒度规则的微妙信号对于分类器捕获的挑战是挑战，为将来的探索提供了见解。

Title: Autoregressive Generation of Static and Growing Trees

• Authors: Hanxiao Wang, Biao Zhang, Jonathan Klein, Dominik L. Michels, Dongming Yan, Peter Wonka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04762
• Pdf URL: https://arxiv.org/pdf/2502.04762
• Copy Paste: [[2502.04762]] Autoregressive Generation of Static and Growing Trees(https://arxiv.org/abs/2502.04762)
• Keywords: generation
• Abstract: We propose a transformer architecture and training strategy for tree generation. The architecture processes data at multiple resolutions and has an hourglass shape, with middle layers processing fewer tokens than outer layers. Similar to convolutional networks, we introduce longer range skip connections to completent this multi-resolution approach. The key advantage of this architecture is the faster processing speed and lower memory consumption. We are therefore able to process more complex trees than would be possible with a vanilla transformer architecture. Furthermore, we extend this approach to perform image-to-tree and point-cloud-to-tree conditional generation and to simulate the tree growth processes, generating 4D trees. Empirical results validate our approach in terms of speed, memory consumption, and generation quality.
• 摘要：我们为树生的变压器架构和培训策略提供了建议。该体系结构以多种分辨率处理数据，并具有沙漏形状，中间层处理的令牌少于外层。与卷积网络类似，我们引入了更长的范围跳过连接到完成此多分辨率方法。该体系结构的关键优势是处理速度更快和较低的内存消耗。因此，我们能够处理比香草变压器体系结构所能更复杂的树木。此外，我们扩展了这种方法，以执行图像对树和点云到树的条件生成并模拟树生长过程，从而生成4D树。经验结果证明了我们在速度，记忆消耗和发电质量方面的方法。

Title: Optimistic Gradient Learning with Hessian Corrections for High-Dimensional Black-Box Optimization

• Authors: Yedidya Kfir, Elad Sarafian, Sarit Kraus, Yoram Louzoun
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04829
• Pdf URL: https://arxiv.org/pdf/2502.04829
• Copy Paste: [[2502.04829]] Optimistic Gradient Learning with Hessian Corrections for High-Dimensional Black-Box Optimization(https://arxiv.org/abs/2502.04829)
• Keywords: generation
• Abstract: Black-box algorithms are designed to optimize functions without relying on their underlying analytical structure or gradient information, making them essential when gradients are inaccessible or difficult to compute. Traditional methods for solving black-box optimization (BBO) problems predominantly rely on non-parametric models and struggle to scale to large input spaces. Conversely, parametric methods that model the function with neural estimators and obtain gradient signals via backpropagation may suffer from significant gradient errors. A recent alternative, Explicit Gradient Learning (EGL), which directly learns the gradient using a first-order Taylor approximation, has demonstrated superior performance over both parametric and non-parametric methods. In this work, we propose two novel gradient learning variants to address the robustness challenges posed by high-dimensional, complex, and highly non-linear problems. Optimistic Gradient Learning (OGL) introduces a bias toward lower regions in the function landscape, while Higher-order Gradient Learning (HGL) incorporates second-order Taylor corrections to improve gradient accuracy. We combine these approaches into the unified OHGL algorithm, achieving state-of-the-art (SOTA) performance on the synthetic COCO suite. Additionally, we demonstrate OHGLs applicability to high-dimensional real-world machine learning (ML) tasks such as adversarial training and code generation. Our results highlight OHGLs ability to generate stronger candidates, offering a valuable tool for ML researchers and practitioners tackling high-dimensional, non-linear optimization challenges
• 摘要：Black-Box算法旨在优化功能，而无需依赖其潜在的分析结构或梯度信息，从而在梯度无法访问或难以计算时至关重要。解决黑盒优化（BBO）问题的传统方法主要取决于非参数模型，并难以扩展到大型输入空间。相反，将功能与神经估计器建模并通过反向传播获得梯度信号的参数方法可能会遭受重大梯度误差。最新的替代性，显式梯度学习（EGL），它直接使用一阶泰勒近似来直接学习梯度，已经证明了与参数和非参数方法相比，表现出色。在这项工作中，我们提出了两个新颖的梯度学习变体，以解决高维，复杂和高度非线性问题所带来的鲁棒性挑战。乐观的梯度学习（OGL）引入了对功能景观中较低区域的偏见，而高阶梯度学习（HGL）融合了二阶泰勒校正，以提高梯度精度。我们将这些方法结合到统一的OHGL算法中，在合成可可套件上实现了最新的（SOTA）性能。此外，我们证明了OHGLS适用于高维实际机器学习（ML）任务，例如对抗培训和代码生成。我们的结果突出了OHGLS产生更强大候选人的能力，为ML研究人员和从业人员提供了有价值的工具

Title: HumanDiT: Pose-Guided Diffusion Transformer for Long-form Human Motion Video Generation

• Authors: Qijun Gan, Yi Ren, Chen Zhang, Zhenhui Ye, Pan Xie, Xiang Yin, Zehuan Yuan, Bingyue Peng, Jianke Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04847
• Pdf URL: https://arxiv.org/pdf/2502.04847
• Copy Paste: [[2502.04847]] HumanDiT: Pose-Guided Diffusion Transformer for Long-form Human Motion Video Generation(https://arxiv.org/abs/2502.04847)
• Keywords: generation
• Abstract: Human motion video generation has advanced significantly, while existing methods still struggle with accurately rendering detailed body parts like hands and faces, especially in long sequences and intricate motions. Current approaches also rely on fixed resolution and struggle to maintain visual consistency. To address these limitations, we propose HumanDiT, a pose-guided Diffusion Transformer (DiT)-based framework trained on a large and wild dataset containing 14,000 hours of high-quality video to produce high-fidelity videos with fine-grained body rendering. Specifically, (i) HumanDiT, built on DiT, supports numerous video resolutions and variable sequence lengths, facilitating learning for long-sequence video generation; (ii) we introduce a prefix-latent reference strategy to maintain personalized characteristics across extended sequences. Furthermore, during inference, HumanDiT leverages Keypoint-DiT to generate subsequent pose sequences, facilitating video continuation from static images or existing videos. It also utilizes a Pose Adapter to enable pose transfer with given sequences. Extensive experiments demonstrate its superior performance in generating long-form, pose-accurate videos across diverse scenarios.
• 摘要：人类运动视频的产生已经大大提高，而现有的方法仍然难以准确渲染详细的身体部位，例如手和面部，尤其是长序列和复杂的动作。当前的方法还依靠固定分辨率和努力保持视觉一致性。为了解决这些局限性，我们提出了Humandit，Humandit是一个姿势引导的扩散变压器（DIT）基于基于野生数据集的框架，该数据集中包含14,000个小时的高质量视频，以制作具有精细颗粒身体渲染的高效率视频。具体而言，（i）基于DIT的Humandit支持众多视频分辨率和可变序列长度，从而促进了长期序列视频的学习； （ii）我们引入了一个前缀叶子参考策略，以维持跨扩展序列的个性化特征。此外，在推断期间，Humandit利用Kepoint-Dit生成后续的姿势序列，从而促进视频延续，从静态图像或现有视频中延续。它还利用姿势适配器以给定序列启用姿势转移。广泛的实验表明，它在在各种情况下产生长形式的姿势精确视频方面表现出色。

Title: Goku: Flow Based Video Generative Foundation Models

• Authors: Shoufa Chen, Chongjian Ge, Yuqi Zhang, Yida Zhang, Fengda Zhu, Hao Yang, Hongxiang Hao, Hui Wu, Zhichao Lai, Yifei Hu, Ting-Che Lin, Shilong Zhang, Fu Li, Chuan Li, Xing Wang, Yanghua Peng, Peize Sun, Ping Luo, Yi Jiang, Zehuan Yuan, Bingyue Peng, Xiaobing Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.04896
• Pdf URL: https://arxiv.org/pdf/2502.04896
• Copy Paste: [[2502.04896]] Goku: Flow Based Video Generative Foundation Models(https://arxiv.org/abs/2502.04896)
• Keywords: generation, generative
• Abstract: This paper introduces Goku, a state-of-the-art family of joint image-and-video generation models leveraging rectified flow Transformers to achieve industry-leading performance. We detail the foundational elements enabling high-quality visual generation, including the data curation pipeline, model architecture design, flow formulation, and advanced infrastructure for efficient and robust large-scale training. The Goku models demonstrate superior performance in both qualitative and quantitative evaluations, setting new benchmarks across major tasks. Specifically, Goku achieves 0.76 on GenEval and 83.65 on DPG-Bench for text-to-image generation, and 84.85 on VBench for text-to-video tasks. We believe that this work provides valuable insights and practical advancements for the research community in developing joint image-and-video generation models.
• 摘要：本文介绍了Goku，这是一个最先进的联合图像和视频生成模型，利用了整流的流动变压器来实现行业领先的绩效。我们详细介绍了实现高质量视觉生成的基础元素，包括数据策划管道，模型体系结构设计，流程配方和高级基础架构，以进行高效且稳健的大规模培训。 Goku模型在定性和定量评估中都表现出卓越的性能，从而在主要任务中设定了新的基准测试。具体而言，Goku在文本到图像生成的DPG基础上达到了Geneval的0.76，在DPG基础上达到了83.65，而在文本到视频任务的VBench上达到了84.85。我们认为，这项工作为研究社区提供了有价值的见解和实践进步，以开发共同的图像和视频生成模型。

Title: Unified Approaches in Self-Supervised Event Stream Modeling: Progress and Prospects

• Authors: Levente Zólyomi, Tianze Wang, Sofiane Ennadir, Oleg Smirnov, Lele Cao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04899
• Pdf URL: https://arxiv.org/pdf/2502.04899
• Copy Paste: [[2502.04899]] Unified Approaches in Self-Supervised Event Stream Modeling: Progress and Prospects(https://arxiv.org/abs/2502.04899)
• Keywords: generation
• Abstract: The proliferation of digital interactions across diverse domains, such as healthcare, e-commerce, gaming, and finance, has resulted in the generation of vast volumes of event stream (ES) data. ES data comprises continuous sequences of timestamped events that encapsulate detailed contextual information relevant to each domain. While ES data holds significant potential for extracting actionable insights and enhancing decision-making, its effective utilization is hindered by challenges such as the scarcity of labeled data and the fragmented nature of existing research efforts. Self-Supervised Learning (SSL) has emerged as a promising paradigm to address these challenges by enabling the extraction of meaningful representations from unlabeled ES data. In this survey, we systematically review and synthesize SSL methodologies tailored for ES modeling across multiple domains, bridging the gaps between domain-specific approaches that have traditionally operated in isolation. We present a comprehensive taxonomy of SSL techniques, encompassing both predictive and contrastive paradigms, and analyze their applicability and effectiveness within different application contexts. Furthermore, we identify critical gaps in current research and propose a future research agenda aimed at developing scalable, domain-agnostic SSL frameworks for ES modeling. By unifying disparate research efforts and highlighting cross-domain synergies, this survey aims to accelerate innovation, improve reproducibility, and expand the applicability of SSL to diverse real-world ES challenges.
• 摘要：跨不同领域（例如医疗保健，电子商务，游戏和金融）的数字互动的扩散导致了大量事件流（ES）数据。 ES数据包括时间戳事件的连续序列，这些序列封装了与每个域相关的详细上下文信息。尽管ES数据具有提取可行的见解和增强决策的巨大潜力，但其有效利用受到挑战的阻碍，例如缺乏标记的数据以及现有研究工作的分散性质。自我监督的学习（SSL）已成为一种有希望的范式，可以通过从未标记的ES数据中提取有意义的表示形式来应对这些挑战。在这项调查中，我们系统地审查并合成了针对跨多个领域建模的SSL方法学，从而弥合了传统上孤立地操作的特定领域方法之间的差距。我们提出了SSL技术的全面分类学，包括预测性和对比范式，并分析其在不同应用程序环境中的适用性和有效性。此外，我们确定了当前研究中的关键差距，并提出了一个未来的研究议程，旨在开发用于ES建模的可扩展的，域形无关的SSL框架。通过统一不同的研究工作并突出跨域协同作用，该调查旨在加速创新，提高可重复性，并扩大SSL对各种现实世界ES挑战的适用性。

Title: Cached Multi-Lora Composition for Multi-Concept Image Generation

• Authors: Xiandong Zou, Mingzhu Shen, Christos-Savvas Bouganis, Yiren Zhao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04923
• Pdf URL: https://arxiv.org/pdf/2502.04923
• Copy Paste: [[2502.04923]] Cached Multi-Lora Composition for Multi-Concept Image Generation(https://arxiv.org/abs/2502.04923)
• Keywords: generation
• Abstract: Low-Rank Adaptation (LoRA) has emerged as a widely adopted technique in text-to-image models, enabling precise rendering of multiple distinct elements, such as characters and styles, in multi-concept image generation. However, current approaches face significant challenges when composing these LoRAs for multi-concept image generation, resulting in diminished generated image quality. In this paper, we initially investigate the role of LoRAs in the denoising process through the lens of the Fourier frequency domain. Based on the hypothesis that applying multiple LoRAs could lead to "semantic conflicts", we find that certain LoRAs amplify high-frequency features such as edges and textures, whereas others mainly focus on low-frequency elements, including the overall structure and smooth color gradients. Building on these insights, we devise a frequency domain based sequencing strategy to determine the optimal order in which LoRAs should be integrated during inference. This strategy offers a methodical and generalizable solution compared to the naive integration commonly found in existing LoRA fusion techniques. To fully leverage our proposed LoRA order sequence determination method in multi-LoRA composition tasks, we introduce a novel, training-free framework, Cached Multi-LoRA (CMLoRA), designed to efficiently integrate multiple LoRAs while maintaining cohesive image generation. With its flexible backbone for multi-LoRA fusion and a non-uniform caching strategy tailored to individual LoRAs, CMLoRA has the potential to reduce semantic conflicts in LoRA composition and improve computational efficiency. Our experimental evaluations demonstrate that CMLoRA outperforms state-of-the-art training-free LoRA fusion methods by a significant margin -- it achieves an average improvement of $2.19\%$ in CLIPScore, and $11.25\%$ in MLLM win rate compared to LoraHub, LoRA Composite, and LoRA Switch.
• 摘要：低级适应性（LORA）已成为文本到图像模型中广泛采用的技术，从而可以在多概念概念图像生成中精确地渲染多个不同的元素，例如字符和样式。但是，当当前的方法为多概念图像生成而编写这些Loras时，面临着巨大的挑战，从而导致生成的图像质量降低。在本文中，我们最初通过傅立叶频域的镜头研究洛拉斯在脱索过程中的作用。基于以下假设：应用多个洛拉斯可能导致“语义冲突”，我们发现某些洛拉斯会放大高频特征，例如边缘和纹理，而其他洛拉斯主要集中于低频元素，包括整体结构和光滑的颜色梯度， 。在这些见解的基础上，我们设计了一种基于频域的测序策略，以确定在推断过程中应集成劳拉的最佳顺序。与现有洛拉融合技术中常见的天真整合相比，该策略提供了有条不紊且可推广的解决方案。为了充分利用我们提出的洛拉顺序序列确定方法，在多LORA组成任务中，我们引入了一个新颖的，无训练的框架，缓存的多洛拉（CMLORA），旨在有效地整合多个洛拉斯，同时保持凝聚力的图像产生。 Cmlora具有柔性的主链，用于多Lora融合和针对单个Loras的非均匀缓存策略，有可能减少Lora组成中的语义冲突并提高计算效率。我们的实验评估表明，CMLORA的表现优于最先进的无培训LORA融合方法，其平均提高了$ 2.19 \％$ $ $ $，而MLLM的胜利率为$ 11.25 \％$ Lorahub，Lora Composite和Lora Switch。

Title: Generative-enhanced optimization for knapsack problems: an industry-relevant study

• Authors: Yelyzaveta Vodovozova, Abhishek Awasthi, Caitlin Jones, Joseph Doetsch, Karen Wintersperger, Florian Krellner, Carlos A. Riofrío
• Subjects: cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2502.04928
• Pdf URL: https://arxiv.org/pdf/2502.04928
• Copy Paste: [[2502.04928]] Generative-enhanced optimization for knapsack problems: an industry-relevant study(https://arxiv.org/abs/2502.04928)
• Keywords: generative
• Abstract: Optimization is a crucial task in various industries such as logistics, aviation, manufacturing, chemical, pharmaceutical, and insurance, where finding the best solution to a problem can result in significant cost savings and increased efficiency. Tensor networks (TNs) have gained prominence in recent years in modeling classical systems with quantum-inspired approaches. More recently, TN generative-enhanced optimization (TN-GEO) has been proposed as a strategy which uses generative modeling to efficiently sample valid solutions with respect to certain constraints of optimization problems. Moreover, it has been shown that symmetric TNs (STNs) can encode certain constraints of optimization problems, thus aiding in their solution process. In this work, we investigate the applicability of TN- and STN-GEO to an industry relevant problem class, a multi-knapsack problem, in which each object must be assigned to an available knapsack. We detail a prescription for practitioners to use the TN-and STN-GEO methodology and study its scaling behavior and dependence on its hyper-parameters. We benchmark 60 different problem instances and find that TN-GEO and STN-GEO produce results of similar quality to simulated annealing.
• 摘要：在物流，航空，制造，化学，药物和保险等各个行业中，优化是一项至关重要的任务，在此找到解决问题的最佳解决方案可以导致大量的成本节省和提高效率。近年来，在用量子启发的方法对古典系统建模方面，张量网络（TNS）已获得突出。最近，已经提出了TN生成增强优化（TN-GEO）作为一种策略，该策略使用生成型建模来有效地针对优化问题的某些约束来采样有效的解决方案。此外，已经表明，对称TNS（STN）可以编码优化问题的某些约束，从而在其解决方案过程中有助于。在这项工作中，我们调查了TN-和STN-GEO对行业相关问题类别的适用性，这是一个多扫描问题，其中每个对象都必须分配给可用的背包。我们详细介绍了从业者使用TN和STN-GEO方法的处方，并研究其缩放行为以及对其超参数的依赖。我们基准了60个不同的问题实例，发现TN-GEO和STN-GEO产生与模拟退火质量相似的结果。

Title: Robust Graph Learning Against Adversarial Evasion Attacks via Prior-Free Diffusion-Based Structure Purification

• Authors: Jiayi Luo, Qingyun Sun, Haonan Yuan, Xingcheng Fu, Jianxin Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.05000
• Pdf URL: https://arxiv.org/pdf/2502.05000
• Copy Paste: [[2502.05000]] Robust Graph Learning Against Adversarial Evasion Attacks via Prior-Free Diffusion-Based Structure Purification(https://arxiv.org/abs/2502.05000)
• Keywords: generation
• Abstract: Adversarial evasion attacks pose significant threats to graph learning, with lines of studies that have improved the robustness of Graph Neural Networks (GNNs). However, existing works rely on priors about clean graphs or attacking strategies, which are often heuristic and inconsistent. To achieve robust graph learning over different types of evasion attacks and diverse datasets, we investigate this problem from a prior-free structure purification perspective. Specifically, we propose a novel Diffusion-based Structure Purification framework named DiffSP, which creatively incorporates the graph diffusion model to learn intrinsic distributions of clean graphs and purify the perturbed structures by removing adversaries under the direction of the captured predictive patterns without relying on priors. DiffSP is divided into the forward diffusion process and the reverse denoising process, during which structure purification is achieved. To avoid valuable information loss during the forward process, we propose an LID-driven nonisotropic diffusion mechanism to selectively inject noise anisotropically. To promote semantic alignment between the clean graph and the purified graph generated during the reverse process, we reduce the generation uncertainty by the proposed graph transfer entropy guided denoising mechanism. Extensive experiments demonstrate the superior robustness of DiffSP against evasion attacks.
• 摘要：对抗性逃避攻击对图形学习构成了重大威胁，从而提高了图神经网络（GNN）的鲁棒性。但是，现有作品依靠有关干净图或攻击策略的先验，这些策略通常是启发式且不一致的。为了在不同类型的逃避攻击和多种数据集上实现强大的图形学习，我们从先前的无结构纯化的角度研究了这个问题。具体而言，我们提出了一个名为diffsp的新型基于扩散的结构纯化框架，该框架创造性地结合了图形扩散模型，以学习清洁图的内在分布，并通过在不依赖priors的捕获的预测模式的指导下删除对手，从而净化扰动的结构。 DIFFSP分为正向扩散过程和反向denoising过程，在此过程中实现结构纯化。为了避免在远期过程中的有价值的信息丢失，我们提出了一种盖子驱动的非偶性扩散机制，以选择性地注入噪声各向异性。为了促进在反向过程中生成的清洁图和纯化的图之间的语义一致性，我们通过提出的图形传输熵引导的denoising机制来减少产生不确定性。广泛的实验表明，DIFFSP对逃避攻击具有出色的鲁棒性。

Title: Beautiful Images, Toxic Words: Understanding and Addressing Offensive Text in Generated Images

• Authors: Aditya Kumar, Tom Blanchard, Adam Dziedzic, Franziska Boenisch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.05066
• Pdf URL: https://arxiv.org/pdf/2502.05066
• Copy Paste: [[2502.05066]] Beautiful Images, Toxic Words: Understanding and Addressing Offensive Text in Generated Images(https://arxiv.org/abs/2502.05066)
• Keywords: generation
• Abstract: State-of-the-art visual generation models, such as Diffusion Models (DMs) and Vision Auto-Regressive Models (VARs), produce highly realistic images. While prior work has successfully mitigated Not Safe For Work (NSFW) content in the visual domain, we identify a novel threat: the generation of NSFW text embedded within images. This includes offensive language, such as insults, racial slurs, and sexually explicit terms, posing significant risks to users. We show that all state-of-the-art DMs (e.g., SD3, Flux, DeepFloyd IF) and VARs (e.g., Infinity) are vulnerable to this issue. Through extensive experiments, we demonstrate that existing mitigation techniques, effective for visual content, fail to prevent harmful text generation while substantially degrading benign text generation. As an initial step toward addressing this threat, we explore safety fine-tuning of the text encoder underlying major DM architectures using a customized dataset. Thereby, we suppress NSFW generation while preserving overall image and text generation quality. Finally, to advance research in this area, we introduce ToxicBench, an open-source benchmark for evaluating NSFW text generation in images. ToxicBench provides a curated dataset of harmful prompts, new metrics, and an evaluation pipeline assessing both NSFW-ness and generation quality. Our benchmark aims to guide future efforts in mitigating NSFW text generation in text-to-image models and is available at this https URL
• 摘要：最先进的视觉生成模型，例如扩散模型（DMS）和视觉自动回归模型（VAR），会产生高度逼真的图像。虽然先前的工作已成功地减轻了视觉域中的工作（NSFW）内容，但我们确定了一种新颖的威胁：嵌入图像中的NSFW文本的产生。这包括令人反感的语言，例如侮辱，种族诽谤和性明确的术语，对用户带来了重大风险。我们表明，所有最先进的DMS（例如SD3，Flux，Deepfloyd If）和Vars（例如，Infinity）都容易受到此问题的影响。通过广泛的实验，我们证明了对视觉内容有效的现有缓解技术无法防止有害文本生成，同时实质上会降低良性文本生成。作为解决这一威胁的第一步，我们探索了使用自定义数据集的主要DM体系结构的文本编码器进行的安全调整。因此，我们抑制了NSFW的生成，同时保留了整体形象和文本生成质量。最后，为了推进该领域的研究，我们介绍了Toxicbench，这是一种开源基准，用于评估图像中的NSFW文本生成。有害的提示，新指标以及评估NSFW-WENS和发电质量的评估管道提供了一个策划的数据集。我们的基准旨在指导未来的努力来减轻文本模型中的NSFW文本生成，并在此HTTPS URL中找到

Title: Graph Contrastive Learning for Connectome Classification

• Authors: Martín Schmidt, Sara Silva, Federico Larroca, Gonzalo Mateos, Pablo Musé
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.05109
• Pdf URL: https://arxiv.org/pdf/2502.05109
• Copy Paste: [[2502.05109]] Graph Contrastive Learning for Connectome Classification(https://arxiv.org/abs/2502.05109)
• Keywords: generation
• Abstract: With recent advancements in non-invasive techniques for measuring brain activity, such as magnetic resonance imaging (MRI), the study of structural and functional brain networks through graph signal processing (GSP) has gained notable prominence. GSP stands as a key tool in unraveling the interplay between the brain's function and structure, enabling the analysis of graphs defined by the connections between regions of interest -- referred to as connectomes in this context. Our work represents a further step in this direction by exploring supervised contrastive learning methods within the realm of graph representation learning. The main objective of this approach is to generate subject-level (i.e., graph-level) vector representations that bring together subjects sharing the same label while separating those with different labels. These connectome embeddings are derived from a graph neural network Encoder-Decoder architecture, which jointly considers structural and functional connectivity. By leveraging data augmentation techniques, the proposed framework achieves state-of-the-art performance in a gender classification task using Human Connectome Project data. More broadly, our connectome-centric methodological advances support the promising prospect of using GSP to discover more about brain function, with potential impact to understanding heterogeneity in the neurodegeneration for precision medicine and diagnosis.
• 摘要：随着用于测量脑活动的非侵入性技术的最新进展，例如磁共振成像（MRI），通过图形信号处理（GSP）研究结构和功能性脑网络（GSP）已显着突出。 GSP是揭开大脑功能和结构之间相互作用的关键工具，从而使图形分析是由感兴趣区域之间的连接定义的图形 - 在此上下文中称为连接组。我们的工作代表了朝着这个方向迈出的进一步步骤，通过探索图表表示学习领域内的对比对比学习方法。这种方法的主要目的是生成主题级别（即图形级）向量表示，这些矢量表示将主体共享相同的标签，同时将具有不同标签的主体分开。这些Connectome嵌入是从图形神经网络编码器架构中得出的，该结构共同考虑结构和功能连接。通过利用数据增强技术，所提出的框架使用人类Connectome项目数据实现了性别分类任务中的最新性能。更广泛地说，我们以连接组为中心的方法学进步支持了使用GSP更多地发现有关脑功能的有前途的前景，并可能影响理解神经变性的异质性进行精确医学和诊断。

Title: Self-supervised Conformal Prediction for Uncertainty Quantification in Imaging Problems

• Authors: Jasper M. Everink, Bernardin Tamo Amougou, Marcelo Pereyra
• Subjects: cs.CV, stat.ME
• Abstract URL: https://arxiv.org/abs/2502.05127
• Pdf URL: https://arxiv.org/pdf/2502.05127
• Copy Paste: [[2502.05127]] Self-supervised Conformal Prediction for Uncertainty Quantification in Imaging Problems(https://arxiv.org/abs/2502.05127)
• Keywords: restoration
• Abstract: Most image restoration problems are ill-conditioned or ill-posed and hence involve significant uncertainty. Quantifying this uncertainty is crucial for reliably interpreting experimental results, particularly when reconstructed images inform critical decisions and science. However, most existing image restoration methods either fail to quantify uncertainty or provide estimates that are highly inaccurate. Conformal prediction has recently emerged as a flexible framework to equip any estimator with uncertainty quantification capabilities that, by construction, have nearly exact marginal coverage. To achieve this, conformal prediction relies on abundant ground truth data for calibration. However, in image restoration problems, reliable ground truth data is often expensive or not possible to acquire. Also, reliance on ground truth data can introduce large biases in situations of distribution shift between calibration and deployment. This paper seeks to develop a more robust approach to conformal prediction for image restoration problems by proposing a self-supervised conformal prediction method that leverages Stein's Unbiased Risk Estimator (SURE) to self-calibrate itself directly from the observed noisy measurements, bypassing the need for ground truth. The method is suitable for any linear imaging inverse problem that is ill-conditioned, and it is especially powerful when used with modern self-supervised image restoration techniques that can also be trained directly from measurement data. The proposed approach is demonstrated through numerical experiments on image denoising and deblurring, where it delivers results that are remarkably accurate and comparable to those obtained by supervised conformal prediction with ground truth data.
• 摘要：大多数图像恢复问题是条件不足或不适合的，因此涉及严重的不确定性。量化这种不确定性对于可靠地解释实验结果至关重要，尤其是当重建图像为关键决策和科学提供依据时。但是，大多数现有的图像恢复方法要么无法量化不确定性，要么提供高度不准确的估计值。保形预测最近已成为一个灵活的框架，可以为任何估计器提供不确定性定量功能，通过构造几乎具有边缘覆盖率。为了实现这一目标，共形预测依靠丰富的地面真相数据进行校准。但是，在图像恢复问题中，可靠的地面真相数据通常很昂贵或无法获取。此外，对地面真相数据的依赖可以在校准和部署之间的分配转移情况下引入巨大的偏见。本文试图通过提出一种自我监督的共形预测方法来开发一种更强大的方法来为图像恢复问题进行保形预测，该方法利用了斯坦因的无偏风险估计器（肯定）直接从观察到的嘈杂测量值中自我校准，从而绕开了对地面真相。该方法适用于任何条件不足的线性成像逆问题，当与现代自我监督的图像恢复技术一起使用时，它特别强大，该技术也可以直接从测量数据中进行训练。通过对图像DeNoising和DeBlurring进行数值实验，证明了所提出的方法，在该实验中，它提供的结果非常准确且可与通过与地面真实数据进行监督的保形预测获得的结果相当。

Title: Hummingbird: High Fidelity Image Generation via Multimodal Context Alignment

• Authors: Minh-Quan Le, Gaurav Mittal, Tianjian Meng, A S M Iftekhar, Vishwas Suryanarayanan, Barun Patra, Dimitris Samaras, Mei Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.05153
• Pdf URL: https://arxiv.org/pdf/2502.05153
• Copy Paste: [[2502.05153]] Hummingbird: High Fidelity Image Generation via Multimodal Context Alignment(https://arxiv.org/abs/2502.05153)
• Keywords: generation
• Abstract: While diffusion models are powerful in generating high-quality, diverse synthetic data for object-centric tasks, existing methods struggle with scene-aware tasks such as Visual Question Answering (VQA) and Human-Object Interaction (HOI) Reasoning, where it is critical to preserve scene attributes in generated images consistent with a multimodal context, i.e. a reference image with accompanying text guidance query. To address this, we introduce Hummingbird, the first diffusion-based image generator which, given a multimodal context, generates highly diverse images w.r.t. the reference image while ensuring high fidelity by accurately preserving scene attributes, such as object interactions and spatial relationships from the text guidance. Hummingbird employs a novel Multimodal Context Evaluator that simultaneously optimizes our formulated Global Semantic and Fine-grained Consistency Rewards to ensure generated images preserve the scene attributes of reference images in relation to the text guidance while maintaining diversity. As the first model to address the task of maintaining both diversity and fidelity given a multimodal context, we introduce a new benchmark formulation incorporating MME Perception and Bongard HOI datasets. Benchmark experiments show Hummingbird outperforms all existing methods by achieving superior fidelity while maintaining diversity, validating Hummingbird's potential as a robust multimodal context-aligned image generator in complex visual tasks.
• 摘要：尽管扩散模型在生成以对象为中心任务的高质量，多样化的合成数据方面具有强大要保留与多模式上下文一致的生成图像中的场景属性，即带有随附文本指南查询的参考图像。为了解决这个问题，我们介绍了蜂鸟，蜂鸟是第一个基于扩散的图像发生器，在给定多模式上下文的情况下，它生成了高度多样化的图像W.R.T.参考图像通过准确保留场景属性（例如对象相互作用和文本指南的空间关系）来确保高保真度。 Hummingbird采用了一种新颖的多模式上下文评估器，同时优化了我们配制的全球​​语义和细粒度的一致性奖励，以确保生成的图像保留有关文本指南的参考图像的场景属性，同时保持多样性。作为解决多模式上下文的第一个确定多样性和忠诚的任务的模型，我们引入了一种新的基准公式，其中包含MME感知和Bongard HOI数据集。基准实验表明，蜂鸟通过在保持多样性的同时实现卓越的保真度来优于所有现有方法，从而在复杂的视觉任务中验证了Hummingbird的潜力作为强大的多模式上下文对立图像发生器。

Title: A Lightweight Method to Disrupt Memorized Sequences in LLM

• Authors: Parjanya Prajakta Prashant, Kaustubh Ponkshe, Babak Salimi
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.05159
• Pdf URL: https://arxiv.org/pdf/2502.05159
• Copy Paste: [[2502.05159]] A Lightweight Method to Disrupt Memorized Sequences in LLM(https://arxiv.org/abs/2502.05159)
• Keywords: generation
• Abstract: Large language models (LLMs) demonstrate impressive capabilities across many tasks yet risk reproducing copyrighted content verbatim, raising legal and ethical concerns. Although methods like differential privacy or neuron editing can reduce memorization, they typically require costly retraining or direct access to model weights and may degrade performance. To address these challenges, we propose TokenSwap, a lightweight, post-hoc approach that replaces the probabilities of grammar-related tokens with those from a small auxiliary model (e.g., DistilGPT-2). We run extensive experiments on commercial grade models such as Pythia-6.9b and LLaMA-3-8b and demonstrate that our method effectively reduces well-known cases of memorized generation by upto 10x with little to no impact on downstream tasks. Our approach offers a uniquely accessible and effective solution to users of real-world systems.
• 摘要：大型语言模型（LLM）在许多任务中表现出令人印象深刻的能力，但风险逐字化受版权保护的内容，引起法律和道德问题。尽管诸如差异隐私或神经元编辑之类的方法可以减少记忆，但它们通常需要昂贵的再培训或直接访问模型重量并可能降低性能。为了应对这些挑战，我们提出了TokensWap，这是一种轻巧的事后方法，可以代替与小型辅助模型（例如，Distilgpt-2）的语法相关令牌的概率。我们对诸如Pythia-6.9b和Llama-3-8B等商业级模型进行了广泛的实验，并证明我们的方法有效地将众所周知的记忆发电案例减少了10倍，对下游任务几乎没有影响。我们的方法为现实系统的用户提供了独特而有效的解决方案。

Title: Multitwine: Multi-Object Compositing with Text and Layout Control

• Authors: Gemma Canet Tarrés, Zhe Lin, Zhifei Zhang, He Zhang, Andrew Gilbert, John Collomosse, Soo Ye Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.05165
• Pdf URL: https://arxiv.org/pdf/2502.05165
• Copy Paste: [[2502.05165]] Multitwine: Multi-Object Compositing with Text and Layout Control(https://arxiv.org/abs/2502.05165)
• Keywords: generation, generative
• Abstract: We introduce the first generative model capable of simultaneous multi-object compositing, guided by both text and layout. Our model allows for the addition of multiple objects within a scene, capturing a range of interactions from simple positional relations (e.g., next to, in front of) to complex actions requiring reposing (e.g., hugging, playing guitar). When an interaction implies additional props, like `taking a selfie', our model autonomously generates these supporting objects. By jointly training for compositing and subject-driven generation, also known as customization, we achieve a more balanced integration of textual and visual inputs for text-driven object compositing. As a result, we obtain a versatile model with state-of-the-art performance in both tasks. We further present a data generation pipeline leveraging visual and language models to effortlessly synthesize multimodal, aligned training data.
• 摘要：我们介绍了第一个能够在文本和布局的指导下同时进行多对象合成的生成模型。我们的模型允许在场景中添加多个对象，从而捕获一系列相互作用，从简单的位置关系（例如，在旁边，在面前）到需要重新安息的复杂动作（例如，拥抱，弹吉他）。当互动意味着其他道具（例如“自拍照”）时，我们的模型会自动生成这些支持对象。通过共同培训组合和主题驱动的生成（也称为定制），我们实现了文本和视觉输入的更加平衡的整合，用于文本驱动对象合成。结果，我们获得了一个多功能模型，在这两个任务中都具有最先进的性能。我们进一步提出了利用视觉和语言模型的数据生成管道，以毫不费力地综合了多模式，对齐的训练数据。

Title: Fillerbuster: Multi-View Scene Completion for Casual Captures

• Authors: Ethan Weber, Norman Müller, Yash Kant, Vasu Agrawal, Michael Zollhöfer, Angjoo Kanazawa, Christian Richardt
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2502.05175
• Pdf URL: https://arxiv.org/pdf/2502.05175
• Copy Paste: [[2502.05175]] Fillerbuster: Multi-View Scene Completion for Casual Captures(https://arxiv.org/abs/2502.05175)
• Keywords: generative
• Abstract: We present Fillerbuster, a method that completes unknown regions of a 3D scene by utilizing a novel large-scale multi-view latent diffusion transformer. Casual captures are often sparse and miss surrounding content behind objects or above the scene. Existing methods are not suitable for handling this challenge as they focus on making the known pixels look good with sparse-view priors, or on creating the missing sides of objects from just one or two photos. In reality, we often have hundreds of input frames and want to complete areas that are missing and unobserved from the input frames. Additionally, the images often do not have known camera parameters. Our solution is to train a generative model that can consume a large context of input frames while generating unknown target views and recovering image poses when desired. We show results where we complete partial captures on two existing datasets. We also present an uncalibrated scene completion task where our unified model predicts both poses and creates new content. Our model is the first to predict many images and poses together for scene completion.
• 摘要：我们提出了FillerBuster，该方法通过利用新型的大型多视图潜在扩散变压器来完成3D场景的未知区域。随意捕获通常很稀疏，错过了物体后面或场景之上的内容。现有方法不适合处理这一挑战，因为它们专注于使已知像素在稀疏视图中看起来不错，或者仅从一两张照片中创建对象的缺失侧面。实际上，我们经常有数百个输入帧，并希望完成输入帧中缺少和未观察到的区域。此外，这些图像通常没有已知的相机参数。我们的解决方案是训练一个生成模型，该模型可以消耗较大的输入框架上下文，同时生成未知的目标视图并在需要时恢复图像姿势。我们显示了在两个现有数据集上完成部分捕获的结果。我们还提出了一个未校准的场景完成任务，我们的统一模型可以预测姿势并创建新内容。我们的模型是第一个预测许多图像并合影以完成场景完成的模型。

Title: AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360{\deg} Unbounded Scene Inpainting

• Authors: Chung-Ho Wu, Yang-Jung Chen, Ying-Huan Chen, Jie-Ying Lee, Bo-Hsu Ke, Chun-Wei Tuan Mu, Yi-Chuan Huang, Chin-Yang Lin, Min-Hung Chen, Yen-Yu Lin, Yu-Lun Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.05176
• Pdf URL: https://arxiv.org/pdf/2502.05176
• Copy Paste: [[2502.05176]] AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360{\deg} Unbounded Scene Inpainting(https://arxiv.org/abs/2502.05176)
• Keywords: generation
• Abstract: Three-dimensional scene inpainting is crucial for applications from virtual reality to architectural visualization, yet existing methods struggle with view consistency and geometric accuracy in 360° unbounded scenes. We present AuraFusion360, a novel reference-based method that enables high-quality object removal and hole filling in 3D scenes represented by Gaussian Splatting. Our approach introduces (1) depth-aware unseen mask generation for accurate occlusion identification, (2) Adaptive Guided Depth Diffusion, a zero-shot method for accurate initial point placement without requiring additional training, and (3) SDEdit-based detail enhancement for multi-view coherence. We also introduce 360-USID, the first comprehensive dataset for 360° unbounded scene inpainting with ground truth. Extensive experiments demonstrate that AuraFusion360 significantly outperforms existing methods, achieving superior perceptual quality while maintaining geometric accuracy across dramatic viewpoint changes. See our project page for video results and the dataset at this https URL.
• 摘要：三维场景介绍对于从虚拟现实到架构可视化的应用至关重要，但是现有的方法在360°无界场景中遇到了视图一致性和几何准确性的困难。我们提出了Aurafusion360，这是一种新型的基于参考的方法，可实现高质量的对象去除和孔填充3D场景，以高斯分裂表示。我们的方法介绍了（1）深度感知的未见掩模的生成，以进行准确的遮挡识别，（2）自适应引导性深度扩散，一种零拍的方法，用于准确的初始点放置而无需额外的训练，以及（3）基于SDEDIT的详细信息增强了以增强的详细信息多视图相干。我们还介绍了360 usid，这是第一个用于360°无界场景与地面真相的综合数据集。广泛的实验表明，Aurafusion360显着胜过现有方法，达到了卓越的感知质量，同时保持跨观点变化的几何精度。请参阅我们的项目页面，以获取此HTTPS URL的视频结果和数据集。

Title: QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation

• Authors: Yue Zhao, Fuzhao Xue, Scott Reed, Linxi Fan, Yuke Zhu, Jan Kautz, Zhiding Yu, Philipp Krähenbühl, De-An Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.05178
• Pdf URL: https://arxiv.org/pdf/2502.05178
• Copy Paste: [[2502.05178]] QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation(https://arxiv.org/abs/2502.05178)
• Keywords: generation
• Abstract: We introduce Quantized Language-Image Pretraining (QLIP), a visual tokenization method that combines state-of-the-art reconstruction quality with state-of-the-art zero-shot image understanding. QLIP trains a binary-spherical-quantization-based autoencoder with reconstruction and language-image alignment objectives. We are the first to show that the two objectives do not need to be at odds. We balance the two loss terms dynamically during training and show that a two-stage training pipeline effectively mixes the large-batch requirements of image-language pre-training with the memory bottleneck imposed by the reconstruction objective. We validate the effectiveness of QLIP for multimodal understanding and text-conditioned image generation with a single model. Specifically, QLIP serves as a drop-in replacement for the visual encoder for LLaVA and the image tokenizer for LlamaGen with comparable or even better performance. Finally, we demonstrate that QLIP enables a unified mixed-modality auto-regressive model for understanding and generation.
• 摘要：我们介绍了量化的语言图像预处理（QLIP），这是一种视觉令牌化方法，将最先进的重建质量与最新的零拍图像理解结合在一起。 QLIP通过重建和语言图像对准目标训练基于二进制定量的自动编码器。我们是第一个表明这两个目标不需要矛盾的人。我们在训练过程中动态平衡了两个损失项，并表明两阶段的训练管道有效地将图像语言预训练的大批量要求与重建目标施加的内存瓶颈混合在一起。我们通过单个模型来验证QLIP对多模式理解和文本条件形成图像生成的有效性。具体而言，QLIP可作为LLAVA视觉编码器的置换式替代品和Llamagen的图像令牌，具有可比的性能甚至更好的性能。最后，我们证明了QLIP可以使统一的混合模式自动回归模型用于理解和产生。

Title: FlashVideo:Flowing Fidelity to Detail for Efficient High-Resolution Video Generation

• Authors: Shilong Zhang, Wenbo Li, Shoufa Chen, Chongjian Ge, Peize Sun, Yida Zhang, Yi Jiang, Zehuan Yuan, Binyue Peng, Ping Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.05179
• Pdf URL: https://arxiv.org/pdf/2502.05179
• Copy Paste: [[2502.05179]] FlashVideo:Flowing Fidelity to Detail for Efficient High-Resolution Video Generation(https://arxiv.org/abs/2502.05179)
• Keywords: generation
• Abstract: DiT diffusion models have achieved great success in text-to-video generation, leveraging their scalability in model capacity and data scale. High content and motion fidelity aligned with text prompts, however, often require large model parameters and a substantial number of function evaluations (NFEs). Realistic and visually appealing details are typically reflected in high resolution outputs, further amplifying computational demands especially for single stage DiT models. To address these challenges, we propose a novel two stage framework, FlashVideo, which strategically allocates model capacity and NFEs across stages to balance generation fidelity and quality. In the first stage, prompt fidelity is prioritized through a low resolution generation process utilizing large parameters and sufficient NFEs to enhance computational efficiency. The second stage establishes flow matching between low and high resolutions, effectively generating fine details with minimal NFEs. Quantitative and visual results demonstrate that FlashVideo achieves state-of-the-art high resolution video generation with superior computational efficiency. Additionally, the two-stage design enables users to preview the initial output before committing to full resolution generation, thereby significantly reducing computational costs and wait times as well as enhancing commercial viability .
• 摘要：DIT扩散模型在文本到视频生成方面取得了巨大成功，利用其在模型容量和数据量表中的可扩展性。但是，高内容和运动保真度与文本提示一致，但是，通常需要大型模型参数和大量功能评估（NFE）。现实和视觉吸引力的细节通常反映在高分辨率输出中，进一步扩大了计算需求，尤其是对于单阶段DIT模型。为了应对这些挑战，我们提出了一个新颖的两个阶段框架FlashVideo，该框架从策略上分配了模型容量和NFE，以平衡发电的忠诚度和质量。在第一阶段，通过使用大型参数和足够的NFE来提高计算效率的低分辨率生成过程，将及时的保真度优先考虑。第二阶段建立了低分辨率和高分辨率之间的流量匹配，从而有效地生成了最小NFE的细节。定量和视觉结果表明，FlashVideo具有具有出色的计算效率的最先进的高分辨率视频生成。此外，两阶段的设计使用户能够预览初始输出，然后再进行完全分辨率生成，从而大大降低了计算成本和等待时间，并增强了商业生存能力。