2024-11-25

Title: FuseGPT: Learnable Layers Fusion of Generative Pre-trained Transformers

• Authors: Zehua Pei, Hui-Ling Zhen, Xianzhi Yu, Sinno Jialin Pan, Mingxuan Yuan, Bei Yu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14507
• Pdf URL: https://arxiv.org/pdf/2411.14507
• Copy Paste: [[2411.14507]] FuseGPT: Learnable Layers Fusion of Generative Pre-trained Transformers(https://arxiv.org/abs/2411.14507)
• Keywords: generative
• Abstract: Generative Pre-trained Transformers (GPTs) have demonstrated remarkable performance across diverse domains through the extensive scaling of model parameters. Recent works observe the redundancy across the transformer blocks and develop compression methods by structured pruning of the unimportant blocks. However, such straightforward elimination will always provide irreversible performance degradation. In this paper, we propose FuseGPT, a novel methodology to recycle the pruned transformer blocks to further recover the model performance. Firstly we introduce a new importance detection metric, Macro Influence (MI), to detect the long-term influence of each transformer block by calculating their loss of information after removal. Then we propose group-level layers fusion, which adopts the parameters in layers of the unimportant blocks and injects them into the corresponding layers inside the neighboring blocks. The fusion is not one-off but through iterative parameter updates by lightweight group-level fine-tuning. Specifically, these injected parameters are frozen but weighted with learnable rank decomposition matrices to reduce the overhead during fine-tuning. Our approach not only works well on large language models but also on large multimodal models. The experiments have shown that, by using modest amounts of data, FuseGPT can outperform previous works in both perplexity and zero-shot task performance.
• 摘要：生成式预训练 Transformer (GPT) 通过大量扩展模型参数，在不同领域表现出色。最近的研究观察到 Transformer 块之间的冗余，并通过对不重要的块进行结构化修剪来开发压缩方法。然而，这种直接的消除总是会导致不可逆转的性能下降。在本文中，我们提出了 FuseGPT，这是一种回收修剪后的 Transformer 块以进一步恢复模型性能的新方法。首先，我们引入了一种新的重要性检测指标——宏观影响 (MI)，通过计算每个 Transformer 块在移除后的信息损失来检测它们的长期影响。然后我们提出了组级层融合，它采用不重要块层中的参数并将它们注入相邻块内的相应层。融合不是一次性的，而是通过轻量级组级微调迭代参数更新。具体来说，这些注入的参数是冻结的，但用可学习的秩分解矩阵加权，以减少微调期间的开销。我们的方法不仅适用于大型语言模型，也适用于大型多模态模型。实验表明，通过使用适量的数据，FuseGPT 可以在困惑度和零样本任务性能方面超越以前的成果。

Title: Variational Autoencoders for Efficient Simulation-Based Inference

• Authors: Mayank Nautiyal, Andrey Shternshis, Andreas Hellander, Prashant Singh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14511
• Pdf URL: https://arxiv.org/pdf/2411.14511
• Copy Paste: [[2411.14511]] Variational Autoencoders for Efficient Simulation-Based Inference(https://arxiv.org/abs/2411.14511)
• Keywords: generative
• Abstract: We present a generative modeling approach based on the variational inference framework for likelihood-free simulation-based inference. The method leverages latent variables within variational autoencoders to efficiently estimate complex posterior distributions arising from stochastic simulations. We explore two variations of this approach distinguished by their treatment of the prior distribution. The first model adapts the prior based on observed data using a multivariate prior network, enhancing generalization across various posterior queries. In contrast, the second model utilizes a standard Gaussian prior, offering simplicity while still effectively capturing complex posterior distributions. We demonstrate the efficacy of these models on well-established benchmark problems, achieving results comparable to flow-based approaches while maintaining computational efficiency and scalability.
• 摘要：我们提出了一种基于变分推理框架的生成建模方法，用于无似然模拟推理。该方法利用变分自动编码器中的潜在变量来有效估计随机模拟产生的复杂后验分布。我们探讨了这种方法的两种变体，它们的区别在于对先验分布的处理。第一个模型使用多元先验网络根据观察到的数据调整先验，增强了对各种后验查询的泛化能力。相比之下，第二个模型使用标准高斯先验，提供简单性，同时仍能有效捕捉复杂的后验分布。我们证明了这些模型在成熟的基准问题上的有效性，实现了与基于流的方法相当的结果，同时保持了计算效率和可扩展性。

Title: Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective

• Authors: Shenglai Zeng, Jiankun Zhang, Bingheng Li, Yuping Lin, Tianqi Zheng, Dante Everaert, Hanqing Lu, Hui Liu, Hui Liu, Yue Xing, Monica Xiao Cheng, Jiliang Tang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14572
• Pdf URL: https://arxiv.org/pdf/2411.14572
• Copy Paste: [[2411.14572]] Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective(https://arxiv.org/abs/2411.14572)
• Keywords: generation
• Abstract: Retrieval-Augmented Generation (RAG) systems have shown promise in enhancing the performance of Large Language Models (LLMs). However, these systems face challenges in effectively integrating external knowledge with the LLM's internal knowledge, often leading to issues with misleading or unhelpful information. This work aims to provide a systematic study on knowledge checking in RAG systems. We conduct a comprehensive analysis of LLM representation behaviors and demonstrate the significance of using representations in knowledge checking. Motivated by the findings, we further develop representation-based classifiers for knowledge filtering. We show substantial improvements in RAG performance, even when dealing with noisy knowledge databases. Our study provides new insights into leveraging LLM representations for enhancing the reliability and effectiveness of RAG systems.
• 摘要：检索增强生成 (RAG) 系统在提高大型语言模型 (LLM) 的性能方面表现出了良好的前景。然而，这些系统在有效地将外部知识与 LLM 的内部知识相结合方面面临挑战，常常导致出现误导性或无用信息的问题。这项工作旨在对 RAG 系统中的知识检查进行系统研究。我们对 LLM 表示行为进行了全面分析，并证明了在知识检查中使用表示的重要性。受这些发现的启发，我们进一步开发了基于表示的知识过滤分类器。即使在处理嘈杂的知识数据库时，我们也显示出 RAG 性能的显着提高。我们的研究为利用 LLM 表示来提高 RAG 系统的可靠性和有效性提供了新的见解。

Title: Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings

• Authors: Pura Peetathawatchai, Wei-Ning Chen, Berivan Isik, Sanmi Koyejo, Albert No
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14639
• Pdf URL: https://arxiv.org/pdf/2411.14639
• Copy Paste: [[2411.14639]] Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings(https://arxiv.org/abs/2411.14639)
• Keywords: generative
• Abstract: We introduce novel methods for adapting diffusion models under differential privacy (DP) constraints, enabling privacy-preserving style and content transfer without fine-tuning. Traditional approaches to private adaptation, such as DP-SGD, incur significant computational overhead and degrade model performance when applied to large, complex models. Our approach instead leverages embedding-based techniques: Universal Guidance and Textual Inversion (TI), adapted with differentially private mechanisms. We apply these methods to Stable Diffusion for style adaptation using two private datasets: a collection of artworks by a single artist and pictograms from the Paris 2024 Olympics. Experimental results show that the TI-based adaptation achieves superior fidelity in style transfer, even under strong privacy guarantees, while both methods maintain high privacy resilience by employing calibrated noise and subsampling strategies. Our findings demonstrate a feasible and efficient pathway for privacy-preserving diffusion model adaptation, balancing data protection with the fidelity of generated images, and offer insights into embedding-driven methods for DP in generative AI applications.
• 摘要：我们引入了在差分隐私 (DP) 约束下调整扩散模型的新方法，无需微调即可实现隐私保护风格和内容传输。传统的隐私适应方法（例如 DP-SGD）在应用于大型复杂模型时会产生大量计算开销并降低模型性能。我们的方法利用基于嵌入的技术：通用指导和文本反转 (TI)，并通过差分隐私机制进行调整。我们将这些方法应用于稳定扩散以使用两个隐私数据集进行风格调整：单个艺术家的艺术作品集和来自 2024 年巴黎奥运会的象形图。实验结果表明，基于 TI 的适应即使在强大的隐私保证下也能实现出色的风格转换保真度，而这两种方法都通过采用校准的噪声和子采样策略保持了较高的隐私弹性。我们的研究结果展示了一种可行且有效的隐私保护扩散模型适应途径，在数据保护与生成图像的保真度之间取得平衡，并为生成 AI 应用中嵌入驱动的 DP 方法提供了见解。

Title: VQalAttent: a Transparent Speech Generation Pipeline based on Transformer-learned VQ-VAE Latent Space

• Authors: Armani Rodriguez, Silvija Kokalj-Filipovic
• Subjects: cs.LG, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.14642
• Pdf URL: https://arxiv.org/pdf/2411.14642
• Copy Paste: [[2411.14642]] VQalAttent: a Transparent Speech Generation Pipeline based on Transformer-learned VQ-VAE Latent Space(https://arxiv.org/abs/2411.14642)
• Keywords: generation, generative
• Abstract: Generating high-quality speech efficiently remains a key challenge for generative models in speech synthesis. This paper introduces VQalAttent, a lightweight model designed to generate fake speech with tunable performance and interpretability. Leveraging the AudioMNIST dataset, consisting of human utterances of decimal digits (0-9), our method employs a two-step architecture: first, a scalable vector quantized autoencoder (VQ-VAE) that compresses audio spectrograms into discrete latent representations, and second, a decoder-only transformer that learns the probability model of these latents. Trained transformer generates similar latent sequences, convertible to audio spectrograms by the VQ-VAE decoder, from which we generate fake utterances. Interpreting statistical and perceptual quality of the fakes, depending on the dimension and the extrinsic information of the latent space, enables guided improvements in larger, commercial generative models. As a valuable tool for understanding and refining audio synthesis, our results demonstrate VQalAttent's capacity to generate intelligible speech samples with limited computational resources, while the modularity and transparency of the training pipeline helps easily correlate the analytics with modular modifications, hence providing insights for the more complex models.
• 摘要：高效生成高质量语音仍然是语音合成生成模型面临的关键挑战。本文介绍了 VQalAttent，这是一种轻量级模型，旨在生成具有可调性能和可解释性的虚假语音。利用由十进制数字（0-9）的人类话语组成的 AudioMNIST 数据集，我们的方法采用了两步架构：首先，可扩展矢量量化自动编码器 (VQ-VAE) 将音频频谱图压缩为离散潜在表示；其次，仅使用解码器的转换器学习这些潜在表示的概率模型。经过训练的转换器会生成类似的潜在序列，VQ-VAE 解码器可将其转换为音频频谱图，然后我们根据这些序列生成虚假话语。根据潜在空间的维度和外部信息解释虚假话语的统计和感知质量，可以引导更大规模的商业生成模型进行改进。作为理解和改进音频合成的宝贵工具，我们的结果证明了 VQalAttent 能够以有限的计算资源生成可理解的语音样本，而训练管道的模块化和透明性有助于轻松地将分析与模块化修改关联起来，从而为更复杂的模型提供见解。

Title: EV-PINN: A Physics-Informed Neural Network for Predicting Electric Vehicle Dynamics

• Authors: Hansol Lim, Jee Won Lee, Jonathan Boyack, Jongseong Brad Choi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14691
• Pdf URL: https://arxiv.org/pdf/2411.14691
• Copy Paste: [[2411.14691]] EV-PINN: A Physics-Informed Neural Network for Predicting Electric Vehicle Dynamics(https://arxiv.org/abs/2411.14691)
• Keywords: generative
• Abstract: An onboard prediction of dynamic parameters (e.g. Aerodynamic drag, rolling resistance) enables accurate path planning for EVs. This paper presents EV-PINN, a Physics-Informed Neural Network approach in predicting instantaneous battery power and cumulative energy consumption during cruising while generalizing to the nonlinear dynamics of an EV. Our method learns real-world parameters such as motor efficiency, regenerative braking efficiency, vehicle mass, coefficient of aerodynamic drag, and coefficient of rolling resistance using automatic differentiation based on dynamics and ensures consistency with ground truth vehicle data. EV-PINN was validated using 15 and 35 minutes of in-situ battery log data from the Tesla Model 3 Long Range and Tesla Model S, respectively. With only vehicle speed and time as inputs, our model achieves high accuracy and generalization to dynamics, with validation losses of 0.002195 and 0.002292, respectively. This demonstrates EV-PINN's effectiveness in estimating parameters and predicting battery usage under actual driving conditions without the need for additional sensors.
• 摘要：车载动态参数（如气动阻力、滚动阻力）预测可为电动汽车实现准确的路径规划。本文介绍了 EV-PINN，这是一种物理信息神经网络方法，用于预测巡航过程中的瞬时电池电量和累积能量消耗，同时推广到电动汽车的非线性动力学。我们的方法使用基于动力学的自动微分来学习真实参数，例如电机效率、再生制动效率、车辆质量、气动阻力系数和滚动阻力系数，并确保与地面真实车辆数据的一致性。EV-PINN 分别使用来自特斯拉 Model 3 Long Range 和特斯拉 Model S 的 15 和 35 分钟现场电池日志数据进行验证。仅以车速和时间作为输入，我们的模型实现了高精度和动力学推广，验证损失分别为 0.002195 和 0.002292。这证明了 EV-PINN 在无需额外传感器的情况下估计参数和预测实际驾驶条件下电池使用情况的有效性。

Title: Any-to-3D Generation via Hybrid Diffusion Supervision

• Authors: Yijun Fan, Yiwei Ma, Jiayi Ji, Xiaoshuai Sun, Rongrong Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14715
• Pdf URL: https://arxiv.org/pdf/2411.14715
• Copy Paste: [[2411.14715]] Any-to-3D Generation via Hybrid Diffusion Supervision(https://arxiv.org/abs/2411.14715)
• Keywords: generation
• Abstract: Recent progress in 3D object generation has been fueled by the strong priors offered by diffusion models. However, existing models are tailored to specific tasks, accommodating only one modality at a time and necessitating retraining to change modalities. Given an image-to-3D model and a text prompt, a naive approach is to convert text prompts to images and then use the image-to-3D model for generation. This approach is both time-consuming and labor-intensive, resulting in unavoidable information loss during modality conversion. To address this, we introduce XBind, a unified framework for any-to-3D generation using cross-modal pre-alignment techniques. XBind integrates an multimodal-aligned encoder with pre-trained diffusion models to generate 3D objects from any modalities, including text, images, and audio. We subsequently present a novel loss function, termed Modality Similarity (MS) Loss, which aligns the embeddings of the modality prompts and the rendered images, facilitating improved alignment of the 3D objects with multiple modalities. Additionally, Hybrid Diffusion Supervision combined with a Three-Phase Optimization process improves the quality of the generated 3D objects. Extensive experiments showcase XBind's broad generation capabilities in any-to-3D scenarios. To our knowledge, this is the first method to generate 3D objects from any modality prompts. Project page: this https URL.
• 摘要：3D 对象生成的最新进展得益于扩散模型提供的强大先验知识。然而，现有模型是针对特定任务量身定制的，一次只能适应一种模态，并且需要重新训练才能更改模态。给定一个图像到 3D 模型和一个文本提示，一种简单的方法是将文本提示转换为图像，然后使用图像到 3D 模型进行生成。这种方法既费时又费力，导致模态转换过程中不可避免地出现信息丢失。为了解决这个问题，我们引入了 XBind，这是一个使用跨模态预对齐技术进行任意到 3D 生成的统一框架。XBind 将多模态对齐编码器与预训练的扩散模型相结合，可以从任何模态（包括文本、图像和音频）生成 3D 对象。随后，我们提出了一种新的损失函数，称为模态相似性 (MS) 损失，它将模态提示和渲染图像的嵌入对齐，从而有助于改善具有多种模态的 3D 对象的对齐。此外，混合扩散监督与三阶段优化过程相结合，提高了生成的 3D 对象的质量。大量实验展示了 XBind 在任意 3D 场景中的广泛生成能力。据我们所知，这是第一种从任何模态提示生成 3D 对象的方法。项目页面：此 https URL。

Title: Enhancing Molecular Design through Graph-based Topological Reinforcement Learning

• Authors: Xiangyu Zhang
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2411.14726
• Pdf URL: https://arxiv.org/pdf/2411.14726
• Copy Paste: [[2411.14726]] Enhancing Molecular Design through Graph-based Topological Reinforcement Learning(https://arxiv.org/abs/2411.14726)
• Keywords: generation
• Abstract: The generation of drug-like molecules is crucial for drug design. Existing reinforcement learning (RL) methods often overlook structural information. However, feature engineering-based methods usually merely focus on binding affinity prediction without substantial molecular modification. To address this, we present Graph-based Topological Reinforcement Learning (GraphTRL), which integrates both chemical and structural data for improved molecular generation. GraphTRL leverages multiscale weighted colored graphs (MWCG) and persistent homology, combined with molecular fingerprints, as the state space for RL. Evaluations show that GraphTRL outperforms existing methods in binding affinity prediction, offering a promising approach to accelerate drug discovery.
• 摘要：类药物分子的生成对于药物设计至关重要。现有的强化学习 (RL) 方法经常忽略结构信息。然而，基于特征工程的方法通常仅关注结合亲和力预测，而没有实质性的分子修改。为了解决这个问题，我们提出了基于图的拓扑强化学习 (GraphTRL)，它整合了化学和结构数据以改进分子生成。GraphTRL 利用多尺度加权彩色图 (MWCG) 和持久同源性，结合分子指纹作为 RL 的状态空间。评估表明，GraphTRL 在结合亲和力预测方面优于现有方法，为加速药物发现提供了一种有希望的方法。

Title: TEXGen: a Generative Diffusion Model for Mesh Textures

• Authors: Xin Yu, Ze Yuan, Yuan-Chen Guo, Ying-Tian Liu, JianHui Liu, Yangguang Li, Yan-Pei Cao, Ding Liang, Xiaojuan Qi
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.14740
• Pdf URL: https://arxiv.org/pdf/2411.14740
• Copy Paste: [[2411.14740]] TEXGen: a Generative Diffusion Model for Mesh Textures(https://arxiv.org/abs/2411.14740)
• Keywords: generative
• Abstract: While high-quality texture maps are essential for realistic 3D asset rendering, few studies have explored learning directly in the texture space, especially on large-scale datasets. In this work, we depart from the conventional approach of relying on pre-trained 2D diffusion models for test-time optimization of 3D textures. Instead, we focus on the fundamental problem of learning in the UV texture space itself. For the first time, we train a large diffusion model capable of directly generating high-resolution texture maps in a feed-forward manner. To facilitate efficient learning in high-resolution UV spaces, we propose a scalable network architecture that interleaves convolutions on UV maps with attention layers on point clouds. Leveraging this architectural design, we train a 700 million parameter diffusion model that can generate UV texture maps guided by text prompts and single-view images. Once trained, our model naturally supports various extended applications, including text-guided texture inpainting, sparse-view texture completion, and text-driven texture synthesis. Project page is at this http URL.
• 摘要：虽然高质量的纹理图对于逼真的 3D 资产渲染至关重要，但很少有研究直接探索纹理空间中的学习，尤其是在大型数据集上。在这项工作中，我们摆脱了依赖预训练的 2D 扩散模型进行 3D 纹理测试时优化的传统方法。相反，我们专注于 UV 纹理空间本身的学习基本问题。我们首次训练了一个能够以前馈方式直接生成高分辨率纹理图的大型扩散模型。为了促进高分辨率 UV 空间中的高效学习，我们提出了一种可扩展的网络架构，将 UV 图上的卷积与点云上的注意层交错。利用这种架构设计，我们训练了一个 7 亿参数扩散模型，该模型可以生成由文本提示和单视图图像引导的 UV 纹理图。经过训练后，我们的模型自然支持各种扩展应用，包括文本引导的纹理修复、稀疏视图纹理完成和文本驱动的纹理合成。项目页面位于此 http URL。

Title: FairAdapter: Detecting AI-generated Images with Improved Fairness

• Authors: Feng Ding, Jun Zhang, Xinan He, Jianfeng Xu
• Subjects: cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.14755
• Pdf URL: https://arxiv.org/pdf/2411.14755
• Copy Paste: [[2411.14755]] FairAdapter: Detecting AI-generated Images with Improved Fairness(https://arxiv.org/abs/2411.14755)
• Keywords: generative
• Abstract: The high-quality, realistic images generated by generative models pose significant challenges for exposing this http URL far, data-driven deep neural networks have been justified as the most efficient forensics tools for the challenges. However, they may be over-fitted to certain semantics, resulting in considerable inconsistency in detection performance across different contents of generated samples. It could be regarded as an issue of detection fairness. In this paper, we propose a novel framework named Fairadapter to tackle the issue. In comparison with existing state-of-the-art methods, our model achieves improved fairness performance. Our project: this https URL
• 摘要：生成模型生成的高质量逼真图像对曝光此 http URL 提出了重大挑战，到目前为止，数据驱动的深度神经网络已被证明是应对挑战的最有效取证工具。然而，它们可能过度拟合某些语义，导致生成样本的不同内容之间的检测性能存在相当大的不一致。这可以看作是一个检测公平性问题。在本文中，我们提出了一个名为 Fairadapter 的新框架来解决这个问题。与现有的最先进方法相比，我们的模型实现了更好的公平性。我们的项目：此 https URL

Title: Efficient Long Video Tokenization via Coordinated-based Patch Reconstruction

• Authors: Huiwon Jang, Sihyun Yu, Jinwoo Shin, Pieter Abbeel, Younggyo Seo
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14762
• Pdf URL: https://arxiv.org/pdf/2411.14762
• Copy Paste: [[2411.14762]] Efficient Long Video Tokenization via Coordinated-based Patch Reconstruction(https://arxiv.org/abs/2411.14762)
• Keywords: generative
• Abstract: Efficient tokenization of videos remains a challenge in training vision models that can process long videos. One promising direction is to develop a tokenizer that can encode long video clips, as it would enable the tokenizer to leverage the temporal coherence of videos better for tokenization. However, training existing tokenizers on long videos often incurs a huge training cost as they are trained to reconstruct all the frames at once. In this paper, we introduce CoordTok, a video tokenizer that learns a mapping from coordinate-based representations to the corresponding patches of input videos, inspired by recent advances in 3D generative models. In particular, CoordTok encodes a video into factorized triplane representations and reconstructs patches that correspond to randomly sampled $(x,y,t)$ coordinates. This allows for training large tokenizer models directly on long videos without requiring excessive training resources. Our experiments show that CoordTok can drastically reduce the number of tokens for encoding long video clips. For instance, CoordTok can encode a 128-frame video with 128$\times$128 resolution into 1280 tokens, while baselines need 6144 or 8192 tokens to achieve similar reconstruction quality. We further show that this efficient video tokenization enables memory-efficient training of a diffusion transformer that can generate 128 frames at once.
• 摘要：在训练能够处理长视频的视觉模型时，视频的有效标记化仍然是一个挑战。一个有希望的方向是开发一个可以编码长视频片段的标记器，因为它将使标记器能够更好地利用视频的时间连贯性进行标记化。然而，在长视频上训练现有的标记器通常会产生巨大的训练成本，因为它们被训练为一次重建所有帧。在本文中，我们介绍了 CoordTok，这是一个视频标记器，它学习从基于坐标的表示到输入视频的相应块的映射，灵感来自 3D 生成模型的最新进展。具体来说，CoordTok 将视频编码为分解的三平面表示，并重建与随机采样的 $(x,y,t)$ 坐标相对应的块。这允许直接在长视频上训练大型标记器模型，而无需过多的训练资源。我们的实验表明，CoordTok 可以大大减少编码长视频片段的标记数量。例如，CoordTok 可以将分辨率为 128$\times$128 的 128 帧视频编码为 1280 个标记，而基线需要 6144 或 8192 个标记才能实现类似的重建质量。我们进一步表明，这种高效的视频标记化可以实现内存高效的扩散变换器训练，该变换器可以一次生成 128 帧。

Title: Reconciling Semantic Controllability and Diversity for Remote Sensing Image Synthesis with Hybrid Semantic Embedding

• Authors: Junde Liu, Danpei Zhao, Bo Yuan, Wentao Li, Tian Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14781
• Pdf URL: https://arxiv.org/pdf/2411.14781
• Copy Paste: [[2411.14781]] Reconciling Semantic Controllability and Diversity for Remote Sensing Image Synthesis with Hybrid Semantic Embedding(https://arxiv.org/abs/2411.14781)
• Keywords: generative
• Abstract: Significant advancements have been made in semantic image synthesis in remote sensing. However, existing methods still face formidable challenges in balancing semantic controllability and diversity. In this paper, we present a Hybrid Semantic Embedding Guided Generative Adversarial Network (HySEGGAN) for controllable and efficient remote sensing image synthesis. Specifically, HySEGGAN leverages hierarchical information from a single source. Motivated by feature description, we propose a hybrid semantic Embedding method, that coordinates fine-grained local semantic layouts to characterize the geometric structure of remote sensing objects without extra information. Besides, a Semantic Refinement Network (SRN) is introduced, incorporating a novel loss function to ensure fine-grained semantic feedback. The proposed approach mitigates semantic confusion and prevents geometric pattern collapse. Experimental results indicate that the method strikes an excellent balance between semantic controllability and diversity. Furthermore, HySEGGAN significantly improves the quality of synthesized images and achieves state-of-the-art performance as a data augmentation technique across multiple datasets for downstream tasks.
• 摘要：遥感领域的语义图像合成已经取得了重大进展。然而，现有的方法在平衡语义可控性和多样性方面仍然面临着巨大的挑战。在本文中，我们提出了一种混合语义嵌入引导生成对抗网络（HySEGGAN），用于可控和高效的遥感图像合成。具体而言，HySEGGAN 利用来自单一来源的分层信息。受特征描述的启发，我们提出了一种混合语义嵌入方法，该方法协调细粒度的局部语义布局来表征遥感对象的几何结构，而无需额外信息。此外，还引入了语义细化网络（SRN），结合了一种新颖的损失函数来确保细粒度的语义反馈。所提出的方法减轻了语义混淆并防止了几何模式崩溃。实验结果表明，该方法在语义可控性和多样性之间取得了极好的平衡。此外，HySEGGAN 显著提高了合成图像的质量，并作为跨下游任务的多个数据集的数据增强技术实现了最先进的性能。

Title: Style-Friendly SNR Sampler for Style-Driven Generation

• Authors: Jooyoung Choi, Chaehun Shin, Yeongtak Oh, Heeseung Kim, Sungroh Yoon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14793
• Pdf URL: https://arxiv.org/pdf/2411.14793
• Copy Paste: [[2411.14793]] Style-Friendly SNR Sampler for Style-Driven Generation(https://arxiv.org/abs/2411.14793)
• Keywords: generation
• Abstract: Recent large-scale diffusion models generate high-quality images but struggle to learn new, personalized artistic styles, which limits the creation of unique style templates. Fine-tuning with reference images is the most promising approach, but it often blindly utilizes objectives and noise level distributions used for pre-training, leading to suboptimal style alignment. We propose the Style-friendly SNR sampler, which aggressively shifts the signal-to-noise ratio (SNR) distribution toward higher noise levels during fine-tuning to focus on noise levels where stylistic features emerge. This enables models to better capture unique styles and generate images with higher style alignment. Our method allows diffusion models to learn and share new "style templates", enhancing personalized content creation. We demonstrate the ability to generate styles such as personal watercolor paintings, minimal flat cartoons, 3D renderings, multi-panel images, and memes with text, thereby broadening the scope of style-driven generation.
• 摘要：最近的大规模传播模型可以生成高质量的图像，但很难学习新的个性化艺术风格，这限制了独特风格模板的创建。使用参考图像进行微调是最有前途的方法，但它通常盲目地利用用于预训练的目标和噪声水平分布，导致风格对齐不理想。我们提出了风格友好的 SNR 采样器，它在微调过程中积极地将信噪比 (SNR) 分布移向更高的噪声水平，以专注于出现风格特征的噪声水平。这使模型能够更好地捕捉独特的风格并生成具有更高风格对齐度的图像。我们的方法允许传播模型学习和共享新的“风格模板”，增强个性化内容创作。我们展示了生成个人水彩画、极简平面卡通、3D 渲染、多面板图像和带有文本的模因等风格的能力，从而扩大了风格驱动生成的范围。

Title: Harlequin: Color-driven Generation of Synthetic Data for Referring Expression Comprehension

• Authors: Luca Parolari, Elena Izzo, Lamberto Ballan
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14807
• Pdf URL: https://arxiv.org/pdf/2411.14807
• Copy Paste: [[2411.14807]] Harlequin: Color-driven Generation of Synthetic Data for Referring Expression Comprehension(https://arxiv.org/abs/2411.14807)
• Keywords: generation
• Abstract: Referring Expression Comprehension (REC) aims to identify a particular object in a scene by a natural language expression, and is an important topic in visual language understanding. State-of-the-art methods for this task are based on deep learning, which generally requires expensive and manually labeled annotations. Some works tackle the problem with limited-supervision learning or relying on Large Vision and Language Models. However, the development of techniques to synthesize labeled data is overlooked. In this paper, we propose a novel framework that generates artificial data for the REC task, taking into account both textual and visual modalities. At first, our pipeline processes existing data to create variations in the annotations. Then, it generates an image using altered annotations as guidance. The result of this pipeline is a new dataset, called Harlequin, made by more than 1M queries. This approach eliminates manual data collection and annotation, enabling scalability and facilitating arbitrary complexity. We pre-train three REC models on Harlequin, then fine-tuned and evaluated on human-annotated datasets. Our experiments show that the pre-training on artificial data is beneficial for performance.
• 摘要：指涉表达理解 (REC) 旨在通过自然语言表达识别场景中的特定对象，是视觉语言理解中的一个重要主题。此任务的最先进的方法基于深度学习，这通常需要昂贵且手动标记的注释。一些工作通过有限监督学习或依赖大型视觉和语言模型来解决这个问题。然而，合成标记数据的技术的发展被忽视了。在本文中，我们提出了一个新颖的框架，该框架为 REC 任务生成人工数据，同时考虑文本和视觉模态。首先，我们的管道处理现有数据以创建注释中的变化。然后，它使用更改后的注释作为指导来生成图像。该管道的结果是一个由超过 100 万个查询组成的新数据集，称为 Harlequin。这种方法消除了手动数据收集和注释，实现了可扩展性并促进了任意复杂性。我们在 Harlequin 上预训练了三个 REC 模型，然后在人工注释的数据集上进行了微调和评估。我们的实验表明，对人工数据进行预训练有利于提高性能。

Title: High-Resolution Image Synthesis via Next-Token Prediction

• Authors: Dengsheng Chen, Jie Hu, Tiezhu Yue, Xiaoming Wei
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14808
• Pdf URL: https://arxiv.org/pdf/2411.14808
• Copy Paste: [[2411.14808]] High-Resolution Image Synthesis via Next-Token Prediction(https://arxiv.org/abs/2411.14808)
• Keywords: generation
• Abstract: Denoising with a Joint-Embedding Predictive Architecture (D-JEPA), an autoregressive model, has demonstrated outstanding performance in class-conditional image generation. However, the application of next-token prediction in high-resolution text-to-image generation remains underexplored. In this paper, we introduce D-JEPA$\cdot$T2I, an extension of D-JEPA incorporating flow matching loss, designed to enable data-efficient continuous resolution learning. D-JEPA$\cdot$T2I leverages a multimodal visual transformer to effectively integrate textual and visual features and adopts Visual Rotary Positional Embedding (VoPE) to facilitate continuous resolution learning. Furthermore, we devise a data feedback mechanism that significantly enhances data utilization efficiency. For the first time, we achieve state-of-the-art \textbf{high-resolution} image synthesis via next-token prediction. The experimental code and pretrained models will be open-sourced at \url{this https URL}.
• 摘要：使用联合嵌入预测架构 (D-JEPA) 进行去噪是一种自回归模型，在类条件图像生成中表现出色。然而，下一个标记预测在高分辨率文本到图像生成中的应用仍未得到充分探索。在本文中，我们介绍了 D-JEPA$\cdot$T2I，这是 D-JEPA 的扩展，结合了流匹配损失，旨在实现数据高效的连续分辨率学习。D-JEPA$\cdot$T2I 利用多模态视觉转换器有效地整合文本和视觉特征，并采用视觉旋转位置嵌入 (VoPE) 来促进连续分辨率学习。此外，我们设计了一种数据反馈机制，可显著提高数据利用效率。我们首次通过下一个标记预测实现了最先进的 \textbf{高分辨率} 图像合成。实验代码和预训练模型将在 \url{此 https URL} 上开源。

Title: Unsupervised Multi-view UAV Image Geo-localization via Iterative Rendering

• Authors: Haoyuan Li, Chang Xu, Wen Yang, Li Mi, Huai Yu, Haijian Zhang
• Subjects: cs.CV, cs.RO, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.14816
• Pdf URL: https://arxiv.org/pdf/2411.14816
• Copy Paste: [[2411.14816]] Unsupervised Multi-view UAV Image Geo-localization via Iterative Rendering(https://arxiv.org/abs/2411.14816)
• Keywords: generation
• Abstract: Unmanned Aerial Vehicle (UAV) Cross-View Geo-Localization (CVGL) presents significant challenges due to the view discrepancy between oblique UAV images and overhead satellite images. Existing methods heavily rely on the supervision of labeled datasets to extract viewpoint-invariant features for cross-view retrieval. However, these methods have expensive training costs and tend to overfit the region-specific cues, showing limited generalizability to new regions. To overcome this issue, we propose an unsupervised solution that lifts the scene representation to 3d space from UAV observations for satellite image generation, providing robust representation against view distortion. By generating orthogonal images that closely resemble satellite views, our method reduces view discrepancies in feature representation and mitigates shortcuts in region-specific image pairing. To further align the rendered image's perspective with the real one, we design an iterative camera pose updating mechanism that progressively modulates the rendered query image with potential satellite targets, eliminating spatial offsets relative to the reference images. Additionally, this iterative refinement strategy enhances cross-view feature invariance through view-consistent fusion across iterations. As such, our unsupervised paradigm naturally avoids the problem of region-specific overfitting, enabling generic CVGL for UAV images without feature fine-tuning or data-driven training. Experiments on the University-1652 and SUES-200 datasets demonstrate that our approach significantly improves geo-localization accuracy while maintaining robustness across diverse regions. Notably, without model fine-tuning or paired training, our method achieves competitive performance with recent supervised methods.
• 摘要：由于倾斜无人机图像和高空卫星图像之间的视角差异，无人机 (UAV) 跨视角地理定位 (CVGL) 面临巨大挑战。现有方法严重依赖标记数据集的监督来提取用于跨视角检索的视点不变特征。然而，这些方法的训练成本高昂，并且往往会过度拟合特定区域的线索，对新区域的通用性有限。为了解决这个问题，我们提出了一种无监督解决方案，将场景表示从无人机观测提升到 3d 空间以生成卫星图像，提供对视角失真的鲁棒表示。通过生成与卫星视图非常相似的正交图像，我们的方法减少了特征表示中的视角差异，并减轻了特定区域图像配对中的捷径。为了进一步将渲染图像的视角与真实视角对齐，我们设计了一种迭代相机姿势更新机制，该机制逐步调节渲染查询图像与潜在卫星目标，消除相对于参考图像的空间偏移。此外，这种迭代细化策略通过跨迭代的视图一致融合增强了跨视图特征不变性。因此，我们的无监督范式自然避免了区域特定过度拟合的问题，无需特征微调或数据驱动训练即可实现无人机图像的通用 CVGL。在 University-1652 和 SUES-200 数据集上进行的实验表明，我们的方法显著提高了地理定位精度，同时保持了跨不同区域的稳健性。值得注意的是，无需模型微调或配对训练，我们的方法就能实现与最近的监督方法相媲美的性能。

Title: Latent Schrodinger Bridge: Prompting Latent Diffusion for Fast Unpaired Image-to-Image Translation

• Authors: Jeongsol Kim, Beomsu Kim, Jong Chul Ye
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14863
• Pdf URL: https://arxiv.org/pdf/2411.14863
• Copy Paste: [[2411.14863]] Latent Schrodinger Bridge: Prompting Latent Diffusion for Fast Unpaired Image-to-Image Translation(https://arxiv.org/abs/2411.14863)
• Keywords: generation
• Abstract: Diffusion models (DMs), which enable both image generation from noise and inversion from data, have inspired powerful unpaired image-to-image (I2I) translation algorithms. However, they often require a larger number of neural function evaluations (NFEs), limiting their practical applicability. In this paper, we tackle this problem with Schrodinger Bridges (SBs), which are stochastic differential equations (SDEs) between distributions with minimal transport cost. We analyze the probability flow ordinary differential equation (ODE) formulation of SBs, and observe that we can decompose its vector field into a linear combination of source predictor, target predictor, and noise predictor. Inspired by this observation, we propose Latent Schrodinger Bridges (LSBs) that approximate the SB ODE via pre-trained Stable Diffusion, and develop appropriate prompt optimization and change of variables formula to match the training and inference between distributions. We demonstrate that our algorithm successfully conduct competitive I2I translation in unsupervised setting with only a fraction of computation cost required by previous DM-based I2I methods.
• 摘要：扩散模型 (DM) 既可以从噪声生成图像，也可以从数据进行反演，它启发了强大的非配对图像到图像 (I2I) 转换算法。然而，它们通常需要大量的神经功能评估 (NFE)，这限制了它们的实际适用性。在本文中，我们使用薛定谔桥 (SB) 来解决这个问题，薛定谔桥是分布之间的随机微分方程 (SDE)，具有最小的传输成本。我们分析了 SB 的概率流常微分方程 (ODE) 公式，并观察到我们可以将其矢量场分解为源预测器、目标预测器和噪声预测器的线性组合。受此观察的启发，我们提出了潜在薛定谔桥 (LSB)，它通过预训练的稳定扩散来近似 SB ODE，并开发了适当的即时优化和变量变化公式以匹配分布之间的训练和推理。我们证明，我们的算法在无监督环境中成功地进行了竞争性 I2I 翻译，且仅需以前基于 DM 的 I2I 方法所需计算成本的一小部分。

Title: Prioritize Denoising Steps on Diffusion Model Preference Alignment via Explicit Denoised Distribution Estimation

• Authors: Dingyuan Shi, Yong Wang, Hangyu Li, Xiangxiang Chu
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14871
• Pdf URL: https://arxiv.org/pdf/2411.14871
• Copy Paste: [[2411.14871]] Prioritize Denoising Steps on Diffusion Model Preference Alignment via Explicit Denoised Distribution Estimation(https://arxiv.org/abs/2411.14871)
• Keywords: generation
• Abstract: Diffusion models have shown remarkable success in text-to-image generation, making alignment methods for these models increasingly important. A key challenge is the sparsity of preference labels, which are typically available only at the terminal of denoising trajectories. This raises the issue of how to assign credit across denoising steps based on these sparse labels. In this paper, we propose Denoised Distribution Estimation (DDE), a novel method for credit assignment. Unlike previous approaches that rely on auxiliary models or hand-crafted schemes, DDE derives its strategy more explicitly. The proposed DDE directly estimates the terminal denoised distribution from the perspective of each step. It is equipped with two estimation strategies and capable of representing the entire denoising trajectory with a single model inference. Theoretically and empirically, we show that DDE prioritizes optimizing the middle part of the denoising trajectory, resulting in a novel and effective credit assignment scheme. Extensive experiments demonstrate that our approach achieves superior performance, both quantitatively and qualitatively.
• 摘要：扩散模型在文本到图像生成中取得了显著的成功，使得这些模型的对齐方法变得越来越重要。一个关键的挑战是偏好标签的稀疏性，这些标签通常仅在去噪轨迹的终端可用。这就提出了一个问题，即如何根据这些稀疏的标签在去噪步骤之间分配信用。在本文中，我们提出了一种新的信用分配方法，即去噪分布估计 (DDE)。与以前依赖辅助模型或手工方案的方法不同，DDE 更明确地推导出其策略。所提出的 DDE 直接从每个步骤的角度估计终端去噪分布。它配备了两种估计策略，能够用一个模型推理来表示整个去噪轨迹。从理论和经验上看，我们表明 DDE 优先优化去噪轨迹的中间部分，从而产生一种新颖有效的信用分配方案。大量实验表明，我们的方法在数量和质量上都取得了优异的性能。

Title: Morph: A Motion-free Physics Optimization Framework for Human Motion Generation

• Authors: Zhuo Li, Mingshuang Luo, Ruibing Hou, Xin Zhao, Hao Liu, Hong Chang, Zimo Liu, Chen Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14951
• Pdf URL: https://arxiv.org/pdf/2411.14951
• Copy Paste: [[2411.14951]] Morph: A Motion-free Physics Optimization Framework for Human Motion Generation(https://arxiv.org/abs/2411.14951)
• Keywords: generation
• Abstract: Human motion generation plays a vital role in applications such as digital humans and humanoid robot control. However, most existing approaches disregard physics constraints, leading to the frequent production of physically implausible motions with pronounced artifacts such as floating and foot sliding. In this paper, we propose \textbf{Morph}, a \textbf{Mo}tion-f\textbf{r}ee \textbf{ph}ysics optimization framework, comprising a Motion Generator and a Motion Physics Refinement module, for enhancing physical plausibility without relying on costly real-world motion data. Specifically, the Motion Generator is responsible for providing large-scale synthetic motion data, while the Motion Physics Refinement Module utilizes these synthetic data to train a motion imitator within a physics simulator, enforcing physical constraints to project the noisy motions into a physically-plausible space. These physically refined motions, in turn, are used to fine-tune the Motion Generator, further enhancing its capability. Experiments on both text-to-motion and music-to-dance generation tasks demonstrate that our framework achieves state-of-the-art motion generation quality while improving physical plausibility drastically.
• 摘要：人体运动生成在数字人和人形机器人控制等应用中起着至关重要的作用。然而，大多数现有方法都忽略了物理约束，导致经常产生物理上不合理的运动，并伴有明显的伪影，例如漂浮和脚部滑动。在本文中，我们提出了 \textbf{Morph}，这是一个 \textbf{Mo}tion-f\textbf{r}ee \textbf{ph}ysics 优化框架，包括运动生成器和运动物理细化模块，用于在不依赖昂贵的现实世界运动数据的情况下增强物理合理性。具体而言，运动生成器负责提供大规模合成运动数据，而运动物理细化模块利用这些合成数据在物理模拟器中训练运动模仿器，强制执行物理约束以将嘈杂的运动投射到物理上合理的空间中。反过来，这些物理上细化的运动又用于微调运动生成器，进一步增强其能力。在文本到动作和音乐到舞蹈生成任务上的实验表明，我们的框架实现了最先进的运动生成质量，同时大幅提高了物理可行性。

Title: FloAt: Flow Warping of Self-Attention for Clothing Animation Generation

• Authors: Swasti Shreya Mishra, Kuldeep Kulkarni, Duygu Ceylan, Balaji Vasan Srinivasan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.15028
• Pdf URL: https://arxiv.org/pdf/2411.15028
• Copy Paste: [[2411.15028]] FloAt: Flow Warping of Self-Attention for Clothing Animation Generation(https://arxiv.org/abs/2411.15028)
• Keywords: generation
• Abstract: We propose a diffusion model-based approach, FloAtControlNet to generate cinemagraphs composed of animations of human clothing. We focus on human clothing like dresses, skirts and pants. The input to our model is a text prompt depicting the type of clothing and the texture of clothing like leopard, striped, or plain, and a sequence of normal maps that capture the underlying animation that we desire in the output. The backbone of our method is a normal-map conditioned ControlNet which is operated in a training-free regime. The key observation is that the underlying animation is embedded in the flow of the normal maps. We utilize the flow thus obtained to manipulate the self-attention maps of appropriate layers. Specifically, the self-attention maps of a particular layer and frame are recomputed as a linear combination of itself and the self-attention maps of the same layer and the previous frame, warped by the flow on the normal maps of the two frames. We show that manipulating the self-attention maps greatly enhances the quality of the clothing animation, making it look more natural as well as suppressing the background artifacts. Through extensive experiments, we show that the method proposed beats all baselines both qualitatively in terms of visual results and user study. Specifically, our method is able to alleviate the background flickering that exists in other diffusion model-based baselines that we consider. In addition, we show that our method beats all baselines in terms of RMSE and PSNR computed using the input normal map sequences and the normal map sequences obtained from the output RGB frames. Further, we show that well-established evaluation metrics like LPIPS, SSIM, and CLIP scores that are generally for visual quality are not necessarily suitable for capturing the subtle motions in human clothing animations.
• 摘要：我们提出了一种基于扩散模型的方法 FloAtControlNet，用于生成由人体服装动画组成的电影摄影。我们专注于人体服装，如连衣裙、短裙和裤子。我们模型的输入是一个文本提示，描述服装的类型和服装的纹理，如豹纹、条纹或纯色，以及一系列法线贴图，这些法线贴图捕捉了我们想要在输出中呈现的底层动画。我们方法的骨干是一个法线贴图条件化 ControlNet，它在无训练模式下运行。关键的观察是底层动画嵌入在法线贴图的流中。我们利用这样获得的流来操纵适当层的自注意力图。具体来说，特定层和帧的自注意力图被重新计算为其自身与同一层和前一帧的自注意力图的线性组合，并被两帧的法线贴图上的流扭曲。我们表明，操纵自注意力图可以大大提高服装动画的质量，使其看起来更自然，并抑制背景伪影。通过大量实验，我们表明，所提出的方法在视觉效果和用户研究方面都优于所有基线。具体来说，我们的方法能够缓解我们所考虑的其他基于扩散模型的基线中存在的背景闪烁。此外，我们表明，在使用输入法线图序列和从输出 RGB 帧获得的法线图序列计算的 RMSE 和 PSNR 方面，我们的方法优于所有基线。此外，我们表明，通常用于视觉质量的完善评估指标（如 LPIPS、SSIM 和 CLIP 分数）不一定适合捕捉人体服装动画中的细微动作。

Title: HeadRouter: A Training-free Image Editing Framework for MM-DiTs by Adaptively Routing Attention Heads

• Authors: Yu Xu, Fan Tang, Juan Cao, Yuxin Zhang, Xiaoyu Kong, Jintao Li, Oliver Deussen, Tong-Yee Lee
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.15034
• Pdf URL: https://arxiv.org/pdf/2411.15034
• Copy Paste: [[2411.15034]] HeadRouter: A Training-free Image Editing Framework for MM-DiTs by Adaptively Routing Attention Heads(https://arxiv.org/abs/2411.15034)
• Keywords: generation
• Abstract: Diffusion Transformers (DiTs) have exhibited robust capabilities in image generation tasks. However, accurate text-guided image editing for multimodal DiTs (MM-DiTs) still poses a significant challenge. Unlike UNet-based structures that could utilize self/cross-attention maps for semantic editing, MM-DiTs inherently lack support for explicit and consistent incorporated text guidance, resulting in semantic misalignment between the edited results and texts. In this study, we disclose the sensitivity of different attention heads to different image semantics within MM-DiTs and introduce HeadRouter, a training-free image editing framework that edits the source image by adaptively routing the text guidance to different attention heads in MM-DiTs. Furthermore, we present a dual-token refinement module to refine text/image token representations for precise semantic guidance and accurate region expression. Experimental results on multiple benchmarks demonstrate HeadRouter's performance in terms of editing fidelity and image quality.
• 摘要：扩散变换器 (DiT) 在图像生成任务中表现出了强大的能力。然而，对多模态 DiT (MM-DiT) 进行准确的文本引导图像编辑仍然是一个重大挑战。与可以利用自/交叉注意力图进行语义编辑的基于 UNet 的结构不同，MM-DiT 本质上缺乏对明确和一致的合并文本指导的支持，导致编辑结果和文本之间的语义不一致。在本研究中，我们揭示了不同注意力头对 MM-DiT 中不同图像语义的敏感性，并介绍了 HeadRouter，这是一个无需训练的图像编辑框架，它通过自适应地将文本指导路由到 MM-DiT 中的不同注意力头来编辑源图像。此外，我们提出了一个双标记细化模块来细化文本/图像标记表示，以实现精确的语义指导和准确的区域表达。多个基准上的实验结果证明了 HeadRouter 在编辑保真度和图像质量方面的性能。

Title: OminiControl: Minimal and Universal Control for Diffusion Transformer

• Authors: Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, Xinchao Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.15098
• Pdf URL: https://arxiv.org/pdf/2411.15098
• Copy Paste: [[2411.15098]] OminiControl: Minimal and Universal Control for Diffusion Transformer(https://arxiv.org/abs/2411.15098)
• Keywords: generation
• Abstract: In this paper, we introduce OminiControl, a highly versatile and parameter-efficient framework that integrates image conditions into pre-trained Diffusion Transformer (DiT) models. At its core, OminiControl leverages a parameter reuse mechanism, enabling the DiT to encode image conditions using itself as a powerful backbone and process them with its flexible multi-modal attention processors. Unlike existing methods, which rely heavily on additional encoder modules with complex architectures, OminiControl (1) effectively and efficiently incorporates injected image conditions with only ~0.1% additional parameters, and (2) addresses a wide range of image conditioning tasks in a unified manner, including subject-driven generation and spatially-aligned conditions such as edges, depth, and more. Remarkably, these capabilities are achieved by training on images generated by the DiT itself, which is particularly beneficial for subject-driven generation. Extensive evaluations demonstrate that OminiControl outperforms existing UNet-based and DiT-adapted models in both subject-driven and spatially-aligned conditional generation. Additionally, we release our training dataset, Subjects200K, a diverse collection of over 200,000 identity-consistent images, along with an efficient data synthesis pipeline to advance research in subject-consistent generation.
• 摘要：在本文中，我们介绍了 OminiControl，这是一个高度通用且参数高效的框架，它将图像条件集成到预先训练的扩散变换器 (DiT) 模型中。OminiControl 的核心是利用参数重用机制，使 DiT 能够使用自身作为强大的骨干对图像条件进行编码，并使用其灵活的多模态注意处理器对其进行处理。与现有方法不同，这些方法严重依赖具有复杂架构的附加编码器模块，OminiControl (1) 有效且高效地整合了注入的图像条件，仅需约 0.1% 的附加参数，并且 (2) 以统一的方式处理广泛的图像调节任务，包括主题驱动生成和空间对齐条件，例如边缘、深度等。值得注意的是，这些功能是通过对 DiT 本身生成的图像进行训练来实现的，这对主题驱动生成特别有益。广泛的评估表明，OminiControl 在主题驱动和空间对齐条件生成方面均优于现有的基于 UNet 和 DiT 的模型。此外，我们还发布了我们的训练数据集 Subjects200K，这是一个包含超过 200,000 个身份一致图像的多样化集合，以及一个高效的数据合成管道，以推进主题一致生成的研究。

Title: AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution

• Authors: Fengyuan Liu, Nikhil Kandpal, Colin Raffel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.15102
• Pdf URL: https://arxiv.org/pdf/2411.15102
• Copy Paste: [[2411.15102]] AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution(https://arxiv.org/abs/2411.15102)
• Keywords: generation
• Abstract: The influence of contextual input on the behavior of large language models (LLMs) has prompted the development of context attribution methods that aim to quantify each context span's effect on an LLM's generations. The leave-one-out (LOO) error, which measures the change in the likelihood of the LLM's response when a given span of the context is removed, provides a principled way to perform context attribution, but can be prohibitively expensive to compute for large models. In this work, we introduce AttriBoT, a series of novel techniques for efficiently computing an approximation of the LOO error for context attribution. Specifically, AttriBoT uses cached activations to avoid redundant operations, performs hierarchical attribution to reduce computation, and emulates the behavior of large target models with smaller proxy models. Taken together, AttriBoT can provide a >300x speedup while remaining more faithful to a target model's LOO error than prior context attribution methods. This stark increase in performance makes computing context attributions for a given response 30x faster than generating the response itself, empowering real-world applications that require computing attributions at scale. We release a user-friendly and efficient implementation of AttriBoT to enable efficient LLM interpretability as well as encourage future development of efficient context attribution methods.
• 摘要：上下文输入对大型语言模型 (LLM) 行为的影响促使人们开发上下文归因方法，旨在量化每个上下文跨度对 LLM 生成的影响。留一法 (LOO) 误差测量当移除给定的上下文跨度时 LLM 响应的可能性的变化，它提供了一种执行上下文归因的原则性方法，但对于大型模型来说，计算成本可能过高。在这项工作中，我们介绍了 AttriBoT，这是一系列用于有效计算上下文归因的 LOO 误差近似值的新技术。具体而言，AttriBoT 使用缓存激活来避免冗余操作，执行分层归因以减少计算，并使用较小的代理模型模拟大型目标模型的行为。总之，与之​​前的上下文归因方法相比，AttriBoT 可以提供 300 倍以上的加速，同时更忠实于目标模型的 LOO 误差。这种显著的性能提升使得计算给定响应的上下文归因比生成响应本身快 30 倍，从而为需要大规模计算归因的实际应用程序提供支持。我们发布了用户友好且高效的 AttriBoT 实现，以实现高效的 LLM 可解释性，并鼓励未来开发高效的上下文归因方法。

Title: Efficient Pruning of Text-to-Image Models: Insights from Pruning Stable Diffusion

• Authors: Samarth N Ramesh, Zhixue Zhao
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.15113
• Pdf URL: https://arxiv.org/pdf/2411.15113
• Copy Paste: [[2411.15113]] Efficient Pruning of Text-to-Image Models: Insights from Pruning Stable Diffusion(https://arxiv.org/abs/2411.15113)
• Keywords: generation
• Abstract: As text-to-image models grow increasingly powerful and complex, their burgeoning size presents a significant obstacle to widespread adoption, especially on resource-constrained devices. This paper presents a pioneering study on post-training pruning of Stable Diffusion 2, addressing the critical need for model compression in text-to-image domain. Our study tackles the pruning techniques for the previously unexplored multi-modal generation models, and particularly examines the pruning impact on the textual component and the image generation component separately. We conduct a comprehensive comparison on pruning the model or the single component of the model in various sparsities. Our results yield previously undocumented findings. For example, contrary to established trends in language model pruning, we discover that simple magnitude pruning outperforms more advanced techniques in text-to-image context. Furthermore, our results show that Stable Diffusion 2 can be pruned to 38.5% sparsity with minimal quality loss, achieving a significant reduction in model size. We propose an optimal pruning configuration that prunes the text encoder to 47.5% and the diffusion generator to 35%. This configuration maintains image generation quality while substantially reducing computational requirements. In addition, our work uncovers intriguing questions about information encoding in text-to-image models: we observe that pruning beyond certain thresholds leads to sudden performance drops (unreadable images), suggesting that specific weights encode critical semantics information. This finding opens new avenues for future research in model compression, interoperability, and bias identification in text-to-image models. By providing crucial insights into the pruning behavior of text-to-image models, our study lays the groundwork for developing more efficient and accessible AI-driven image generation systems
• 摘要：随着文本转图像模型变得越来越强大和复杂，其不断增长的规模对广泛采用构成了重大障碍，尤其是在资源受限的设备上。本文介绍了一项关于稳定扩散 2 的训练后剪枝的开创性研究，解决了文本转图像领域对模型压缩的迫切需求。我们的研究解决了以前未探索过的多模态生成模型的剪枝技术，并特别分别研究了剪枝对文本组件和图像生成组件的影响。我们对在不同稀疏度下剪枝模型或模型的单个组件进行了全面比较。我们的结果产生了以前未记录的发现。例如，与语言模型剪枝的既定趋势相反，我们发现简单的幅度剪枝在文本转图像环境中优于更先进的技术。此外，我们的结果表明，稳定扩散 2 可以剪枝至 38.5% 的稀疏度，同时质量损失最小，从而显着减小模型大小。我们提出了一种最佳修剪配置，将文本编码器修剪至 47.5%，将扩散生成器修剪至 35%。此配置在保持图像生成质量的同时，大幅降低了计算要求。此外，我们的工作揭示了有关文本到图像模型中信息编码的有趣问题：我们观察到，修剪超过某些阈值会导致性能突然下降（图像无法读取），这表明特定权重编码了关键的语义信息。这一发现为未来在文本到图像模型中的模型压缩、互操作性和偏差识别方面的研究开辟了新途径。通过提供有关文本到图像模型修剪行为的关键见解，我们的研究为开发更高效、更易于访问的 AI 驱动图像生成系统奠定了基础

Title: VideoRepair: Improving Text-to-Video Generation via Misalignment Evaluation and Localized Refinement

• Authors: Daeun Lee, Jaehong Yoon, Jaemin Cho, Mohit Bansal
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.15115
• Pdf URL: https://arxiv.org/pdf/2411.15115
• Copy Paste: [[2411.15115]] VideoRepair: Improving Text-to-Video Generation via Misalignment Evaluation and Localized Refinement(https://arxiv.org/abs/2411.15115)
• Keywords: generation
• Abstract: Recent text-to-video (T2V) diffusion models have demonstrated impressive generation capabilities across various domains. However, these models often generate videos that have misalignments with text prompts, especially when the prompts describe complex scenes with multiple objects and attributes. To address this, we introduce VideoRepair, a novel model-agnostic, training-free video refinement framework that automatically identifies fine-grained text-video misalignments and generates explicit spatial and textual feedback, enabling a T2V diffusion model to perform targeted, localized refinements. VideoRepair consists of four stages: In (1) video evaluation, we detect misalignments by generating fine-grained evaluation questions and answering those questions with MLLM. In (2) refinement planning, we identify accurately generated objects and then create localized prompts to refine other areas in the video. Next, in (3) region decomposition, we segment the correctly generated area using a combined grounding module. We regenerate the video by adjusting the misaligned regions while preserving the correct regions in (4) localized refinement. On two popular video generation benchmarks (EvalCrafter and T2V-CompBench), VideoRepair substantially outperforms recent baselines across various text-video alignment metrics. We provide a comprehensive analysis of VideoRepair components and qualitative examples.
• 摘要：最近的文本到视频 (T2V) 传播模型已在各个领域展示了令人印象深刻的生成能力。然而，这些模型生成的视频通常与文本提示不一致，尤其是当提示描述具有多个对象和属性的复杂场景时。为了解决这个问题，我们引入了 VideoRepair，这是一种新颖的与模型无关、无需训练的视频细化框架，可自动识别细粒度的文本视频错位并生成明确的空间和文本反馈，使 T2V 传播模型能够执行有针对性的局部细化。VideoRepair 包含四个阶段：在 (1) 视频评估中，我们通过生成细粒度的评估问题并使用 MLLM 回答这些问题来检测错位。在 (2) 细化规划中，我们识别准确生成的对象，然后创建局部提示以细化视频中的其他区域。接下来，在 (3) 区域分解中，我们使用组合接地模块分割正确生成的区域。我们通过调整错位区域来重新生成视频，同时在 (4) 局部细化中保留正确的区域。在两个流行的视频生成基准测试（EvalCrafter 和 T2V-CompBench）中，VideoRepair 在各种文本视频对齐指标方面的表现都远远优于近期的基准测试。我们对 VideoRepair 组件和定性示例进行了全面的分析。

Title: ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation

• Authors: Xiaoman Zhang, Hong-Yu Zhou, Xiaoli Yang, Oishi Banerjee, Julián N. Acosta, Josh Miller, Ouwen Huang, Pranav Rajpurkar
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.15122
• Pdf URL: https://arxiv.org/pdf/2411.15122
• Copy Paste: [[2411.15122]] ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation(https://arxiv.org/abs/2411.15122)
• Keywords: generation
• Abstract: AI-driven models have demonstrated significant potential in automating radiology report generation for chest X-rays. However, there is no standardized benchmark for objectively evaluating their performance. To address this, we present ReXrank, this https URL, a public leaderboard and challenge for assessing AI-powered radiology report generation. Our framework incorporates ReXGradient, the largest test dataset consisting of 10,000 studies, and three public datasets (MIMIC-CXR, IU-Xray, CheXpert Plus) for report generation assessment. ReXrank employs 8 evaluation metrics and separately assesses models capable of generating only findings sections and those providing both findings and impressions sections. By providing this standardized evaluation framework, ReXrank enables meaningful comparisons of model performance and offers crucial insights into their robustness across diverse clinical settings. Beyond its current focus on chest X-rays, ReXrank's framework sets the stage for comprehensive evaluation of automated reporting across the full spectrum of medical imaging.
• 摘要：人工智能驱动的模型在自动生成胸部 X 光片放射学报告方面表现出巨大潜力。然而，目前还没有标准化的基准来客观评估它们的表现。为了解决这个问题，我们推出了 ReXrank，这个 https URL，一个用于评估人工智能驱动的放射学报告生成的公共排行榜和挑战。我们的框架结合了 ReXGradient，这是由 10,000 项研究组成的最大测试数据集，以及三个用于报告生成评估的公共数据集（MIMIC-CXR、IU-Xray、CheXpert Plus）。ReXrank 采用 8 个评估指标，分别评估仅能生成发现部分的模型和同时提供发现和印象部分的模型。通过提供这种标准化评估框架，ReXrank 可以对模型性能进行有意义的比较，并提供有关其在不同临床环境中的稳健性的关键见解。除了目前对胸部 X 光片的关注之外，ReXrank 的框架还为全面评估整个医学成像领域的自动报告奠定了基础。

Title: Material Anything: Generating Materials for Any 3D Object via Diffusion

• Authors: Xin Huang, Tengfei Wang, Ziwei Liu, Qing Wang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.15138
• Pdf URL: https://arxiv.org/pdf/2411.15138
• Copy Paste: [[2411.15138]] Material Anything: Generating Materials for Any 3D Object via Diffusion(https://arxiv.org/abs/2411.15138)
• Keywords: generation
• Abstract: We present Material Anything, a fully-automated, unified diffusion framework designed to generate physically-based materials for 3D objects. Unlike existing methods that rely on complex pipelines or case-specific optimizations, Material Anything offers a robust, end-to-end solution adaptable to objects under diverse lighting conditions. Our approach leverages a pre-trained image diffusion model, enhanced with a triple-head architecture and rendering loss to improve stability and material quality. Additionally, we introduce confidence masks as a dynamic switcher within the diffusion model, enabling it to effectively handle both textured and texture-less objects across varying lighting conditions. By employing a progressive material generation strategy guided by these confidence masks, along with a UV-space material refiner, our method ensures consistent, UV-ready material outputs. Extensive experiments demonstrate our approach outperforms existing methods across a wide range of object categories and lighting conditions.
• 摘要：我们提出了 Material Anything，这是一个全自动的统一扩散框架，旨在为 3D 对象生成基于物理的材质。与依赖复杂流程或针对特定情况进行优化的现有方法不同，Material Anything 提供了一种强大的端到端解决方案，可适应不同光照条件下的物体。我们的方法利用预先训练的图像扩散模型，并通过三头架构和渲染损失进行增强，以提高稳定性和材质质量。此外，我们在扩散模型中引入了置信蒙版作为动态切换器，使其能够在不同的光照条件下有效处理有纹理和无纹理的物体。通过采用由这些置信蒙版指导的渐进式材质生成策略以及 UV 空间材质细化器，我们的方法可确保一致的、可用于 UV 的材质输出。大量实验表明，我们的方法在各种物体类别和光照条件下都优于现有方法。

Title: DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving

• Authors: Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, Xinggang Wang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.15139
• Pdf URL: https://arxiv.org/pdf/2411.15139
• Copy Paste: [[2411.15139]] DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving(https://arxiv.org/abs/2411.15139)
• Keywords: generative
• Abstract: Recently, the diffusion model has emerged as a powerful generative technique for robotic policy learning, capable of modeling multi-mode action distributions. Leveraging its capability for end-to-end autonomous driving is a promising direction. However, the numerous denoising steps in the robotic diffusion policy and the more dynamic, open-world nature of traffic scenes pose substantial challenges for generating diverse driving actions at a real-time speed. To address these challenges, we propose a novel truncated diffusion policy that incorporates prior multi-mode anchors and truncates the diffusion schedule, enabling the model to learn denoising from anchored Gaussian distribution to the multi-mode driving action distribution. Additionally, we design an efficient cascade diffusion decoder for enhanced interaction with conditional scene context. The proposed model, DiffusionDrive, demonstrates 10$\times$ reduction in denoising steps compared to vanilla diffusion policy, delivering superior diversity and quality in just 2 steps. On the planning-oriented NAVSIM dataset, with the aligned ResNet-34 backbone, DiffusionDrive achieves 88.1 PDMS without bells and whistles, setting a new record, while running at a real-time speed of 45 FPS on an NVIDIA 4090. Qualitative results on challenging scenarios further confirm that DiffusionDrive can robustly generate diverse plausible driving actions. Code and model will be available at this https URL.
• 摘要：最近，扩散模型已成为一种强大的机器人策略学习生成技术，能够对多模式动作分布进行建模。利用其能力实现端到端自动驾驶是一个很有前途的方向。然而，机器人扩散策略中的众多去噪步骤以及交通场景更具动态性和开放性，对实时生成多样化驾驶动作构成了巨大挑战。为了应对这些挑战，我们提出了一种新颖的截断扩散策略，该策略结合了先前的多模式锚点并截断了扩散计划，使模型能够从锚定高斯分布到多模式驾驶动作分布学习去噪。此外，我们设计了一个高效的级联扩散解码器，以增强与条件场景上下文的交互。与普通扩散策略相比，所提出的模型 DiffusionDrive 的去噪步骤减少了 10$\times$，仅用 2 个步骤就实现了卓越的多样性和质量。在面向规划的 NAVSIM 数据集上，借助对齐的 ResNet-34 主干，DiffusionDrive 无需任何花哨的配置即可实现 88.1 PDMS，创下新纪录，同时在 NVIDIA 4090 上以 45 FPS 的实时速度运行。在具有挑战性的场景中的定性结果进一步证实，DiffusionDrive 可以稳健地生成各种合理的驾驶行为。代码和模型将在此 https URL 上提供。