2025-01-09

Title: BiasGuard: Guardrailing Fairness in Machine Learning Production Systems

• Authors: Nurit Cohen-Inger, Seffi Cohen, Neomi Rabaev, Lior Rokach, Bracha Shapira
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.04142
• Pdf URL: https://arxiv.org/pdf/2501.04142
• Copy Paste: [[2501.04142]] BiasGuard: Guardrailing Fairness in Machine Learning Production Systems(https://arxiv.org/abs/2501.04142)
• Keywords: generative
• Abstract: As machine learning (ML) systems increasingly impact critical sectors such as hiring, financial risk assessments, and criminal justice, the imperative to ensure fairness has intensified due to potential negative implications. While much ML fairness research has focused on enhancing training data and processes, addressing the outputs of already deployed systems has received less attention. This paper introduces 'BiasGuard', a novel approach designed to act as a fairness guardrail in production ML systems. BiasGuard leverages Test-Time Augmentation (TTA) powered by Conditional Generative Adversarial Network (CTGAN), a cutting-edge generative AI model, to synthesize data samples conditioned on inverted protected attribute values, thereby promoting equitable outcomes across diverse groups. This method aims to provide equal opportunities for both privileged and unprivileged groups while significantly enhancing the fairness metrics of deployed systems without the need for retraining. Our comprehensive experimental analysis across diverse datasets reveals that BiasGuard enhances fairness by 31% while only reducing accuracy by 0.09% compared to non-mitigated benchmarks. Additionally, BiasGuard outperforms existing post-processing methods in improving fairness, positioning it as an effective tool to safeguard against biases when retraining the model is impractical.
• 摘要：随着机器学习 (ML) 系统对招聘、金融风险评估和刑事司法等关键领域的影响日益加深，由于潜在的负面影响，确保公平的必要性也日益增强。虽然许多 ML 公平性研究都集中在增强训练数据和流程上，但解决已部署系统的输出问题却受到较少关注。本文介绍了一种新方法“BiasGuard”，旨在充当生产 ML 系统中的公平护栏。BiasGuard 利用由条件生成对抗网络 (CTGAN)（一种尖端的生成式 AI 模型）提供支持的测试时间增强 (TTA) 来合成以倒置受保护属性值为条件的数据样本，从而促进不同群体之间的公平结果。该方法旨在为特权群体和非特权群体提供平等的机会，同时显着提高已部署系统的公平性指标，而无需重新训练。我们对各种数据集进行的全面实验分析表明，与未缓解的基准相比，BiasGuard 将公平性提高了 31%，而准确率仅降低了 0.09%。此外，BiasGuard 在提高公平性方面的表现优于现有的后处理方法，当重新训练模型不切实际时，它可以成为一种有效的工具来防止偏见。

Title: Chirpy3D: Continuous Part Latents for Creative 3D Bird Generation

• Authors: Kam Woh Ng, Jing Yang, Jia Wei Sii, Jiankang Deng, Chee Seng Chan, Yi-Zhe Song, Tao Xiang, Xiatian Zhu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.04144
• Pdf URL: https://arxiv.org/pdf/2501.04144
• Copy Paste: [[2501.04144]] Chirpy3D: Continuous Part Latents for Creative 3D Bird Generation(https://arxiv.org/abs/2501.04144)
• Keywords: generation
• Abstract: In this paper, we push the boundaries of fine-grained 3D generation into truly creative territory. Current methods either lack intricate details or simply mimic existing objects -- we enable both. By lifting 2D fine-grained understanding into 3D through multi-view diffusion and modeling part latents as continuous distributions, we unlock the ability to generate entirely new, yet plausible parts through interpolation and sampling. A self-supervised feature consistency loss further ensures stable generation of these unseen parts. The result is the first system capable of creating novel 3D objects with species-specific details that transcend existing examples. While we demonstrate our approach on birds, the underlying framework extends beyond things that can chirp! Code will be released at this https URL.
• 摘要：在本文中，我们将细粒度 3D 生成的边界推向了真正具有创造性的领域。当前的方法要么缺乏复杂的细节，要么只是模仿现有对象——我们同时实现了这两种方法。通过多视图扩散将 2D 细粒度理解提升到 3D，并将部分潜在值建模为连续分布，我们能够通过插值和采样生成全新但合理的部分。自监督特征一致性损失进一步确保了这些看不见的部分的稳定生成。结果是第一个能够创建具有超越现有示例的物种特定细节的新型 3D 对象的系统。虽然我们在鸟类身上展示了我们的方法，但底层框架超越了会鸣叫的东西！代码将在此 https URL 上发布。

Title: KGIF: Optimizing Relation-Aware Recommendations with Knowledge Graph Information Fusion

• Authors: Dong Hyun Jeon, Wenbo Sun, Houbing Herbert Song, Dongfang Liu, Velasquez Alvaro, Yixin Chloe Xie, Shuteng Niu
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.04161
• Pdf URL: https://arxiv.org/pdf/2501.04161
• Copy Paste: [[2501.04161]] KGIF: Optimizing Relation-Aware Recommendations with Knowledge Graph Information Fusion(https://arxiv.org/abs/2501.04161)
• Keywords: generation
• Abstract: While deep-learning-enabled recommender systems demonstrate strong performance benchmarks, many struggle to adapt effectively in real-world environments due to limited use of user-item relationship data and insufficient transparency in recommendation generation. Traditional collaborative filtering approaches fail to integrate multifaceted item attributes, and although Factorization Machines account for item-specific details, they overlook broader relational patterns. Collaborative knowledge graph-based models have progressed by embedding user-item interactions with item-attribute relationships, offering a holistic perspective on interconnected entities. However, these models frequently aggregate attribute and interaction data in an implicit manner, leaving valuable relational nuances underutilized. This study introduces the Knowledge Graph Attention Network with Information Fusion (KGIF), a specialized framework designed to merge entity and relation embeddings explicitly through a tailored self-attention mechanism. The KGIF framework integrates reparameterization via dynamic projection vectors, enabling embeddings to adaptively represent intricate relationships within knowledge graphs. This explicit fusion enhances the interplay between user-item interactions and item-attribute relationships, providing a nuanced balance between user-centric and item-centric representations. An attentive propagation mechanism further optimizes knowledge graph embeddings, capturing multi-layered interaction patterns. The contributions of this work include an innovative method for explicit information fusion, improved robustness for sparse knowledge graphs, and the ability to generate explainable recommendations through interpretable path visualization.
• 摘要：虽然支持深度学习的推荐系统表现出了强大的性能基准，但由于用户-项目关系数据的使用有限以及推荐生成透明度不足，许多推荐系统难以在现实环境中有效适应。传统的协同过滤方法无法整合多方面的项目属性，尽管分解机考虑了项目特定的细节，但它们忽略了更广泛的关系模式。基于协作知识图谱的模型通过将用户-项目交互与项目-属性关系嵌入在一起而取得了进展，为互联实体提供了整体视角。然而，这些模型经常以隐式方式聚合属性和交互数据，导致有价值的关系细微差别得不到充分利用。本研究介绍了具有信息融合的知识图谱注意网络 (KGIF)，这是一种专门的框架，旨在通过量身定制的自注意力机制显式合并实体和关系嵌入。KGIF 框架通过动态投影向量集成了重新参数化，使嵌入能够自适应地表示知识图谱中的复杂关系。这种显式融合增强了用户-项目交互和项目-属性关系之间的相互作用，在以用户为中心和以项目为中心的表示之间提供了细微的平衡。注意力传播机制进一步优化了知识图谱嵌入，捕获了多层交互模式。这项工作的贡献包括显式信息融合的创新方法、稀疏知识图谱的增强鲁棒性以及通过可解释的路径可视化生成可解释建议的能力。

Title: Fixed Points of Deep Neural Networks: Emergence, Stability, and Applications

• Authors: L. Berlyand, V. Slavin
• Subjects: cs.LG, cs.AI, math.NA
• Abstract URL: https://arxiv.org/abs/2501.04182
• Pdf URL: https://arxiv.org/pdf/2501.04182
• Copy Paste: [[2501.04182]] Fixed Points of Deep Neural Networks: Emergence, Stability, and Applications(https://arxiv.org/abs/2501.04182)
• Keywords: restoration
• Abstract: We present numerical and analytical results on the formation and stability of a family of fixed points of deep neural networks (DNNs). Such fixed points appear in a class of DNNs when dimensions of input and output vectors are the same. We demonstrate examples of applications of such networks in supervised, semi-supervised and unsupervised learning such as encoding/decoding of images, restoration of damaged images among others. We present several numerical and analytical results. First, we show that for untrained DNN's with weights and biases initialized by normally distributed random variables the only one fixed point exists. This result holds for DNN with any depth (number of layers) $L$, any layer width $N$, and sigmoid-type activation functions. Second, it has been shown that for a DNN whose parameters (weights and biases) are initialized by ``light-tailed'' distribution of weights (e.g. normal distribution), after training the distribution of these parameters become ``heavy-tailed''. This motivates our study of DNNs with ``heavy-tailed'' initialization. For such DNNs we show numerically %existence and stability that training leads to emergence of $Q(N,L)$ fixed points, where $Q(N,L)$ is a positive integer which depends on the number of layers $L$ and layer width $N$. We further observe numerically that for fixed $N = N_0$ the function $Q(N_0, L)$ is non-monotone, that is it initially grows as $L$ increases and then decreases to 1. This non-monotone behavior of $Q(N_0, L)$ is also obtained by analytical derivation of equation for Empirical Spectral Distribution (ESD) of input-output Jacobian followed by numerical solution of this equation.
• 摘要：我们给出了关于深度神经网络 (DNN) 不动点家族的形成和稳定性的数值和分析结果。当输入和输出向量的维度相同时，此类不动点会出现在一类 DNN 中。我们展示了此类网络在监督、半监督和无监督学习中的应用示例，例如图像的编码/解码、受损图像的修复等。我们给出了几个数值和分析结果。首先，我们表明，对于权重和偏差由正态分布的随机变量初始化的未经训练的 DNN，只存在一个不动点。此结果适用于具有任何深度（层数）$L$、任何层宽度 $N$ 和 S 型激活函数的 DNN。其次，已经证明，对于其参数（权重和偏差）由权重的“轻尾”分布（例如正态分布）初始化的 DNN，经过训练后，这些参数的分布将变为“重尾”。这促使我们研究具有“重尾”初始化的 DNN。对于此类 DNN，我们通过数值证明了其存在性和稳定性，即训练会导致出现 $Q(N,L)$ 个不动点，其中 $Q(N,L)$ 是一个正整数，取决于层数 $L$ 和层宽 $N$。我们进一步通过数值观察到，对于固定的 $N = N_0$，函数 $Q(N_0, L)$ 是非单调的，即它最初随着 $L$ 的增加而增长，然后减小到 1。$Q(N_0, L)$ 的这种非单调行为也是通过对输入输出雅可比矩阵的经验谱分布 (ESD) 方程进行解析推导，然后对该方程进行数值解而获得的。

Title: Generative Dataset Distillation Based on Self-knowledge Distillation

• Authors: Longzhen Li, Guang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04202
• Pdf URL: https://arxiv.org/pdf/2501.04202
• Copy Paste: [[2501.04202]] Generative Dataset Distillation Based on Self-knowledge Distillation(https://arxiv.org/abs/2501.04202)
• Keywords: generative
• Abstract: Dataset distillation is an effective technique for reducing the cost and complexity of model training while maintaining performance by compressing large datasets into smaller, more efficient versions. In this paper, we present a novel generative dataset distillation method that can improve the accuracy of aligning prediction logits. Our approach integrates self-knowledge distillation to achieve more precise distribution matching between the synthetic and original data, thereby capturing the overall structure and relationships within the data. To further improve the accuracy of alignment, we introduce a standardization step on the logits before performing distribution matching, ensuring consistency in the range of logits. Through extensive experiments, we demonstrate that our method outperforms existing state-of-the-art methods, resulting in superior distillation performance.
• 摘要：数据集蒸馏是一种有效的技术，它可以将大数据集压缩成更小、更高效的版本，从而降低模型训练的成本和复杂性，同时保持性能。在本文中，我们提出了一种新颖的生成数据集蒸馏方法，可以提高对齐预测逻辑的准确性。我们的方法集成了自我知识蒸馏，以实现合成数据和原始数据之间更精确的分布匹配，从而捕获数据中的整体结构和关系。为了进一步提高对齐的准确性，我们在执行分布匹配之前对逻辑引入了一个标准化步骤，确保逻辑范围内的一致性。通过大量实验，我们证明了我们的方法优于现有的最先进方法，从而实现了卓越的蒸馏性能。

Title: LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition

• Authors: Bowen Hao, Dongliang Zhou, Xiaojie Li, Xingyu Zhang, Liang Xie, Jianlong Wu, Erwei Yin
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2501.04204
• Pdf URL: https://arxiv.org/pdf/2501.04204
• Copy Paste: [[2501.04204]] LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition(https://arxiv.org/abs/2501.04204)
• Keywords: generation
• Abstract: Visual speech recognition (VSR), commonly known as lip reading, has garnered significant attention due to its wide-ranging practical applications. The advent of deep learning techniques and advancements in hardware capabilities have significantly enhanced the performance of lip reading models. Despite these advancements, existing datasets predominantly feature stable video recordings with limited variability in lip movements. This limitation results in models that are highly sensitive to variations encountered in real-world scenarios. To address this issue, we propose a novel framework, LipGen, which aims to improve model robustness by leveraging speech-driven synthetic visual data, thereby mitigating the constraints of current datasets. Additionally, we introduce an auxiliary task that incorporates viseme classification alongside attention mechanisms. This approach facilitates the efficient integration of temporal information, directing the model's focus toward the relevant segments of speech, thereby enhancing discriminative capabilities. Our method demonstrates superior performance compared to the current state-of-the-art on the lip reading in the wild (LRW) dataset and exhibits even more pronounced advantages under challenging conditions.
• 摘要：视觉语音识别 (VSR)，俗称唇读，因其广泛的实际应用而备受关注。深度学习技术的出现和硬件功能的进步显著提高了唇读模型的性能。尽管取得了这些进步，但现有数据集主要以稳定的视频记录为特色，唇部运动的变化有限。这种限制导致模型对现实场景中遇到的变化高度敏感。为了解决这个问题，我们提出了一个新框架 LipGen，旨在通过利用语音驱动的合成视觉数据来提高模型的鲁棒性，从而减轻当前数据集的限制。此外，我们引入了一个辅助任务，将视位分类与注意力机制结合起来。这种方法有助于有效整合时间信息，将模型的注意力引导到相关的语音片段上，从而增强判别能力。与目前最先进的唇读 (LRW) 数据集相比，我们的方法表现出卓越的性能，并且在具有挑战性的条件下表现出更明显的优势。

Title: Physics-Informed Super-Resolution Diffusion for 6D Phase Space Diagnostics

• Authors: Alexander Scheinker
• Subjects: cs.LG, math.DS, physics.acc-ph
• Abstract URL: https://arxiv.org/abs/2501.04305
• Pdf URL: https://arxiv.org/pdf/2501.04305
• Copy Paste: [[2501.04305]] Physics-Informed Super-Resolution Diffusion for 6D Phase Space Diagnostics(https://arxiv.org/abs/2501.04305)
• Keywords: super-resolution
• Abstract: Adaptive physics-informed super-resolution diffusion is developed for non-invasive virtual diagnostics of the 6D phase space density of charged particle beams. An adaptive variational autoencoder (VAE) embeds initial beam condition images and scalar measurements to a low-dimensional latent space from which a 326 pixel 6D tensor representation of the beam's 6D phase space density is generated. Projecting from a 6D tensor generates physically consistent 2D projections. Physics-guided super-resolution diffusion transforms low-resolution images of the 6D density to high resolution 256x256 pixel images. Un-supervised adaptive latent space tuning enables tracking of time-varying beams without knowledge of time-varying initial conditions. The method is demonstrated with experimental data and multi-particle simulations at the HiRES UED. The general approach is applicable to a wide range of complex dynamic systems evolving in high-dimensional phase space. The method is shown to be robust to distribution shift without re-training.
• 摘要：开发了自适应物理信息超分辨率扩散，用于对带电粒子束的 6D 相空间密度进行非侵入式虚拟诊断。自适应变分自动编码器 (VAE) 将初始光束条件图像和标量测量嵌入到低维潜在空间，从中生成光束 6D 相空间密度的 326 像素 6D 张量表示。从 6D 张量投影会生成物理上一致的 2D 投影。物理引导的超分辨率扩散将 6D 密度的低分辨率图像转换为高分辨率 256x256 像素图像。无监督自适应潜在空间调整能够在不了解随时间变化的初始条件的情况下跟踪随时间变化的光束。该方法已通过 HiRES UED 的实验数据和多粒子模拟进行了演示。该通用方法适用于在高维相空间中演化的各种复杂动态系统。该方法被证明对分布偏移具有鲁棒性，无需重新训练。

Title: Instructive3D: Editing Large Reconstruction Models with Text Instructions

• Authors: Kunal Kathare, Ankit Dhiman, K Vikas Gowda, Siddharth Aravindan, Shubham Monga, Basavaraja Shanthappa Vandrotti, Lokesh R Boregowda
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04374
• Pdf URL: https://arxiv.org/pdf/2501.04374
• Copy Paste: [[2501.04374]] Instructive3D: Editing Large Reconstruction Models with Text Instructions(https://arxiv.org/abs/2501.04374)
• Keywords: generation
• Abstract: Transformer based methods have enabled users to create, modify, and comprehend text and image data. Recently proposed Large Reconstruction Models (LRMs) further extend this by providing the ability to generate high-quality 3D models with the help of a single object image. These models, however, lack the ability to manipulate or edit the finer details, such as adding standard design patterns or changing the color and reflectance of the generated objects, thus lacking fine-grained control that may be very helpful in domains such as augmented reality, animation and gaming. Naively training LRMs for this purpose would require generating precisely edited images and 3D object pairs, which is computationally expensive. In this paper, we propose Instructive3D, a novel LRM based model that integrates generation and fine-grained editing, through user text prompts, of 3D objects into a single model. We accomplish this by adding an adapter that performs a diffusion process conditioned on a text prompt specifying edits in the triplane latent space representation of 3D object models. Our method does not require the generation of edited 3D objects. Additionally, Instructive3D allows us to perform geometrically consistent modifications, as the edits done through user-defined text prompts are applied to the triplane latent representation thus enhancing the versatility and precision of 3D objects generated. We compare the objects generated by Instructive3D and a baseline that first generates the 3D object meshes using a standard LRM model and then edits these 3D objects using text prompts when images are provided from the Objaverse LVIS dataset. We find that Instructive3D produces qualitatively superior 3D objects with the properties specified by the edit prompts.
• 摘要：基于 Transformer 的方法使用户能够创建、修改和理解文本和图像数据。最近提出的大型重建模型 (LRM) 进一步扩展了这一功能，它提供了在单个对象图像的帮助下生成高质量 3D 模型的能力。然而，这些模型缺乏操纵或编辑更精细细节的能力，例如添加标准设计模式或更改生成对象的颜色和反射率，因此缺乏在增强现实、动画和游戏等领域可能非常有用的细粒度控制。为此目的简单地训练 LRM 需要生成精确编辑的图像和 3D 对象对，这在计算上是昂贵的。在本文中，我们提出了 Instructive3D，这是一种基于 LRM 的新型模型，它将通过用户文本提示将 3D 对象的生成和细粒度编辑集成到一个模型中。我们通过添加一个适配器来实现这一点，该适配器执行以文本提示为条件的扩散过程，该提示指定 3D 对象模型的三平面潜在空间表示中的编辑。我们的方法不需要生成经过编辑的 3D 对象。此外，Instructive3D 允许我们执行几何一致的修改，因为通过用户定义的文本提示进行的编辑会应用于三平面潜在表示，从而增强生成的 3D 对象的多功能性和精度。我们将 Instructive3D 生成的对象与基线进行比较，基线首先使用标准 LRM 模型生成 3D 对象网格，然后在 Objaverse LVIS 数据集提供图像时使用文本提示编辑这些 3D 对象。我们发现 Instructive3D 生成的 3D 对象具有编辑提示指定的属性，质量上乘。

Title: On Computational Limits and Provably Efficient Criteria of Visual Autoregressive Models: A Fine-Grained Complexity Analysis

• Authors: Yekun Ke, Xiaoyu Li, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song
• Subjects: cs.LG, cs.AI, cs.CC, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04377
• Pdf URL: https://arxiv.org/pdf/2501.04377
• Copy Paste: [[2501.04377]] On Computational Limits and Provably Efficient Criteria of Visual Autoregressive Models: A Fine-Grained Complexity Analysis(https://arxiv.org/abs/2501.04377)
• Keywords: generation
• Abstract: Recently, Visual Autoregressive ($\mathsf{VAR}$) Models introduced a groundbreaking advancement in the field of image generation, offering a scalable approach through a coarse-to-fine "next-scale prediction" paradigm. However, the state-of-the-art algorithm of $\mathsf{VAR}$ models in [Tian, Jiang, Yuan, Peng and Wang, NeurIPS 2024] takes $O(n^4)$ time, which is computationally inefficient. In this work, we analyze the computational limits and efficiency criteria of $\mathsf{VAR}$ Models through a fine-grained complexity lens. Our key contribution is identifying the conditions under which $\mathsf{VAR}$ computations can achieve sub-quadratic time complexity. Specifically, we establish a critical threshold for the norm of input matrices used in $\mathsf{VAR}$ attention mechanisms. Above this threshold, assuming the Strong Exponential Time Hypothesis ($\mathsf{SETH}$) from fine-grained complexity theory, a sub-quartic time algorithm for $\mathsf{VAR}$ models is impossible. To substantiate our theoretical findings, we present efficient constructions leveraging low-rank approximations that align with the derived criteria. This work initiates the study of the computational efficiency of the $\mathsf{VAR}$ model from a theoretical perspective. Our technique will shed light on advancing scalable and efficient image generation in $\mathsf{VAR}$ frameworks.
• 摘要：最近，视觉自回归（$\mathsf{VAR}$）模型在图像生成领域取得了突破性进展，通过由粗到细的“下一尺度预测”范式提供了一种可扩展的方法。然而，[Tian, Jiang, Yuan, Peng and Wang, NeurIPS 2024] 中最先进的 $\mathsf{VAR}$ 模型算法需要 $O(n^4)$ 时间，计算效率低下。在这项工作中，我们通过细粒度复杂性镜头分析了 $\mathsf{VAR}$ 模型的计算限制和效率标准。我们的主要贡献是确定 $\mathsf{VAR}$ 计算可以实现亚二次时间复杂度的条件。具体而言，我们为 $\mathsf{VAR}$ 注意力机制中使用的输入矩阵范数建立了一个临界阈值。高于此阈值，假设细粒度复杂性理论中的强指数时间假设 ($\mathsf{SETH}$)，$\mathsf{VAR}$ 模型的亚四次时间算法是不可能的。为了证实我们的理论发现，我们提出了利用符合导出标准的低秩近似的有效构造。这项工作从理论角度启动了对 $\mathsf{VAR}$ 模型计算效率的研究。我们的技术将为在 $\mathsf{VAR}$ 框架中推进可扩展且高效的图像生成提供启示。

Title: iFADIT: Invertible Face Anonymization via Disentangled Identity Transform

• Authors: Lin Yuan, Kai Liang, Xiong Li, Tao Wu, Nannan Wang, Xinbo Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04390
• Pdf URL: https://arxiv.org/pdf/2501.04390
• Copy Paste: [[2501.04390]] iFADIT: Invertible Face Anonymization via Disentangled Identity Transform(https://arxiv.org/abs/2501.04390)
• Keywords: generative
• Abstract: Face anonymization aims to conceal the visual identity of a face to safeguard the individual's privacy. Traditional methods like blurring and pixelation can largely remove identifying features, but these techniques significantly degrade image quality and are vulnerable to deep reconstruction attacks. Generative models have emerged as a promising solution for anonymizing faces while preserving a natural this http URL, many still face limitations in visual quality and often overlook the potential to recover the original face from the anonymized version, which can be valuable in specific contexts such as image forensics. This paper proposes a novel framework named iFADIT, an acronym for Invertible Face Anonymization via Disentangled Identity this http URL framework features a disentanglement architecture coupled with a secure flow-based model: the former decouples identity information from non-identifying attributes, while the latter transforms the decoupled identity into an anonymized version in an invertible manner controlled by a secret key. The anonymized face can then be reconstructed based on a pre-trained StyleGAN that ensures high image quality and realistic facial details. Recovery of the original face (aka de-anonymization) is possible upon the availability of the matching secret, by inverting the anonymization process based on the same set of model parameters. Furthermore, a dedicated secret-key mechanism along with a dual-phase training strategy is devised to ensure the desired properties of face anonymization. Qualitative and quantitative experiments demonstrate the superiority of the proposed approach in anonymity, reversibility, security, diversity, and interpretability over competing methods.
• 摘要：人脸匿名化旨在隐藏人脸的视觉身份，以保护个人隐私。模糊和像素化等传统方法可以在很大程度上消除识别特征，但这些技术会显著降低图像质量，并且容易受到深度重建攻击。生成模型已成为一种有前途的解决方案，可在保留自然人脸的同时匿名化人脸，但许多模型仍然面临视觉质量的限制，并且经常忽略从匿名版本恢复原始人脸的潜力，这在图像取证等特定情况下非常有用。本文提出了一个名为 iFADIT 的新框架，即通过解缠身份实现可逆人脸匿名化。该框架采用解缠架构和安全的基于流的模型：前者将身份信息与非识别属性解耦，而后者以可逆方式将解耦后的身份转换为匿名版本，由密钥控制。然后可以基于预先训练的 StyleGAN 重建匿名人脸，以确保高图像质量和逼真的面部细节。通过基于同一组模型参数反转匿名化过程，可以在获得匹配密钥后恢复原始人脸（又称去匿名化）。此外，还设计了专用密钥机制以及双相训练策略，以确保人脸匿名化的所需属性。定性和定量实验证明了所提出的方法在匿名性、可逆性、安全性、多样性和可解释性方面优于竞争方法。

Title: MB-TaylorFormer V2: Improved Multi-branch Linear Transformer Expanded by Taylor Formula for Image Restoration

• Authors: Zhi Jin, Yuwei Qiu, Kaihao Zhang, Hongdong Li, Wenhan Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04486
• Pdf URL: https://arxiv.org/pdf/2501.04486
• Copy Paste: [[2501.04486]] MB-TaylorFormer V2: Improved Multi-branch Linear Transformer Expanded by Taylor Formula for Image Restoration(https://arxiv.org/abs/2501.04486)
• Keywords: restoration
• Abstract: Recently, Transformer networks have demonstrated outstanding performance in the field of image restoration due to the global receptive field and adaptability to input. However, the quadratic computational complexity of Softmax-attention poses a significant limitation on its extensive application in image restoration tasks, particularly for high-resolution images. To tackle this challenge, we propose a novel variant of the Transformer. This variant leverages the Taylor expansion to approximate the Softmax-attention and utilizes the concept of norm-preserving mapping to approximate the remainder of the first-order Taylor expansion, resulting in a linear computational complexity. Moreover, we introduce a multi-branch architecture featuring multi-scale patch embedding into the proposed Transformer, which has four distinct advantages: 1) various sizes of the receptive field; 2) multi-level semantic information; 3) flexible shapes of the receptive field; 4) accelerated training and inference speed. Hence, the proposed model, named the second version of Taylor formula expansion-based Transformer (for short MB-TaylorFormer V2) has the capability to concurrently process coarse-to-fine features, capture long-distance pixel interactions with limited computational cost, and improve the approximation of the Taylor expansion remainder. Experimental results across diverse image restoration benchmarks demonstrate that MB-TaylorFormer V2 achieves state-of-the-art performance in multiple image restoration tasks, such as image dehazing, deraining, desnowing, motion deblurring, and denoising, with very little computational overhead. The source code is available at this https URL.
• 摘要：近年来，Transformer 网络凭借其全局感受野和对输入的自适应性在图像恢复领域表现出色。然而，Softmax-attention 的二次计算复杂度严重限制了其在图像恢复任务中的广泛应用，特别是对于高分辨率图像。为了应对这一挑战，我们提出了一种新颖的 Transformer 变体。该变体利用泰勒展开式来近似 Softmax-attention，并利用保范数映射的概念来近似一阶泰勒展开式的余数，从而实现线性计算复杂度。此外，我们在所提出的 Transformer 中引入了一种具有多尺度块嵌入的多分支架构，该架构具有四个明显的优势：1）各种大小的感受野；2）多级语义信息；3）灵活的感受野形状；4）加快训练和推理速度。因此，提出的模型被称为基于泰勒公式展开的 Transformer 的第二版（简称 MB-TaylorFormer V2），它能够同时处理由粗到细的特征，以有限的计算成本捕获长距离像素交互，并改进泰勒展开余数的近似值。在各种图像恢复基准上进行的实验结果表明，MB-TaylorFormer V2 在图像去雾、去雨、去雪、运动去模糊和去噪等多项图像恢复任务中实现了最先进的性能，而且计算开销非常小。源代码可从此 https URL 获取。

Title: Integrating remote sensing data assimilation, deep learning and large language model for interactive wheat breeding yield prediction

• Authors: Guofeng Yang, Nanfei Jin, Wenjie Ai, Zhonghua Zheng, Yuhong He, Yong He
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04487
• Pdf URL: https://arxiv.org/pdf/2501.04487
• Copy Paste: [[2501.04487]] Integrating remote sensing data assimilation, deep learning and large language model for interactive wheat breeding yield prediction(https://arxiv.org/abs/2501.04487)
• Keywords: generation
• Abstract: Yield is one of the core goals of crop breeding. By predicting the potential yield of different breeding materials, breeders can screen these materials at various growth stages to select the best performing. Based on unmanned aerial vehicle remote sensing technology, high-throughput crop phenotyping data in breeding areas is collected to provide data support for the breeding decisions of breeders. However, the accuracy of current yield predictions still requires improvement, and the usability and user-friendliness of yield forecasting tools remain suboptimal. To address these challenges, this study introduces a hybrid method and tool for crop yield prediction, designed to allow breeders to interactively and accurately predict wheat yield by chatting with a large language model (LLM). First, the newly designed data assimilation algorithm is used to assimilate the leaf area index into the WOFOST model. Then, selected outputs from the assimilation process, along with remote sensing inversion results, are used to drive the time-series temporal fusion transformer model for wheat yield prediction. Finally, based on this hybrid method and leveraging an LLM with retrieval augmented generation technology, we developed an interactive yield prediction Web tool that is user-friendly and supports sustainable data updates. This tool integrates multi-source data to assist breeding decision-making. This study aims to accelerate the identification of high-yield materials in the breeding process, enhance breeding efficiency, and enable more scientific and smart breeding decisions.
• 摘要：产量是作物育种的核心目标之一。通过预测不同育种材料的潜在产量，育种者可以在不同生长阶段对这些材料进行筛选，以选出表现最佳的材料。基于无人机遥感技术，收集育种区高通量作物表型数据，为育种者的育种决策提供数据支持。然而，目前产量预测的准确性仍有待提高，产量预测工具的可用性和用户友好性仍然不够理想。为了应对这些挑战，本研究引入了一种用于作物产量预测的混合方法和工具，旨在让育种者通过与大型语言模型 (LLM) 聊天，以交互方式准确地预测小麦产量。首先，使用新设计的数据同化算法将叶面积指数同化到 WOFOST 模型中。然后，从同化过程中选定的输出与遥感反演结果一起用于驱动时间序列融合变压器模型以进行小麦产量预测。最后，基于该混合方法，利用LLM与检索增强生成技术，开发了一款用户友好、支持持续数据更新的交互式产量预测Web工具，该工具整合了多源数据，辅助育种决策，旨在加速育种过程中高产材料的识别，提高育种效率，使育种决策更加科学、智能。

Title: Improving Image Captioning by Mimicking Human Reformulation Feedback at Inference-time

• Authors: Uri Berger, Omri Abend, Lea Frermann, Gabriel Stanovsky
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04513
• Pdf URL: https://arxiv.org/pdf/2501.04513
• Copy Paste: [[2501.04513]] Improving Image Captioning by Mimicking Human Reformulation Feedback at Inference-time(https://arxiv.org/abs/2501.04513)
• Keywords: generative
• Abstract: Incorporating automatically predicted human feedback into the process of training generative models has attracted substantial recent interest, while feedback at inference time has received less attention. The typical feedback at training time, i.e., preferences of choice given two samples, does not naturally transfer to the inference phase. We introduce a novel type of feedback -- caption reformulations -- and train models to mimic reformulation feedback based on human annotations. Our method does not require training the image captioning model itself, thereby demanding substantially less computational effort. We experiment with two types of reformulation feedback: first, we collect a dataset of human reformulations that correct errors in the generated captions. We find that incorporating reformulation models trained on this data into the inference phase of existing image captioning models results in improved captions, especially when the original captions are of low quality. We apply our method to non-English image captioning, a domain where robust models are less prevalent, and gain substantial improvement. Second, we apply reformulations to style transfer. Quantitative evaluations reveal state-of-the-art performance on German image captioning and English style transfer, while human validation with a detailed comparative framework exposes the specific axes of improvement.
• 摘要：将自动预测的人工反馈纳入生成模型的训练过程最近引起了广泛关注，而推理时的反馈则受到较少关注。训练时的典型反馈，即给定两个样本的选择偏好，不会自然转移到推理阶段。我们引入了一种新型反馈——字幕重构——并训练模型以模仿基于人工注释的重构反馈。我们的方法不需要训练图像字幕模型本身，因此所需的计算工作量大大减少。我们尝试了两种类型的重构反馈：首先，我们收集一个人工重构的数据集，以纠正生成的字幕中的错误。我们发现，将基于这些数据训练的重构模型纳入现有图像字幕模型的推理阶段可以改进字幕，尤其是在原始字幕质量较低的情况下。我们将我们的方法应用于非英语图像字幕（该领域中稳健模型不太流行）并获得了显着的改进。其次，我们将重构应用于风格转换。定量评估揭示了德语图像字幕和英语风格转换的最新性能，而具有详细比较框架的人工验证则揭示了改进的具体轴线。

Title: Boosting Salient Object Detection with Knowledge Distillated from Large Foundation Models

• Authors: Miaoyang He, Shuyong Gao, Tsui Qin Mok, Weifeng Ge, Wengqiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04582
• Pdf URL: https://arxiv.org/pdf/2501.04582
• Copy Paste: [[2501.04582]] Boosting Salient Object Detection with Knowledge Distillated from Large Foundation Models(https://arxiv.org/abs/2501.04582)
• Keywords: generation
• Abstract: Salient Object Detection (SOD) aims to identify and segment prominent regions within a scene. Traditional models rely on manually annotated pseudo labels with precise pixel-level accuracy, which is time-consuming. We developed a low-cost, high-precision annotation method by leveraging large foundation models to address the challenges. Specifically, we use a weakly supervised approach to guide large models in generating pseudo-labels through textual prompts. Since large models do not effectively focus on the salient regions of images, we manually annotate a subset of text to fine-tune the model. Based on this approach, which enables precise and rapid generation of pseudo-labels, we introduce a new dataset, BDS-TR. Compared to the previous DUTS-TR dataset, BDS-TR is more prominent in scale and encompasses a wider variety of categories and scenes. This expansion will enhance our model's applicability across a broader range of scenarios and provide a more comprehensive foundational dataset for future SOD research. Additionally, we present an edge decoder based on dynamic upsampling, which focuses on object edges while gradually recovering image feature resolution. Comprehensive experiments on five benchmark datasets demonstrate that our method significantly outperforms state-of-the-art approaches and also surpasses several existing fully-supervised SOD methods. The code and results will be made available.
• 摘要：显著性目标检测 (SOD) 旨在识别和分割场景中的显著区域。传统模型依赖于手动标注的伪标签，精度达到像素级，非常耗时。我们利用大型基础模型开发了一种低成本、高精度的标注方法来解决这些挑战。具体来说，我们使用弱监督方法通过文本提示引导大型模型生成伪标签。由于大型模型无法有效地关注图像的显著区域，我们手动标注文本子集来微调模型。基于这种能够精确快速生成伪标签的方法，我们引入了一个新数据集 BDS-TR。与之前的 DUTS-TR 数据集相比，BDS-TR 规模更大，涵盖的类别和场景种类更多。这种扩展将增强我们模型在更广泛场景中的适用性，并为未来的 SOD 研究提供更全面的基础数据集。此外，我们提出了一种基于动态上采样的边缘解码器，它专注于对象边缘，同时逐渐恢复图像特征分辨率。在五个基准数据集上进行的综合实验表明，我们的方法明显优于最先进的方法，并且还超越了几种现有的全监督 SOD 方法。代码和结果将公开。

Title: Enhancing Low-Cost Video Editing with Lightweight Adaptors and Temporal-Aware Inversion

• Authors: Yangfan He, Sida Li, Kun Li, Jianhui Wang, Binxu Li, Tianyu Shi, Jun Yin, Miao Zhang, Xueqian Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04606
• Pdf URL: https://arxiv.org/pdf/2501.04606
• Copy Paste: [[2501.04606]] Enhancing Low-Cost Video Editing with Lightweight Adaptors and Temporal-Aware Inversion(https://arxiv.org/abs/2501.04606)
• Keywords: generation
• Abstract: Recent advancements in text-to-image (T2I) generation using diffusion models have enabled cost-effective video-editing applications by leveraging pre-trained models, eliminating the need for resource-intensive training. However, the frame-independence of T2I generation often results in poor temporal consistency. Existing methods address this issue through temporal layer fine-tuning or inference-based temporal propagation, but these approaches suffer from high training costs or limited temporal coherence. To address these challenges, we propose a General and Efficient Adapter (GE-Adapter) that integrates temporal-spatial and semantic consistency with Baliteral DDIM inversion. This framework introduces three key components: (1) Frame-based Temporal Consistency Blocks (FTC Blocks) to capture frame-specific features and enforce smooth inter-frame transitions via temporally-aware loss functions; (2) Channel-dependent Spatial Consistency Blocks (SCD Blocks) employing bilateral filters to enhance spatial coherence by reducing noise and artifacts; and (3) Token-based Semantic Consistency Module (TSC Module) to maintain semantic alignment using shared prompt tokens and frame-specific tokens. Our method significantly improves perceptual quality, text-image alignment, and temporal coherence, as demonstrated on the MSR-VTT dataset. Additionally, it achieves enhanced fidelity and frame-to-frame coherence, offering a practical solution for T2V editing.
• 摘要：使用扩散模型的文本到图像 (T2I) 生成的最新进展通过利用预先训练的模型实现了具有成本效益的视频编辑应用程序，从而消除了资源密集型训练的需要。然而，T2I 生成的帧独立性通常会导致较差的时间一致性。现有方法通过时间层微调或基于推理的时间传播来解决该问题，但是这些方法的训练成本高或时间连贯性有限。为了应对这些挑战，我们提出了一种通用高效适配器 (GE-Adapter)，将时空和语义一致性与 Baliteral DDIM 反演相结合。该框架引入了三个关键组件：(1) 基于帧的时间一致性块 (FTC Blocks)，用于捕获特定于帧的特征并通过时间感知损失函数实现平滑的帧间转换；(2) 采用双边滤波器的通道相关空间一致性块 (SCD Blocks)，通过降低噪音和伪影来增强空间连贯性； （3）基于标记的语义一致性模块（TSC 模块），使用共享提示标记和特定于帧的标记来保持语义对齐。我们的方法显著提高了感知质量、文本图像对齐和时间连贯性，如 MSR-VTT 数据集所示。此外，它还实现了增强的保真度和帧间连贯性，为 T2V 编辑提供了实用的解决方案。

Title: Disentangled Clothed Avatar Generation with Layered Representation

• Authors: Weitian Zhang, Sijing Wu, Manwen Liao, Yichao Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04631
• Pdf URL: https://arxiv.org/pdf/2501.04631
• Copy Paste: [[2501.04631]] Disentangled Clothed Avatar Generation with Layered Representation(https://arxiv.org/abs/2501.04631)
• Keywords: generation
• Abstract: Clothed avatar generation has wide applications in virtual and augmented reality, filmmaking, and more. Previous methods have achieved success in generating diverse digital avatars, however, generating avatars with disentangled components (\eg, body, hair, and clothes) has long been a challenge. In this paper, we propose LayerAvatar, the first feed-forward diffusion-based method for generating component-disentangled clothed avatars. To achieve this, we first propose a layered UV feature plane representation, where components are distributed in different layers of the Gaussian-based UV feature plane with corresponding semantic labels. This representation supports high-resolution and real-time rendering, as well as expressive animation including controllable gestures and facial expressions. Based on the well-designed representation, we train a single-stage diffusion model and introduce constrain terms to address the severe occlusion problem of the innermost human body layer. Extensive experiments demonstrate the impressive performances of our method in generating disentangled clothed avatars, and we further explore its applications in component transfer. The project page is available at: this https URL
• 摘要：着装虚拟形象生成在虚拟和增强现实、电影制作等领域有着广泛的应用。先前的方法在生成各种数字化形象方面取得了成功，然而，生成具有解缠结组件（例如身体、头发和衣服）的虚拟形象一直是一个挑战。在本文中，我们提出了 LayerAvatar，这是第一个基于前馈扩散的生成解缠结着装虚拟形象的方法。为了实现这一点，我们首先提出了一种分层的 UV 特征平面表示，其中组件分布在基于高斯的 UV 特征平面的不同层中，并带有相应的语义标签。这种表示支持高分辨率和实时渲染，以及包括可控制手势和面部表情在内的富有表现力的动画。基于精心设计的表示，我们训练了一个单阶段扩散模型并引入约束项来解决最内层人体的严重遮挡问题。大量实验证明了我们的方法在生成解缠结着装虚拟形象方面具有令人印象深刻的性能，我们进一步探索了它在组件迁移中的应用。项目页面位于：此 https URL

Title: A Statistical Theory of Contrastive Pre-training and Multimodal Generative AI

• Authors: Kazusato Oko, Licong Lin, Yuhang Cai, Song Mei
• Subjects: cs.LG, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.04641
• Pdf URL: https://arxiv.org/pdf/2501.04641
• Copy Paste: [[2501.04641]] A Statistical Theory of Contrastive Pre-training and Multimodal Generative AI(https://arxiv.org/abs/2501.04641)
• Keywords: generative
• Abstract: Multi-modal generative AI systems, such as those combining vision and language, rely on contrastive pre-training to learn representations across different modalities. While their practical benefits are widely acknowledged, a rigorous theoretical understanding of the contrastive pre-training framework remains limited. This paper develops a theoretical framework to explain the success of contrastive pre-training in downstream tasks, such as zero-shot classification, conditional diffusion models, and vision-language models. We introduce the concept of approximate sufficient statistics, a generalization of the classical sufficient statistics, and show that near-minimizers of the contrastive pre-training loss are approximately sufficient, making them adaptable to diverse downstream tasks. We further propose the Joint Generative Hierarchical Model for the joint distribution of images and text, showing that transformers can efficiently approximate relevant functions within this model via belief propagation. Building on this framework, we derive sample complexity guarantees for multi-modal learning based on contrastive pre-trained representations. Numerical simulations validate these theoretical findings, demonstrating the strong generalization performance of contrastively pre-trained transformers in various multi-modal tasks.
• 摘要：多模态生成式 AI 系统（例如将视觉和语言相结合的系统）依赖于对比预训练来学习跨不同模态的表征。虽然它们的实际好处得到了广泛认可，但对比预训练框架的严格理论理解仍然有限。本文开发了一个理论框架来解释对比预训练在下游任务（例如零样本分类、条件扩散模型和视觉语言模型）中的成功。我们引入了近似充分统计的概念，这是经典充分统计的概括，并表明对比预训练损失的近似最小化器是近似充分的，使其能够适应各种下游任务。我们进一步提出了用于图像和文本联合分布的联合生成分层模型，表明 Transformer 可以通过信念传播有效地近似该模型中的相关函数。在此框架的基础上，我们推导出基于对比预训练表征的多模态学习的样本复杂性保证。数值模拟验证了这些理论发现，证明了对比预训练的 Transformer 在各种多模态任务中的强大泛化性能。

Title: Enhancing Virtual Try-On with Synthetic Pairs and Error-Aware Noise Scheduling

• Authors: Nannan Li, Kevin J. Shih, Bryan A. Plummer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04666
• Pdf URL: https://arxiv.org/pdf/2501.04666
• Copy Paste: [[2501.04666]] Enhancing Virtual Try-On with Synthetic Pairs and Error-Aware Noise Scheduling(https://arxiv.org/abs/2501.04666)
• Keywords: generation
• Abstract: Given an isolated garment image in a canonical product view and a separate image of a person, the virtual try-on task aims to generate a new image of the person wearing the target garment. Prior virtual try-on works face two major challenges in achieving this goal: a) the paired (human, garment) training data has limited availability; b) generating textures on the human that perfectly match that of the prompted garment is difficult, often resulting in distorted text and faded textures. Our work explores ways to tackle these issues through both synthetic data as well as model refinement. We introduce a garment extraction model that generates (human, synthetic garment) pairs from a single image of a clothed individual. The synthetic pairs can then be used to augment the training of virtual try-on. We also propose an Error-Aware Refinement-based Schrödinger Bridge (EARSB) that surgically targets localized generation errors for correcting the output of a base virtual try-on model. To identify likely errors, we propose a weakly-supervised error classifier that localizes regions for refinement, subsequently augmenting the Schrödinger Bridge's noise schedule with its confidence heatmap. Experiments on VITON-HD and DressCode-Upper demonstrate that our synthetic data augmentation enhances the performance of prior work, while EARSB improves the overall image quality. In user studies, our model is preferred by the users in an average of 59% of cases.
• 摘要：给定一个标准产品视图中的孤立服装图像和一个单独的人物图像，虚拟试穿任务旨在生成穿着目标服装的人的新图像。之前的虚拟试穿工作在实现这一目标时面临两大挑战：a) 配对的 (人类，服装) 训练数据有限；b) 生成与提示服装完美匹配的人体纹理很困难，通常会导致文本扭曲和纹理褪色。我们的工作探索了通过合成数据和模型细化解决这些问题的方法。我们引入了一个服装提取模型，该模型可以从穿着衣服的个体的单张图像生成 (人类，合成服装) 对。然后可以使用合成对来增强虚拟试穿的训练。我们还提出了一种基于错误感知细化的薛定谔桥 (EARSB)，它可以精准地瞄准局部生成错误，以纠正基础虚拟试穿模型的输出。为了识别可能的错误，我们提出了一个弱监督错误分类器，它可以定位要细化的区域，随后使用其置信度热图增强 Schrödinger Bridge 的噪声计划。在 VITON-HD 和 DressCode-Upper 上的实验表明，我们的合成数据增强提高了先前工作的性能，而 EARSB 提高了整体图像质量。在用户研究中，我们的模型在平均 59% 的情况下受到用户的青睐。

Title: SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images

• Authors: Zixuan Huang, Mark Boss, Aaryaman Vasishta, James M. Rehg, Varun Jampani
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.04689
• Pdf URL: https://arxiv.org/pdf/2501.04689
• Copy Paste: [[2501.04689]] SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images(https://arxiv.org/abs/2501.04689)
• Keywords: generation, generative
• Abstract: We study the problem of single-image 3D object reconstruction. Recent works have diverged into two directions: regression-based modeling and generative modeling. Regression methods efficiently infer visible surfaces, but struggle with occluded regions. Generative methods handle uncertain regions better by modeling distributions, but are computationally expensive and the generation is often misaligned with visible surfaces. In this paper, we present SPAR3D, a novel two-stage approach aiming to take the best of both directions. The first stage of SPAR3D generates sparse 3D point clouds using a lightweight point diffusion model, which has a fast sampling speed. The second stage uses both the sampled point cloud and the input image to create highly detailed meshes. Our two-stage design enables probabilistic modeling of the ill-posed single-image 3D task while maintaining high computational efficiency and great output fidelity. Using point clouds as an intermediate representation further allows for interactive user edits. Evaluated on diverse datasets, SPAR3D demonstrates superior performance over previous state-of-the-art methods, at an inference speed of 0.7 seconds. Project page with code and model: this https URL
• 摘要：我们研究单图像 3D 物体重建问题。最近的研究分为两个方向：基于回归的建模和生成建模。回归方法可以有效地推断可见表面，但在处理遮挡区域时会遇到困难。生成方法通过建模分布可以更好地处理不确定区域，但计算成本高，并且生成通常与可见表面不一致。在本文中，我们提出了 SPAR3D，这是一种新颖的两阶段方法，旨在兼顾两个方向的优点。SPAR3D 的第一阶段使用轻量级点扩散模型生成稀疏 3D 点云，该模型具有快速的采样速度。第二阶段使用采样点云和输入图像来创建高度详细的网格。我们的两阶段设计能够对不适定的单图像 3D 任务进行概率建模，同时保持高计算效率和出色的输出保真度。使用点云作为中间表示进一步允许交互式用户编辑。经过对各种数据集的评估，SPAR3D 表现出比之前最先进的方法更优越的性能，推理速度为 0.7 秒。包含代码和模型的项目页面：此 https URL

Title: Re-ranking the Context for Multimodal Retrieval Augmented Generation

• Authors: Matin Mortaheb, Mohammad A. Amir Khojastepour, Srimat T. Chakradhar, Sennur Ulukus
• Subjects: cs.LG, cs.CV, cs.IR, cs.IT
• Abstract URL: https://arxiv.org/abs/2501.04695
• Pdf URL: https://arxiv.org/pdf/2501.04695
• Copy Paste: [[2501.04695]] Re-ranking the Context for Multimodal Retrieval Augmented Generation(https://arxiv.org/abs/2501.04695)
• Keywords: generation
• Abstract: Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating external knowledge to generate a response within a context with improved accuracy and reduced hallucinations. However, multi-modal RAG systems face unique challenges: (i) the retrieval process may select irrelevant entries to user query (e.g., images, documents), and (ii) vision-language models or multi-modal language models like GPT-4o may hallucinate when processing these entries to generate RAG output. In this paper, we aim to address the first challenge, i.e, improving the selection of relevant context from the knowledge-base in retrieval phase of the multi-modal RAG. Specifically, we leverage the relevancy score (RS) measure designed in our previous work for evaluating the RAG performance to select more relevant entries in retrieval process. The retrieval based on embeddings, say CLIP-based embedding, and cosine similarity usually perform poorly particularly for multi-modal data. We show that by using a more advanced relevancy measure, one can enhance the retrieval process by selecting more relevant pieces from the knowledge-base and eliminate the irrelevant pieces from the context by adaptively selecting up-to-$k$ entries instead of fixed number of entries. Our evaluation using COCO dataset demonstrates significant enhancement in selecting relevant context and accuracy of the generated response.
• 摘要：检索增强生成 (RAG) 通过整合外部知识来增强大型语言模型 (LLM)，从而提高准确性并减少幻觉。然而，多模态 RAG 系统面临着独特的挑战：(i) 检索过程可能会选择与用户查询无关的条目（例如图像、文档），以及 (ii) 视觉语言模型或多模态语言模型（如 GPT-4o）在处理这些条目以生成 RAG 输出时可能会产生幻觉。在本文中，我们旨在解决第一个挑战，即在多模态 RAG 的检索阶段改进从知识库中选择相关上下文的方法。具体而言，我们利用我们之前工作中设计的用于评估 RAG 性能的相关性分数 (RS) 度量来在检索过程中选择更多相关条目。基于嵌入的检索（例如基于 CLIP 的嵌入）和余弦相似度通常表现不佳，尤其是对于多模态数据。我们表明，通过使用更先进的相关性度量，人们可以通过从知识库中选择更多相关部分来增强检索过程，并通过自适应地选择最多 $k$ 个条目（而不是固定数量的条目）来从上下文中消除不相关的部分。我们使用 COCO 数据集进行的评估表明，在选择相关上下文和生成响应的准确性方面有显著增强。

Title: ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuning

• Authors: Yuzhou Huang, Ziyang Yuan, Quande Liu, Qiulin Wang, Xintao Wang, Ruimao Zhang, Pengfei Wan, Di Zhang, Kun Gai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04698
• Pdf URL: https://arxiv.org/pdf/2501.04698
• Copy Paste: [[2501.04698]] ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuning(https://arxiv.org/abs/2501.04698)
• Keywords: generation
• Abstract: Text-to-video generation has made remarkable advancements through diffusion models. However, Multi-Concept Video Customization (MCVC) remains a significant challenge. We identify two key challenges in this task: 1) the identity decoupling problem, where directly adopting existing customization methods inevitably mix attributes when handling multiple concepts simultaneously, and 2) the scarcity of high-quality video-entity pairs, which is crucial for training such a model that represents and decouples various concepts well. To address these challenges, we introduce ConceptMaster, an innovative framework that effectively tackles the critical issues of identity decoupling while maintaining concept fidelity in customized videos. Specifically, we introduce a novel strategy of learning decoupled multi-concept embeddings that are injected into the diffusion models in a standalone manner, which effectively guarantees the quality of customized videos with multiple identities, even for highly similar visual concepts. To further overcome the scarcity of high-quality MCVC data, we carefully establish a data construction pipeline, which enables systematic collection of precise multi-concept video-entity data across diverse concepts. A comprehensive benchmark is designed to validate the effectiveness of our model from three critical dimensions: concept fidelity, identity decoupling ability, and video generation quality across six different concept composition scenarios. Extensive experiments demonstrate that our ConceptMaster significantly outperforms previous approaches for this task, paving the way for generating personalized and semantically accurate videos across multiple concepts.
• 摘要：通过传播模型，文本到视频的生成取得了显著的进步。然而，多概念视频定制 (MCVC) 仍然是一个重大挑战。我们确定了这项任务中的两个关键挑战：1) 身份解耦问题，直接采用现有的定制方法在同时处理多个概念时不可避免地会混合属性；2) 高质量视频实体对的稀缺性，这对于训练这种能够很好地表示和解耦各种概念的模型至关重要。为了应对这些挑战，我们引入了 ConceptMaster，这是一个创新的框架，可以有效地解决身份解耦的关键问题，同时保持定制视频中的概念保真度。具体来说，我们引入了一种学习解耦多概念嵌入的新策略，这些嵌入以独立的方式注入到传播模型中，这有效地保证了具有多个身份的定制视频的质量，即使对于高度相似的视觉概念也是如此。为了进一步克服高质量 MCVC 数据的稀缺性，我们精心建立了一个数据构建管道，可以系统地收集跨不同概念的精确多概念视频实体数据。我们设计了一个全面的基准来从三个关键维度验证我们模型的有效性：概念保真度、身份解耦能力和跨六种不同概念组合场景的视频生成质量。大量实验表明，我们的 ConceptMaster 明显优于之前完成此任务的方法，为跨多个概念生成个性化且语义准确的视频铺平了道路。

Title: EditAR: Unified Conditional Generation with Autoregressive Models

• Authors: Jiteng Mu, Nuno Vasconcelos, Xiaolong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04699
• Pdf URL: https://arxiv.org/pdf/2501.04699
• Copy Paste: [[2501.04699]] EditAR: Unified Conditional Generation with Autoregressive Models(https://arxiv.org/abs/2501.04699)
• Keywords: generation
• Abstract: Recent progress in controllable image generation and editing is largely driven by diffusion-based methods. Although diffusion models perform exceptionally well in specific tasks with tailored designs, establishing a unified model is still challenging. In contrast, autoregressive models inherently feature a unified tokenized representation, which simplifies the creation of a single foundational model for various tasks. In this work, we propose EditAR, a single unified autoregressive framework for a variety of conditional image generation tasks, e.g., image editing, depth-to-image, edge-to-image, segmentation-to-image. The model takes both images and instructions as inputs, and predicts the edited images tokens in a vanilla next-token paradigm. To enhance the text-to-image alignment, we further propose to distill the knowledge from foundation models into the autoregressive modeling process. We evaluate its effectiveness across diverse tasks on established benchmarks, showing competitive performance to various state-of-the-art task-specific methods. Project page: this https URL
• 摘要：可控图像生成和编辑的最新进展主要由基于扩散的方法推动。尽管扩散模型在具有定制设计的特定任务中表现异常出色，但建立统一的模型仍然具有挑战性。相比之下，自回归模型本质上具有统一的标记化表示，这简化了为各种任务创建单个基础模型的过程。在这项工作中，我们提出了 EditAR，这是一个用于各种条件图像生成任务的单一统一的自回归框架，例如图像编辑、深度到图像、边缘到图像、分割到图像。该模型将图像和指令作为输入，并在 vanilla next-token 范式中预测已编辑图像的标记。为了增强文本到图像的对齐，我们进一步建议将基础模型中的知识提炼到自回归建模过程中。我们在既定的基准上评估了它在各种任务中的有效性，显示出与各种最先进的任务特定方法具有竞争力的性能。项目页面：此 https URL