2025-01-23

Title: ImageRef-VL: Enabling Contextual Image Referencing in Vision-Language Models

• Authors: Jingwei Yi, Junhao Yin, Ju Xu, Peng Bao, Yongliang Wang, Wei Fan, Hao Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.12418
• Pdf URL: https://arxiv.org/pdf/2501.12418
• Copy Paste: [[2501.12418]] ImageRef-VL: Enabling Contextual Image Referencing in Vision-Language Models(https://arxiv.org/abs/2501.12418)
• Keywords: generation
• Abstract: Vision-Language Models (VLMs) have demonstrated remarkable capabilities in understanding multimodal inputs and have been widely integrated into Retrieval-Augmented Generation (RAG) based conversational systems. While current VLM-powered chatbots can provide textual source references in their responses, they exhibit significant limitations in referencing contextually relevant images during conversations. In this paper, we introduce Contextual Image Reference -- the ability to appropriately reference relevant images from retrieval documents based on conversation context -- and systematically investigate VLMs' capability in this aspect. We conduct the first evaluation for contextual image referencing, comprising a dedicated testing dataset and evaluation metrics. Furthermore, we propose ImageRef-VL, a method that significantly enhances open-source VLMs' image referencing capabilities through instruction fine-tuning on a large-scale, manually curated multimodal conversation dataset. Experimental results demonstrate that ImageRef-VL not only outperforms proprietary models but also achieves an 88% performance improvement over state-of-the-art open-source VLMs in contextual image referencing tasks. Our code is available at this https URL.
• 摘要：视觉语言模型 (VLM) 在理解多模态输入方面表现出了卓越的能力，并已广泛集成到基于检索增强生成 (RAG) 的对话系统中。虽然目前由 VLM 驱动的聊天机器人可以在响应中提供文本源引用，但它们在对话过程中引用上下文相关图像方面表现出很大的局限性。在本文中，我们引入了上下文图像引用——根据对话上下文适当引用检索文档中的相关图像的能力——并系统地研究了 VLM 在这方面的能力。我们对上下文图像引用进行了首次评估，包括专用的测试数据集和评估指标。此外，我们提出了 ImageRef-VL，这种方法通过对大规模、手动策划的多模态对话数据集进行指令微调，显著增强了开源 VLM 的图像引用能力。实验结果表明，ImageRef-VL 不仅优于专有模型，而且在上下文图像引用任务中比最先进的开源 VLM 实现了 88% 的性能提升。我们的代码可在此 https URL 上获取。

Title: Owls are wise and foxes are unfaithful: Uncovering animal stereotypes in vision-language models

• Authors: Tabinda Aman, Mohammad Nadeem, Shahab Saquib Sohail, Mohammad Anas, Erik Cambria
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.12433
• Pdf URL: https://arxiv.org/pdf/2501.12433
• Copy Paste: [[2501.12433]] Owls are wise and foxes are unfaithful: Uncovering animal stereotypes in vision-language models(https://arxiv.org/abs/2501.12433)
• Keywords: generation
• Abstract: Animal stereotypes are deeply embedded in human culture and language. They often shape our perceptions and expectations of various species. Our study investigates how animal stereotypes manifest in vision-language models during the task of image generation. Through targeted prompts, we explore whether DALL-E perpetuates stereotypical representations of animals, such as "owls as wise," "foxes as unfaithful," etc. Our findings reveal significant stereotyped instances where the model consistently generates images aligned with cultural biases. The current work is the first of its kind to examine animal stereotyping in vision-language models systematically and to highlight a critical yet underexplored dimension of bias in AI-generated visual content.
• 摘要：动物刻板印象深深植根于人类文化和语言中。它们常常塑造我们对各种物种的看法和期望。我们的研究调查了动物刻板印象在图像生成任务中如何在视觉语言模型中体现。通过有针对性的提示，我们探索 DALL-E 是否会延续对动物的刻板印象，例如“猫头鹰是聪明的”、“狐狸是不忠诚的”等。我们的研究结果揭示了显著的刻板印象，即模型始终生成与文化偏见相符的图像。当前的研究是首次系统地研究视觉语言模型中的动物刻板印象，并强调人工智能生成的视觉内容中一个关键但尚未得到充分探索的偏见维度。

Title: Federated Discrete Denoising Diffusion Model for Molecular Generation with OpenFL

• Authors: Kevin Ta, Patrick Foley, Mattson Thieme, Abhishek Pandey, Prashant Shah
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.12523
• Pdf URL: https://arxiv.org/pdf/2501.12523
• Copy Paste: [[2501.12523]] Federated Discrete Denoising Diffusion Model for Molecular Generation with OpenFL(https://arxiv.org/abs/2501.12523)
• Keywords: generation
• Abstract: Generating unique molecules with biochemically desired properties to serve as viable drug candidates is a difficult task that requires specialized domain expertise. In recent years, diffusion models have shown promising results in accelerating the drug design process through AI-driven molecular generation. However, training these models requires massive amounts of data, which are often isolated in proprietary silos. OpenFL is a federated learning framework that enables privacy-preserving collaborative training across these decentralized data sites. In this work, we present a federated discrete denoising diffusion model that was trained using OpenFL. The federated model achieves comparable performance with a model trained on centralized data when evaluating the uniqueness and validity of the generated molecules. This demonstrates the utility of federated learning in the drug design process. OpenFL is available at: this https URL
• 摘要：生成具有生物化学所需特性的独特分子以作为可行的候选药物是一项艰巨的任务，需要专门领域的专业知识。近年来，扩散模型通过人工智能驱动的分子生成在加速药物设计过程中显示出良好的效果。然而，训练这些模型需要大量数据，而这些数据通常被隔离在专有孤岛中。OpenFL 是一个联邦学习框架，可以在这些分散的数据站点之间进行隐私保护的协作训练。在这项工作中，我们提出了一个使用 OpenFL 训练的联邦离散去噪扩散模型。在评估生成的分子的独特性和有效性时，联邦模型实现了与在集中数据上训练的模型相当的性能。这证明了联邦学习在药物设计过程中的实用性。OpenFL 可在以下网址获得：此 https URL

Title: FedGrAINS: Personalized SubGraph Federated Learning with Adaptive Neighbor Sampling

• Authors: Emir Ceyani, Han Xie, Baturalp Buyukates, Carl Yang, Salman Avestimehr
• Subjects: cs.LG, cs.AI, cs.DC, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.12592
• Pdf URL: https://arxiv.org/pdf/2501.12592
• Copy Paste: [[2501.12592]] FedGrAINS: Personalized SubGraph Federated Learning with Adaptive Neighbor Sampling(https://arxiv.org/abs/2501.12592)
• Keywords: generative
• Abstract: Graphs are crucial for modeling relational and biological data. As datasets grow larger in real-world scenarios, the risk of exposing sensitive information increases, making privacy-preserving training methods like federated learning (FL) essential to ensure data security and compliance with privacy regulations. Recently proposed personalized subgraph FL methods have become the de-facto standard for training personalized Graph Neural Networks (GNNs) in a federated manner while dealing with the missing links across clients' subgraphs due to privacy restrictions. However, personalized subgraph FL faces significant challenges due to the heterogeneity in client subgraphs, such as degree distributions among the nodes, which complicate federated training of graph models. To address these challenges, we propose \textit{FedGrAINS}, a novel data-adaptive and sampling-based regularization method for subgraph FL. FedGrAINS leverages generative flow networks (GFlowNets) to evaluate node importance concerning clients' tasks, dynamically adjusting the message-passing step in clients' GNNs. This adaptation reflects task-optimized sampling aligned with a trajectory balance objective. Experimental results demonstrate that the inclusion of \textit{FedGrAINS} as a regularizer consistently improves the FL performance compared to baselines that do not leverage such regularization.
• 摘要：图对于关系数据和生物数据的建模至关重要。随着现实场景中数据集的不断增大，泄露敏感信息的风险也随之增加，因此，联邦学习 (FL) 等隐私保护训练方法对于确保数据安全和遵守隐私法规至关重要。最近提出的个性化子图 FL 方法已成为以联邦方式训练个性化图神经网络 (GNN) 的事实标准，同时处理由于隐私限制而导致的客户端子图中缺失的链接。然而，由于客户端子图中的异质性（例如节点之间的度分布），个性化子图 FL 面临着重大挑战，这使图模型的联邦训练变得复杂。为了应对这些挑战，我们提出了 \textit{FedGrAINS}，这是一种用于子图 FL 的新型数据自适应和基于采样的正则化方法。FedGrAINS 利用生成流网络 (GFlowNets) 来评估与客户端任务有关的节点重要性，动态调整客户端 GNN 中的消息传递步骤。这种调整反映了与轨迹平衡目标相一致的任务优化采样。实验结果表明，与不利用此类正则化的基线相比，将 \textit{FedGrAINS} 作为正则化器可以持续提高 FL 性能。

Title: EchoLM: Accelerating LLM Serving with Real-time Knowledge Distillation

• Authors: Yifan Yu, Yu Gan, Lily Tasi, Nikhil Sarda, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Henry M. Levy, David Culler
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.12689
• Pdf URL: https://arxiv.org/pdf/2501.12689
• Copy Paste: [[2501.12689]] EchoLM: Accelerating LLM Serving with Real-time Knowledge Distillation(https://arxiv.org/abs/2501.12689)
• Keywords: generation
• Abstract: Large language models (LLMs) have excelled in various applications, yet serving them at scale is challenging due to their substantial resource demands and high latency. Our real-world studies reveal that over 60% of user requests to LLMs have semantically similar counterparts, suggesting the potential for knowledge sharing among requests. However, naively caching and reusing past responses leads to large quality degradation. In this paper, we introduce EchoLM, an in-context caching system that leverages historical requests as examples to guide response generation, enabling selective offloading of requests to more efficient LLMs. However, enabling this real-time knowledge transfer leads to intricate tradeoffs between response quality, latency, and system throughput at scale. For a new request, EchoLM identifies similar, high-utility examples and efficiently prepends them to the input for better response. At scale, EchoLM adaptively routes requests to LLMs of varying capabilities, accounting for response quality and serving loads. EchoLM employs a cost-aware cache replay mechanism to improve example quality and coverage offline, maximizing cache utility and runtime efficiency. Evaluations on millions of open-source requests demonstrate that EchoLM has a throughput improvement of 1.4-5.9x while reducing latency by 28-71% without hurting response quality on average.
• 摘要：大型语言模型 (LLM) 在各种应用中表现出色，但由于其巨大的资源需求和高延迟，大规模服务于它们具有挑战性。我们的实际研究表明，超过 60% 的用户对 LLM 的请求具有语义上相似的对应项，这表明请求之间可能存在知识共享。然而，单纯地缓存和重用过去的响应会导致质量大幅下降。在本文中，我们介绍了 EchoLM，这是一个上下文缓存系统，它利用历史请求作为示例来指导响应生成，从而能够选择性地将请求卸载到更高效的 LLM。然而，实现这种实时知识传输会导致在大规模响应质量、延迟和系统吞吐量之间进行复杂的权衡。对于新请求，EchoLM 会识别类似的、高实用性的示例，并有效地将它们添加到输入中以获得更好的响应。在大规模情况下，EchoLM 会自适应地将请求路由到具有不同功能的 LLM，同时考虑响应质量和服务负载。 EchoLM 采用成本感知缓存重放机制来提高离线示例质量和覆盖率，从而最大限度地提高缓存效用和运行时效率。对数百万个开源请求的评估表明，EchoLM 的吞吐量提高了 1.4-5.9 倍，同时将延迟降低了 28-71%，而平均响应质量没有受到影响。

Title: REX: Causal Discovery based on Machine Learning and Explainability techniques

• Authors: Jesus Renero, Idoia Ochoa, Roberto Maestre
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.12706
• Pdf URL: https://arxiv.org/pdf/2501.12706
• Copy Paste: [[2501.12706]] REX: Causal Discovery based on Machine Learning and Explainability techniques(https://arxiv.org/abs/2501.12706)
• Keywords: generation
• Abstract: Explainability techniques hold significant potential for enhancing the causal discovery process, which is crucial for understanding complex systems in areas like healthcare, economics, and artificial intelligence. However, no causal discovery methods currently incorporate explainability into their models to derive causal graphs. Thus, in this paper we explore this innovative approach, as it offers substantial potential and represents a promising new direction worth investigating. Specifically, we introduce REX, a causal discovery method that leverages machine learning (ML) models coupled with explainability techniques, specifically Shapley values, to identify and interpret significant causal relationships among variables. Comparative evaluations on synthetic datasets comprising continuous tabular data reveal that REX outperforms state-of-the-art causal discovery methods across diverse data generation processes, including non-linear and additive noise models. Moreover, REX was tested on the Sachs single-cell protein-signaling dataset, achieving a precision of 0.952 and recovering key causal relationships with no incorrect edges. Taking together, these results showcase REX's effectiveness in accurately recovering true causal structures while minimizing false positive predictions, its robustness across diverse datasets, and its applicability to real-world problems. By combining ML and explainability techniques with causal discovery, REX bridges the gap between predictive modeling and causal inference, offering an effective tool for understanding complex causal structures. REX is publicly available at this https URL.
• 摘要：可解释性技术在增强因果发现过程方面具有巨大潜力，这对于理解医疗保健、经济学和人工智能等领域的复杂系统至关重要。然而，目前还没有因果发现方法将可解释性纳入其模型中以得出因果图。因此，在本文中，我们探索了这种创新方法，因为它具有巨大的潜力，代表了一个值得研究的有前途的新方向。具体来说，我们介绍了 REX，这是一种因果发现方法，它利用机器学习 (ML) 模型与可解释性技术（特别是 Shapley 值）相结合来识别和解释变量之间的重要因果关系。对包含连续表格数据的合成数据集的比较评估表明，REX 在包括非线性和加性噪声模型在内的各种数据生成过程中的表现均优于最先进的因果发现方法。此外，REX 在 Sachs 单细胞蛋白质信号数据集上进行了测试，精度达到 0.952，并恢复了关键因果关系，没有错误的边缘。综合起来，这些结果展示了 REX 在准确恢复真实因果结构的同时最大限度地减少误报预测的有效性、其在不同数据集中的稳健性以及其对实际问题的适用性。通过将 ML 和可解释性技术与因果发现相结合，REX 弥合了预测建模和因果推理之间的差距，为理解复杂的因果结构提供了有效的工具。REX 可在此 https URL 上公开获取。

Title: Certified Guidance for Planning with Deep Generative Models

• Authors: Francesco Giacomarra, Mehran Hosseini, Nicola Paoletti, Francesca Cairoli
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.12815
• Pdf URL: https://arxiv.org/pdf/2501.12815
• Copy Paste: [[2501.12815]] Certified Guidance for Planning with Deep Generative Models(https://arxiv.org/abs/2501.12815)
• Keywords: generative
• Abstract: Deep generative models, such as generative adversarial networks and diffusion models, have recently emerged as powerful tools for planning tasks and behavior synthesis in autonomous systems. Various guidance strategies have been introduced to steer the generative process toward outputs that are more likely to satisfy the planning objectives. These strategies avoid the need for model retraining but do not provide any guarantee that the generated outputs will satisfy the desired planning objectives. To address this limitation, we introduce certified guidance, an approach that modifies a generative model, without retraining it, into a new model guaranteed to satisfy a given specification with probability one. We focus on Signal Temporal Logic specifications, which are rich enough to describe nontrivial planning tasks. Our approach leverages neural network verification techniques to systematically explore the latent spaces of the generative models, identifying latent regions that are certifiably correct with respect to the STL property of interest. We evaluate the effectiveness of our method on four planning benchmarks using GANs and diffusion models. Our results confirm that certified guidance produces generative models that are always correct, unlike existing guidance methods that are not certified.
• 摘要：深度生成模型（例如生成对抗网络和扩散模型）最近已成为自主系统中规划任务和行为合成的强大工具。已经引入了各种指导策略来引导生成过程朝着更有可能满足规划目标的输出方向发展。这些策略避免了模型重新训练的需要，但不能保证生成的输出将满足所需的规划目标。为了解决这一限制，我们引入了经过认证的指导，这是一种修改生成模型而不对其进行重新训练的方法，将其变成一个保证以概率 1 满足给定规范的新模型。我们专注于信号时序逻辑规范，这些规范足够丰富，可以描述非平凡的规划任务。我们的方法利用神经网络验证技术系统地探索生成模型的潜在空间，识别出相对于感兴趣的 STL 属性可证明正确的潜在区域。我们使用 GAN 和扩散模型在四个规划基准上评估了我们方法的有效性。我们的研究结果证实，经过认证的指导产生的生成模型始终是正确的，这与未经认证的现有指导方法不同。

Title: AMM-Diff: Adaptive Multi-Modality Diffusion Network for Missing Modality Imputation

• Authors: Aghiles Kebaili, Jérôme Lapuyade-Lahorgue, Pierre Vera, Su Ruan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.12840
• Pdf URL: https://arxiv.org/pdf/2501.12840
• Copy Paste: [[2501.12840]] AMM-Diff: Adaptive Multi-Modality Diffusion Network for Missing Modality Imputation(https://arxiv.org/abs/2501.12840)
• Keywords: generative
• Abstract: In clinical practice, full imaging is not always feasible, often due to complex acquisition protocols, stringent privacy regulations, or specific clinical needs. However, missing MR modalities pose significant challenges for tasks like brain tumor segmentation, especially in deep learning-based segmentation, as each modality provides complementary information crucial for improving accuracy. A promising solution is missing data imputation, where absent modalities are generated from available ones. While generative models have been widely used for this purpose, most state-of-the-art approaches are limited to single or dual target translations, lacking the adaptability to generate missing modalities based on varying input configurations. To address this, we propose an Adaptive Multi-Modality Diffusion Network (AMM-Diff), a novel diffusion-based generative model capable of handling any number of input modalities and generating the missing ones. We designed an Image-Frequency Fusion Network (IFFN) that learns a unified feature representation through a self-supervised pretext task across the full input modalities and their selected high-frequency Fourier components. The proposed diffusion model leverages this representation, encapsulating prior knowledge of the complete modalities, and combines it with an adaptive reconstruction strategy to achieve missing modality completion. Experimental results on the BraTS 2021 dataset demonstrate the effectiveness of our approach.
• 摘要：在临床实践中，完整成像并不总是可行的，通常是由于复杂的采集协议、严格的隐私法规或特定的临床需求。然而，缺失的 MR 模态对脑肿瘤分割等任务构成了重大挑战，尤其是在基于深度学习的分割中，因为每种模态都提供了对提高准确性至关重要的互补信息。一个有希望的解决方案是缺失数据插补，即从可用模态中生成缺失的模态。虽然生成模型已广泛用于此目的，但大多数最先进的方法仅限于单目标或双目标翻译，缺乏根据不同输入配置生成缺失模态的适应性。为了解决这个问题，我们提出了一种自适应多模态扩散网络 (AMM-Diff)，这是一种基于扩散的新型生成模型，能够处理任意数量的输入模态并生成缺失的模态。我们设计了一个图像频率融合网络 (IFFN)，它通过自监督的借口任务在完整输入模态及其选定的高频傅里叶分量中学习统一的特征表示。所提出的扩散模型利用这种表示，封装了完整模态的先验知识，并将其与自适应重建策略相结合，以实现缺失模态的补全。在 BraTS 2021 数据集上的实验结果证明了我们方法的有效性。

Title: PreciseCam: Precise Camera Control for Text-to-Image Generation

• Authors: Edurne Bernal-Berdun, Ana Serrano, Belen Masia, Matheus Gadelha, Yannick Hold-Geoffroy, Xin Sun, Diego Gutierrez
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.12910
• Pdf URL: https://arxiv.org/pdf/2501.12910
• Copy Paste: [[2501.12910]] PreciseCam: Precise Camera Control for Text-to-Image Generation(https://arxiv.org/abs/2501.12910)
• Keywords: generation
• Abstract: Images as an artistic medium often rely on specific camera angles and lens distortions to convey ideas or emotions; however, such precise control is missing in current text-to-image models. We propose an efficient and general solution that allows precise control over the camera when generating both photographic and artistic images. Unlike prior methods that rely on predefined shots, we rely solely on four simple extrinsic and intrinsic camera parameters, removing the need for pre-existing geometry, reference 3D objects, and multi-view data. We also present a novel dataset with more than 57,000 images, along with their text prompts and ground-truth camera parameters. Our evaluation shows precise camera control in text-to-image generation, surpassing traditional prompt engineering approaches. Our data, model, and code are publicly available at this https URL.
• 摘要：图像作为一种艺术媒介，通常依赖于特定的相机角度和镜头扭曲来传达想法或情感；然而，目前的文本转图像模型缺乏这种精确的控制。我们提出了一种高效而通用的解决方案，可以在生成摄影和艺术图像时精确控制相机。与依赖预定义镜头的先前方法不同，我们仅依赖四个简单的外部和内部相机参数，无需预先存在的几何体、参考 3D 对象和多视图数据。我们还提供了一个包含 57,000 多张图像的新数据集，以及它们的文本提示和真实相机参数。我们的评估表明，文本转图像生成中的相机控制非常精确，超越了传统的提示工程方法。我们的数据、模型和代码可在此 https URL 上公开获取。

Title: 3D Object Manipulation in a Single Image using Generative Models

• Authors: Ruisi Zhao, Zechuan Zhang, Zongxin Yang, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.12935
• Pdf URL: https://arxiv.org/pdf/2501.12935
• Copy Paste: [[2501.12935]] 3D Object Manipulation in a Single Image using Generative Models(https://arxiv.org/abs/2501.12935)
• Keywords: generation, generative
• Abstract: Object manipulation in images aims to not only edit the object's presentation but also gift objects with motion. Previous methods encountered challenges in concurrently handling static editing and dynamic generation, while also struggling to achieve fidelity in object appearance and scene lighting. In this work, we introduce \textbf{OMG3D}, a novel framework that integrates the precise geometric control with the generative power of diffusion models, thus achieving significant enhancements in visual performance. Our framework first converts 2D objects into 3D, enabling user-directed modifications and lifelike motions at the geometric level. To address texture realism, we propose CustomRefiner, a texture refinement module that pre-train a customized diffusion model, aligning the details and style of coarse renderings of 3D rough model with the original image, further refine the texture. Additionally, we introduce IllumiCombiner, a lighting processing module that estimates and corrects background lighting to match human visual perception, resulting in more realistic shadow effects. Extensive experiments demonstrate the outstanding visual performance of our approach in both static and dynamic scenarios. Remarkably, all these steps can be done using one NVIDIA 3090. Project page is at this https URL
• 摘要：图像中的对象操作不仅旨在编辑对象的呈现，还旨在为对象赋予运动。以前的方法在同时处理静态编辑和动态生成方面遇到了挑战，同时还难以实现对象外观和场景照明的保真度。在这项工作中，我们引入了 \textbf{OMG3D}，这是一个新颖的框架，它将精确的几何控制与扩散模型的生成能力相结合，从而显著提高了视觉性能。我们的框架首先将 2D 对象转换为 3D，从而实现用户指导的几何级修改和逼真的运动。为了解决纹理真实感问题，我们提出了 CustomRefiner，这是一个纹理细化模块，可以预先训练定制的扩散模型，将 3D 粗糙模型的粗略渲染的细节和样式与原始图像对齐，进一步细化纹理。此外，我们引入了 IllumiCombiner，这是一个照明处理模块，可以估计和校正背景照明以匹配人类视觉感知，从而产生更逼真的阴影效果。大量实验证明了我们的方法在静态和动态场景中的出色视觉性能。值得注意的是，所有这些步骤都可以使用一个 NVIDIA 3090 完成。项目页面位于此 https URL

Title: LiT: Delving into a Simplified Linear Diffusion Transformer for Image Generation

• Authors: Jiahao Wang, Ning Kang, Lewei Yao, Mengzhao Chen, Chengyue Wu, Songyang Zhang, Shuchen Xue, Yong Liu, Taiqiang Wu, Xihui Liu, Kaipeng Zhang, Shifeng Zhang, Wenqi Shao, Zhenguo Li, Ping Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.12976
• Pdf URL: https://arxiv.org/pdf/2501.12976
• Copy Paste: [[2501.12976]] LiT: Delving into a Simplified Linear Diffusion Transformer for Image Generation(https://arxiv.org/abs/2501.12976)
• Keywords: generation
• Abstract: In commonly used sub-quadratic complexity modules, linear attention benefits from simplicity and high parallelism, making it promising for image synthesis tasks. However, the architectural design and learning strategy for linear attention remain underexplored in this field. In this paper, we offer a suite of ready-to-use solutions for efficient linear diffusion Transformers. Our core contributions include: (1) Simplified Linear Attention using few heads, observing the free-lunch effect of performance without latency increase. (2) Weight inheritance from a fully pre-trained diffusion Transformer: initializing linear Transformer using pre-trained diffusion Transformer and loading all parameters except for those related to linear attention. (3) Hybrid knowledge distillation objective: using a pre-trained diffusion Transformer to help the training of the student linear Transformer, supervising not only the predicted noise but also the variance of the reverse diffusion process. These guidelines lead to our proposed Linear Diffusion Transformer (LiT), an efficient text-to-image Transformer that can be deployed offline on a laptop. Experiments show that in class-conditional 256*256 and 512*512 ImageNet benchmark LiT achieves highly competitive FID while reducing training steps by 80% and 77% compared to DiT. LiT also rivals methods based on Mamba or Gated Linear Attention. Besides, for text-to-image generation, LiT allows for the rapid synthesis of up to 1K resolution photorealistic images. Project page: this https URL.
• 摘要：在常用的亚二次复杂度模块中，线性注意受益于简单性和高并行性，使其有望在图像合成任务中大放异彩。然而，线性注意的架构设计和学习策略在该领域仍未得到充分探索。在本文中，我们为高效的线性扩散Transformers提供了一套现成的解决方案。我们的核心贡献包括：（1）使用少量头部的简化线性注意，在不增加延迟的情况下观察到性能的免费午餐效应。（2）从完全预训练的扩散Transformer继承权重：使用预训练的扩散Transformer初始化线性Transformer并加载除与线性注意相关的参数之外的所有参数。（3）混合知识蒸馏目标：使用预训练的扩散Transformer帮助学生线性Transformer的训练，不仅监督预测噪声，还监督逆扩散过程的方差。这些指导方针促成了我们提出的线性扩散Transformer（LiT），这是一种高效的文本到图像Transformer，可以在笔记本电脑上离线部署。实验表明，在类条件 256*256 和 512*512 ImageNet 基准测试中，LiT 实现了极具竞争力的 FID，同时与 DiT 相比，训练步骤减少了 80% 和 77%。LiT 还可以与基于 Mamba 或 Gated Linear Attention 的方法相媲美。此外，对于文本到图像的生成，LiT 可以快速合成高达 1K 分辨率的照片级逼真图像。项目页面：这个 https URL。

Title: UniUIR: Considering Underwater Image Restoration as An All-in-One Learner

• Authors: Xu Zhang, Huan Zhang, Guoli Wang, Qian Zhang, Lefei Zhang, Bo Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.12981
• Pdf URL: https://arxiv.org/pdf/2501.12981
• Copy Paste: [[2501.12981]] UniUIR: Considering Underwater Image Restoration as An All-in-One Learner(https://arxiv.org/abs/2501.12981)
• Keywords: restoration
• Abstract: Existing underwater image restoration (UIR) methods generally only handle color distortion or jointly address color and haze issues, but they often overlook the more complex degradations that can occur in underwater scenes. To address this limitation, we propose a Universal Underwater Image Restoration method, termed as UniUIR, considering the complex scenario of real-world underwater mixed distortions as an all-in-one manner. To decouple degradation-specific issues and explore the inter-correlations among various degradations in UIR task, we designed the Mamba Mixture-of-Experts module. This module enables each expert to identify distinct types of degradation and collaboratively extract task-specific priors while maintaining global feature representation based on linear complexity. Building upon this foundation, to enhance degradation representation and address the task conflicts that arise when handling multiple types of degradation, we introduce the spatial-frequency prior generator. This module extracts degradation prior information in both spatial and frequency domains, and adaptively selects the most appropriate task-specific prompts based on image content, thereby improving the accuracy of image restoration. Finally, to more effectively address complex, region-dependent distortions in UIR task, we incorporate depth information derived from a large-scale pre-trained depth prediction model, thereby enabling the network to perceive and leverage depth variations across different image regions to handle localized degradation. Extensive experiments demonstrate that UniUIR can produce more attractive results across qualitative and quantitative comparisons, and shows strong generalization than state-of-the-art methods.
• 摘要：现有的水下图像恢复 (UIR) 方法通常只处理颜色失真或同时解决颜色和雾度问题，但它们往往忽略了水下场景中可能发生的更复杂的退化。为了解决这一限制，我们提出了一种通用水下图像恢复方法，称为 UniUIR，将现实世界水下混合失真的复杂场景视为一种一体化的方式。为了解耦特定于退化的问题并探索 UIR 任务中各种退化之间的相互关系，我们设计了 Mamba Mixture-of-Experts 模块。该模块使每个专家能够识别不同类型的退化并协作提取特定于任务的先验，同时保持基于线性复杂度的全局特征表示。在此基础上，为了增强退化表示并解决处理多种类型退化时出现的任务冲突，我们引入了空间频率先验生成器。该模块在空间和频域中提取退化先验信息，并根据图像内容自适应地选择最合适的特定于任务的提示，从而提高图像恢复的准确性。最后，为了更有效地解决 UIR 任务中复杂的、区域相关的失真问题，我们结合了来自大规模预训练深度预测模型的深度信息，从而使网络能够感知和利用不同图像区域的深度变化来处理局部退化。大量实验表明，UniUIR 可以在定性和定量比较中产生更具吸引力的结果，并且比最先进的方法表现出更强的泛化能力。

Title: Deep Learning-Based Image Recovery and Pose Estimation for Resident Space Objects

• Authors: Louis Aberdeen, Mark Hansen, Melvyn L. Smith, Lyndon Smith
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.13009
• Pdf URL: https://arxiv.org/pdf/2501.13009
• Copy Paste: [[2501.13009]] Deep Learning-Based Image Recovery and Pose Estimation for Resident Space Objects(https://arxiv.org/abs/2501.13009)
• Keywords: restoration
• Abstract: As the density of spacecraft in Earth's orbit increases, their recognition, pose and trajectory identification becomes crucial for averting potential collisions and executing debris removal operations. However, training models able to identify a spacecraft and its pose presents a significant challenge due to a lack of available image data for model training. This paper puts forth an innovative framework for generating realistic synthetic datasets of Resident Space Object (RSO) imagery. Using the International Space Station (ISS) as a test case, it goes on to combine image regression with image restoration methodologies to estimate pose from blurred images. An analysis of the proposed image recovery and regression techniques was undertaken, providing insights into the performance, potential enhancements and limitations when applied to real imagery of RSOs. The image recovery approach investigated involves first applying image deconvolution using an effective point spread function, followed by detail object extraction with a U-Net. Interestingly, using only U-Net for image reconstruction the best pose performance was attained, reducing the average Mean Squared Error in image recovery by 97.28% and the average angular error by 71.9%. The successful application of U-Net image restoration combined with the Resnet50 regression network for pose estimation of the International Space Station demonstrates the value of a diverse set of evaluation tools for effective solutions to real-world problems such as the analysis of distant objects in Earth's orbit.
• 摘要：随着地球轨道上航天器的密度增加，它们的识别、姿势和轨迹识别对于避免潜在碰撞和执行碎片清除操作至关重要。然而，由于缺乏可用于模型训练的图像数据，训练能够识别航天器及其姿势的模型是一项重大挑战。本文提出了一个创新框架，用于生成驻留空间物体 (RSO) 图像的真实合成数据集。使用国际空间站 (ISS) 作为测试案例，它继续将图像回归与图像恢复方法相结合，以从模糊图像中估计姿势。对提出的图像恢复和回归技术进行了分析，深入了解了应用于 RSO 真实图像时的性能、潜在增强和局限性。所研究的图像恢复方法首先使用有效的点扩展函数应用图像反卷积，然后使用 U-Net 提取细节对象。有趣的是，仅使用 U-Net 进行图像重建即可获得最佳姿势性能，将图像恢复中的平均均方误差降低 97.28%，平均角度误差降低 71.9%。 U-Net图像修复结合Resnet50回归网络在国际空间站姿态估计中的成功应用，证明了多样化评估工具对于有效解决地球轨道上遥远物体分析等现实问题的价值。

Title: A Probabilistic Model for Self-Supervised Learning

• Authors: Maximilian Fleissner, Pascal Esser, Debarghya Ghoshdastidar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13031
• Pdf URL: https://arxiv.org/pdf/2501.13031
• Copy Paste: [[2501.13031]] A Probabilistic Model for Self-Supervised Learning(https://arxiv.org/abs/2501.13031)
• Keywords: generative
• Abstract: Self-supervised learning (SSL) aims to find meaningful representations from unlabeled data by encoding semantic similarities through data augmentations. Despite its current popularity, theoretical insights about SSL are still scarce. For example, it is not yet known whether commonly used SSL loss functions can be related to a statistical model, much in the same as OLS, generalized linear models or PCA naturally emerge as maximum likelihood estimates of an underlying generative process. In this short paper, we consider a latent variable statistical model for SSL that exhibits an interesting property: Depending on the informativeness of the data augmentations, the MLE of the model either reduces to PCA, or approaches a simple non-contrastive loss. We analyze the model and also empirically illustrate our findings.
• 摘要：自监督学习 (SSL) 旨在通过数据增强对语义相似性进行编码，从未标记数据中找到有意义的表示。尽管 SSL 目前非常流行，但关于 SSL 的理论见解仍然很少。例如，目前尚不清楚常用的 SSL 损失函数是否可以与统计模型相关，就像 OLS、广义线性模型或 PCA 自然地作为底层生成过程的最大似然估计一样。在这篇短文中，我们考虑了一个 SSL 的潜在变量统计模型，该模型表现出一个有趣的特性：根据数据增强的信息量，模型的 MLE 要么简化为 PCA，要么接近简单的非对比损失。我们分析了该模型，并通过实证说明了我们的发现。

Title: STMDNet: A Lightweight Directional Framework for Motion Pattern Recognition of Tiny Targets

• Authors: Mingshuo Xu, Hao Luan, Zhou Daniel Hao, Jigen Peng, Shigang Yue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13054
• Pdf URL: https://arxiv.org/pdf/2501.13054
• Copy Paste: [[2501.13054]] STMDNet: A Lightweight Directional Framework for Motion Pattern Recognition of Tiny Targets(https://arxiv.org/abs/2501.13054)
• Keywords: generation
• Abstract: Recognizing motions of tiny targets - only few dozen pixels - in cluttered backgrounds remains a fundamental challenge when standard feature-based or deep learning methods fail under scarce visual cues. We propose STMDNet, a model-based computational framework to Recognize motions of tiny targets at variable velocities under low-sampling frequency scenarios. STMDNet designs a novel dual-dynamics-and-correlation mechanism, harnessing ipsilateral excitation to integrate target cues and leakage-enhancing-type contralateral inhibition to suppress large-object and background motion interference. Moreover, we develop the first collaborative directional encoding-decoding strategy that determines the motion direction from only one correlation per spatial location, cutting computational costs to one-eighth of prior methods. Further, simply substituting the backbone of a strong STMD model with STMDNet raises AUC by 24%, yielding an enhanced STMDNet-F. Evaluations on real-world low sampling frequency datasets show state-of-the-art results, surpassing the deep learning baseline. Across diverse speeds, STMDNet-F improves mF1 by 19%, 16%, and 8% at 240Hz, 120Hz, and 60Hz, respectively, while STMDNet achieves 87 FPS on a single CPU thread. These advances highlight STMDNet as a next-generation backbone for tiny target motion pattern recognition and underscore its broader potential to revitalize model-based visual approaches in motion detection.
• 摘要：当基于特征的标准方法或深度学习方法在稀缺的视觉线索下失效时，在杂乱的背景中识别微小目标（只有几十个像素）的运动仍然是一项基本挑战。我们提出了 STMDNet，这是一个基于模型的计算框架，用于在低采样频率场景下识别可变速度下的微小目标的运动。STMDNet 设计了一种新颖的双动力学和相关机制，利用同侧激励来整合目标线索和泄漏增强型对侧抑制来抑制大物体和背景运动干扰。此外，我们开发了第一个协作方向编码解码策略，该策略仅从每个空间位置的一个相关性来确定运动方向，将计算成本降低到先前方法的八分之一。此外，只需用 STMDNet 替换强大的 STMD 模型的主干即可将 AUC 提高 24%，从而产生增强的 STMDNet-F。对现实世界低采样频率数据集的评估显示了最先进的结果，超越了深度学习基线。在不同速度下，STMDNet-F 在 240Hz、120Hz 和 60Hz 下分别将 mF1 提高了 19%、16% 和 8%，而 STMDNet 在单个 CPU 线程上实现了 87 FPS。这些进步凸显了 STMDNet 作为微小目标运动模式识别的下一代骨干，并强调了其在振兴基于模型的运动检测视觉方法方面的更广泛潜力。

Title: Orchid: Image Latent Diffusion for Joint Appearance and Geometry Generation

• Authors: Akshay Krishnan, Xinchen Yan, Vincent Casser, Abhijit Kundu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13087
• Pdf URL: https://arxiv.org/pdf/2501.13087
• Copy Paste: [[2501.13087]] Orchid: Image Latent Diffusion for Joint Appearance and Geometry Generation(https://arxiv.org/abs/2501.13087)
• Keywords: generation
• Abstract: Diffusion models are state-of-the-art for image generation. Trained on large datasets, they capture expressive image priors that have been used for tasks like inpainting, depth, and (surface) normal prediction. However, these models are typically trained for one specific task, e.g., a separate model for each of color, depth, and normal prediction. Such models do not leverage the intrinsic correlation between appearance and geometry, often leading to inconsistent predictions. In this paper, we propose using a novel image diffusion prior that jointly encodes appearance and geometry. We introduce a diffusion model Orchid, comprising a Variational Autoencoder (VAE) to encode color, depth, and surface normals to a latent space, and a Latent Diffusion Model (LDM) for generating these joint latents. Orchid directly generates photo-realistic color images, relative depth, and surface normals from user-provided text, and can be used to create image-aligned partial 3D scenes seamlessly. It can also perform image-conditioned tasks like joint monocular depth and normal prediction and is competitive in accuracy to state-of-the-art methods designed for those tasks alone. Lastly, our model learns a joint prior that can be used zero-shot as a regularizer for many inverse problems that entangle appearance and geometry. For example, we demonstrate its effectiveness in color-depth-normal inpainting, showcasing its applicability to problems in 3D generation from sparse views.
• 摘要：扩散模型是图像生成领域最先进的模型。它们在大型数据集上进行训练，可以捕获富有表现力的图像先验，这些先验已用于修复、深度和（表面）法线预测等任务。然而，这些模型通常针对一项特定任务进行训练，例如，针对颜色、深度和法线预测分别使用一个模型。这样的模型没有利用外观和几何之间的内在相关性，通常会导致预测不一致。在本文中，我们提出使用一种新颖的图像扩散先验，联合编码外观和几何。我们引入了一个扩散模型 Orchid，它包括一个变分自动编码器 (VAE)，用于将颜色、深度和表面法线编码到潜在空间，以及一个潜在扩散模型 (LDM)，用于生成这些联合潜在值。Orchid 直接从用户提供的文本生成照片般逼真的彩色图像、相对深度和表面法线，并可用于无缝创建图像对齐的部分 3D 场景。它还可以执行图像条件任务，如联合单目深度和法线预测，其准确率可与专为这些任务而设计的最先进的方法相媲美。最后，我们的模型学习了一种联合先验，可以零样本用作许多涉及外观和几何的逆问题的正则化器。例如，我们展示了它在颜色深度法线修复中的有效性，展示了它对从稀疏视图生成 3D 问题的适用性。

Title: Robust Representation Consistency Model via Contrastive Denoising

• Authors: Jiachen Lei, Julius Berner, Jiongxiao Wang, Zhongzhu Chen, Zhongjia Ba, Kui Ren, Jun Zhu, Anima Anandkumar
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13094
• Pdf URL: https://arxiv.org/pdf/2501.13094
• Copy Paste: [[2501.13094]] Robust Representation Consistency Model via Contrastive Denoising(https://arxiv.org/abs/2501.13094)
• Keywords: generative
• Abstract: Robustness is essential for deep neural networks, especially in security-sensitive applications. To this end, randomized smoothing provides theoretical guarantees for certifying robustness against adversarial perturbations. Recently, diffusion models have been successfully employed for randomized smoothing to purify noise-perturbed samples before making predictions with a standard classifier. While these methods excel at small perturbation radii, they struggle with larger perturbations and incur a significant computational overhead during inference compared to classical methods. To address this, we reformulate the generative modeling task along the diffusion trajectories in pixel space as a discriminative task in the latent space. Specifically, we use instance discrimination to achieve consistent representations along the trajectories by aligning temporally adjacent points. After fine-tuning based on the learned representations, our model enables implicit denoising-then-classification via a single prediction, substantially reducing inference costs. We conduct extensive experiments on various datasets and achieve state-of-the-art performance with minimal computation budget during inference. For example, our method outperforms the certified accuracy of diffusion-based methods on ImageNet across all perturbation radii by 5.3% on average, with up to 11.6% at larger radii, while reducing inference costs by 85$\times$ on average. Codes are available at: this https URL.
• 摘要：鲁棒性对于深度神经网络至关重要，尤其是在安全敏感的应用中。为此，随机平滑为证明对抗性扰动的鲁棒性提供了理论保证。最近，扩散模型已成功用于随机平滑，以在使用标准分类器进行预测之前净化噪声扰动样本。虽然这些方法在小扰动半径下表现出色，但它们在处理较大的扰动时会遇到困难，并且与传统方法相比，在推理过程中会产生大量的计算开销。为了解决这个问题，我们将像素空间中沿扩散轨迹的生成建模任务重新表述为潜在空间中的判别任务。具体而言，我们使用实例判别通过对齐时间相邻的点来实现沿轨迹的一致表示。在基于学习到的表示进行微调后，我们的模型可以通过单个预测实现隐式去噪然后分类，从而大大降低了推理成本。我们对各种数据集进行了广泛的实验，并在推理过程中以最小的计算预算实现了最先进的性能。例如，我们的方法在所有扰动半径上的平均准确度比基于扩散的方法高出 5.3%，在更大的半径上最高可提高 11.6%，同时平均将推理成本降低了 85$\times$。代码可从此 https URL 获取。

Title: Neural Radiance Fields for the Real World: A Survey

• Authors: Wenhui Xiao, Remi Chierchia, Rodrigo Santa Cruz, Xuesong Li, David Ahmedt-Aristizabal, Olivier Salvado, Clinton Fookes, Leo Lebrat
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.13104
• Pdf URL: https://arxiv.org/pdf/2501.13104
• Copy Paste: [[2501.13104]] Neural Radiance Fields for the Real World: A Survey(https://arxiv.org/abs/2501.13104)
• Keywords: generation
• Abstract: Neural Radiance Fields (NeRFs) have remodeled 3D scene representation since release. NeRFs can effectively reconstruct complex 3D scenes from 2D images, advancing different fields and applications such as scene understanding, 3D content generation, and robotics. Despite significant research progress, a thorough review of recent innovations, applications, and challenges is lacking. This survey compiles key theoretical advancements and alternative representations and investigates emerging challenges. It further explores applications on reconstruction, highlights NeRFs' impact on computer vision and robotics, and reviews essential datasets and toolkits. By identifying gaps in the literature, this survey discusses open challenges and offers directions for future research.
• 摘要：自发布以来，神经辐射场 (NeRF) 已经重塑了 3D 场景表示。NeRF 可以有效地从 2D 图像重建复杂的 3D 场景，推动场景理解、3D 内容生成和机器人等不同领域和应用的发展。尽管研究取得了重大进展，但缺乏对近期创新、应用和挑战的全面回顾。本综述汇编了关键的理论进步和替代表示，并研究了新出现的挑战。它进一步探讨了重建方面的应用，强调了 NeRF 对计算机视觉和机器人的影响，并回顾了重要的数据集和工具包。通过确定文献中的差距，本综述讨论了尚未解决的挑战并为未来的研究提供了方向。

Title: Accelerate High-Quality Diffusion Models with Inner Loop Feedback

• Authors: Matthew Gwilliam, Han Cai, Di Wu, Abhinav Shrivastava, Zhiyu Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13107
• Pdf URL: https://arxiv.org/pdf/2501.13107
• Copy Paste: [[2501.13107]] Accelerate High-Quality Diffusion Models with Inner Loop Feedback(https://arxiv.org/abs/2501.13107)
• Keywords: generation, quality assessment
• Abstract: We propose Inner Loop Feedback (ILF), a novel approach to accelerate diffusion models' inference. ILF trains a lightweight module to predict future features in the denoising process by leveraging the outputs from a chosen diffusion backbone block at a given time step. This approach exploits two key intuitions; (1) the outputs of a given block at adjacent time steps are similar, and (2) performing partial computations for a step imposes a lower burden on the model than skipping the step entirely. Our method is highly flexible, since we find that the feedback module itself can simply be a block from the diffusion backbone, with all settings copied. Its influence on the diffusion forward can be tempered with a learnable scaling factor from zero initialization. We train this module using distillation losses; however, unlike some prior work where a full diffusion backbone serves as the student, our model freezes the backbone, training only the feedback module. While many efforts to optimize diffusion models focus on achieving acceptable image quality in extremely few steps (1-4 steps), our emphasis is on matching best case results (typically achieved in 20 steps) while significantly reducing runtime. ILF achieves this balance effectively, demonstrating strong performance for both class-to-image generation with diffusion transformer (DiT) and text-to-image generation with DiT-based PixArt-alpha and PixArt-sigma. The quality of ILF's 1.7x-1.8x speedups are confirmed by FID, CLIP score, CLIP Image Quality Assessment, ImageReward, and qualitative comparisons.
• 摘要：我们提出了内环反馈 (ILF)，这是一种加速扩散模型推理的新方法。ILF 通过在给定的时间步骤利用所选扩散主干块的输出来训练轻量级模块，以预测去噪过程中的未来特征。这种方法利用了两个关键直觉；(1) 给定块在相邻时间步骤的输出相似，(2) 对某个步骤执行部分计算比完全跳过该步骤对模型的负担更小。我们的方法非常灵活，因为我们发现反馈模块本身可以简单地是来自扩散主干的一个块，所有设置都复制了。它对扩散前向的影响可以通过零初始化的可学习缩放因子来缓和。我们使用蒸馏损失来训练这个模块；然而，与一些以前的工作不同，其中完整的扩散主干充当学生，我们的模型冻结了主干，只训练反馈模块。虽然许多优化扩散模型的努力都集中在以极少的步骤（1-4 步）实现可接受的图像质量，但我们的重点是匹配最佳情况结果（通常在 20 步中实现），同时显着减少运行时间。 ILF 有效地实现了这种平衡，在使用扩散变换器 (DiT) 进行类别到图像生成以及使用基于 DiT 的 PixArt-alpha 和 PixArt-sigma 进行文本到图像生成方面均表现出色。ILF 的 1.7 倍至 1.8 倍加速效果已通过 FID、CLIP 分数、CLIP 图像质量评估、ImageReward 和定性比较得到证实。