2025-01-14

Title: Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models

• Authors: Xin Ding, Shijie Cao, Ting Cao, Zhibo Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06218
• Pdf URL: https://arxiv.org/pdf/2501.06218
• Copy Paste: [[2501.06218]] Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models(https://arxiv.org/abs/2501.06218)
• Keywords: generative
• Abstract: Vision generative models have recently made significant advancements along two primary paradigms: diffusion-style and language-style, both of which have demonstrated excellent scaling laws. Quantization is crucial for efficiently deploying these models, as it reduces memory and computation costs. In this work, we systematically investigate the impact of quantization on these two paradigms. Surprisingly, despite achieving comparable performance in full precision, language-style models consistently outperform diffusion-style models across various quantization settings. This observation suggests that language-style models have superior bit-level scaling laws, offering a better tradeoff between model quality and total bits. To dissect this phenomenon, we conduct extensive experiments and find that the primary reason is the discrete representation space of language-style models, which is more tolerant of information loss during quantization. Furthermore, our analysis indicates that improving the bit-level scaling law of quantized vision generative models is challenging, with model distillation identified as a highly effective approach. Specifically, we propose TopKLD to optimize the transfer of distilled knowledge by balancing ``implicit knowledge'' and ``explicit knowledge'' during the distillation process. This approach elevates the bit-level scaling laws by one level across both integer and floating-point quantization settings.
• 摘要：视觉生成模型最近在两个主要范式上取得了重大进展：扩散风格和语言风格，两者都表现出了出色的缩放定律。量化对于有效部署这些模型至关重要，因为它可以降低内存和计算成本。在这项工作中，我们系统地研究了量化对这两个范式的影响。令人惊讶的是，尽管在完全精度下实现了可比的性能，但语言风格模型在各种量化设置中始终优于扩散风格模型。这一观察结果表明，语言风格模型具有优越的位级缩放定律，在模型质量和总位数之间提供了更好的权衡。为了剖析这一现象，我们进行了大量实验，发现主要原因是语言风格模型的离散表示空间，它对量化过程中的信息丢失更具容忍度。此外，我们的分析表明，改进量化视觉生成模型的位级缩放定律具有挑战性，模型蒸馏被认为是一种非常有效的方法。具体来说，我们提出 TopKLD 通过在蒸馏过程中平衡“隐性知识”和“显性知识”来优化蒸馏知识的传递。这种方法在整数和浮点量化设置中将位级缩放定律提升了一个级别。

Title: Mechanics and Design of Metastructured Auxetic Patches with Bio-inspired Materials

• Authors: Yingbin Chen, Milad Arzani, Xuan Mu, Sophia Jin, Shaoping Xiao
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2501.06233
• Pdf URL: https://arxiv.org/pdf/2501.06233
• Copy Paste: [[2501.06233]] Mechanics and Design of Metastructured Auxetic Patches with Bio-inspired Materials(https://arxiv.org/abs/2501.06233)
• Keywords: generation, generative
• Abstract: Metastructured auxetic patches, characterized by negative Poisson's ratios, offer unique mechanical properties that closely resemble the behavior of human tissues and organs. As a result, these patches have gained significant attention for their potential applications in organ repair and tissue regeneration. This study focuses on neural networks-based computational modeling of auxetic patches with a sinusoidal metastructure fabricated from silk fibroin, a bio-inspired material known for its biocompatibility and strength. The primary objective of this research is to introduce a novel, data-driven framework for patch design. To achieve this, we conducted experimental fabrication and mechanical testing to determine material properties and validate the corresponding finite element models. Finite element simulations were then employed to generate the necessary data, while greedy sampling, an active learning technique, was utilized to reduce the computational cost associated with data labeling. Two neural networks were trained to accurately predict Poisson's ratios and stresses for strains up to 15\%, respectively. Both models achieved $R^2$ scores exceeding 0.995, which indicates highly reliable predictions. Building on this, we developed a neural network-based design model capable of tailoring patch designs to achieve specific mechanical properties. This model demonstrated superior performance when compared to traditional optimization methods, such as genetic algorithms, by providing more efficient and precise design solutions. The proposed framework represents a significant advancement in the design of bio-inspired metastructures for medical applications, paving the way for future innovations in tissue engineering and regenerative medicine.
• 摘要：超结构膨胀补片具有负泊松比的特征，具有与人体组织和器官行为极为相似的独特机械性能。因此，这些补片因其在器官修复和组织再生方面的潜在应用而备受关注。本研究重点关注基于神经网络的膨胀补片计算建模，该补片具有由丝素蛋白制成的正弦超结构，丝素蛋白是一种以生物相容性和强度而闻名的仿生材料。本研究的主要目标是引入一种新颖的、数据驱动的补片设计框架。为了实现这一目标，我们进行了实验制造和机械测试，以确定材料特性并验证相应的有限元模型。然后采用有限元模拟来生成必要的数据，同时利用主动学习技术贪婪采样来降低与数据标记相关的计算成本。训练了两个神经网络，以准确预测高达 15% 的应变的泊松比和应力。两种模型的 $R^2$ 得分均超过 0.995，这表明预测高度可靠。在此基础上，我们开发了一种基于神经网络的设计模型，该模型能够定制贴片设计以实现特定的机械性能。与遗传算法等传统优化方法相比，该模型通过提供更高效、更精确的设计解决方案，表现出了卓越的性能。所提出的框架代表了医疗应用的仿生超结构设计的重大进步，为组织工程和再生医学的未来创新铺平了道路。

Title: Data-Driven Radio Propagation Modeling using Graph Neural Networks

• Authors: Adrien Bufort, Laurent Lebocq, Stefan Cathabard
• Subjects: cs.LG, cs.AI, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.06236
• Pdf URL: https://arxiv.org/pdf/2501.06236
• Copy Paste: [[2501.06236]] Data-Driven Radio Propagation Modeling using Graph Neural Networks(https://arxiv.org/abs/2501.06236)
• Keywords: generative
• Abstract: Modeling radio propagation is essential for wireless network design and performance optimization. Traditional methods rely on physics models of radio propagation, which can be inaccurate or inflexible. In this work, we propose using graph neural networks to learn radio propagation behaviors directly from real-world network data. Our approach converts the radio propagation environment into a graph representation, with nodes corresponding to locations and edges representing spatial and ray-tracing relationships between locations. The graph is generated by converting images of the environment into a graph structure, with specific relationships between nodes. The model is trained on this graph representation, using sensor measurements as target data. We demonstrate that the graph neural network, which learns to predict radio propagation directly from data, achieves competitive performance compared to traditional heuristic models. This data-driven approach outperforms classic numerical solvers in terms of both speed and accuracy. To the best of our knowledge, we are the first to apply graph neural networks to real-world radio propagation data to generate coverage maps, enabling generative models of signal propagation with point measurements only.
• 摘要：对无线电传播进行建模对于无线网络设计和性能优化至关重要。传统方法依赖于无线电传播的物理模型，这些模型可能不准确或不灵活。在这项工作中，我们提出使用图神经网络直接从现实世界的网络数据中学习无线电传播行为。我们的方法将无线电传播环境转换为图形表示，其中节点对应于位置，边缘表示位置之间的空间和光线追踪关系。该图是通过将环境图像转换为具有节点之间特定关系的图形结构而生成的。该模型在此图形表示上进行训练，使用传感器测量值作为目标数据。我们证明，与传统启发式模型相比，学习直接从数据预测无线电传播的图神经网络实现了具有竞争力的性能。这种数据驱动的方法在速度和准确性方面都优于经典的数值求解器。据我们所知，我们是第一个将图神经网络应用于现实世界的无线电传播数据以生成覆盖图的人，仅使用点测量就可以生成信号传播的生成模型。

Title: Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models

• Authors: Roberto-Rafael Maura-Rivero, Chirag Nagpal, Roma Patel, Francesco Visin
• Subjects: cs.LG, cs.AI, cs.CL, econ.GN
• Abstract URL: https://arxiv.org/abs/2501.06248
• Pdf URL: https://arxiv.org/pdf/2501.06248
• Copy Paste: [[2501.06248]] Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models(https://arxiv.org/abs/2501.06248)
• Keywords: generation
• Abstract: Current methods that train large language models (LLMs) with reinforcement learning feedback, often resort to averaging outputs of multiple rewards functions during training. This overlooks crucial aspects of individual reward dimensions and inter-reward dependencies that can lead to sub-optimal outcomes in generations. In this work, we show how linear aggregation of rewards exhibits some vulnerabilities that can lead to undesired properties of generated text. We then propose a transformation of reward functions inspired by economic theory of utility functions (specifically Inada conditions), that enhances sensitivity to low reward values while diminishing sensitivity to already high values. We compare our approach to the existing baseline methods that linearly aggregate rewards and show how the Inada-inspired reward feedback is superior to traditional weighted averaging. We quantitatively and qualitatively analyse the difference in the methods, and see that models trained with Inada-transformations score as more helpful while being less harmful.
• 摘要：目前使用强化学习反馈训练大型语言模型 (LLM) 的方法，通常会在训练期间求助于对多个奖励函数的输出进行平均。这忽略了单个奖励维度和奖励间依赖性的关键方面，而这些方面可能会导致生成结果不理想。在这项工作中，我们展示了奖励的线性聚合如何表现出一些漏洞，这些漏洞可能会导致生成文本的不良属性。然后，我们提出了一种受效用函数经济理论（特别是 Inada 条件）启发的奖励函数变换，它增强了对低奖励值的敏感性，同时降低了对已经很高的值的敏感性。我们将我们的方法与现有的线性聚合奖励的基线方法进行比较，并展示了受 Inada 启发的奖励反馈如何优于传统的加权平均。我们定量和定性地分析了这些方法的差异，发现使用 Inada 变换训练的模型得分更有帮助，同时危害更小。

Title: Generative AI for Cel-Animation: A Survey

• Authors: Yunlong Tang, Junjia Guo, Pinxin Liu, Zhiyuan Wang, Hang Hua, Jia-Xing Zhong, Yunzhong Xiao, Chao Huang, Luchuan Song, Susan Liang, Yizhi Song, Liu He, Jing Bi, Mingqian Feng, Xinyang Li, Zeliang Zhang, Chenliang Xu
• Subjects: cs.CV, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.06250
• Pdf URL: https://arxiv.org/pdf/2501.06250
• Copy Paste: [[2501.06250]] Generative AI for Cel-Animation: A Survey(https://arxiv.org/abs/2501.06250)
• Keywords: generation, generative
• Abstract: Traditional Celluloid (Cel) Animation production pipeline encompasses multiple essential steps, including storyboarding, layout design, keyframe animation, inbetweening, and colorization, which demand substantial manual effort, technical expertise, and significant time investment. These challenges have historically impeded the efficiency and scalability of Cel-Animation production. The rise of generative artificial intelligence (GenAI), encompassing large language models, multimodal models, and diffusion models, offers innovative solutions by automating tasks such as inbetween frame generation, colorization, and storyboard creation. This survey explores how GenAI integration is revolutionizing traditional animation workflows by lowering technical barriers, broadening accessibility for a wider range of creators through tools like AniDoc, ToonCrafter, and AniSora, and enabling artists to focus more on creative expression and artistic innovation. Despite its potential, issues such as maintaining visual consistency, ensuring stylistic coherence, and addressing ethical considerations continue to pose challenges. Furthermore, this paper discusses future directions and explores potential advancements in AI-assisted animation. For further exploration and resources, please visit our GitHub repository: this https URL
• 摘要：传统的赛璐珞 (Cel) 动画制作流程包含多个基本步骤，包括故事板、布局设计、关键帧动画、中间帧和着色，这些步骤需要大量的手动工作、技术专长和大量时间投入。这些挑战在历史上阻碍了 Cel-Animation 制作的效率和可扩展性。生成人工智能 (GenAI) 的兴起，包括大型语言模型、多模态模型和扩散模型，通过自动执行诸如中间帧生成、着色和故事板创建等任务，提供了创新的解决方案。本调查探讨了 GenAI 集成如何通过降低技术壁垒、通过 AniDoc、ToonCrafter 和 AniSora 等工具为更广泛的创作者扩大可访问性以及使艺术家能够更多地专注于创意表达和艺术创新来彻底改变传统动画工作流程。尽管它具有潜力，但诸如保持视觉一致性、确保风格连贯性和解决道德问题等问题仍然构成挑战。此外，本文讨论了未来的方向并探索了人工智能辅助动画的潜在进步。如需进一步探索和获取资源，请访问我们的 GitHub 存储库：此 https URL

Title: Quantum Down Sampling Filter for Variational Auto-encoder

• Authors: Farina Riaz, Fakhar Zaman, Hajime Suzuki, Sharif Abuadbba, David Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06259
• Pdf URL: https://arxiv.org/pdf/2501.06259
• Copy Paste: [[2501.06259]] Quantum Down Sampling Filter for Variational Auto-encoder(https://arxiv.org/abs/2501.06259)
• Keywords: generation, generative
• Abstract: Variational Autoencoders (VAEs) are essential tools in generative modeling and image reconstruction, with their performance heavily influenced by the encoder-decoder architecture. This study aims to improve the quality of reconstructed images by enhancing their resolution and preserving finer details, particularly when working with low-resolution inputs (16x16 pixels), where traditional VAEs often yield blurred or in-accurate results. To address this, we propose a hybrid model that combines quantum computing techniques in the VAE encoder with convolutional neural networks (CNNs) in the decoder. By upscaling the resolution from 16x16 to 32x32 during the encoding process, our approach evaluates how the model reconstructs images with enhanced resolution while maintaining key features and structures. This method tests the model's robustness in handling image reconstruction and its ability to preserve essential details despite training on lower-resolution data. We evaluate our proposed down sampling filter for Quantum VAE (Q-VAE) on the MNIST and USPS datasets and compare it with classical VAEs and a variant called Classical Direct Passing VAE (CDP-VAE), which uses windowing pooling filters in the encoding process. Performance is assessed using metrics such as the Fréchet Inception Distance (FID) and Mean Squared Error (MSE), which measure the fidelity of reconstructed images. Our results demonstrate that the Q-VAE consistently outperforms both the Classical VAE and CDP-VAE, achieving significantly lower FID and MSE scores. Additionally, CDP-VAE yields better performance than C-VAE. These findings highlight the potential of quantum-enhanced VAEs to improve image reconstruction quality by enhancing resolution and preserving essential features, offering a promising direction for future applications in computer vision and synthetic data generation.
• 摘要：变分自动编码器 (VAE) 是生成建模和图像重建中必不可少的工具，其性能在很大程度上受编码器-解码器架构的影响。本研究旨在通过提高分辨率和保留更精细的细节来提高重建图像的质量，特别是在处理低分辨率输入（16x16 像素）时，传统 VAE 通常会产生模糊或不准确的结果。为了解决这个问题，我们提出了一种混合模型，将 VAE 编码器中的量子计算技术与解码器中的卷积神经网络 (CNN) 相结合。通过在编码过程中将分辨率从 16x16 升级到 32x32，我们的方法评估了模型如何在保持关键特征和结构的同时以增强的分辨率重建图像。该方法测试了模型在处理图像重建方面的稳健性以及它在使用低分辨率数据进行训练时保留基本细节的能力。我们在 MNIST 和 USPS 数据集上评估了我们提出的用于量子 VAE (Q-VAE) 的下采样滤波器，并将其与经典 VAE 和称为经典直接传递 VAE (CDP-VAE) 的变体进行了比较，后者在编码过程中使用窗口池化滤波器。使用诸如 Fréchet 初始距离 (FID) 和均方误差 (MSE) 等指标来评估性能，这些指标衡量重建图像的保真度。我们的结果表明，Q-VAE 的表现始终优于经典 VAE 和 CDP-VAE，实现了明显更低的 FID 和 MSE 分数。此外，CDP-VAE 的性能优于 C-VAE。这些发现凸显了量子增强 VAE 通过提高分辨率和保留基本特征来改善图像重建质量的潜力，为未来在计算机视觉和合成数据生成领域的应用提供了一个有希望的方向。

Title: MEt3R: Measuring Multi-View Consistency in Generated Images

• Authors: Mohammad Asim, Christopher Wewer, Thomas Wimmer, Bernt Schiele, Jan Eric Lenssen
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.06336
• Pdf URL: https://arxiv.org/pdf/2501.06336
• Copy Paste: [[2501.06336]] MEt3R: Measuring Multi-View Consistency in Generated Images(https://arxiv.org/abs/2501.06336)
• Keywords: generation, generative
• Abstract: We introduce MEt3R, a metric for multi-view consistency in generated images. Large-scale generative models for multi-view image generation are rapidly advancing the field of 3D inference from sparse observations. However, due to the nature of generative modeling, traditional reconstruction metrics are not suitable to measure the quality of generated outputs and metrics that are independent of the sampling procedure are desperately needed. In this work, we specifically address the aspect of consistency between generated multi-view images, which can be evaluated independently of the specific scene. Our approach uses DUSt3R to obtain dense 3D reconstructions from image pairs in a feed-forward manner, which are used to warp image contents from one view into the other. Then, feature maps of these images are compared to obtain a similarity score that is invariant to view-dependent effects. Using MEt3R, we evaluate the consistency of a large set of previous methods for novel view and video generation, including our open, multi-view latent diffusion model.
• 摘要：我们引入了 MEt3R，这是生成图像中多视图一致性的度量。用于多视图图像生成的大规模生成模型正在迅速推动从稀疏观测中进行 3D 推理的领域。然而，由于生成模型的性质，传统的重建指标不适合衡量生成输出的质量，迫切需要独立于采样过程的指标。在这项工作中，我们专门解决了生成的多视图图像之间的一致性问题，可以独立于特定场景进行评估。我们的方法使用 DUSt3R 以前馈方式从图像对中获得密集的 3D 重建，这些重建用于将图像内容从一个视图扭曲到另一个视图。然后，比较这些图像的特征图以获得与视图相关效应不变的相似度得分。使用 MEt3R，我们评估了大量先前用于新视图和视频生成的方法的一致性，包括我们的开放多视图潜在扩散模型。

Title: Has an AI model been trained on your images?

• Authors: Matyas Bohacek, Hany Farid
• Subjects: cs.CV, cs.AI, cs.CR, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06399
• Pdf URL: https://arxiv.org/pdf/2501.06399
• Copy Paste: [[2501.06399]] Has an AI model been trained on your images?(https://arxiv.org/abs/2501.06399)
• Keywords: generative
• Abstract: From a simple text prompt, generative-AI image models can create stunningly realistic and creative images bounded, it seems, by only our imagination. These models have achieved this remarkable feat thanks, in part, to the ingestion of billions of images collected from nearly every corner of the internet. Many creators have understandably expressed concern over how their intellectual property has been ingested without their permission or a mechanism to opt out of training. As a result, questions of fair use and copyright infringement have quickly emerged. We describe a method that allows us to determine if a model was trained on a specific image or set of images. This method is computationally efficient and assumes no explicit knowledge of the model architecture or weights (so-called black-box membership inference). We anticipate that this method will be crucial for auditing existing models and, looking ahead, ensuring the fairer development and deployment of generative AI models.
• 摘要：通过简单的文本提示，生成式 AI 图像模型可以创建出令人惊叹的逼真且富有创意的图像，而这些图像似乎只受我们的想象力的限制。这些模型之所以取得如此非凡的成就，部分原因在于它们吸收了从互联网几乎每个角落收集的数十亿张图像。许多创作者都表达了担忧，担心他们的知识产权在未经他们许可或没有选择退出训练的机制的情况下被吸收。因此，合理使用和版权侵权问题迅速浮出水面。我们描述了一种方法，可以让我们确定模型是否是在特定图像或图像集上进行训练的。该方法具有计算效率，并且不需要明确了解模型架构或权重（所谓的黑盒成员推理）。我们预计，这种方法对于审核现有模型以及展望未来确保更公平地开发和部署生成式 AI 模型至关重要。

Title: Task Delay and Energy Consumption Minimization for Low-altitude MEC via Evolutionary Multi-objective Deep Reinforcement Learning

• Authors: Geng Sun, Weilong Ma, Jiahui Li, Zemin Sun, Jiacheng Wang, Dusit Niyato, Shiwen Mao
• Subjects: cs.LG, cs.NE, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.06410
• Pdf URL: https://arxiv.org/pdf/2501.06410
• Copy Paste: [[2501.06410]] Task Delay and Energy Consumption Minimization for Low-altitude MEC via Evolutionary Multi-objective Deep Reinforcement Learning(https://arxiv.org/abs/2501.06410)
• Keywords: generation
• Abstract: The low-altitude economy (LAE), driven by unmanned aerial vehicles (UAVs) and other aircraft, has revolutionized fields such as transportation, agriculture, and environmental monitoring. In the upcoming six-generation (6G) era, UAV-assisted mobile edge computing (MEC) is particularly crucial in challenging environments such as mountainous or disaster-stricken areas. The computation task offloading problem is one of the key issues in UAV-assisted MEC, primarily addressing the trade-off between minimizing the task delay and the energy consumption of the UAV. In this paper, we consider a UAV-assisted MEC system where the UAV carries the edge servers to facilitate task offloading for ground devices (GDs), and formulate a calculation delay and energy consumption multi-objective optimization problem (CDECMOP) to simultaneously improve the performance and reduce the cost of the system. Then, by modeling the formulated problem as a multi-objective Markov decision process (MOMDP), we propose a multi-objective deep reinforcement learning (DRL) algorithm within an evolutionary framework to dynamically adjust the weights and obtain non-dominated policies. Moreover, to ensure stable convergence and improve performance, we incorporate a target distribution learning (TDL) algorithm. Simulation results demonstrate that the proposed algorithm can better balance multiple optimization objectives and obtain superior non-dominated solutions compared to other methods.
• 摘要：由无人机 (UAV) 和其他飞机驱动的低空经济 (LAE) 彻底改变了交通、农业和环境监测等领域。在即将到来的第六代 (6G) 时代，无人机辅助移动边缘计算 (MEC) 在山区或灾区等具有挑战性的环境中尤为重要。计算任务卸载问题是无人机辅助 MEC 中的关键问题之一，主要解决最小化任务延迟和无人机能耗之间的权衡。在本文中，我们考虑了一种无人机辅助 MEC 系统，其中无人机携带边缘服务器以方便地面设备 (GD) 的任务卸载，并制定了计算延迟和能耗多目标优化问题 (CDECMOP) 以同时提高性能并降低系统成本。然后，通过将公式化问题建模为多目标马尔可夫决策过程 (MOMDP)，我们提出了一种进化框架内的多目标深度强化学习 (DRL) 算法，以动态调整权重并获得非支配策略。此外，为了确保稳定收敛并提高性能，我们结合了目标分布学习 (TDL) 算法。仿真结果表明，与其他方法相比，所提出的算法可以更好地平衡多个优化目标并获得更好的非支配解决方案。

Title: Qffusion: Controllable Portrait Video Editing via Quadrant-Grid Attention Learning

• Authors: Maomao Li, Lijian Lin, Yunfei Liu, Ye Zhu, Yu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06438
• Pdf URL: https://arxiv.org/pdf/2501.06438
• Copy Paste: [[2501.06438]] Qffusion: Controllable Portrait Video Editing via Quadrant-Grid Attention Learning(https://arxiv.org/abs/2501.06438)
• Keywords: generation, generative
• Abstract: This paper presents Qffusion, a dual-frame-guided framework for portrait video editing. Specifically, we consider a design principle of ``animation for editing'', and train Qffusion as a general animation framework from two still reference images while we can use it for portrait video editing easily by applying modified start and end frames as references during inference. Leveraging the powerful generative power of Stable Diffusion, we propose a Quadrant-grid Arrangement (QGA) scheme for latent re-arrangement, which arranges the latent codes of two reference images and that of four facial conditions into a four-grid fashion, separately. Then, we fuse features of these two modalities and use self-attention for both appearance and temporal learning, where representations at different times are jointly modeled under QGA. Our Qffusion can achieve stable video editing without additional networks or complex training stages, where only the input format of Stable Diffusion is modified. Further, we propose a Quadrant-grid Propagation (QGP) inference strategy, which enjoys a unique advantage on stable arbitrary-length video generation by processing reference and condition frames recursively. Through extensive experiments, Qffusion consistently outperforms state-of-the-art techniques on portrait video editing.
• 摘要：本文介绍了一种用于肖像视频编辑的双帧引导框架 Qffusion。具体来说，我们考虑了“用于编辑的动画”的设计原则，并从两个静态参考图像训练 Qffusion 作为通用动画框架，同时我们可以通过在推理过程中应用修改后的开始和结束帧作为参考，轻松地将其用于肖像视频编辑。利用稳定扩散的强大生成能力，我们提出了一种用于潜在重新排列的象限网格排列 (QGA) 方案，该方案分别将两个参考图像和四个面部条件的潜在代码排列成四网格方式。然后，我们融合这两种模态的特征，并使用自注意力进行外观和时间学习，其中不同时间的表示在 QGA 下联合建模。我们的 Qffusion 无需额外的网络或复杂的训练阶段即可实现稳定的视频编辑，其中仅修改了稳定扩散的输入格式。此外，我们提出了一种象限网格传播 (QGP) 推理策略，该策略通过递归处理参考帧和条件帧，在稳定生成任意长度的视频方面具有独特优势。通过大量实验，Qffusion 在肖像视频编辑方面的表现始终优于最先进的技术。

Title: Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation

• Authors: Xiaoying Xing, Avinab Saha, Junfeng He, Susan Hao, Paul Vicol, Moonkyung Ryu, Gang Li, Sahil Singla, Sarah Young, Yinxiao Li, Feng Yang, Deepak Ramachandran
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06481
• Pdf URL: https://arxiv.org/pdf/2501.06481
• Copy Paste: [[2501.06481]] Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation(https://arxiv.org/abs/2501.06481)
• Keywords: generation
• Abstract: Text-to-image (T2I) generation has made significant advances in recent years, but challenges still remain in the generation of perceptual artifacts, misalignment with complex prompts, and safety. The prevailing approach to address these issues involves collecting human feedback on generated images, training reward models to estimate human feedback, and then fine-tuning T2I models based on the reward models to align them with human preferences. However, while existing reward fine-tuning methods can produce images with higher rewards, they may change model behavior in unexpected ways. For example, fine-tuning for one quality aspect (e.g., safety) may degrade other aspects (e.g., prompt alignment), or may lead to reward hacking (e.g., finding a way to increase rewards without having the intended effect). In this paper, we propose Focus-N-Fix, a region-aware fine-tuning method that trains models to correct only previously problematic image regions. The resulting fine-tuned model generates images with the same high-level structure as the original model but shows significant improvements in regions where the original model was deficient in safety (over-sexualization and violence), plausibility, or other criteria. Our experiments demonstrate that Focus-N-Fix improves these localized quality aspects with little or no degradation to others and typically imperceptible changes in the rest of the image. Disclaimer: This paper contains images that may be overly sexual, violent, offensive, or harmful.
• 摘要：近年来，文本到图像 (T2I) 生成取得了重大进展，但在感知伪影生成、与复杂提示不一致以及安全性方面仍然存在挑战。解决这些问题的主流方法包括收集人类对生成图像的反馈，训练奖励模型以估计人类反馈，然后根据奖励模型对 T2I 模型进行微调，使其与人类偏好保持一致。然而，虽然现有的奖励微调方法可以生成具有更高奖励的图像，但它们可能会以意想不到的方式改变模型行为。例如，针对一个质量方面（例如安全性）的微调可能会降低其他方面（例如提示对齐）的质量，或者可能导致奖励黑客攻击（例如，找到一种增加奖励的方法而没有达到预期的效果）。在本文中，我们提出了 Focus-N-Fix，这是一种区域感知微调方法，可训练模型仅纠正以前有问题的图像区域。经过微调的模型生成的图像具有与原始模型相同的高级结构，但在原始模型缺乏安全性（过度色情和暴力）、可信度或其他标准的区域显示出显著的改进。我们的实验表明，Focus-N-Fix 改善了这些局部质量方面，而对其他方面几乎没有影响，并且通常对图像的其余部分的影响微乎其微。免责声明：本文包含的图像可能过于色情、暴力、冒犯或有害。

Title: NVS-SQA: Exploring Self-Supervised Quality Representation Learning for Neurally Synthesized Scenes without References

• Authors: Qiang Qu, Yiran Shen, Xiaoming Chen, Yuk Ying Chung, Weidong Cai, Tongliang Liu
• Subjects: cs.CV, cs.AI, cs.HC, cs.MM, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.06488
• Pdf URL: https://arxiv.org/pdf/2501.06488
• Copy Paste: [[2501.06488]] NVS-SQA: Exploring Self-Supervised Quality Representation Learning for Neurally Synthesized Scenes without References(https://arxiv.org/abs/2501.06488)
• Keywords: quality assessment
• Abstract: Neural View Synthesis (NVS), such as NeRF and 3D Gaussian Splatting, effectively creates photorealistic scenes from sparse viewpoints, typically evaluated by quality assessment methods like PSNR, SSIM, and LPIPS. However, these full-reference methods, which compare synthesized views to reference views, may not fully capture the perceptual quality of neurally synthesized scenes (NSS), particularly due to the limited availability of dense reference views. Furthermore, the challenges in acquiring human perceptual labels hinder the creation of extensive labeled datasets, risking model overfitting and reduced generalizability. To address these issues, we propose NVS-SQA, a NSS quality assessment method to learn no-reference quality representations through self-supervision without reliance on human labels. Traditional self-supervised learning predominantly relies on the "same instance, similar representation" assumption and extensive datasets. However, given that these conditions do not apply in NSS quality assessment, we employ heuristic cues and quality scores as learning objectives, along with a specialized contrastive pair preparation process to improve the effectiveness and efficiency of learning. The results show that NVS-SQA outperforms 17 no-reference methods by a large margin (i.e., on average 109.5% in SRCC, 98.6% in PLCC, and 91.5% in KRCC over the second best) and even exceeds 16 full-reference methods across all evaluation metrics (i.e., 22.9% in SRCC, 19.1% in PLCC, and 18.6% in KRCC over the second best).
• 摘要：神经视图合成 (NVS)，例如 NeRF 和 3D Gaussian Splatting，可有效地从稀疏视点创建逼真的场景，通常通过 PSNR、SSIM 和 LPIPS 等质量评估方法进行评估。然而，这些将合成视图与参考视图进行比较的全参考方法可能无法完全捕捉神经合成场景 (NSS) 的感知质量，特别是由于密集参考视图的可用性有限。此外，获取人类感知标签的挑战阻碍了创建大量标记数据集，存在模型过度拟合和通用性降低的风险。为了解决这些问题，我们提出了 NVS-SQA，这是一种 NSS 质量评估方法，可通过自我监督学习无参考质量表示，而无需依赖人类标签。传统的自监督学习主要依赖于“相同实例，相似表示”假设和大量数据集。然而，鉴于这些条件不适用于 NSS 质量评估，我们使用启发式线索和质量分数作为学习目标，并采用专门的对比对准备过程来提高学习的有效性和效率。结果表明，NVS-SQA 的表现远胜于 17 种无参考方法（即，SRCC 平均高出第二名 109.5%，PLCC 高出第二名 98.6%，KRCC 高出第二名 91.5%），甚至在所有评估指标上都超过了 16 种全参考方法（即，SRCC 高出第二名 22.9%，PLCC 高出第二名 19.1%，KRCC 高出第二名 18.6%）。

Title: DivTrackee versus DynTracker: Promoting Diversity in Anti-Facial Recognition against Dynamic FR Strategy

• Authors: Wenshu Fan, Minxing Zhang, Hongwei Li, Wenbo Jiang, Hanxiao Chen, Xiangyu Yue, Michael Backes, Xiao Zhang
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.06533
• Pdf URL: https://arxiv.org/pdf/2501.06533
• Copy Paste: [[2501.06533]] DivTrackee versus DynTracker: Promoting Diversity in Anti-Facial Recognition against Dynamic FR Strategy(https://arxiv.org/abs/2501.06533)
• Keywords: generation
• Abstract: The widespread adoption of facial recognition (FR) models raises serious concerns about their potential misuse, motivating the development of anti-facial recognition (AFR) to protect user facial privacy. In this paper, we argue that the static FR strategy, predominantly adopted in prior literature for evaluating AFR efficacy, cannot faithfully characterize the actual capabilities of determined trackers who aim to track a specific target identity. In particular, we introduce \emph{\ourAttack}, a dynamic FR strategy where the model's gallery database is iteratively updated with newly recognized target identity images. Surprisingly, such a simple approach renders all the existing AFR protections ineffective. To mitigate the privacy threats posed by DynTracker, we advocate for explicitly promoting diversity in the AFR-protected images. We hypothesize that the lack of diversity is the primary cause of the failure of existing AFR methods. Specifically, we develop \emph{DivTrackee}, a novel method for crafting diverse AFR protections that builds upon a text-guided image generation framework and diversity-promoting adversarial losses. Through comprehensive experiments on various facial image benchmarks and feature extractors, we demonstrate DynTracker's strength in breaking existing AFR methods and the superiority of DivTrackee in preventing user facial images from being identified by dynamic FR strategies. We believe our work can act as an important initial step towards developing more effective AFR methods for protecting user facial privacy against determined trackers.
• 摘要：面部识别 (FR) 模型的广泛采用引发了人们对其潜在滥用的严重担忧，从而推动了反面部识别 (AFR) 的开发，以保护用户的面部隐私。在本文中，我们认为，先前文献中主要采用的静态 FR 策略用于评估 AFR 功效，无法忠实地描述旨在跟踪特定目标身份的确定跟踪器的实际能力。特别是，我们引入了 \emph{\ourAttack}，这是一种动态 FR 策略，其中模型的图库数据库会使用新识别的目标身份图像进行迭代更新。令人惊讶的是，这种简单的方法使所有现有的 AFR 保护都无效。为了减轻 DynTracker 带来的隐私威胁，我们主张明确促进受 AFR 保护的图像的多样性。我们假设缺乏多样性是现有 AFR 方法失败的主要原因。具体来说，我们开发了 \emph{DivTrackee}，这是一种基于文本引导图像生成框架和促进多样性的对抗性损失来制定多样化 AFR 保护的新方法。通过对各种面部图像基准和特征提取器进行全面实验，我们展示了 DynTracker 在打破现有 AFR 方法方面的实力以及 DivTrackee 在防止用户面部图像被动态 FR 策略识别方面的优势。我们相信我们的工作可以作为开发更有效的 AFR 方法以保护用户面部隐私免受确定的跟踪者侵害的重要第一步。

Title: EmoXpt: Analyzing Emotional Variances in Human Comments and LLM-Generated Responses

• Authors: Shireesh Reddy Pyreddy, Tarannum Shaila Zaman
• Subjects: cs.LG, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.06597
• Pdf URL: https://arxiv.org/pdf/2501.06597
• Copy Paste: [[2501.06597]] EmoXpt: Analyzing Emotional Variances in Human Comments and LLM-Generated Responses(https://arxiv.org/abs/2501.06597)
• Keywords: generative
• Abstract: The widespread adoption of generative AI has generated diverse opinions, with individuals expressing both support and criticism of its applications. This study investigates the emotional dynamics surrounding generative AI by analyzing human tweets referencing terms such as ChatGPT, OpenAI, Copilot, and LLMs. To further understand the emotional intelligence of ChatGPT, we examine its responses to selected tweets, highlighting differences in sentiment between human comments and LLM-generated responses. We introduce EmoXpt, a sentiment analysis framework designed to assess both human perspectives on generative AI and the sentiment embedded in ChatGPT's responses. Unlike prior studies that focus exclusively on human sentiment, EmoXpt uniquely evaluates the emotional expression of ChatGPT. Experimental results demonstrate that LLM-generated responses are notably more efficient, cohesive, and consistently positive than human responses.
• 摘要：生成式人工智能的广泛应用引发了不同的意见，人们对其应用既有支持也有批评。本研究通过分析人类推文中引用 ChatGPT、OpenAI、Copilot 和 LLM 等术语的推文，调查了围绕生成式人工智能的情感动态。为了进一步了解 ChatGPT 的情商，我们研究了它对选定推文的回复，突出了人类评论和 LLM 生成的回复之间的情感差异。我们推出了 EmoXpt，这是一个情绪分析框架，旨在评估人类对生成式人工智能的看法以及 ChatGPT 回复中蕴含的情绪。与之前只关注人类情绪的研究不同，EmoXpt 以独特的方式评估了 ChatGPT 的情感表达。实验结果表明，LLM 生成的回复明显比人类回复更高效、更有凝聚力、更积极。

Title: Personalized Preference Fine-tuning of Diffusion Models

• Authors: Meihua Dang, Anikait Singh, Linqi Zhou, Stefano Ermon, Jiaming Song
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06655
• Pdf URL: https://arxiv.org/pdf/2501.06655
• Copy Paste: [[2501.06655]] Personalized Preference Fine-tuning of Diffusion Models(https://arxiv.org/abs/2501.06655)
• Keywords: generation
• Abstract: RLHF techniques like DPO can significantly improve the generation quality of text-to-image diffusion models. However, these methods optimize for a single reward that aligns model generation with population-level preferences, neglecting the nuances of individual users' beliefs or values. This lack of personalization limits the efficacy of these models. To bridge this gap, we introduce PPD, a multi-reward optimization objective that aligns diffusion models with personalized preferences. With PPD, a diffusion model learns the individual preferences of a population of users in a few-shot way, enabling generalization to unseen users. Specifically, our approach (1) leverages a vision-language model (VLM) to extract personal preference embeddings from a small set of pairwise preference examples, and then (2) incorporates the embeddings into diffusion models through cross attention. Conditioning on user embeddings, the text-to-image models are fine-tuned with the DPO objective, simultaneously optimizing for alignment with the preferences of multiple users. Empirical results demonstrate that our method effectively optimizes for multiple reward functions and can interpolate between them during inference. In real-world user scenarios, with as few as four preference examples from a new user, our approach achieves an average win rate of 76\% over Stable Cascade, generating images that more accurately reflect specific user preferences.
• 摘要：RLHF 技术（如 DPO）可以显著提高文本到图像传播模型的生成质量。然而，这些方法针对单一奖励进行优化，将模型生成与人口层面的偏好保持一致，而忽略了个人用户的信念或价值观的细微差别。这种个性化的缺乏限制了这些模型的有效性。为了弥补这一差距，我们引入了 PPD，这是一种多奖励优化目标，可将传播模型与个性化偏好保持一致。借助 PPD，传播模型可以以少量样本的方式学习用户群体的个人偏好，从而能够推广到未见过的用户。具体来说，我们的方法 (1) 利用视觉语言模型 (VLM) 从一小组成对偏好示例中提取个人偏好嵌入，然后 (2) 通过交叉注意将嵌入合并到传播模型中。以用户嵌入为条件，文本到图像模型使用 DPO 目标进行微调，同时优化以与多个用户的偏好保持一致。实证结果表明，我们的方法可以有效优化多个奖励函数，并可以在推理过程中对它们进行插值。在现实世界的用户场景中，只需从新用户那里获得四个偏好示例，我们的方法就能比 Stable Cascade 实现 76% 的平均胜率，从而生成更准确地反映特定用户偏好的图像。

Title: Challenging reaction prediction models to generalize to novel chemistry

• Authors: John Bradshaw, Anji Zhang, Babak Mahjour, David E. Graff, Marwin H.S. Segler, Connor W. Coley
• Subjects: cs.LG, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2501.06669
• Pdf URL: https://arxiv.org/pdf/2501.06669
• Copy Paste: [[2501.06669]] Challenging reaction prediction models to generalize to novel chemistry(https://arxiv.org/abs/2501.06669)
• Keywords: generation
• Abstract: Deep learning models for anticipating the products of organic reactions have found many use cases, including validating retrosynthetic pathways and constraining synthesis-based molecular design tools. Despite compelling performance on popular benchmark tasks, strange and erroneous predictions sometimes ensue when using these models in practice. The core issue is that common benchmarks test models in an in-distribution setting, whereas many real-world uses for these models are in out-of-distribution settings and require a greater degree of extrapolation. To better understand how current reaction predictors work in out-of-distribution domains, we report a series of more challenging evaluations of a prototypical SMILES-based deep learning model. First, we illustrate how performance on randomly sampled datasets is overly optimistic compared to performance when generalizing to new patents or new authors. Second, we conduct time splits that evaluate how models perform when tested on reactions published in years after those in their training set, mimicking real-world deployment. Finally, we consider extrapolation across reaction classes to reflect what would be required for the discovery of novel reaction types. This panel of tasks can reveal the capabilities and limitations of today's reaction predictors, acting as a crucial first step in the development of tomorrow's next-generation models capable of reaction discovery.
• 摘要：用于预测有机反应产物的深度学习模型已发现了许多用例，包括验证逆合成途径和约束基于合成的分子设计工具。尽管在流行的基准测试任务上表现出色，但在实践中使用这些模型时有时会出现奇怪和错误的预测。核心问题是常见的基准测试模型是在分布内设置中进行的，而这些模型的许多实际用途是在分布外设置中进行的，需要更大程度的外推。为了更好地了解当前反应预测器在分布外域中的工作方式，我们报告了一系列更具挑战性的基于 SMILES 的原型深度学习模型的评估。首先，我们说明了随机抽样数据集上的表现与推广到新专利或新作者时的表现相比过于乐观。其次，我们进行时间分割，以评估模型在测试训练集中发表的反应时的表现，这些反应发表在训练集发表的几年后，模拟现实世界的部署。最后，我们考虑跨反应类别进行外推，以反映发现新反应类型所需的条件。这组任务可以揭示当今反应预测器的能力和局限性，作为开发未来能够发现反应的下一代模型的关键的第一步。

Title: PGP-SAM: Prototype-Guided Prompt Learning for Efficient Few-Shot Medical Image Segmentation

• Authors: Zhonghao Yan, Zijin Yin, Tianyu Lin, Xiangzhu Zeng, Kongming Liang, Zhanyu Ma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06692
• Pdf URL: https://arxiv.org/pdf/2501.06692
• Copy Paste: [[2501.06692]] PGP-SAM: Prototype-Guided Prompt Learning for Efficient Few-Shot Medical Image Segmentation(https://arxiv.org/abs/2501.06692)
• Keywords: generation
• Abstract: The Segment Anything Model (SAM) has demonstrated strong and versatile segmentation capabilities, along with intuitive prompt-based interactions. However, customizing SAM for medical image segmentation requires massive amounts of pixel-level annotations and precise point- or box-based prompt designs. To address these challenges, we introduce PGP-SAM, a novel prototype-based few-shot tuning approach that uses limited samples to replace tedious manual prompts. Our key idea is to leverage inter- and intra-class prototypes to capture class-specific knowledge and relationships. We propose two main components: (1) a plug-and-play contextual modulation module that integrates multi-scale information, and (2) a class-guided cross-attention mechanism that fuses prototypes and features for automatic prompt generation. Experiments on a public multi-organ dataset and a private ventricle dataset demonstrate that PGP-SAM achieves superior mean Dice scores compared with existing prompt-free SAM variants, while using only 10\% of the 2D slices.
• 摘要：任何分割模型 (SAM) 都表现出强大而多功能的分割功能，以及直观的基于提示的交互。然而，定制 SAM 用于医学图像分割需要大量像素级注释和精确的基于点或框的提示设计。为了应对这些挑战，我们引入了 PGP-SAM，这是一种新颖的基于原型的少量调整方法，它使用有限的样本来取代繁琐的手动提示。我们的主要思想是利用类间和类内原型来捕获特定于类的知识和关系。我们提出了两个主要组件：(1) 集成多尺度信息的即插即用上下文调制模块，以及 (2) 融合原型和特征以自动生成提示的类引导交叉注意机制。在公共多器官数据集和私人心室数据集上的实验表明，与现有的无提示 SAM 变体相比，PGP-SAM 实现了更高的平均 Dice 分数，同时仅使用 10\% 的 2D 切片。

Title: F3D-Gaus: Feed-forward 3D-aware Generation on ImageNet with Cycle-Consistent Gaussian Splatting

• Authors: Yuxin Wang, Qianyi Wu, Dan Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06714
• Pdf URL: https://arxiv.org/pdf/2501.06714
• Copy Paste: [[2501.06714]] F3D-Gaus: Feed-forward 3D-aware Generation on ImageNet with Cycle-Consistent Gaussian Splatting(https://arxiv.org/abs/2501.06714)
• Keywords: generation
• Abstract: This paper tackles the problem of generalizable 3D-aware generation from monocular datasets, e.g., ImageNet. The key challenge of this task is learning a robust 3D-aware representation without multi-view or dynamic data, while ensuring consistent texture and geometry across different viewpoints. Although some baseline methods are capable of 3D-aware generation, the quality of the generated images still lags behind state-of-the-art 2D generation approaches, which excel in producing high-quality, detailed images. To address this severe limitation, we propose a novel feed-forward pipeline based on pixel-aligned Gaussian Splatting, coined as F3D-Gaus, which can produce more realistic and reliable 3D renderings from monocular inputs. In addition, we introduce a self-supervised cycle-consistent constraint to enforce cross-view consistency in the learned 3D representation. This training strategy naturally allows aggregation of multiple aligned Gaussian primitives and significantly alleviates the interpolation limitations inherent in single-view pixel-aligned Gaussian Splatting. Furthermore, we incorporate video model priors to perform geometry-aware refinement, enhancing the generation of fine details in wide-viewpoint scenarios and improving the model's capability to capture intricate 3D textures. Extensive experiments demonstrate that our approach not only achieves high-quality, multi-view consistent 3D-aware generation from monocular datasets, but also significantly improves training and inference efficiency.
• 摘要：本文解决了从单目数据集（例如 ImageNet）进行可推广的 3D 感知生成的问题。这项任务的关键挑战是在没有多视图或动态数据的情况下学习稳健的 3D 感知表示，同时确保不同视点之间的纹理和几何形状一致。尽管一些基线方法能够实现 3D 感知生成，但生成的图像质量仍然落后于最先进的 2D 生成方法，后者擅长生成高质量、详细的图像。为了解决这一严重限制，我们提出了一种基于像素对齐高斯 Splatting 的新型前馈管道，称为 F3D-Gaus，它可以从单目输入生成更逼真、更可靠的 3D 渲染。此外，我们引入了自监督循环一致约束，以在学习的 3D 表示中强制实现跨视图一致性。这种训练策略自然允许聚合多个对齐的高斯基元，并显著缓解单视图像素对齐高斯 Splatting 固有的插值限制。此外，我们结合视频模型先验知识进行几何感知细化，增强了广视角场景中精细细节的生成，并提高了模型捕捉复杂 3D 纹理的能力。大量实验表明，我们的方法不仅可以从单目数据集实现高质量、多视角一致的 3D 感知生成，还可以显著提高训练和推理效率。

Title: DRDT3: Diffusion-Refined Decision Test-Time Training Model

• Authors: Xingshuai Huang, Di Wu, Benoit Boulet
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06718
• Pdf URL: https://arxiv.org/pdf/2501.06718
• Copy Paste: [[2501.06718]] DRDT3: Diffusion-Refined Decision Test-Time Training Model(https://arxiv.org/abs/2501.06718)
• Keywords: generation, generative
• Abstract: Decision Transformer (DT), a trajectory modeling method, has shown competitive performance compared to traditional offline reinforcement learning (RL) approaches on various classic control tasks. However, it struggles to learn optimal policies from suboptimal, reward-labeled trajectories. In this study, we explore the use of conditional generative modeling to facilitate trajectory stitching given its high-quality data generation ability. Additionally, recent advancements in Recurrent Neural Networks (RNNs) have shown their linear complexity and competitive sequence modeling performance over Transformers. We leverage the Test-Time Training (TTT) layer, an RNN that updates hidden states during testing, to model trajectories in the form of DT. We introduce a unified framework, called Diffusion-Refined Decision TTT (DRDT3), to achieve performance beyond DT models. Specifically, we propose the Decision TTT (DT3) module, which harnesses the sequence modeling strengths of both self-attention and the TTT layer to capture recent contextual information and make coarse action predictions. We further integrate DT3 with the diffusion model using a unified optimization objective. With experiments on multiple tasks of Gym and AntMaze in the D4RL benchmark, our DT3 model without diffusion refinement demonstrates improved performance over standard DT, while DRDT3 further achieves superior results compared to state-of-the-art conventional offline RL and DT-based methods.
• 摘要：决策变换器 (DT) 是一种轨迹建模方法，与传统的离线强化学习 (RL) 方法相比，它在各种经典控制任务上表现出了极具竞争力的性能。然而，它很难从次优的奖励标记轨迹中学习最优策略。在本研究中，我们探索了使用条件生成模型来促进轨迹拼接，因为它具有高质量的数据生成能力。此外，循环神经网络 (RNN) 的最新进展表明，它们具有线性复杂性和优于 Transformers 的序列建模性能。我们利用测试时间训练 (TTT) 层（一种在测试期间更新隐藏状态的 RNN）以 DT 的形式建模轨迹。我们引入了一个统一的框架，称为扩散细化决策 TTT (DRDT3)，以实现超越 DT 模型的性能。具体来说，我们提出了决策 TTT (DT3) 模块，它利用自注意力和 TTT 层的序列建模优势来捕获最近的上下文信息并做出粗略的动作预测。我们进一步使用统一的优化目标将 DT3 与扩散模型集成。通过在 D4RL 基准中对 Gym 和 AntMaze 的多个任务进行实验，我们的无扩散细化的 DT3 模型比标准 DT 表现出更好的性能，而 DRDT3 与最先进的传统离线 RL 和基于 DT 的方法相比也取得了优异的结果。

Title: Multi-Label Scene Classification in Remote Sensing Benefits from Image Super-Resolution

• Authors: Ashitha Mudraje, Brian B. Moser, Stanislav Frolov, Andreas Dengel
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06720
• Pdf URL: https://arxiv.org/pdf/2501.06720
• Copy Paste: [[2501.06720]] Multi-Label Scene Classification in Remote Sensing Benefits from Image Super-Resolution(https://arxiv.org/abs/2501.06720)
• Keywords: super-resolution
• Abstract: Satellite imagery is a cornerstone for numerous Remote Sensing (RS) applications; however, limited spatial resolution frequently hinders the precision of such systems, especially in multi-label scene classification tasks as it requires a higher level of detail and feature differentiation. In this study, we explore the efficacy of image Super-Resolution (SR) as a pre-processing step to enhance the quality of satellite images and thus improve downstream classification performance. We investigate four SR models - SRResNet, HAT, SeeSR, and RealESRGAN - and evaluate their impact on multi-label scene classification across various CNN architectures, including ResNet-50, ResNet-101, ResNet-152, and Inception-v4. Our results show that applying SR significantly improves downstream classification performance across various metrics, demonstrating its ability to preserve spatial details critical for multi-label tasks. Overall, this work offers valuable insights into the selection of SR techniques for multi-label prediction in remote sensing and presents an easy-to-integrate framework to improve existing RS systems.
• 摘要：卫星图像是众多遥感 (RS) 应用的基石；然而，有限的空间分辨率经常会阻碍此类系统的精度，尤其是在多标签场景分类任务中，因为它需要更高级别的细节和特征区分。在本研究中，我们探索了图像超分辨率 (SR) 作为预处理步骤的有效性，以提高卫星图像的质量，从而提高下游分类性能。我们研究了四种 SR 模型 - SRResNet、HAT、SeeSR 和 RealESRGAN - 并评估了它们对各种 CNN 架构（包括 ResNet-50、ResNet-101、ResNet-152 和 Inception-v4）中多标签场景分类的影响。我们的结果表明，应用 SR 可显著提高各种指标的下游分类性能，表明其能够保留对多标签任务至关重要的空间细节。总体而言，这项工作为选择用于遥感多标签预测的 SR 技术提供了宝贵的见解，并提出了一个易于集成的框架来改进现有的 RS 系统。

Title: ODPG: Outfitting Diffusion with Pose Guided Condition

• Authors: Seohyun Lee, Jintae Park, Sanghyeok Park
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06769
• Pdf URL: https://arxiv.org/pdf/2501.06769
• Copy Paste: [[2501.06769]] ODPG: Outfitting Diffusion with Pose Guided Condition(https://arxiv.org/abs/2501.06769)
• Keywords: generative
• Abstract: Virtual Try-On (VTON) technology allows users to visualize how clothes would look on them without physically trying them on, gaining traction with the rise of digitalization and online shopping. Traditional VTON methods, often using Generative Adversarial Networks (GANs) and Diffusion models, face challenges in achieving high realism and handling dynamic poses. This paper introduces Outfitting Diffusion with Pose Guided Condition (ODPG), a novel approach that leverages a latent diffusion model with multiple conditioning inputs during the denoising process. By transforming garment, pose, and appearance images into latent features and integrating these features in a UNet-based denoising model, ODPG achieves non-explicit synthesis of garments on dynamically posed human images. Our experiments on the FashionTryOn and a subset of the DeepFashion dataset demonstrate that ODPG generates realistic VTON images with fine-grained texture details across various poses, utilizing an end-to-end architecture without the need for explicit garment warping processes. Future work will focus on generating VTON outputs in video format and on applying our attention mechanism, as detailed in the Method section, to other domains with limited data.
• 摘要：虚拟试穿 (VTON) 技术让用户无需实际试穿即可看到衣服穿在身上的效果，随着数字化和网上购物的兴起，该技术越来越受到关注。传统的 VTON 方法通常使用生成对抗网络 (GAN) 和扩散模型，在实现高真实感和处理动态姿势方面面临挑战。本文介绍了一种新颖的方法，即姿势引导条件的服装扩散 (ODPG)，它在去噪过程中利用具有多个条件输入的潜在扩散模型。通过将服装、姿势和外观图像转换为潜在特征，并将这些特征集成到基于 UNet 的去噪模型中，ODPG 可以在动态姿势的人体图像上实现服装的非显式合成。我们在 FashionTryOn 和 DeepFashion 数据集的一个子集上进行的实验表明，ODPG 利用端到端架构生成具有各种姿势的细粒度纹理细节的逼真的 VTON 图像，而无需显式的服装扭曲过程。未来的工作将集中于生成视频格式的 VTON 输出，并将我们的注意力机制（如方法部分所述）应用于数据有限的其他领域。

Title: SuperNeRF-GAN: A Universal 3D-Consistent Super-Resolution Framework for Efficient and Enhanced 3D-Aware Image Synthesis

• Authors: Peng Zheng, Linzhi Huang, Yizhou Yu, Yi Chang, Yilin Wang, Rui Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06770
• Pdf URL: https://arxiv.org/pdf/2501.06770
• Copy Paste: [[2501.06770]] SuperNeRF-GAN: A Universal 3D-Consistent Super-Resolution Framework for Efficient and Enhanced 3D-Aware Image Synthesis(https://arxiv.org/abs/2501.06770)
• Keywords: super-resolution, generation
• Abstract: Neural volume rendering techniques, such as NeRF, have revolutionized 3D-aware image synthesis by enabling the generation of images of a single scene or object from various camera poses. However, the high computational cost of NeRF presents challenges for synthesizing high-resolution (HR) images. Most existing methods address this issue by leveraging 2D super-resolution, which compromise 3D-consistency. Other methods propose radiance manifolds or two-stage generation to achieve 3D-consistent HR synthesis, yet they are limited to specific synthesis tasks, reducing their universality. To tackle these challenges, we propose SuperNeRF-GAN, a universal framework for 3D-consistent super-resolution. A key highlight of SuperNeRF-GAN is its seamless integration with NeRF-based 3D-aware image synthesis methods and it can simultaneously enhance the resolution of generated images while preserving 3D-consistency and reducing computational cost. Specifically, given a pre-trained generator capable of producing a NeRF representation such as tri-plane, we first perform volume rendering to obtain a low-resolution image with corresponding depth and normal map. Then, we employ a NeRF Super-Resolution module which learns a network to obtain a high-resolution NeRF. Next, we propose a novel Depth-Guided Rendering process which contains three simple yet effective steps, including the construction of a boundary-correct multi-depth map through depth aggregation, a normal-guided depth super-resolution and a depth-guided NeRF rendering. Experimental results demonstrate the superior efficiency, 3D-consistency, and quality of our approach. Additionally, ablation studies confirm the effectiveness of our proposed components.
• 摘要：神经体积渲染技术（例如 NeRF）通过实现从各种相机姿势生成单个场景或物体的图像，彻底改变了 3D 感知图像合成。然而，NeRF 的高计算成本给合成高分辨率 (HR) 图像带来了挑战。大多数现有方法通过利用 2D 超分辨率来解决此问题，但这会损害 3D 一致性。其他方法提出使用辐射流形或两阶段生成来实现 3D 一致的 HR 合成，但它们仅限于特定的合成任务，从而降低了其通用性。为了应对这些挑战，我们提出了 SuperNeRF-GAN，这是一个用于 3D 一致超分辨率的通用框架。SuperNeRF-GAN 的一个主要亮点是它与基于 NeRF 的 3D 感知图像合成方法的无缝集成，并且可以同时提高生成图像的分辨率，同时保持 3D 一致性并降低计算成本。具体来说，给定一个能够生成 NeRF 表示（例如三平面）的预训练生成器，我们首先执行体积渲染以获得具有相应深度和法线图的低分辨率图像。然后，我们使用 NeRF 超分辨率模块来学习网络以获得高分辨率 NeRF。接下来，我们提出了一种新颖的深度引导渲染过程，其中包含三个简单但有效的步骤，包括通过深度聚合构建边界正确的多深度图、法线引导的深度超分辨率和深度引导的 NeRF 渲染。实验结果证明了我们的方法具有卓越的效率、3D 一致性和质量。此外，消融研究证实了我们提出的组件的有效性。

Title: Pareto Set Learning for Multi-Objective Reinforcement Learning

• Authors: Erlong Liu, Yu-Chang Wu, Xiaobin Huang, Chengrui Gao, Ren-Jian Wang, Ke Xue, Chao Qian
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06773
• Pdf URL: https://arxiv.org/pdf/2501.06773
• Copy Paste: [[2501.06773]] Pareto Set Learning for Multi-Objective Reinforcement Learning(https://arxiv.org/abs/2501.06773)
• Keywords: generation
• Abstract: Multi-objective decision-making problems have emerged in numerous real-world scenarios, such as video games, navigation and robotics. Considering the clear advantages of Reinforcement Learning (RL) in optimizing decision-making processes, researchers have delved into the development of Multi-Objective RL (MORL) methods for solving multi-objective decision problems. However, previous methods either cannot obtain the entire Pareto front, or employ only a single policy network for all the preferences over multiple objectives, which may not produce personalized solutions for each preference. To address these limitations, we propose a novel decomposition-based framework for MORL, Pareto Set Learning for MORL (PSL-MORL), that harnesses the generation capability of hypernetwork to produce the parameters of the policy network for each decomposition weight, generating relatively distinct policies for various scalarized subproblems with high efficiency. PSL-MORL is a general framework, which is compatible for any RL algorithm. The theoretical result guarantees the superiority of the model capacity of PSL-MORL and the optimality of the obtained policy network. Through extensive experiments on diverse benchmarks, we demonstrate the effectiveness of PSL-MORL in achieving dense coverage of the Pareto front, significantly outperforming state-of-the-art MORL methods in the hypervolume and sparsity indicators.
• 摘要：多目标决策问题已出现在许多现实世界场景中，例如视频游戏、导航和机器人技术。考虑到强化学习 (RL) 在优化决策过程方面的明显优势，研究人员已深入研究开发用于解决多目标决策问题的多目标 RL (MORL) 方法。然而，以前的方法要么无法获得整个帕累托前沿，要么只对多个目标的所有偏好采用单一策略网络，这可能无法为每个偏好产生个性化的解决方案。为了解决这些限制，我们提出了一种基于分解的新型 MORL 框架，即用于 MORL 的帕累托集学习 (PSL-MORL)，该框架利用超网络的生成能力为每个分解权重生成策略网络的参数，从而高效地为各种标量子问题生成相对不同的策略。PSL-MORL 是一个通用框架，适用于任何 RL 算法。理论结果保证了 PSL-MORL 模型容量的优越性和获得的策略网络的最优性。通过在不同基准上进行大量实验，我们证明了 PSL-MORL 在实现帕累托前沿密集覆盖方面的有效性，在超体积和稀疏性指标方面明显优于最先进的 MORL 方法。

Title: GeoPix: Multi-Modal Large Language Model for Pixel-level Image Understanding in Remote Sensing

• Authors: Ruizhe Ou, Yuan Hu, Fan Zhang, Jiaxin Chen, Yu Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06828
• Pdf URL: https://arxiv.org/pdf/2501.06828
• Copy Paste: [[2501.06828]] GeoPix: Multi-Modal Large Language Model for Pixel-level Image Understanding in Remote Sensing(https://arxiv.org/abs/2501.06828)
• Keywords: generation
• Abstract: Multi-modal large language models (MLLMs) have achieved remarkable success in image- and region-level remote sensing (RS) image understanding tasks, such as image captioning, visual question answering, and visual grounding. However, existing RS MLLMs lack the pixel-level dialogue capability, which involves responding to user instructions with segmentation masks for specific instances. In this paper, we propose GeoPix, a RS MLLM that extends image understanding capabilities to the pixel level. This is achieved by equipping the MLLM with a mask predictor, which transforms visual features from the vision encoder into masks conditioned on the LLM's segmentation token embeddings. To facilitate the segmentation of multi-scale objects in RS imagery, a class-wise learnable memory module is integrated into the mask predictor to capture and store class-wise geo-context at the instance level across the entire dataset. In addition, to address the absence of large-scale datasets for training pixel-level RS MLLMs, we construct the GeoPixInstruct dataset, comprising 65,463 images and 140,412 instances, with each instance annotated with text descriptions, bounding boxes, and masks. Furthermore, we develop a two-stage training strategy to balance the distinct requirements of text generation and masks prediction in multi-modal multi-task optimization. Extensive experiments verify the effectiveness and superiority of GeoPix in pixel-level segmentation tasks, while also maintaining competitive performance in image- and region-level benchmarks.
• 摘要：多模态大型语言模型 (MLLM) 在图像和区域级遥感 (RS) 图像理解任务（例如图像字幕、视觉问答和视觉基础）中取得了显著成功。然而，现有的 RS MLLM 缺乏像素级对话能力，这涉及使用特定实例的分割掩码来响应用户指令。在本文中，我们提出了 GeoPix，一种将图像理解能力扩展到像素级的 RS MLLM。这是通过为 MLLM 配备掩码预测器来实现的，该预测器将来自视觉编码器的视觉特征转换为以 LLM 的分割标记嵌入为条件的掩码。为了促进 RS 图像中多尺度对象的分割，在掩码预测器中集成了一个类别可学习记忆模块，以捕获和存储整个数据集中实例级别的类别地理上下文。此外，为了解决缺乏用于训练像素级 RS MLLM 的大规模数据集的问题，我们构建了 GeoPixInstruct 数据集，其中包含 65,463 张图像和 140,412 个实例，每个实例都带有文本描述、边界框和蒙版注释。此外，我们开发了一种两阶段训练策略，以平衡多模态多任务优化中文本生成和蒙版预测的不同要求。大量实验验证了 GeoPix 在像素级分割任务中的有效性和优越性，同时在图像和区域级基准测试中保持了竞争性性能。

Title: Transfer Learning of Tabular Data by Finetuning Large Language Models

• Authors: Shourav B. Rabbani, Ibna Kowsar, Manar D. Samad
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06863
• Pdf URL: https://arxiv.org/pdf/2501.06863
• Copy Paste: [[2501.06863]] Transfer Learning of Tabular Data by Finetuning Large Language Models(https://arxiv.org/abs/2501.06863)
• Keywords: generative
• Abstract: Despite the artificial intelligence (AI) revolution, deep learning has yet to achieve much success with tabular data due to heterogeneous feature space and limited sample sizes without viable transfer learning. The new era of generative AI, powered by large language models (LLM), brings unprecedented learning opportunities to diverse data and domains. This paper investigates the effectiveness of an LLM application programming interface (API) and transfer learning of LLM in tabular data classification. LLM APIs respond to input text prompts with tokenized data and instructions, whereas transfer learning finetunes an LLM for a target classification task. This paper proposes an end-to-end finetuning of LLM to demonstrate cross-data transfer learning on ten benchmark data sets when large pre-trained tabular data models do not exist to facilitate transfer learning. The proposed LLM finetuning method outperforms state-of-the-art machine and deep learning methods on tabular data with less than ten features - a standard feature size for tabular data sets. The transfer learning approach uses a fraction of the computational cost of other deep learning or API-based solutions while ensuring competitive or superior classification performance.
• 摘要：尽管人工智能 (AI) 革命不断，但由于特征空间异构、样本量有限且缺乏可行的迁移学习，深度学习在表格数据方面尚未取得很大成功。由大型语言模型 (LLM) 驱动的生成式人工智能新时代为各种数据和领域带来了前所未有的学习机会。本文研究了 LLM 应用程序编程接口 (API) 和 LLM 迁移学习在表格数据分类中的有效性。LLM API 使用标记数据和指令响应输入文本提示，而迁移学习则针对目标分类任务对 LLM 进行微调。本文提出了一种端到端的 LLM 微调，以在不存在大型预训练表格数据模型来促进迁移学习的情况下，在十个基准数据集上展示跨数据迁移学习。所提出的 LLM 微调方法在具有少于十个特征（表格数据集的标准特征大小）的表格数据上的表现优于最先进的机器和深度学习方法。迁移学习方法仅使用其他深度学习或基于 API 的解决方案的计算成本的一小部分，同时确保具有竞争力或卓越的分类性能。

Title: Synthetic Prior for Few-Shot Drivable Head Avatar Inversion

• Authors: Wojciech Zielonka, Stephan J. Garbin, Alexandros Lattas, George Kopanas, Paulo Gotardo, Thabo Beeler, Justus Thies, Timo Bolkart
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06903
• Pdf URL: https://arxiv.org/pdf/2501.06903
• Copy Paste: [[2501.06903]] Synthetic Prior for Few-Shot Drivable Head Avatar Inversion(https://arxiv.org/abs/2501.06903)
• Keywords: generative
• Abstract: We present SynShot, a novel method for the few-shot inversion of a drivable head avatar based on a synthetic prior. We tackle two major challenges. First, training a controllable 3D generative network requires a large number of diverse sequences, for which pairs of images and high-quality tracked meshes are not always available. Second, state-of-the-art monocular avatar models struggle to generalize to new views and expressions, lacking a strong prior and often overfitting to a specific viewpoint distribution. Inspired by machine learning models trained solely on synthetic data, we propose a method that learns a prior model from a large dataset of synthetic heads with diverse identities, expressions, and viewpoints. With few input images, SynShot fine-tunes the pretrained synthetic prior to bridge the domain gap, modeling a photorealistic head avatar that generalizes to novel expressions and viewpoints. We model the head avatar using 3D Gaussian splatting and a convolutional encoder-decoder that outputs Gaussian parameters in UV texture space. To account for the different modeling complexities over parts of the head (e.g., skin vs hair), we embed the prior with explicit control for upsampling the number of per-part primitives. Compared to state-of-the-art monocular methods that require thousands of real training images, SynShot significantly improves novel view and expression synthesis.
• 摘要：我们提出了 SynShot，一种基于合成先验的可驾驶头部化身的少样本反演新方法。我们解决了两个主要挑战。首先，训练可控的 3D 生成网络需要大量不同的序列，而这些序列并不总是有图像对和高质量的跟踪网格。其次，最先进的单目化身模型难以推广到新的视角和表情，缺乏强大的先验，并且经常过度拟合特定的视点分布。受到仅基于合成数据训练的机器学习模型的启发，我们提出了一种方法，该方法从具有不同身份、表情和视点的大型合成头部数据集中学习先验模型。使用少量输入图像，SynShot 对预训练的合成先验进行微调以弥合领域差距，建模出一个可推广到新表情和视点的逼真的头部化身。我们使用 3D 高斯溅射和卷积编码器-解码器对头部头像进行建模，该编码器-解码器在 UV 纹理空间中输出高斯参数。为了解决头部各个部分（例如皮肤与头发）的建模复杂性差异，我们嵌入了先验，并明确控制对每个部分基元的数量进行上采样。与需要数千张真实训练图像的最先进的单目方法相比，SynShot 显著改善了新颖的视图和表情合成。

Title: Deep Learning and Foundation Models for Weather Prediction: A Survey

• Authors: Jimeng Shi, Azam Shirali, Bowen Jin, Sizhe Zhou, Wei Hu, Rahuul Rangaraj, Shaowen Wang, Jiawei Han, Zhaonan Wang, Upmanu Lall, Yanzhao Wu, Leonardo Bobadilla, Giri Narasimhan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06907
• Pdf URL: https://arxiv.org/pdf/2501.06907
• Copy Paste: [[2501.06907]] Deep Learning and Foundation Models for Weather Prediction: A Survey(https://arxiv.org/abs/2501.06907)
• Keywords: generative
• Abstract: Physics-based numerical models have been the bedrock of atmospheric sciences for decades, offering robust solutions but often at the cost of significant computational resources. Deep learning (DL) models have emerged as powerful tools in meteorology, capable of analyzing complex weather and climate data by learning intricate dependencies and providing rapid predictions once trained. While these models demonstrate promising performance in weather prediction, often surpassing traditional physics-based methods, they still face critical challenges. This paper presents a comprehensive survey of recent deep learning and foundation models for weather prediction. We propose a taxonomy to classify existing models based on their training paradigms: deterministic predictive learning, probabilistic generative learning, and pre-training and fine-tuning. For each paradigm, we delve into the underlying model architectures, address major challenges, offer key insights, and propose targeted directions for future research. Furthermore, we explore real-world applications of these methods and provide a curated summary of open-source code repositories and widely used datasets, aiming to bridge research advancements with practical implementations while fostering open and trustworthy scientific practices in adopting cutting-edge artificial intelligence for weather prediction. The related sources are available at this https URL DL-Foundation-Models-Weather.
• 摘要：几十年来，基于物理的数值模型一直是大气科学的基石，它提供了强大的解决方案，但通常需要耗费大量的计算资源。深度学习 (DL) 模型已成为气象学的强大工具，能够通过学习复杂的依赖关系来分析复杂的天气和气候数据，并在训练后提供快速预测。虽然这些模型在天气预报中表现出色，通常超越传统的基于物理的方法，但它们仍然面临着严峻的挑战。本文全面介绍了最近的深度学习和天气预报基础模型。我们提出了一种分类法，根据现有模型的训练范式对其进行分类：确定性预测学习、概率生成学习以及预训练和微调。对于每个范式，我们深入研究底层模型架构，解决主要挑战，提供关键见解，并提出未来研究的目标方向。此外，我们探索了这些方法的实际应用，并提供了开源代码存储库和广泛使用的数据集的精选摘要，旨在将研究进展与实际实施联系起来，同时促进采用尖端人工智能进行天气预报的开放和值得信赖的科学实践。相关资源可在此 https URL DL-Foundation-Models-Weather 上找到。

Title: Evaluating unsupervised contrastive learning framework for MRI sequences classification

• Authors: Yuli Wang, Kritika Iyer, Sep Farhand, Yoshihisa Shinagawa
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.06938
• Pdf URL: https://arxiv.org/pdf/2501.06938
• Copy Paste: [[2501.06938]] Evaluating unsupervised contrastive learning framework for MRI sequences classification(https://arxiv.org/abs/2501.06938)
• Keywords: generation
• Abstract: The automatic identification of Magnetic Resonance Imaging (MRI) sequences can streamline clinical workflows by reducing the time radiologists spend manually sorting and identifying sequences, thereby enabling faster diagnosis and treatment planning for patients. However, the lack of standardization in the parameters of MRI scans poses challenges for automated systems and complicates the generation and utilization of datasets for machine learning research. To address this issue, we propose a system for MRI sequence identification using an unsupervised contrastive deep learning framework. By training a convolutional neural network based on the ResNet-18 architecture, our system classifies nine common MRI sequence types as a 9-class classification problem. The network was trained using an in-house internal dataset and validated on several public datasets, including BraTS, ADNI, Fused Radiology-Pathology Prostate Dataset, the Breast Cancer Dataset (ACRIN), among others, encompassing diverse acquisition protocols and requiring only 2D slices for training. Our system achieves a classification accuracy of over 0.95 across the nine most common MRI sequence types.
• 摘要：自动识别磁共振成像 (MRI) 序列可以简化临床工作流程，减少放射科医生手动分类和识别序列的时间，从而更快地为患者制定诊断和治疗计划。然而，MRI 扫描参数缺乏标准化，这对自动化系统构成了挑战，并使机器学习研究数据集的生成和利用变得复杂。为了解决这个问题，我们提出了一种使用无监督对比深度学习框架进行 MRI 序列识别的系统。通过训练基于 ResNet-18 架构的卷积神经网络，我们的系统将九种常见的 MRI 序列类型归类为 9 类分类问题。该网络使用内部数据集进行训练，并在多个公共数据集上进行验证，包括 BraTS、ADNI、融合放射病理学前列腺数据集、乳腺癌数据集 (ACRIN) 等，涵盖了多种采集协议，并且只需要 2D 切片即可进行训练。我们的系统在九种最常见的 MRI 序列类型中实现了超过 0.95 的分类准确率。

Title: Comparison of Autoencoders for tokenization of ASL datasets

• Authors: Vouk Praun-Petrovic, Aadhvika Koundinya, Lavanya Prahallad
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06942
• Pdf URL: https://arxiv.org/pdf/2501.06942
• Copy Paste: [[2501.06942]] Comparison of Autoencoders for tokenization of ASL datasets(https://arxiv.org/abs/2501.06942)
• Keywords: generation, generative
• Abstract: Generative AI, powered by large language models (LLMs), has revolutionized applications across text, audio, images, and video. This study focuses on developing and evaluating encoder-decoder architectures for the American Sign Language (ASL) image dataset, consisting of 87,000 images across 29 hand sign classes. Three approaches were compared: Feedforward Autoencoders, Convolutional Autoencoders, and Diffusion Autoencoders. The Diffusion Autoencoder outperformed the others, achieving the lowest mean squared error (MSE) and highest Mean Opinion Score (MOS) due to its probabilistic noise modeling and iterative denoising capabilities. The Convolutional Autoencoder demonstrated effective spatial feature extraction but lacked the robustness of the diffusion process, while the Feedforward Autoencoder served as a baseline with limitations in handling complex image data. Objective and subjective evaluations confirmed the superiority of the Diffusion Autoencoder for high-fidelity image reconstruction, emphasizing its potential in multimodal AI applications such as sign language recognition and generation. This work provides critical insights into designing robust encoder-decoder systems to advance multimodal AI capabilities.
• 摘要：以大型语言模型 (LLM) 为驱动力的生成式 AI 彻底改变了文本、音频、图像和视频领域的应用。本研究重点开发和评估美国手语 (ASL) 图像数据集的编码器-解码器架构，该数据集包含 29 个手势类别的 87,000 张图像。比较了三种方法：前馈自动编码器、卷积自动编码器和扩散自动编码器。扩散自动编码器的表现优于其他自动编码器，由于其概率噪声建模和迭代去噪功能，实现了最低均方误差 (MSE) 和最高平均意见分数 (MOS)。卷积自动编码器展示了有效的空间特征提取，但缺乏扩散过程的稳健性，而前馈自动编码器作为基线，在处理复杂图像数据方面存在局限性。客观和主观评估证实了扩散自动编码器在高保真图像重建方面的优势，强调了其在手语识别和生成等多模态 AI 应用中的潜力。这项工作为设计强大的编码器-解码器系统以提升多模式人工智能能力提供了关键见解。

Title: Likelihood Training of Cascaded Diffusion Models via Hierarchical Volume-preserving Maps

• Authors: Henry Li, Ronen Basri, Yuval Kluger
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06999
• Pdf URL: https://arxiv.org/pdf/2501.06999
• Copy Paste: [[2501.06999]] Likelihood Training of Cascaded Diffusion Models via Hierarchical Volume-preserving Maps(https://arxiv.org/abs/2501.06999)
• Keywords: generative
• Abstract: Cascaded models are multi-scale generative models with a marked capacity for producing perceptually impressive samples at high resolutions. In this work, we show that they can also be excellent likelihood models, so long as we overcome a fundamental difficulty with probabilistic multi-scale models: the intractability of the likelihood function. Chiefly, in cascaded models each intermediary scale introduces extraneous variables that cannot be tractably marginalized out for likelihood evaluation. This issue vanishes by modeling the diffusion process on latent spaces induced by a class of transformations we call hierarchical volume-preserving maps, which decompose spatially structured data in a hierarchical fashion without introducing local distortions in the latent space. We demonstrate that two such maps are well-known in the literature for multiscale modeling: Laplacian pyramids and wavelet transforms. Not only do such reparameterizations allow the likelihood function to be directly expressed as a joint likelihood over the scales, we show that the Laplacian pyramid and wavelet transform also produces significant improvements to the state-of-the-art on a selection of benchmarks in likelihood modeling, including density estimation, lossless compression, and out-of-distribution detection. Investigating the theoretical basis of our empirical gains we uncover deep connections to score matching under the Earth Mover's Distance (EMD), which is a well-known surrogate for perceptual similarity. Code can be found at \href{this https URL}{this https url}.
• 摘要：级联模型是多尺度生成模型，具有生成高分辨率感知样本的显著能力。在这项研究中，我们表明，只要我们克服概率多尺度模型的一个基本困难：似然函数的难解性，它们也可以成为出色的似然模型。主要是，在级联模型中，每个中间尺度都会引入外部变量，这些变量无法通过边缘化来评估似然度。通过在由我们称为分层体积保持映射的一类变换引起的潜在空间上对扩散过程进行建模，这个问题就消失了，这些变换以分层方式分解空间结构化数据，而不会在潜在空间中引入局部扭曲。我们证明了文献中两种这样的映射在多尺度建模方面是众所周知的：拉普拉斯金字塔和小波变换。此类重新参数化不仅允许将似然函数直接表示为尺度上的联合似然，我们还表明，拉普拉斯金字塔和小波变换还可以在似然建模的一系列基准上显著提高最新水平，包括密度估计、无损压缩和分布外检测。通过研究我们经验成果的理论基础，我们发现了与地球移动距离 (EMD) 下的分数匹配的深层联系，EMD 是感知相似性的众所周知的替代指标。代码可以在 \href{this https URL}{this https url} 找到。

Title: Detection of AI Deepfake and Fraud in Online Payments Using GAN-Based Models

• Authors: Zong Ke, Shicheng Zhou, Yining Zhou, Chia Hong Chang, Rong Zhang
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07033
• Pdf URL: https://arxiv.org/pdf/2501.07033
• Copy Paste: [[2501.07033]] Detection of AI Deepfake and Fraud in Online Payments Using GAN-Based Models(https://arxiv.org/abs/2501.07033)
• Keywords: generative
• Abstract: This study explores the use of Generative Adversarial Networks (GANs) to detect AI deepfakes and fraudulent activities in online payment systems. With the growing prevalence of deepfake technology, which can manipulate facial features in images and videos, the potential for fraud in online transactions has escalated. Traditional security systems struggle to identify these sophisticated forms of fraud. This research proposes a novel GAN-based model that enhances online payment security by identifying subtle manipulations in payment images. The model is trained on a dataset consisting of real-world online payment images and deepfake images generated using advanced GAN architectures, such as StyleGAN and DeepFake. The results demonstrate that the proposed model can accurately distinguish between legitimate transactions and deepfakes, achieving a high detection rate above 95%. This approach significantly improves the robustness of payment systems against AI-driven fraud. The paper contributes to the growing field of digital security, offering insights into the application of GANs for fraud detection in financial services. Keywords- Payment Security, Image Recognition, Generative Adversarial Networks, AI Deepfake, Fraudulent Activities
• 摘要：本研究探讨了如何使用生成对抗网络 (GAN) 检测在线支付系统中的 AI 深度伪造和欺诈活动。随着可以操纵图像和视频中面部特征的深度伪造技术的日益普及，在线交易中发生欺诈的可能性也随之增加。传统的安全系统难以识别这些复杂的欺诈形式。本研究提出了一种基于 GAN 的新型模型，通过识别支付图像中的细微操纵来增强在线支付安全性。该模型在由真实世界的在线支付图像和使用高级 GAN 架构（例如 StyleGAN 和 DeepFake）生成的深度伪造图像组成的数据集上进行训练。结果表明，所提出的模型可以准确区分合法交易和深度伪造，实现 95% 以上的高检测率。这种方法显著提高了支付系统对 AI 驱动欺诈的鲁棒性。本文为不断发展的数字安全领域做出了贡献，为 GAN 在金融服务欺诈检测中的应用提供了见解。关键词-支付安全、图像识别、生成对抗网络、AI Deepfake、欺诈活动

Title: SFC-GAN: A Generative Adversarial Network for Brain Functional and Structural Connectome Translation

• Authors: Yee-Fan Tan, Jun Lin Liow, Pei-Sze Tan, Fuad Noman, Raphael C.-W. Phan, Hernando Ombao, Chee-Ming Ting
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07055
• Pdf URL: https://arxiv.org/pdf/2501.07055
• Copy Paste: [[2501.07055]] SFC-GAN: A Generative Adversarial Network for Brain Functional and Structural Connectome Translation(https://arxiv.org/abs/2501.07055)
• Keywords: generative
• Abstract: Modern brain imaging technologies have enabled the detailed reconstruction of human brain connectomes, capturing structural connectivity (SC) from diffusion MRI and functional connectivity (FC) from functional MRI. Understanding the intricate relationships between SC and FC is vital for gaining deeper insights into the brain's functional and organizational mechanisms. However, obtaining both SC and FC modalities simultaneously remains challenging, hindering comprehensive analyses. Existing deep generative models typically focus on synthesizing a single modality or unidirectional translation between FC and SC, thereby missing the potential benefits of bi-directional translation, especially in scenarios where only one connectome is available. Therefore, we propose Structural-Functional Connectivity GAN (SFC-GAN), a novel framework for bidirectional translation between SC and FC. This approach leverages the CycleGAN architecture, incorporating convolutional layers to effectively capture the spatial structures of brain connectomes. To preserve the topological integrity of these connectomes, we employ a structure-preserving loss that guides the model in capturing both global and local connectome patterns while maintaining symmetry. Our framework demonstrates superior performance in translating between SC and FC, outperforming baseline models in similarity and graph property evaluations compared to ground truth data, each translated modality can be effectively utilized for downstream classification.
• 摘要：现代脑成像技术已经能够详细重建人类大脑连接组，从扩散 MRI 中捕获结构连接 (SC)，从功能 MRI 中捕获功能连接 (FC)。了解 SC 和 FC 之间的复杂关系对于深入了解大脑的功能和组织机制至关重要。然而，同时获得 SC 和 FC 模态仍然具有挑战性，阻碍了全面的分析。现有的深度生成模型通常专注于合成 FC 和 SC 之间的单一模态或单向翻译，从而忽略了双向翻译的潜在好处，尤其是在只有一个连接组可用的情况下。因此，我们提出了结构功能连接 GAN (SFC-GAN)，这是一种在 SC 和 FC 之间进行双向翻译的新框架。这种方法利用 CycleGAN 架构，结合卷积层来有效捕捉大脑连接组的空间结构。为了保持这些连接组的拓扑完整性，我们采用了结构保留损失，指导模型在保持对称性的同时捕捉全局和局部连接组模式。我们的框架在 SC 和 FC 之间的转换中表现出卓越的性能，与地面真实数据相比，在相似性和图形属性评估方面优于基线模型，每种转换后的模态都可以有效地用于下游分类。

Title: Enhancing Image Generation Fidelity via Progressive Prompts

• Authors: Zhen Xiong, Yuqi Li, Chuanguang Yang, Tiao Tan, Zhihong Zhu, Siyuan Li, Yue Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07070
• Pdf URL: https://arxiv.org/pdf/2501.07070
• Copy Paste: [[2501.07070]] Enhancing Image Generation Fidelity via Progressive Prompts(https://arxiv.org/abs/2501.07070)
• Keywords: generation
• Abstract: The diffusion transformer (DiT) architecture has attracted significant attention in image generation, achieving better fidelity, performance, and diversity. However, most existing DiT - based image generation methods focus on global - aware synthesis, and regional prompt control has been less explored. In this paper, we propose a coarse - to - fine generation pipeline for regional prompt - following generation. Specifically, we first utilize the powerful large language model (LLM) to generate both high - level descriptions of the image (such as content, topic, and objects) and low - level descriptions (such as details and style). Then, we explore the influence of cross - attention layers at different depths. We find that deeper layers are always responsible for high - level content control, while shallow layers handle low - level content control. Various prompts are injected into the proposed regional cross - attention control for coarse - to - fine generation. By using the proposed pipeline, we enhance the controllability of DiT - based image generation. Extensive quantitative and qualitative results show that our pipeline can improve the performance of the generated images.
• 摘要：扩散变换器 (DiT) 架构在图像生成中引起了广泛关注，实现了更好的保真度、性能和多样性。然而，大多数现有的基于 DiT 的图像生成方法侧重于全局感知合成，而区域提示控制的探索较少。在本文中，我们提出了一种用于区域提示生成的由粗到细的生成流程。具体来说，我们首先利用强大的大型语言模型 (LLM) 来生成图像的高级描述（例如内容、主题和对象）和低级描述（例如细节和风格）。然后，我们探索不同深度的交叉注意层的影响。我们发现较深的层始终负责高级内容控制，而浅层负责低级内容控制。各种提示被注入到所提出的区域交叉注意控制中，以实现由粗到细的生成。通过使用所提出的流程，我们增强了基于 DiT 的图像生成的可控性。大量的定量和定性结果表明，我们的流程可以提高生成图像的性能。

Title: D3MES: Diffusion Transformer with multihead equivariant self-attention for 3D molecule generation

• Authors: Zhejun Zhang, Yuanping Chen, Shibing Chu
• Subjects: cs.LG, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2501.07077
• Pdf URL: https://arxiv.org/pdf/2501.07077
• Copy Paste: [[2501.07077]] D3MES: Diffusion Transformer with multihead equivariant self-attention for 3D molecule generation(https://arxiv.org/abs/2501.07077)
• Keywords: generation, generative
• Abstract: Understanding and predicting the diverse conformational states of molecules is crucial for advancing fields such as chemistry, material science, and drug development. Despite significant progress in generative models, accurately generating complex and biologically or material-relevant molecular structures remains a major challenge. In this work, we introduce a diffusion model for three-dimensional (3D) molecule generation that combines a classifiable diffusion model, Diffusion Transformer, with multihead equivariant self-attention. This method addresses two key challenges: correctly attaching hydrogen atoms in generated molecules through learning representations of molecules after hydrogen atoms are removed; and overcoming the limitations of existing models that cannot generate molecules across multiple classes simultaneously. The experimental results demonstrate that our model not only achieves state-of-the-art performance across several key metrics but also exhibits robustness and versatility, making it highly suitable for early-stage large-scale generation processes in molecular design, followed by validation and further screening to obtain molecules with specific properties.
• 摘要：理解和预测分子的多种构象状态对于化学、材料科学和药物开发等领域的发展至关重要。尽管生成模型取得了重大进展，但准确生成复杂且与生物或材料相关的分子结构仍然是一项重大挑战。在这项工作中，我们引入了一种用于三维 (3D) 分子生成的扩散模型，该模型结合了可分类扩散模型 Diffusion Transformer 和多头等变自注意力。该方法解决了两个关键挑战：通过学习去除氢原子后的分子表示正确地将氢原子附着在生成的分子中；并克服了现有模型无法同时生成多个类别的分子的局限性。实验结果表明，我们的模型不仅在几个关键指标上实现了最先进的性能，而且还表现出稳健性和多功能性，使其非常适合分子设计中早期的大规模生成过程，随后进行验证和进一步筛选以获得具有特定性质的分子。

Title: Video Quality Assessment for Online Processing: From Spatial to Temporal Sampling

• Authors: Jiebin Yan, Lei Wu, Yuming Fang, Xuelin Liu, Xue Xia, Weide Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07087
• Pdf URL: https://arxiv.org/pdf/2501.07087
• Copy Paste: [[2501.07087]] Video Quality Assessment for Online Processing: From Spatial to Temporal Sampling(https://arxiv.org/abs/2501.07087)
• Keywords: quality assessment
• Abstract: With the rapid development of multimedia processing and deep learning technologies, especially in the field of video understanding, video quality assessment (VQA) has achieved significant progress. Although researchers have moved from designing efficient video quality mapping models to various research directions, in-depth exploration of the effectiveness-efficiency trade-offs of spatio-temporal modeling in VQA models is still less sufficient. Considering the fact that videos have highly redundant information, this paper investigates this problem from the perspective of joint spatial and temporal sampling, aiming to seek the answer to how little information we should keep at least when feeding videos into the VQA models while with acceptable performance sacrifice. To this end, we drastically sample the video's information from both spatial and temporal dimensions, and the heavily squeezed video is then fed into a stable VQA model. Comprehensive experiments regarding joint spatial and temporal sampling are conducted on six public video quality databases, and the results demonstrate the acceptable performance of the VQA model when throwing away most of the video information. Furthermore, with the proposed joint spatial and temporal sampling strategy, we make an initial attempt to design an online VQA model, which is instantiated by as simple as possible a spatial feature extractor, a temporal feature fusion module, and a global quality regression module. Through quantitative and qualitative experiments, we verify the feasibility of online VQA model by simplifying itself and reducing input.
• 摘要：随着多媒体处理和深度学习技术的快速发展，特别是在视频理解领域的发展，视频质量评估（VQA）取得了长足的进步。尽管研究人员已经从设计高效的视频质量映射模型转向了各个研究方向，但对 VQA 模型中时空建模的有效性-效率权衡的深入探索仍然不够。考虑到视频具有高度冗余的信息，本文从时空联合采样的角度研究该问题，旨在寻求在可接受的性能牺牲下，在将视频输入 VQA 模型时至少应该保留多少信息。为此，我们从空间和时间维度对视频的信息进行大幅度采样，然后将经过大量压缩的视频输入到稳定的 VQA 模型中。在六个公开的视频质量数据库上进行了全面的时空联合采样实验，结果表明，在丢弃大部分视频信息的情况下，VQA 模型的性能仍然可接受。此外，基于提出的联合空间和时间采样策略，我们初步尝试设计一个在线 VQA 模型，该模型由尽可能简单的空间特征提取器、时间特征融合模块和全局质量回归模块实例化。通过定量和定性实验，我们通过简化自身和减少输入来验证在线 VQA 模型的可行性。

Title: Eye Sclera for Fair Face Image Quality Assessment

• Authors: Wassim Kabbani, Kiran Raja, Raghavendra Ramachandra, Christoph Busch
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07158
• Pdf URL: https://arxiv.org/pdf/2501.07158
• Copy Paste: [[2501.07158]] Eye Sclera for Fair Face Image Quality Assessment(https://arxiv.org/abs/2501.07158)
• Keywords: quality assessment
• Abstract: Fair operational systems are crucial in gaining and maintaining society's trust in face recognition systems (FRS). FRS start with capturing an image and assessing its quality before using it further for enrollment or verification. Fair Face Image Quality Assessment (FIQA) schemes therefore become equally important in the context of fair FRS. This work examines the sclera as a quality assessment region for obtaining a fair FIQA. The sclera region is agnostic to demographic variations and skin colour for assessing the quality of a face image. We analyze three skin tone related ISO/IEC face image quality assessment measures and assess the sclera region as an alternative area for assessing FIQ. Our analysis of the face dataset of individuals from different demographic groups representing different skin tones indicates sclera as an alternative to measure dynamic range, over- and under-exposure of face using sclera region alone. The sclera region being agnostic to skin tone, i.e., demographic factors, provides equal utility as a fair FIQA as shown by our Error-vs-Discard Characteristic (EDC) curve analysis.
• 摘要：公平的操作系统对于获得和维持社会对人脸识别系统 (FRS) 的信任至关重要。FRS 首先捕获图像并评估其质量，然后再将其用于注册或验证。因此，公平人脸图像质量评估 (FIQA) 方案在公平 FRS 的背景下变得同样重要。这项工作将巩膜作为获得公平 FIQA 的质量评估区域进行研究。巩膜区域与评估人脸图像质量的人口统计学变化和肤色无关。我们分析了三种与肤色相关的 ISO/IEC 人脸图像质量评估指标，并评估巩膜区域作为评估 FIQ 的替代区域。我们对代表不同肤色的不同人口统计学群体的个体人脸数据集的分析表明，巩膜是仅使用巩膜区域来测量动态范围、面部过度曝光和曝光不足的替代方案。巩膜区域与肤色（即人口统计因素）无关，但如我们的错误与丢弃特性 (EDC) 曲线分析所示，它与公平的 FIQA 具有同等的效用。

Title: Radial Distortion in Face Images: Detection and Impact

• Authors: Wassim Kabbani, Tristan Le Pessot, Kiran Raja, Raghavendra Ramachandra, Christoph Busch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07179
• Pdf URL: https://arxiv.org/pdf/2501.07179
• Copy Paste: [[2501.07179]] Radial Distortion in Face Images: Detection and Impact(https://arxiv.org/abs/2501.07179)
• Keywords: quality assessment
• Abstract: Acquiring face images of sufficiently high quality is important for online ID and travel document issuance applications using face recognition systems (FRS). Low-quality, manipulated (intentionally or unintentionally), or distorted images degrade the FRS performance and facilitate documents' misuse. Securing quality for enrolment images, especially in the unsupervised self-enrolment scenario via a smartphone, becomes important to assure FRS performance. In this work, we focus on the less studied area of radial distortion (a.k.a., the fish-eye effect) in face images and its impact on FRS performance. We introduce an effective radial distortion detection model that can detect and flag radial distortion in the enrolment scenario. We formalize the detection model as a face image quality assessment (FIQA) algorithm and provide a careful inspection of the effect of radial distortion on FRS performance. Evaluation results show excellent detection results for the proposed models, and the study on the impact on FRS uncovers valuable insights into how to best use these models in operational systems.
• 摘要：对于使用人脸识别系统 (FRS) 的在线身份证和旅行证件签发应用程序来说，获取足够高质量的人脸图像非常重要。低质量、被操纵（有意或无意）或扭曲的图像会降低 FRS 性能并导致证件被滥用。确保注册图像的质量，尤其是在通过智能手机进行无监督的自我注册场景中，对于确保 FRS 性能非常重要。在这项工作中，我们专注于人脸图像中研究较少的径向失真（又称鱼眼效应）领域及其对 FRS 性能的影响。我们引入了一种有效的径向失真检测模型，可以检测和标记注册场景中的径向失真。我们将检测模型形式化为人脸图像质量评估 (FIQA) 算法，并仔细检查径向失真对 FRS 性能的影响。评估结果表明，所提出的模型具有出色的检测结果，而对 FRS 影响的研究为如何在操作系统中最好地使用这些模型提供了宝贵的见解。

Title: FaceOracle: Chat with a Face Image Oracle

• Authors: Wassim Kabbani, Kiran Raja, Raghavendra Ramachandra, Christoph Busch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07202
• Pdf URL: https://arxiv.org/pdf/2501.07202
• Copy Paste: [[2501.07202]] FaceOracle: Chat with a Face Image Oracle(https://arxiv.org/abs/2501.07202)
• Keywords: quality assessment
• Abstract: A face image is a mandatory part of ID and travel documents. Obtaining high-quality face images when issuing such documents is crucial for both human examiners and automated face recognition systems. In several international standards, face image quality requirements are intricate and defined in detail. Identifying and understanding non-compliance or defects in the submitted face images is crucial for both issuing authorities and applicants. In this work, we introduce FaceOracle, an LLM-powered AI assistant that helps its users analyze a face image in a natural conversational manner using standard compliant algorithms. Leveraging the power of LLMs, users can get explanations of various face image quality concepts as well as interpret the outcome of face image quality assessment (FIQA) algorithms. We implement a proof-of-concept that demonstrates how experts at an issuing authority could integrate FaceOracle into their workflow to analyze, understand, and communicate their decisions more efficiently, resulting in enhanced productivity.
• 摘要：人脸图像是身份证和旅行证件的必备部分。在签发此类证件时，获取高质量的人脸图像对于人工检查人员和自动人脸识别系统都至关重要。在多项国际标准中，人脸图像质量要求错综复杂且定义详细。识别和了解所提交人脸图像中的不合规或缺陷对于签发机构和申请人都至关重要。在这项工作中，我们介绍了 FaceOracle，这是一个由 LLM 驱动的 AI 助手，可帮助用户使用符合标准的算法以自然对话的方式分析人脸图像。利用 LLM 的强大功能，用户可以获得各种人脸图像质量概念的解释，并解释人脸图像质量评估 (FIQA) 算法的结果。我们实施了一个概念验证，展示了签发机构的专家如何将 FaceOracle 集成到他们的工作流程中，以更有效地分析、理解和传达他们的决策，从而提高工作效率。

Title: A data-driven approach to discover and quantify systemic lupus erythematosus etiological heterogeneity from electronic health records

• Authors: Marco Barbero Mota, John M. Still, Jorge L. Gamboa, Eric V. Strobl, Charles M. Stein, Vivian K. Kawai, Thomas A. Lasko
• Subjects: cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2501.07206
• Pdf URL: https://arxiv.org/pdf/2501.07206
• Copy Paste: [[2501.07206]] A data-driven approach to discover and quantify systemic lupus erythematosus etiological heterogeneity from electronic health records(https://arxiv.org/abs/2501.07206)
• Keywords: generation
• Abstract: Systemic lupus erythematosus (SLE) is a complex heterogeneous disease with many manifestational facets. We propose a data-driven approach to discover probabilistic independent sources from multimodal imperfect EHR data. These sources represent exogenous variables in the data generation process causal graph that estimate latent root causes of the presence of SLE in the health record. We objectively evaluated the sources against the original variables from which they were discovered by training supervised models to discriminate SLE from negative health records using a reduced set of labelled instances. We found 19 predictive sources with high clinical validity and whose EHR signatures define independent factors of SLE heterogeneity. Using the sources as input patient data representation enables models to provide with rich explanations that better capture the clinical reasons why a particular record is (not) an SLE case. Providers may be willing to trade patient-level interpretability for discrimination especially in challenging cases.
• 摘要：系统性红斑狼疮 (SLE) 是一种复杂的异质性疾病，具有多种表现形式。我们提出了一种数据驱动的方法，从多模态不完善的 EHR 数据中发现概率独立源。这些来源代表数据生成过程因果图中的外生变量，用于估计健康记录中存在 SLE 的潜在根本原因。我们通过训练监督模型，使用一组精简的标记实例区分 SLE 与负面健康记录，客观地评估了这些来源与发现它们的原始变量之间的关系。我们发现了 19 个具有高临床效度的预测源，它们的 EHR 特征定义了 SLE 异质性的独立因素。使用这些源作为输入患者数据表示，使模型能够提供丰富的解释，从而更好地捕捉特定记录是（不是）SLE 病例的临床原因。提供者可能愿意用患者级别的可解释性来换取歧视，尤其是在具有挑战性的病例中。

Title: MOS-Attack: A Scalable Multi-objective Adversarial Attack Framework

• Authors: Ping Guo, Cheng Gong, Xi Lin, Fei Liu, Zhichao Lu, Qingfu Zhang, Zhenkun Wang
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07251
• Pdf URL: https://arxiv.org/pdf/2501.07251
• Copy Paste: [[2501.07251]] MOS-Attack: A Scalable Multi-objective Adversarial Attack Framework(https://arxiv.org/abs/2501.07251)
• Keywords: generation
• Abstract: Crafting adversarial examples is crucial for evaluating and enhancing the robustness of Deep Neural Networks (DNNs), presenting a challenge equivalent to maximizing a non-differentiable 0-1 loss function. However, existing single objective methods, namely adversarial attacks focus on a surrogate loss function, do not fully harness the benefits of engaging multiple loss functions, as a result of insufficient understanding of their synergistic and conflicting nature. To overcome these limitations, we propose the Multi-Objective Set-based Attack (MOS Attack), a novel adversarial attack framework leveraging multiple loss functions and automatically uncovering their interrelations. The MOS Attack adopts a set-based multi-objective optimization strategy, enabling the incorporation of numerous loss functions without additional parameters. It also automatically mines synergistic patterns among various losses, facilitating the generation of potent adversarial attacks with fewer objectives. Extensive experiments have shown that our MOS Attack outperforms single-objective attacks. Furthermore, by harnessing the identified synergistic patterns, MOS Attack continues to show superior results with a reduced number of loss functions.
• 摘要：制作对抗样本对于评估和增强深度神经网络 (DNN) 的鲁棒性至关重要，其挑战性相当于最大化不可微的 0-1 损失函数。然而，现有的单一目标方法，即专注于替代损失函数的对抗攻击，由于对它们的协同和冲突性质了解不足，无法充分利用使用多个损失函数的好处。为了克服这些限制，我们提出了基于多目标集合的攻击 (MOS 攻击)，这是一种利用多个损失函数并自动发现它们相互关系的新型对抗攻击框架。MOS 攻击采用基于集合的多目标优化策略，无需额外参数即可合并众多损失函数。它还自动挖掘各种损失之间的协同模式，从而有助于以更少的目标生成强大的对抗攻击。大量实验表明，我们的 MOS 攻击优于单目标攻击。此外，通过利用已识别的协同模式，MOS Attack 继续以减少的损失函数数量显示出优异的效果。

Title: Skip Mamba Diffusion for Monocular 3D Semantic Scene Completion

• Authors: Li Liang, Naveed Akhtar, Jordan Vice, Xiangrui Kong, Ajmal Saeed Mian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07260
• Pdf URL: https://arxiv.org/pdf/2501.07260
• Copy Paste: [[2501.07260]] Skip Mamba Diffusion for Monocular 3D Semantic Scene Completion(https://arxiv.org/abs/2501.07260)
• Keywords: generative
• Abstract: 3D semantic scene completion is critical for multiple downstream tasks in autonomous systems. It estimates missing geometric and semantic information in the acquired scene data. Due to the challenging real-world conditions, this task usually demands complex models that process multi-modal data to achieve acceptable performance. We propose a unique neural model, leveraging advances from the state space and diffusion generative modeling to achieve remarkable 3D semantic scene completion performance with monocular image input. Our technique processes the data in the conditioned latent space of a variational autoencoder where diffusion modeling is carried out with an innovative state space technique. A key component of our neural network is the proposed Skimba (Skip Mamba) denoiser, which is adept at efficiently processing long-sequence data. The Skimba diffusion model is integral to our 3D scene completion network, incorporating a triple Mamba structure, dimensional decomposition residuals and varying dilations along three directions. We also adopt a variant of this network for the subsequent semantic segmentation stage of our method. Extensive evaluation on the standard SemanticKITTI and SSCBench-KITTI360 datasets show that our approach not only outperforms other monocular techniques by a large margin, it also achieves competitive performance against stereo methods. The code is available at this https URL
• 摘要：3D 语义场景完成对于自主系统中的多个下游任务至关重要。它估计获取的场景数据中缺失的几何和语义信息。由于现实条件具有挑战性，此任务通常需要处理多模态数据的复杂模型才能实现可接受的性能。我们提出了一种独特的神经模型，利用状态空间和扩散生成建模的进步，通过单目图像输入实现卓越的 3D 语义场景完成性能。我们的技术在变分自动编码器的条件潜在空间中处理数据，其中使用创新的状态空间技术进行扩散建模。我们神经网络的一个关键组件是提出的 Skimba（Skip Mamba）降噪器，它擅长高效处理长序列数据。Skimba 扩散模型是我们 3D 场景完成网络不可或缺的一部分，它结合了三重 Mamba 结构、维度分解残差和沿三个方向的不同扩张。我们还采用了该网络的变体，用于我们方法的后续语义分割阶段。对标准 SemanticKITTI 和 SSCBench-KITTI360 数据集的广泛评估表明，我们的方法不仅比其他单目技术表现优异，而且与立体方法相比也具有竞争力。代码可从此 https URL 获取

Title: Foundation Models at Work: Fine-Tuning for Fairness in Algorithmic Hiring

• Authors: Buse Sibel Korkmaz, Rahul Nair, Elizabeth M. Daly, Evangelos Anagnostopoulos, Christos Varytimidis, Antonio del Rio Chanona
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07324
• Pdf URL: https://arxiv.org/pdf/2501.07324
• Copy Paste: [[2501.07324]] Foundation Models at Work: Fine-Tuning for Fairness in Algorithmic Hiring(https://arxiv.org/abs/2501.07324)
• Keywords: generative
• Abstract: Foundation models require fine-tuning to ensure their generative outputs align with intended results for specific tasks. Automating this fine-tuning process is challenging, as it typically needs human feedback that can be expensive to acquire. We present AutoRefine, a method that leverages reinforcement learning for targeted fine-tuning, utilizing direct feedback from measurable performance improvements in specific downstream tasks. We demonstrate the method for a problem arising in algorithmic hiring platforms where linguistic biases influence a recommendation system. In this setting, a generative model seeks to rewrite given job specifications to receive more diverse candidate matches from a recommendation engine which matches jobs to candidates. Our model detects and regulates biases in job descriptions to meet diversity and fairness criteria. The experiments on a public hiring dataset and a real-world hiring platform showcase how large language models can assist in identifying and mitigation biases in the real world.
• 摘要：基础模型需要进行微调，以确保其生成输出与特定任务的预期结果一致。自动化这一微调过程具有挑战性，因为它通常需要人工反馈，而获取这些反馈的成本可能很高。我们提出了 AutoRefine，这是一种利用强化学习进行有针对性微调的方法，利用特定下游任务中可衡量的性能改进的直接反馈。我们展示了该方法，用于解决算法招聘平台中出现的一个问题，其中语言偏见会影响推荐系统。在这种情况下，生成模型试图重写给定的职位规范，以从将职位与候选人匹配的推荐引擎中接收更多样化的候选人匹配。我们的模型检测和调节职位描述中的偏见，以满足多样性和公平性标准。在公共招聘数据集和现实世界招聘平台上进行的实验展示了大型语言模型如何帮助识别和缓解现实世界中的偏见。

Title: Deep Generative Clustering with VAEs and Expectation-Maximization

• Authors: Michael Adipoetra, Ségolène Martin
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.07358
• Pdf URL: https://arxiv.org/pdf/2501.07358
• Copy Paste: [[2501.07358]] Deep Generative Clustering with VAEs and Expectation-Maximization(https://arxiv.org/abs/2501.07358)
• Keywords: generation, generative
• Abstract: We propose a novel deep clustering method that integrates Variational Autoencoders (VAEs) into the Expectation-Maximization (EM) framework. Our approach models the probability distribution of each cluster with a VAE and alternates between updating model parameters by maximizing the Evidence Lower Bound (ELBO) of the log-likelihood and refining cluster assignments based on the learned distributions. This enables effective clustering and generation of new samples from each cluster. Unlike existing VAE-based methods, our approach eliminates the need for a Gaussian Mixture Model (GMM) prior or additional regularization techniques. Experiments on MNIST and FashionMNIST demonstrate superior clustering performance compared to state-of-the-art methods.
• 摘要：我们提出了一种新颖的深度聚类方法，将变分自动编码器 (VAE) 集成到期望最大化 (EM) 框架中。我们的方法使用 VAE 为每个聚类建模概率分布，并通过最大化对数似然的证据下限 (ELBO) 和基于学习到的分布优化聚类分配来交替更新模型参数。这可以实现有效的聚类并从每个聚类中生成新样本。与现有的基于 VAE 的方法不同，我们的方法无需先验高斯混合模型 (GMM) 或额外的正则化技术。在 MNIST 和 FashionMNIST 上进行的实验表明，与最先进的方法相比，我们的聚类性能更出色。

Title: OCORD: Open-Campus Object Removal Dataset

• Authors: Shuo Zhang, Runpu Wei, Kongming Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07397
• Pdf URL: https://arxiv.org/pdf/2501.07397
• Copy Paste: [[2501.07397]] OCORD: Open-Campus Object Removal Dataset(https://arxiv.org/abs/2501.07397)
• Keywords: generation, generative
• Abstract: The rapid advancements in generative models, particularly diffusion-based techniques, have revolutionized image inpainting tasks by enabling the generation of high-fidelity and diverse content. However, object removal remains under-explored as a specific subset of inpainting, facing challenges such as inadequate semantic understanding and the unintended generation of artifacts. Existing datasets for object removal often rely on synthetic data, which fails to align with real-world scenarios, limiting model performance. Although some real-world datasets address these issues partially, they suffer from scalability, annotation inefficiencies, and limited realism in physical phenomena such as lighting and shadows. To address these limitations, this paper introduces a novel approach to object removal by constructing a high-resolution real-world dataset through long-duration video capture with fixed camera settings. Leveraging advanced tools such as Grounding-DINO, Segment-Anything-Model, and MASA for automated annotation, we provides image, background, and mask pairs while significantly reducing annotation time and labor. With our efficient annotation pipeline, we release the first fully open, high-resolution real-world dataset for object removal, and improved performance in object removal tasks through fine-tuning of pre-trained diffusion models.
• 摘要：生成模型（尤其是基于扩散的技术）的快速发展，通过生成高保真和多样化的内容，彻底改变了图像修复任务。然而，作为修复的一个特定子集，物体移除仍未得到充分探索，面临着语义理解不足和意外生成伪影等挑战。现有的物体移除数据集通常依赖于合成数据，这些数据与真实场景不一致，限制了模型性能。尽管一些真实世界数据集部分解决了这些问题，但它们存在可扩展性、注释效率低下以及照明和阴影等物理现象的真实感有限等问题。为了解决这些限制，本文介绍了一种新颖的物体移除方法，即通过固定摄像头设置的长时间视频捕获构建高分辨率真实世界数据集。利用 Grounding-DINO、Segment-Anything-Model 和 MASA 等高级工具进行自动注释，我们提供图像、背景和蒙版对，同时显著减少注释时间和劳动量。通过我们高效的注释管道，我们发布了第一个完全开放、高分辨率的真实世界物体移除数据集，并通过对预先训练的扩散模型进行微调提高了物体移除任务的性能。

Title: Diff-Ensembler: Learning to Ensemble 2D Diffusion Models for Volume-to-Volume Medical Image Translation

• Authors: Xiyue Zhu, Dou Hoon Kwark, Ruike Zhu, Kaiwen Hong, Yiqi Tao, Shirui Luo, Yudu Li, Zhi-Pei Liang, Volodymyr Kindratenko
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07430
• Pdf URL: https://arxiv.org/pdf/2501.07430
• Copy Paste: [[2501.07430]] Diff-Ensembler: Learning to Ensemble 2D Diffusion Models for Volume-to-Volume Medical Image Translation(https://arxiv.org/abs/2501.07430)
• Keywords: super-resolution
• Abstract: Despite success in volume-to-volume translations in medical images, most existing models struggle to effectively capture the inherent volumetric distribution using 3D representations. The current state-of-the-art approach combines multiple 2D-based networks through weighted averaging, thereby neglecting the 3D spatial structures. Directly training 3D models in medical imaging presents significant challenges due to high computational demands and the need for large-scale datasets. To address these challenges, we introduce Diff-Ensembler, a novel hybrid 2D-3D model for efficient and effective volumetric translations by ensembling perpendicularly trained 2D diffusion models with a 3D network in each diffusion step. Moreover, our model can naturally be used to ensemble diffusion models conditioned on different modalities, allowing flexible and accurate fusion of input conditions. Extensive experiments demonstrate that Diff-Ensembler attains superior accuracy and volumetric realism in 3D medical image super-resolution and modality translation. We further demonstrate the strength of our model's volumetric realism using tumor segmentation as a downstream task.
• 摘要：尽管在医学图像的体积到体积转换方面取得了成功，但大多数现有模型仍难以使用 3D 表示有效捕捉固有的体积分布。当前最先进的方法通过加权平均组合多个基于 2D 的网络，从而忽略了 3D 空间结构。由于计算需求高且需要大规模数据集，直接训练医学成像中的 3D 模型面临着重大挑战。为了应对这些挑战，我们引入了 Diff-Ensembler，这是一种新颖的混合 2D-3D 模型，通过在每个扩散步骤中将垂直训练的 2D 扩散模型与 3D 网络组合来实现高效的体积转换。此外，我们的模型可以自然地用于集成基于不同模态的扩散模型，从而允许灵活和准确地融合输入条件。大量实验表明，Diff-Ensembler 在 3D 医学图像超分辨率和模态转换中实现了卓越的精度和体积真实感。我们进一步通过使用肿瘤分割作为下游任务来证明我们模型的体积真实感的强度。

Title: Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards

• Authors: Yangsibo Huang, Milad Nasr, Anastasios Angelopoulos, Nicholas Carlini, Wei-Lin Chiang, Christopher A. Choquette-Choo, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Ken Ziyu Liu, Ion Stoica, Florian Tramer, Chiyuan Zhang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.07493
• Pdf URL: https://arxiv.org/pdf/2501.07493
• Copy Paste: [[2501.07493]] Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards(https://arxiv.org/abs/2501.07493)
• Keywords: generation
• Abstract: It is now common to evaluate Large Language Models (LLMs) by having humans manually vote to evaluate model outputs, in contrast to typical benchmarks that evaluate knowledge or skill at some particular task. Chatbot Arena, the most popular benchmark of this type, ranks models by asking users to select the better response between two randomly selected models (without revealing which model was responsible for the generations). These platforms are widely trusted as a fair and accurate measure of LLM capabilities. In this paper, we show that if bot protection and other defenses are not implemented, these voting-based benchmarks are potentially vulnerable to adversarial manipulation. Specifically, we show that an attacker can alter the leaderboard (to promote their favorite model or demote competitors) at the cost of roughly a thousand votes (verified in a simulated, offline version of Chatbot Arena). Our attack consists of two steps: first, we show how an attacker can determine which model was used to generate a given reply with more than $95\%$ accuracy; and then, the attacker can use this information to consistently vote for (or against) a target model. Working with the Chatbot Arena developers, we identify, propose, and implement mitigations to improve the robustness of Chatbot Arena against adversarial manipulation, which, based on our analysis, substantially increases the cost of such attacks. Some of these defenses were present before our collaboration, such as bot protection with Cloudflare, malicious user detection, and rate limiting. Others, including reCAPTCHA and login are being integrated to strengthen the security in Chatbot Arena.
• 摘要：现在，评估大型语言模型 (LLM) 的常用方法是让人类手动投票来评估模型输出，而不是评估某些特定任务的知识或技能的典型基准。Chatbot Arena 是此类基准中最受欢迎的，它通过要求用户在两个随机选择的模型中选择更好的响应（不透露哪个模型负责生成）来对模型进行排名。这些平台被广泛认为是公平和准确的 LLM 能力衡量标准。在本文中，我们表明，如果不实施机器人保护和其他防御措施，这些基于投票的基准可能会受到对抗性操纵。具体来说，我们表明攻击者可以改变排行榜（以推广他们最喜欢的模型或降级竞争对手），代价是大约一千张选票（在 Chatbot Arena 的模拟离线版本中得到验证）。我们的攻击包括两个步骤：首先，我们展示攻击者如何确定使用哪个模型以超过 $95\%$ 的准确率生成给定的回复；然后，攻击者可以使用此信息始终如一地投票支持（或反对）目标模型。我们与 Chatbot Arena 开发人员合作，确定、提出并实施缓解措施，以提高 Chatbot Arena 抵御对抗性操纵的稳健性，根据我们的分析，对抗性操纵会大大增加此类攻击的成本。其中一些防御措施在我们合作之前就已存在，例如使用 Cloudflare 的机器人保护、恶意用户检测和速率限制。其他措施（包括 reCAPTCHA 和登录）正在集成以加强 Chatbot Arena 的安全性。

Title: RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment

• Authors: Difei Gu, Yunhe Gao, Yang Zhou, Mu Zhou, Dimitris Metaxas
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07525
• Pdf URL: https://arxiv.org/pdf/2501.07525
• Copy Paste: [[2501.07525]] RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment(https://arxiv.org/abs/2501.07525)
• Keywords: generation, generative
• Abstract: Automated chest radiographs interpretation requires both accurate disease classification and detailed radiology report generation, presenting a significant challenge in the clinical workflow. Current approaches either focus on classification accuracy at the expense of interpretability or generate detailed but potentially unreliable reports through image captioning techniques. In this study, we present RadAlign, a novel framework that combines the predictive accuracy of vision-language models (VLMs) with the reasoning capabilities of large language models (LLMs). Inspired by the radiologist's workflow, RadAlign first employs a specialized VLM to align visual features with key medical concepts, achieving superior disease classification with an average AUC of 0.885 across multiple diseases. These recognized medical conditions, represented as text-based concepts in the aligned visual-language space, are then used to prompt LLM-based report generation. Enhanced by a retrieval-augmented generation mechanism that grounds outputs in similar historical cases, RadAlign delivers superior report quality with a GREEN score of 0.678, outperforming state-of-the-art methods' 0.634. Our framework maintains strong clinical interpretability while reducing hallucinations, advancing automated medical imaging and report analysis through integrated predictive and generative AI. Code is available at this https URL.
• 摘要：自动胸部 X 光片解释需要准确的疾病分类和详细的放射学报告生成，这对临床工作流程提出了重大挑战。当前的方法要么以牺牲可解释性为代价来关注分类准确性，要么通过图像字幕技术生成详细但可能不可靠的报告。在本研究中，我们提出了 RadAlign，这是一个新颖的框架，它将视觉语言模型 (VLM) 的预测准确性与大型语言模型 (LLM) 的推理能力相结合。受放射科医生工作流程的启发，RadAlign 首先采用专门的 VLM 将视觉特征与关键医学概念对齐，实现了出色的疾病分类，在多种疾病中的平均 AUC 为 0.885。这些已识别的医疗状况在对齐的视觉语言空间中表示为基于文本的概念，然后用于提示基于 LLM 的报告生成。 RadAlign 采用了检索增强生成机制，该机制将输出结果与类似的历史案例联系起来，可提供卓越的报告质量，GREEN 评分为 0.678，优于最先进方法的 0.634。我们的框架在减少幻觉的同时保持了强大的临床可解释性，并通过集成的预测和生成 AI 推进了自动化医学成像和报告分析。代码可在此 https URL 上获取。

Title: Training-Free Motion-Guided Video Generation with Enhanced Temporal Consistency Using Motion Consistency Loss

• Authors: Xinyu Zhang, Zicheng Duan, Dong Gong, Lingqiao Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07563
• Pdf URL: https://arxiv.org/pdf/2501.07563
• Copy Paste: [[2501.07563]] Training-Free Motion-Guided Video Generation with Enhanced Temporal Consistency Using Motion Consistency Loss(https://arxiv.org/abs/2501.07563)
• Keywords: generation
• Abstract: In this paper, we address the challenge of generating temporally consistent videos with motion guidance. While many existing methods depend on additional control modules or inference-time fine-tuning, recent studies suggest that effective motion guidance is achievable without altering the model architecture or requiring extra training. Such approaches offer promising compatibility with various video generation foundation models. However, existing training-free methods often struggle to maintain consistent temporal coherence across frames or to follow guided motion accurately. In this work, we propose a simple yet effective solution that combines an initial-noise-based approach with a novel motion consistency loss, the latter being our key innovation. Specifically, we capture the inter-frame feature correlation patterns of intermediate features from a video diffusion model to represent the motion pattern of the reference video. We then design a motion consistency loss to maintain similar feature correlation patterns in the generated video, using the gradient of this loss in the latent space to guide the generation process for precise motion control. This approach improves temporal consistency across various motion control tasks while preserving the benefits of a training-free setup. Extensive experiments show that our method sets a new standard for efficient, temporally coherent video generation.
• 摘要：在本文中，我们解决了生成具有运动引导的时间一致视频的挑战。虽然许多现有方法依赖于额外的控制模块或推理时间微调，但最近的研究表明，无需改变模型架构或额外训练即可实现有效的运动引导。这些方法与各种视频生成基础模型具有良好的兼容性。然而，现有的无需训练的方法往往难以保持帧间一致的时间连贯性或准确跟随引导运动。在这项工作中，我们提出了一种简单而有效的解决方案，将基于初始噪声的方法与新颖的运动一致性损失相结合，后者是我们的主要创新。具体来说，我们从视频扩散模型中捕获中间特征的帧间特征相关模式来表示参考视频的运动模式。然后，我们设计了一个运动一致性损失以在生成的视频中保持相似的特征相关模式，使用潜在空间中这种损失的梯度来指导生成过程以实现精确的运动控制。这种方法提高了各种运动控制任务的时间一致性，同时保留了无需训练设置的优势。大量实验表明，我们的方法为高效、时间连贯的视频生成设立了新的标准。

Title: UnCommon Objects in 3D

• Authors: Xingchen Liu, Piyush Tayal, Jianyuan Wang, Jesus Zarzar, Tom Monnier, Konstantinos Tertikas, Jiali Duan, Antoine Toisoul, Jason Y. Zhang, Natalia Neverova, Andrea Vedaldi, Roman Shapovalov, David Novotny
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.07574
• Pdf URL: https://arxiv.org/pdf/2501.07574
• Copy Paste: [[2501.07574]] UnCommon Objects in 3D(https://arxiv.org/abs/2501.07574)
• Keywords: generative
• Abstract: We introduce Uncommon Objects in 3D (uCO3D), a new object-centric dataset for 3D deep learning and 3D generative AI. uCO3D is the largest publicly-available collection of high-resolution videos of objects with 3D annotations that ensures full-360$^{\circ}$ coverage. uCO3D is significantly more diverse than MVImgNet and CO3Dv2, covering more than 1,000 object categories. It is also of higher quality, due to extensive quality checks of both the collected videos and the 3D annotations. Similar to analogous datasets, uCO3D contains annotations for 3D camera poses, depth maps and sparse point clouds. In addition, each object is equipped with a caption and a 3D Gaussian Splat reconstruction. We train several large 3D models on MVImgNet, CO3Dv2, and uCO3D and obtain superior results using the latter, showing that uCO3D is better for learning applications.
• 摘要：我们引入了 3D 中不常见物体 (uCO3D)，这是一个以物体为中心的新数据集，用于 3D 深度学习和 3D 生成式 AI。uCO3D 是最大的公开可用高分辨率物体视频集合，带有 3D 注释，可确保全方位 360 度覆盖。uCO3D 比 MVImgNet 和 CO3Dv2 更加多样化，涵盖了 1,000 多个物体类别。由于对收集的视频和 3D 注释进行了广泛的质量检查，因此它的质量也更高。与类似数据集类似，uCO3D 包含 3D 相机姿势、深度图和稀疏点云的注释。此外，每个对象都配备了标题和 3D 高斯 Splat 重建。我们在 MVImgNet、CO3Dv2 和 uCO3D 上训练了几个大型 3D 模型，并使用后者获得了更好的结果，表明 uCO3D 更适合学习应用。