2024-12-25

Title: LaMI-GO: Latent Mixture Integration for Goal-Oriented Communications Achieving High Spectrum Efficiency

• Authors: Achintha Wijesinghe, Suchinthaka Wanninayaka, Weiwei Wang, Yu-Chieh Chao, Songyang Zhang, Zhi Ding
• Subjects: cs.LG, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.17839
• Pdf URL: https://arxiv.org/pdf/2412.17839
• Copy Paste: [[2412.17839]] LaMI-GO: Latent Mixture Integration for Goal-Oriented Communications Achieving High Spectrum Efficiency(https://arxiv.org/abs/2412.17839)
• Keywords: generative
• Abstract: The recent rise of semantic-style communications includes the development of goal-oriented communications (GOCOMs) remarkably efficient multimedia information transmissions. The concept of GO-COMS leverages advanced artificial intelligence (AI) tools to address the rising demand for bandwidth efficiency in applications, such as edge computing and Internet-of-Things (IoT). Unlike traditional communication systems focusing on source data accuracy, GO-COMs provide intelligent message delivery catering to the special needs critical to accomplishing downstream tasks at the receiver. In this work, we present a novel GO-COM framework, namely LaMI-GO that utilizes emerging generative AI for better quality-of-service (QoS) with ultra-high communication efficiency. Specifically, we design our LaMI-GO system backbone based on a latent diffusion model followed by a vector-quantized generative adversarial network (VQGAN) for efficient latent embedding and information representation. The system trains a common feature codebook the receiver side. Our experimental results demonstrate substantial improvement in perceptual quality, accuracy of downstream tasks, and bandwidth consumption over the state-of-the-art GOCOM systems and establish the power of our proposed LaMI-GO communication framework.
• 摘要：语义式通信的近期兴起包括目标导向通信 (GOCOM) 的发展，这是一种非常高效的多媒体信息传输。GO-COMS 的概念利用先进的人工智能 (AI) 工具来满足边缘计算和物联网 (IoT) 等应用中对带宽效率日益增长的需求。与专注于源数据准确性的传统通信系统不同，GO-COM 提供智能消息传递，以满足在接收器完成下游任务的关键特殊需求。在这项工作中，我们提出了一个新颖的 GO-COM 框架，即 LaMI-GO，它利用新兴的生成式 AI 实现更好的服务质量 (QoS) 和超高的通信效率。具体来说，我们基于潜在扩散模型设计了我们的 LaMI-GO 系统主干，然后是矢量量化生成对抗网络 (VQGAN)，以实现高效的潜在嵌入和信息表示。该系统在接收端训练一个通用特征码本。我们的实验结果表明，与最先进的 GOCOM 系统相比，感知质量、下游任务的准确性和带宽消耗有了显著改善，并确立了我们提出的 LaMI-GO 通信框架的强大功能。

Title: Graph Structure Refinement with Energy-based Contrastive Learning

• Authors: Xianlin Zeng, Yufeng Wang, Yuqi Sun, Guodong Guo, Wenrui Ding, Baochang Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17856
• Pdf URL: https://arxiv.org/pdf/2412.17856
• Copy Paste: [[2412.17856]] Graph Structure Refinement with Energy-based Contrastive Learning(https://arxiv.org/abs/2412.17856)
• Keywords: generative
• Abstract: Graph Neural Networks (GNNs) have recently gained widespread attention as a successful tool for analyzing graph-structured data. However, imperfect graph structure with noisy links lacks enough robustness and may damage graph representations, therefore limiting the GNNs' performance in practical tasks. Moreover, existing generative architectures fail to fit discriminative graph-related tasks. To tackle these issues, we introduce an unsupervised method based on a joint of generative training and discriminative training to learn graph structure and representation, aiming to improve the discriminative performance of generative models. We propose an Energy-based Contrastive Learning (ECL) guided Graph Structure Refinement (GSR) framework, denoted as ECL-GSR. To our knowledge, this is the first work to combine energy-based models with contrastive learning for GSR. Specifically, we leverage ECL to approximate the joint distribution of sample pairs, which increases the similarity between representations of positive pairs while reducing the similarity between negative ones. Refined structure is produced by augmenting and removing edges according to the similarity metrics among node representations. Extensive experiments demonstrate that ECL-GSR outperforms \textit{the state-of-the-art on eight benchmark datasets} in node classification. ECL-GSR achieves \textit{faster training with fewer samples and memories} against the leading baseline, highlighting its simplicity and efficiency in downstream tasks.
• 摘要：图神经网络 (GNN) 近年来作为分析图结构数据的成功工具而受到广泛关注。然而，不完善的图结构和噪声链接缺乏足够的鲁棒性，可能会损坏图表示，因此限制了 GNN 在实际任务中的表现。此外，现有的生成架构无法适应判别性图相关任务。为了解决这些问题，我们引入了一种基于生成训练和判别训练联合的无监督方法来学习图结构和表示，旨在提高生成模型的判别性能。我们提出了一种基于能量的对比学习 (ECL) 引导的图结构细化 (GSR) 框架，表示为 ECL-GSR。据我们所知，这是第一项将基于能量的模型与对比学习相结合用于 GSR 的工作。具体而言，我们利用 ECL 来近似样本对的联合分布，这增加了正对表示之间的相似性，同时降低了负对之间的相似性。根据节点表示之间的相似性度量，通过增加和移除边来生成精细结构。大量实验表明，ECL-GSR 在节点分类方面的表现优于 \textit{八个基准数据集上的最新成果}。与领先的基线相比，ECL-GSR 实现了 \textit{使用更少的样本和内存进行更快的训练}，突出了其在下游任务中的简单性和效率。

Title: Trading Devil RL: Backdoor attack via Stock market, Bayesian Optimization and Reinforcement Learning

• Authors: Orson Mengara
• Subjects: cs.LG, cs.CE, physics.comp-ph, physics.soc-ph
• Abstract URL: https://arxiv.org/abs/2412.17908
• Pdf URL: https://arxiv.org/pdf/2412.17908
• Copy Paste: [[2412.17908]] Trading Devil RL: Backdoor attack via Stock market, Bayesian Optimization and Reinforcement Learning(https://arxiv.org/abs/2412.17908)
• Keywords: generative
• Abstract: With the rapid development of generative artificial intelligence, particularly large language models, a number of sub-fields of deep learning have made significant progress and are now very useful in everyday applications. For example, well-known financial institutions simulate a wide range of scenarios for various models created by their research teams using reinforcement learning, both before production and after regular operations. In this work, we propose a backdoor attack that focuses solely on data poisoning. This particular backdoor attack is classified as an attack without prior consideration or trigger, and we name it FinanceLLMsBackRL. Our aim is to examine the potential effects of large language models that use reinforcement learning systems for text production or speech recognition, finance, physics, or the ecosystem of contemporary artificial intelligence models.
• 摘要：随着生成式人工智能（尤其是大型语言模型）的快速发展，深度学习的许多子领域取得了重大进展，如今在日常应用中非常有用。例如，知名金融机构在生产前和正常运营后，使用强化学习模拟其研究团队创建的各种模型的各种场景。在这项工作中，我们提出了一种仅针对数据中毒的后门攻击。这种特定的后门攻击被归类为没有事先考虑或触发的攻击，我们将其命名为 FinanceLLMsBackRL。我们的目标是研究使用强化学习系统进行文本生成或语音识别、金融、物理或当代人工智能模型生态系统的大型语言模型的潜在影响。

Title: Hyperbolic Chamfer Distance for Point Cloud Completion and Beyond

• Authors: Fangzhou Lin, Songlin Hou, Haotian Liu, Shang Gao, Kazunori D Yamada, Haichong K. Zhang, Ziming Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17951
• Pdf URL: https://arxiv.org/pdf/2412.17951
• Copy Paste: [[2412.17951]] Hyperbolic Chamfer Distance for Point Cloud Completion and Beyond(https://arxiv.org/abs/2412.17951)
• Keywords: generative
• Abstract: Chamfer Distance (CD) is widely used as a metric to quantify difference between two point clouds. In point cloud completion, Chamfer Distance (CD) is typically used as a loss function in deep learning frameworks. However, it is generally acknowledged within the field that Chamfer Distance (CD) is vulnerable to the presence of outliers, which can consequently lead to the convergence on suboptimal models. In divergence from the existing literature, which largely concentrates on resolving such concerns in the realm of Euclidean space, we put forth a notably uncomplicated yet potent metric specifically designed for point cloud completion tasks: {Hyperbolic Chamfer Distance (HyperCD)}. This metric conducts Chamfer Distance computations within the parameters of hyperbolic space. During the backpropagation process, HyperCD systematically allocates greater weight to matched point pairs exhibiting reduced Euclidean distances. This mechanism facilitates the preservation of accurate point pair matches while permitting the incremental adjustment of suboptimal matches, thereby contributing to enhanced point cloud completion outcomes. Moreover, measure the shape dissimilarity is not solely work for point cloud completion task, we further explore its applications in other generative related tasks, including single image reconstruction from point cloud, and upsampling. We demonstrate state-of-the-art performance on the point cloud completion benchmark datasets, PCN, ShapeNet-55, and ShapeNet-34, and show from visualization that HyperCD can significantly improve the surface smoothness, we also provide the provide experimental results beyond completion task.
• 摘要：倒角距离 (CD) 被广泛用作量化两个点云之间差异的指标。在点云修复中，倒角距离 (CD) 通常用作深度学习框架中的损失函数。然而，业内普遍认为倒角距离 (CD) 容易受到异常值的影响，从而导致收敛于次优模型。与现有文献主要集中在解决欧几里得空间领域的此类问题不同，我们提出了一个专门为点云修复任务设计的非常简单但有效的指标：双曲倒角距离 (HyperCD) 。该指标在双曲空间参数内进行倒角距离计算。在反向传播过程中，HyperCD 系统地为表现出减小的欧几里得距离的匹配点对分配更大的权重。这种机制有利于保留精确的点对匹配，同时允许对次优匹配进行增量调整，从而有助于增强点云补全结果。此外，测量形状差异性不仅仅适用于点云补全任务，我们进一步探索其在其他生成相关任务中的应用，包括从点云重建单幅图像和上采样。我们在点云补全基准数据集 PCN、ShapeNet-55 和 ShapeNet-34 上展示了最先进的性能，并从可视化中表明 HyperCD 可以显著提高表面平滑度，我们还提供了补全任务之外的实验结果。

Title: ArchComplete: Autoregressive 3D Architectural Design Generation with Hierarchical Diffusion-Based Upsampling

• Authors: S. Rasoulzadeh, M. Bank, M. Wimmer, I. Kovacic, K. Schinegger, S. Rutzinger
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17957
• Pdf URL: https://arxiv.org/pdf/2412.17957
• Copy Paste: [[2412.17957]] ArchComplete: Autoregressive 3D Architectural Design Generation with Hierarchical Diffusion-Based Upsampling(https://arxiv.org/abs/2412.17957)
• Keywords: generation, generative
• Abstract: $\textit{ArchComplete}$ is a two-stage dense voxel-based 3D generative pipeline developed to tackle the high complexity in architectural geometries and topologies, assisting with ideation and geometric detailisation in the early design process. In stage 1, a $\textit{3D Voxel VQGAN}$ model is devised, whose composition is then modelled with an autoregressive transformer for generating coarse models. Subsequently, in stage 2, $\textit{Hierarchical Voxel Upsampling Networks}$ consisting of a set of 3D conditional denoising diffusion probabilistic models are defined to augment the coarse shapes with fine geometric details. The first stage is trained on a dataset of house models with fully modelled exteriors and interiors with a novel 2.5D perceptual loss to capture input complexities across multiple abstraction levels, while the second stage trains on randomly cropped local volumetric patches, requiring significantly less compute and memory. For inference, the pipeline first autoregressively generates house models at a resolution of $64^3$ and then progressively refines them to resolution of $256^3$ with voxel sizes as small as $18\text{cm}$. ArchComplete supports a range of interaction modes solving a variety of tasks, including interpolation, variation generation, unconditional synthesis, and two conditional synthesis tasks: shape completion and plan-drawing completion, as well as geometric detailisation. The results demonstrate notable improvements against state-of-the-art on established metrics.
• 摘要：$\textit{ArchComplete}$ 是一个两阶段的基于密集体素的 3D 生成流程，旨在解决建筑几何和拓扑的高复杂性，帮助在早期设计过程中进行构思和几何细节化。在第 1 阶段，设计了一个 $\textit{3D Voxel VQGAN}$ 模型，然后使用自回归变换器对其组成进行建模，以生成粗略模型。随后，在第 2 阶段，定义由一组 3D 条件去噪扩散概率模型组成的 $\textit{分层体素上采样网络}$，以用精细的几何细节增强粗糙形状。第一阶段在具有完全建模的外部和内部的房屋模型数据集上进行训练，采用新颖的 2.5D 感知损失来捕捉跨多个抽象级别的输入复杂性，而第二阶段在随机裁剪的局部体积块上进行训练，需要的计算和内存明显更少。为了进行推理，该流程首先自回归生成分辨率为 $64^3$ 的房屋模型，然后逐步将其细化到分辨率为 $256^3$，体素大小最小为 $18\text{cm}$。ArchComplete 支持一系列解决各种任务的交互模式，包括插值、变化生成、无条件合成和两个条件合成任务：形状完成和平面图完成，以及几何细节化。结果表明，在既定指标上，与最先进的技术相比有显著的改进。

Title: Data-driven Modeling of Parameterized Nonlinear Fluid Dynamical Systems with a Dynamics-embedded Conditional Generative Adversarial Network

• Authors: Abdolvahhab Rostamijavanani, Shanwu Li, Yongchao Yang
• Subjects: cs.LG, nlin.CD
• Abstract URL: https://arxiv.org/abs/2412.17978
• Pdf URL: https://arxiv.org/pdf/2412.17978
• Copy Paste: [[2412.17978]] Data-driven Modeling of Parameterized Nonlinear Fluid Dynamical Systems with a Dynamics-embedded Conditional Generative Adversarial Network(https://arxiv.org/abs/2412.17978)
• Keywords: generative
• Abstract: This work presents a data-driven solution to accurately predict parameterized nonlinear fluid dynamical systems using a dynamics-generator conditional GAN (Dyn-cGAN) as a surrogate model. The Dyn-cGAN includes a dynamics block within a modified conditional GAN, enabling the simultaneous identification of temporal dynamics and their dependence on system parameters. The learned Dyn-cGAN model takes into account the system parameters to predict the flow fields of the system accurately. We evaluate the effectiveness and limitations of the developed Dyn-cGAN through numerical studies of various parameterized nonlinear fluid dynamical systems, including flow over a cylinder and a 2-D cavity problem, with different Reynolds numbers. Furthermore, we examine how Reynolds number affects the accuracy of the predictions for both case studies. Additionally, we investigate the impact of the number of time steps involved in the process of dynamics block training on the accuracy of predictions, and we find that an optimal value exists based on errors and mutual information relative to the ground truth.
• 摘要：本研究提出了一种数据驱动的解决方案，使用动态生成器条件 GAN (Dyn-cGAN) 作为替代模型，准确预测参数化的非线性流体动力系统。Dyn-cGAN 在经过修改的条件 GAN 中包含一个动态块，从而能够同时识别时间动态及其对系统参数的依赖性。学习到的 Dyn-cGAN 模型会考虑系统参数，以准确预测系统的流场。我们通过对具有不同雷诺数的各种参数化的非线性流体动力系统（包括圆柱体上的流动和二维空腔问题）进行数值研究，评估了开发的 Dyn-cGAN 的有效性和局限性。此外，我们研究了雷诺数如何影响这两个案例研究的预测准确性。此外，我们研究了动态块训练过程中涉及的时间步数对预测准确性的影响，我们发现基于相对于基本事实的误差和相互信息，存在一个最佳值。

Title: AA-SGAN: Adversarially Augmented Social GAN with Synthetic Data

• Authors: Mirko Zaffaroni, Federico Signoretta, Marco Grangetto, Attilio Fiandrotti
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18038
• Pdf URL: https://arxiv.org/pdf/2412.18038
• Copy Paste: [[2412.18038]] AA-SGAN: Adversarially Augmented Social GAN with Synthetic Data(https://arxiv.org/abs/2412.18038)
• Keywords: generative
• Abstract: Accurately predicting pedestrian trajectories is crucial in applications such as autonomous driving or service robotics, to name a few. Deep generative models achieve top performance in this task, assuming enough labelled trajectories are available for training. To this end, large amounts of synthetically generated, labelled trajectories exist (e.g., generated by video games). However, such trajectories are not meant to represent pedestrian motion realistically and are ineffective at training a predictive model. We propose a method and an architecture to augment synthetic trajectories at training time and with an adversarial approach. We show that trajectory augmentation at training time unleashes significant gains when a state-of-the-art generative model is evaluated over real-world trajectories.
• 摘要：准确预测行人轨迹对于自动驾驶或服务机器人等应用至关重要。假设有足够的标记轨迹可供训练，深度生成模型在此任务中可实现最佳性能。为此，存在大量合成生成的标记轨迹（例如，由视频游戏生成）。但是，此类轨迹并非旨在真实地表示行人运动，并且无法有效训练预测模型。我们提出了一种方法和一种架构，用于在训练时和使用对抗性方法增强合成轨迹。我们表明，当使用现实世界轨迹评估最先进的生成模型时，在训练时增强轨迹可带来显着的收益。

Title: An Ensemble Approach to Short-form Video Quality Assessment Using Multimodal LLM

• Authors: Wen Wen, Yilin Wang, Neil Birkbeck, Balu Adsumilli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18060
• Pdf URL: https://arxiv.org/pdf/2412.18060
• Copy Paste: [[2412.18060]] An Ensemble Approach to Short-form Video Quality Assessment Using Multimodal LLM(https://arxiv.org/abs/2412.18060)
• Keywords: quality assessment
• Abstract: The rise of short-form videos, characterized by diverse content, editing styles, and artifacts, poses substantial challenges for learning-based blind video quality assessment (BVQA) models. Multimodal large language models (MLLMs), renowned for their superior generalization capabilities, present a promising solution. This paper focuses on effectively leveraging a pretrained MLLM for short-form video quality assessment, regarding the impacts of pre-processing and response variability, and insights on combining the MLLM with BVQA models. We first investigated how frame pre-processing and sampling techniques influence the MLLM's performance. Then, we introduced a lightweight learning-based ensemble method that adaptively integrates predictions from the MLLM and state-of-the-art BVQA models. Our results demonstrated superior generalization performance with the proposed ensemble approach. Furthermore, the analysis of content-aware ensemble weights highlighted that some video characteristics are not fully represented by existing BVQA models, revealing potential directions to improve BVQA models further.
• 摘要：短视频的兴起，以多样化的内容、编辑风格和伪像为特征，对基于学习的盲视频质量评估 (BVQA) 模型提出了巨大的挑战。多模态大型语言模型 (MLLM) 以其卓越的泛化能力而闻名，提供了一种有前途的解决方案。本文重点介绍如何有效地利用预训练的 MLLM 进行短视频质量评估，涉及预处理和响应变化的影响，以及将 MLLM 与 BVQA 模型相结合的见解。我们首先研究了帧预处理和采样技术如何影响 MLLM 的性能。然后，我们引入了一种基于学习的轻量级集成方法，该方法自适应地集成了 MLLM 和最先进的 BVQA 模型的预测。我们的结果表明，所提出的集成方法具有卓越的泛化性能。此外，对内容感知集成权重的分析强调，现有 BVQA 模型无法完全代表某些视频特征，从而揭示了进一步改进 BVQA 模型的潜在方向。

Title: Beyond the Known: Enhancing Open Set Domain Adaptation with Unknown Exploration

• Authors: Lucas Fernando Alvarenga e Silva, Samuel Felipe dos Santos, Nicu Sebe, Jurandy Almeida
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18105
• Pdf URL: https://arxiv.org/pdf/2412.18105
• Copy Paste: [[2412.18105]] Beyond the Known: Enhancing Open Set Domain Adaptation with Unknown Exploration(https://arxiv.org/abs/2412.18105)
• Keywords: generative
• Abstract: Convolutional neural networks (CNNs) can learn directly from raw data, resulting in exceptional performance across various research areas. However, factors present in non-controllable environments such as unlabeled datasets with varying levels of domain and category shift can reduce model accuracy. The Open Set Domain Adaptation (OSDA) is a challenging problem that arises when both of these issues occur together. Existing OSDA approaches in literature only align known classes or use supervised training to learn unknown classes as a single new category. In this work, we introduce a new approach to improve OSDA techniques by extracting a set of high-confidence unknown instances and using it as a hard constraint to tighten the classification boundaries. Specifically, we use a new loss constraint that is evaluated in three different ways: (1) using pristine negative instances directly; (2) using data augmentation techniques to create randomly transformed negatives; and (3) with generated synthetic negatives containing adversarial features. We analyze different strategies to improve the discriminator and the training of the Generative Adversarial Network (GAN) used to generate synthetic negatives. We conducted extensive experiments and analysis on OVANet using three widely-used public benchmarks, the Office-31, Office-Home, and VisDA datasets. We were able to achieve similar H-score to other state-of-the-art methods, while increasing the accuracy on unknown categories.
• 摘要：卷积神经网络 (CNN) 可以直接从原始数据中学习，从而在各个研究领域中表现出色。然而，不可控环境中存在的因素（例如具有不同程度的域和类别偏移的未标记数据集）会降低模型准确性。开放集域自适应 (OSDA) 是一个具有挑战性的问题，当这两个问题同时出现时就会出现。文献中现有的 OSDA 方法仅对齐已知类别，或使用监督训练将未知类别学习为单个新类别。在这项工作中，我们引入了一种新方法来改进 OSDA 技术，方法是提取一组高置信度的未知实例并将其用作硬约束来收紧分类边界。具体而言，我们使用一种新的损失约束，该约束以三种不同的方式进行评估：(1) 直接使用原始负例；(2) 使用数据增强技术创建随机转换的负例；(3) 使用包含对抗特征的生成合成负例。我们分析了不同的策略来改进用于生成合成负例的鉴别器和生成对抗网络 (GAN) 的训练。我们使用三个广泛使用的公共基准数据集（Office-31、Office-Home 和 VisDA 数据集）对 OVANet 进行了广泛的实验和分析。我们能够实现与其他最先进方法相似的 H 分数，同时提高未知类别的准确率。

Title: Dense-Face: Personalized Face Generation Model via Dense Annotation Prediction

• Authors: Xiao Guo, Manh Tran, Jiaxin Cheng, Xiaoming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18149
• Pdf URL: https://arxiv.org/pdf/2412.18149
• Copy Paste: [[2412.18149]] Dense-Face: Personalized Face Generation Model via Dense Annotation Prediction(https://arxiv.org/abs/2412.18149)
• Keywords: generation
• Abstract: The text-to-image (T2I) personalization diffusion model can generate images of the novel concept based on the user input text caption. However, existing T2I personalized methods either require test-time fine-tuning or fail to generate images that align well with the given text caption. In this work, we propose a new T2I personalization diffusion model, Dense-Face, which can generate face images with a consistent identity as the given reference subject and align well with the text caption. Specifically, we introduce a pose-controllable adapter for the high-fidelity image generation while maintaining the text-based editing ability of the pre-trained stable diffusion (SD). Additionally, we use internal features of the SD UNet to predict dense face annotations, enabling the proposed method to gain domain knowledge in face generation. Empirically, our method achieves state-of-the-art or competitive generation performance in image-text alignment, identity preservation, and pose control.
• 摘要：文本到图像 (T2I) 个性化扩散模型可以根据用户输入的文本标题生成新概念的图像。然而，现有的 T2I 个性化方法要么需要测试时微调，要么无法生成与给定文本标题很好地对齐的图像。在这项工作中，我们提出了一种新的 T2I 个性化扩散模型 Dense-Face，它可以生成与给定参考主体具有一致身份的人脸图像，并与文本标题很好地对齐。具体来说，我们引入了一个姿势可控适配器来生成高保真图像，同时保持预训练稳定扩散 (SD) 的基于文本的编辑能力。此外，我们使用 SD UNet 的内部特征来预测密集的人脸注释，使所提出的方法能够获得人脸生成领域的知识。从经验上看，我们的方法在图像文本对齐、身份保存和姿势控制方面实现了最先进或具有竞争力的生成性能。

Title: EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation

• Authors: Shuhao Han, Haotian Fan, Jiachen Fu, Liang Li, Tao Li, Junhui Cui, Yunqiu Wang, Yang Tai, Jingwei Sun, Chunle Guo, Chongyi Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18150
• Pdf URL: https://arxiv.org/pdf/2412.18150
• Copy Paste: [[2412.18150]] EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation(https://arxiv.org/abs/2412.18150)
• Keywords: generation, generative
• Abstract: Recently, Text-to-Image (T2I) generation models have achieved significant advancements. Correspondingly, many automated metrics have emerged to evaluate the image-text alignment capabilities of generative models. However, the performance comparison among these automated metrics is limited by existing small datasets. Additionally, these datasets lack the capacity to assess the performance of automated metrics at a fine-grained level. In this study, we contribute an EvalMuse-40K benchmark, gathering 40K image-text pairs with fine-grained human annotations for image-text alignment-related tasks. In the construction process, we employ various strategies such as balanced prompt sampling and data re-annotation to ensure the diversity and reliability of our benchmark. This allows us to comprehensively evaluate the effectiveness of image-text alignment metrics for T2I models. Meanwhile, we introduce two new methods to evaluate the image-text alignment capabilities of T2I models: FGA-BLIP2 which involves end-to-end fine-tuning of a vision-language model to produce fine-grained image-text alignment scores and PN-VQA which adopts a novel positive-negative VQA manner in VQA models for zero-shot fine-grained evaluation. Both methods achieve impressive performance in image-text alignment evaluations. We also use our methods to rank current AIGC models, in which the results can serve as a reference source for future study and promote the development of T2I generation. The data and code will be made publicly available.
• 摘要：最近，文本到图像 (T2I) 生成模型取得了重大进展。相应地，许多自动化指标也应运而生，用于评估生成模型的图像-文本对齐能力。然而，这些自动化指标之间的性能比较受到现有小型数据集的限制。此外，这些数据集缺乏在细粒度级别上评估自动化指标性能的能力。在本研究中，我们贡献了一个 EvalMuse-40K 基准，为与图像-文本对齐相关的任务收集了 40K 个带有细粒度人工注释的图像-文本对。在构建过程中，我们采用了平衡快速采样和数据重新注释等各种策略来确保基准的多样性和可靠性。这使我们能够全面评估 T2I 模型的图像-文本对齐指标的有效性。同时，我们引入了两种新方法来评估 T2I 模型的图像-文本对齐能力：FGA-BLIP2 涉及对视觉语言模型进行端到端微调以产生细粒度的图像-文本对齐分数，而 PN-VQA 则在 VQA 模型中采用一种新颖的正负 VQA 方式进行零样本细粒度评估。这两种方法在图像-文本对齐评估中都取得了令人印象深刻的表现。我们还使用我们的方法对当前的 AIGC 模型进行排名，其中的结果可以作为未来研究的参考来源并促进 T2I 生成的发展。数据和代码将公开提供。

Title: DepthLab: From Partial to Complete

• Authors: Zhiheng Liu, Ka Leong Cheng, Qiuyu Wang, Shuzhe Wang, Hao Ouyang, Bin Tan, Kai Zhu, Yujun Shen, Qifeng Chen, Ping Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18153
• Pdf URL: https://arxiv.org/pdf/2412.18153
• Copy Paste: [[2412.18153]] DepthLab: From Partial to Complete(https://arxiv.org/abs/2412.18153)
• Keywords: generation
• Abstract: Missing values remain a common challenge for depth data across its wide range of applications, stemming from various causes like incomplete data acquisition and perspective alteration. This work bridges this gap with DepthLab, a foundation depth inpainting model powered by image diffusion priors. Our model features two notable strengths: (1) it demonstrates resilience to depth-deficient regions, providing reliable completion for both continuous areas and isolated points, and (2) it faithfully preserves scale consistency with the conditioned known depth when filling in missing values. Drawing on these advantages, our approach proves its worth in various downstream tasks, including 3D scene inpainting, text-to-3D scene generation, sparse-view reconstruction with DUST3R, and LiDAR depth completion, exceeding current solutions in both numerical performance and visual quality. Our project page with source code is available at this https URL.
• 摘要：缺失值仍然是深度数据在其广泛应用中面临的常见挑战，其原因多种多样，例如数据采集不完整和视角改变。这项工作通过 DepthLab 弥补了这一空白，DepthLab 是一个由图像扩散先验驱动的基础深度修复模型。我们的模型具有两个显着的优势：（1）它对深度不足区域具有弹性，为连续区域和孤立点提供可靠的补全，（2）在填充缺失值时，它忠实地保持了与条件已知深度的比例一致性。利用这些优势，我们的方法在各种下游任务中证明了其价值，包括 3D 场景修复、文本到 3D 场景生成、使用 DUST3R 进行稀疏视图重建和 LiDAR 深度补全，在数值性能和视觉质量方面都超越了当前的解决方案。我们的项目页面及其源代码可在此 https URL 上找到。

Title: Semantics Disentanglement and Composition for Versatile Codec toward both Human-eye Perception and Machine Vision Task

• Authors: Jinming Liu, Yuntao Wei, Junyan Lin, Shengyang Zhao, Heming Sun, Zhibo Chen, Wenjun Zeng, Xin Jin
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.18158
• Pdf URL: https://arxiv.org/pdf/2412.18158
• Copy Paste: [[2412.18158]] Semantics Disentanglement and Composition for Versatile Codec toward both Human-eye Perception and Machine Vision Task(https://arxiv.org/abs/2412.18158)
• Keywords: generative
• Abstract: While learned image compression methods have achieved impressive results in either human visual perception or machine vision tasks, they are often specialized only for one domain. This drawback limits their versatility and generalizability across scenarios and also requires retraining to adapt to new applications-a process that adds significant complexity and cost in real-world scenarios. In this study, we introduce an innovative semantics DISentanglement and COmposition VERsatile codec (DISCOVER) to simultaneously enhance human-eye perception and machine vision tasks. The approach derives a set of labels per task through multimodal large models, which grounding models are then applied for precise localization, enabling a comprehensive understanding and disentanglement of image components at the encoder side. At the decoding stage, a comprehensive reconstruction of the image is achieved by leveraging these encoded components alongside priors from generative models, thereby optimizing performance for both human visual perception and machine-based analytical tasks. Extensive experimental evaluations substantiate the robustness and effectiveness of DISCOVER, demonstrating superior performance in fulfilling the dual objectives of human and machine vision requirements.
• 摘要：虽然学习图像压缩方法在人类视觉感知或机器视觉任务中取得了令人印象深刻的成果，但它们通常只针对一个领域。这一缺点限制了它们在各种场景中的多功能性和通用性，并且需要重新训练以适应新的应用——这一过程在现实场景中增加了显著的复杂性和成本。在本研究中，我们引入了一种创新的语义解缠结和组合多功能编解码器 (DISCOVER)，以同时增强人眼感知和机器视觉任务。该方法通过多模态大型模型为每个任务派生一组标签，然后应用这些基础模型进行精确定位，从而实现对编码器端图像组件的全面理解和解缠结。在解码阶段，通过利用这些编码组件以及生成模型的先验来实现图像的全面重建，从而优化人类视觉感知和基于机器的分析任务的性能。大量的实验评估证实了 DISCOVER 的稳健性和有效性，证明了在满足人类和机器视觉要求的双重目标方面具有卓越的性能。

Title: Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence

• Authors: Yinbin Han, Meisam Razaviyayn, Renyuan Xu
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2412.18164
• Pdf URL: https://arxiv.org/pdf/2412.18164
• Copy Paste: [[2412.18164]] Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence(https://arxiv.org/abs/2412.18164)
• Keywords: generative
• Abstract: Diffusion models have emerged as powerful tools for generative modeling, demonstrating exceptional capability in capturing target data distributions from large datasets. However, fine-tuning these massive models for specific downstream tasks, constraints, and human preferences remains a critical challenge. While recent advances have leveraged reinforcement learning algorithms to tackle this problem, much of the progress has been empirical, with limited theoretical understanding. To bridge this gap, we propose a stochastic control framework for fine-tuning diffusion models. Building on denoising diffusion probabilistic models as the pre-trained reference dynamics, our approach integrates linear dynamics control with Kullback-Leibler regularization. We establish the well-posedness and regularity of the stochastic control problem and develop a policy iteration algorithm (PI-FT) for numerical solution. We show that PI-FT achieves global convergence at a linear rate. Unlike existing work that assumes regularities throughout training, we prove that the control and value sequences generated by the algorithm maintain the regularity. Additionally, we explore extensions of our framework to parametric settings and continuous-time formulations.
• 摘要：扩散模型已成为生成建模的强大工具，展示了从大型数据集中捕获目标数据分布的卓越能力。然而，针对特定的下游任务、约束和人类偏好对这些大型模型进行微调仍然是一个关键挑战。虽然最近的进展利用了强化学习算法来解决这个问题，但大部分进展都是经验性的，理论理解有限。为了弥补这一差距，我们提出了一个用于微调扩散模型的随机控制框架。以去噪扩散概率模型作为预训练参考动力学为基础，我们的方法将线性动力学控制与 Kullback-Leibler 正则化相结合。我们建立了随机控制问题的适定性和规律性，并开发了一种用于数值解的策略迭代算法 (PI-FT)。我们表明 PI-FT 以线性速率实现全局收敛。与现有在整个训练过程中假设规律性的工作不同，我们证明算法生成的控制和值序列保持了规律性。此外，我们还探索了我们的框架向参数设置和连续时间公式的扩展。

Title: TextMatch: Enhancing Image-Text Consistency Through Multimodal Optimization

• Authors: Yucong Luo, Mingyue Cheng, Jie Ouyang, Xiaoyu Tao, Qi Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18185
• Pdf URL: https://arxiv.org/pdf/2412.18185
• Copy Paste: [[2412.18185]] TextMatch: Enhancing Image-Text Consistency Through Multimodal Optimization(https://arxiv.org/abs/2412.18185)
• Keywords: generation, generative
• Abstract: Text-to-image generative models excel in creating images from text but struggle with ensuring alignment and consistency between outputs and prompts. This paper introduces TextMatch, a novel framework that leverages multimodal optimization to address image-text discrepancies in text-to-image (T2I) generation and editing. TextMatch employs a scoring strategy powered by large language models (LLMs) and visual question-answering (VQA) models to evaluate semantic consistency between prompts and generated images. By integrating multimodal in-context learning and chain of thought reasoning, our method dynamically refines prompts through iterative optimization. This process ensures that the generated images better capture user intent of, resulting in higher fidelity and relevance. Extensive experiments demonstrate that TextMatch significantly improves text-image consistency across multiple benchmarks, establishing a reliable framework for advancing the capabilities of text-to-image generative models. Our code is available at this https URL.
• 摘要：文本到图像生成模型擅长从文本创建图像，但在确保输出和提示之间的对齐和一致性方面却举步维艰。本文介绍了 TextMatch，这是一个利用多模态优化来解决文本到图像 (T2I) 生成和编辑中的图像文本差异的新框架。TextMatch 采用由大型语言模型 (LLM) 和视觉问答 (VQA) 模型提供支持的评分策略来评估提示和生成的图像之间的语义一致性。通过整合多模态上下文学习和思路链推理，我们的方法通过迭代优化动态细化提示。此过程可确保生成的图像更好地捕捉用户意图，从而实现更高的保真度和相关性。大量实验表明，TextMatch 显著提高了多个基准测试中的文本图像一致性，为提高文本到图像生成模型的功能建立了可靠的框架。我们的代码可在此 https URL 上找到。

Title: Accelerating AIGC Services with Latent Action Diffusion Scheduling in Edge Networks

• Authors: Changfu Xu, Jianxiong Guo, Wanyu Lin, Haodong Zou, Wentao Fan, Tian Wang, Xiaowen Chu, Jiannong Cao
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.18212
• Pdf URL: https://arxiv.org/pdf/2412.18212
• Copy Paste: [[2412.18212]] Accelerating AIGC Services with Latent Action Diffusion Scheduling in Edge Networks(https://arxiv.org/abs/2412.18212)
• Keywords: generation
• Abstract: Artificial Intelligence Generated Content (AIGC) has gained significant popularity for creating diverse content. Current AIGC models primarily focus on content quality within a centralized framework, resulting in a high service delay and negative user experiences. However, not only does the workload of an AIGC task depend on the AIGC model's complexity rather than the amount of data, but the large model and its multi-layer encoder structure also result in a huge demand for computational and memory resources. These unique characteristics pose new challenges in its modeling, deployment, and scheduling at edge networks. Thus, we model an offloading problem among edges for providing real AIGC services and propose LAD-TS, a novel Latent Action Diffusion-based Task Scheduling method that orchestrates multiple edge servers for expedited AIGC services. The LAD-TS generates a near-optimal offloading decision by leveraging the diffusion model's conditional generation capability and the reinforcement learning's environment interaction ability, thereby minimizing the service delays under multiple resource constraints. Meanwhile, a latent action diffusion strategy is designed to guide decision generation by utilizing historical action probability, enabling rapid achievement of near-optimal decisions. Furthermore, we develop DEdgeAI, a prototype edge system with a refined AIGC model deployment to implement and evaluate our LAD-TS method. DEdgeAI provides a real AIGC service for users, demonstrating up to 29.18% shorter service delays than the current five representative AIGC platforms. We release our open-source code at this https URL.
• 摘要：人工智能生成内容 (AIGC) 因创建多样化内容而广受欢迎。当前的 AIGC 模型主要关注集中式框架内的内容质量，导致服务延迟高和用户体验不佳。然而，AIGC 任务的工作量不仅取决于 AIGC 模型的复杂性而非数据量，而且大型模型及其多层编码器结构也导致对计算和内存资源的巨大需求。这些独特的特性对其在边缘网络的建模、部署和调度提出了新的挑战。因此，我们为提供真正的 AIGC 服务，在边缘之间建模卸载问题，并提出了 LAD-TS，这是一种新颖的基于潜在动作扩散的任务调度方法，可协调多个边缘服务器以加快 AIGC 服务。LAD-TS 通过利用扩散模型的条件生成能力和强化学习的环境交互能力来生成近乎最优的卸载决策，从而在多种资源约束下最大限度地减少服务延迟。同时，设计了一种潜在动作扩散策略来利用历史动作概率来指导决策生成，从而能够快速实现近乎最优的决策。此外，我们开发了 DEdgeAI，这是一个原型边缘系统，具有完善的 AIGC 模型部署，用于实现和评估我们的 LAD-TS 方法。DEdgeAI 为用户提供了真正的 AIGC 服务，与当前五个代表性 AIGC 平台相比，其服务延迟缩短了 29.18%。我们在此 https URL 上发布了我们的开源代码。

Title: ICM-Assistant: Instruction-tuning Multimodal Large Language Models for Rule-based Explainable Image Content Moderation

• Authors: Mengyang Wu, Yuzhi Zhao, Jialun Cao, Mingjie Xu, Zhongming Jiang, Xuehui Wang, Qinbin Li, Guangneng Hu, Shengchao Qin, Chi-Wing Fu
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18216
• Pdf URL: https://arxiv.org/pdf/2412.18216
• Copy Paste: [[2412.18216]] ICM-Assistant: Instruction-tuning Multimodal Large Language Models for Rule-based Explainable Image Content Moderation(https://arxiv.org/abs/2412.18216)
• Keywords: generation
• Abstract: Controversial contents largely inundate the Internet, infringing various cultural norms and child protection standards. Traditional Image Content Moderation (ICM) models fall short in producing precise moderation decisions for diverse standards, while recent multimodal large language models (MLLMs), when adopted to general rule-based ICM, often produce classification and explanation results that are inconsistent with human moderators. Aiming at flexible, explainable, and accurate ICM, we design a novel rule-based dataset generation pipeline, decomposing concise human-defined rules and leveraging well-designed multi-stage prompts to enrich short explicit image annotations. Our ICM-Instruct dataset includes detailed moderation explanation and moderation Q-A pairs. Built upon it, we create our ICM-Assistant model in the framework of rule-based ICM, making it readily applicable in real practice. Our ICM-Assistant model demonstrates exceptional performance and flexibility. Specifically, it significantly outperforms existing approaches on various sources, improving both the moderation classification (36.8\% on average) and moderation explanation quality (26.6\% on average) consistently over existing MLLMs. Code/Data is available at this https URL.
• 摘要：大量有争议的内容充斥互联网，侵犯了各种文化规范和儿童保护标准。传统的图像内容审核 (ICM) 模型无法针对各种标准做出精准的审核决策，而最近的多模态大型语言模型 (MLLM) 在应用于基于规则的通用 ICM 时，通常会产生与人类审核员不一致的分类和解释结果。为了实现灵活、可解释和准确的 ICM，我们设计了一种新颖的基于规则的数据集生成流程，分解简明的人为定义规则并利用精心设计的多阶段提示来丰富简短的显式图像注释。我们的 ICM-Instruct 数据集包括详细的审核解释和审核问答对。在此基础上，我们在基于规则的 ICM 框架中创建了我们的 ICM-Assistant 模型，使其可轻松应用于实际。我们的 ICM-Assistant 模型表现出卓越的性能和灵活性。具体而言，它在各种来源上的表现都显著优于现有方法，与现有 MLLM 相比，其审核分类质量（平均 36.8% ）和审核解释质量（平均 26.6% ）均持续提高。代码/数据可在此 https URL 上获取。

Title: Sch\"odinger Bridge Type Diffusion Models as an Extension of Variational Autoencoders

• Authors: Kentaro Kaba, Reo Shimizu, Masayuki Ohzeki, Yuki Sughiyama
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18237
• Pdf URL: https://arxiv.org/pdf/2412.18237
• Copy Paste: [[2412.18237]] Sch\"odinger Bridge Type Diffusion Models as an Extension of Variational Autoencoders(https://arxiv.org/abs/2412.18237)
• Keywords: generative
• Abstract: Generative diffusion models use time-forward and backward stochastic differential equations to connect the data and prior distributions. While conventional diffusion models (e.g., score-based models) only learn the backward process, more flexible frameworks have been proposed to also learn the forward process by employing the Schrödinger bridge (SB). However, due to the complexity of the mathematical structure behind SB-type models, we can not easily give an intuitive understanding of their objective function. In this work, we propose a unified framework to construct diffusion models by reinterpreting the SB-type models as an extension of variational autoencoders. In this context, the data processing inequality plays a crucial role. As a result, we find that the objective function consists of the prior loss and drift matching parts.
• 摘要：生成扩散模型使用时间前向和后向随机微分方程来连接数据和先验分布。虽然传统的扩散模型（例如基于分数的模型）只学习后向过程，但已经提出了更灵活的框架来通过使用薛定谔桥（SB）来学习前向过程。然而，由于 SB 型模型背后的数学结构的复杂性，我们无法轻易地直观地理解它们的目标函数。在这项工作中，我们提出了一个统一的框架来构建扩散模型，方法是将 SB 型模型重新解释为变分自动编码器的扩展。在这种情况下，数据处理不等式起着至关重要的作用。结果，我们发现目标函数由先验损失和漂移匹配部分组成。

Title: AdaCo: Overcoming Visual Foundation Model Noise in 3D Semantic Segmentation via Adaptive Label Correction

• Authors: Pufan Zou, Shijia Zhao, Weijie Huang, Qiming Xia, Chenglu Wen, Wei Li, Cheng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18255
• Pdf URL: https://arxiv.org/pdf/2412.18255
• Copy Paste: [[2412.18255]] AdaCo: Overcoming Visual Foundation Model Noise in 3D Semantic Segmentation via Adaptive Label Correction(https://arxiv.org/abs/2412.18255)
• Keywords: generation
• Abstract: Recently, Visual Foundation Models (VFMs) have shown a remarkable generalization performance in 3D perception tasks. However, their effectiveness in large-scale outdoor datasets remains constrained by the scarcity of accurate supervision signals, the extensive noise caused by variable outdoor conditions, and the abundance of unknown objects. In this work, we propose a novel label-free learning method, Adaptive Label Correction (AdaCo), for 3D semantic segmentation. AdaCo first introduces the Cross-modal Label Generation Module (CLGM), providing cross-modal supervision with the formidable interpretive capabilities of the VFMs. Subsequently, AdaCo incorporates the Adaptive Noise Corrector (ANC), updating and adjusting the noisy samples within this supervision iteratively during training. Moreover, we develop an Adaptive Robust Loss (ARL) function to modulate each sample's sensitivity to noisy supervision, preventing potential underfitting issues associated with robust loss. Our proposed AdaCo can effectively mitigate the performance limitations of label-free learning networks in 3D semantic segmentation tasks. Extensive experiments on two outdoor benchmark datasets highlight the superior performance of our method.
• 摘要：最近，视觉基础模型 (VFM) 在 3D 感知任务中表现出了出色的泛化性能。然而，它们在大型户外数据集中的有效性仍然受到精确监督信号稀缺、多变的户外条件造成的大量噪声以及未知物体丰富的限制。在这项工作中，我们提出了一种用于 3D 语义分割的新型无标签学习方法，自适应标签校正 (AdaCo)。AdaCo 首先引入了跨模态标签生成模块 (CLGM)，为跨模态监督提供了 VFM 强大的解释能力。随后，AdaCo 结合了自适应噪声校正器 (ANC)，在训练期间迭代更新和调整此监督中的噪声样本。此外，我们开发了一个自适应稳健损失 (ARL) 函数来调节每个样本对噪声监督的敏感度，防止与稳健损失相关的潜在欠拟合问题。我们提出的 AdaCo 可以有效地缓解无标签学习网络在 3D 语义分割任务中的性能限制。在两个户外基准数据集上进行的大量实验凸显了我们方法的卓越性能。

Title: UNet--: Memory-Efficient and Feature-Enhanced Network Architecture based on U-Net with Reduced Skip-Connections

• Authors: Lingxiao Yin, Wei Tao, Dongyue Zhao, Tadayuki Ito, Kinya Osa, Masami Kato, Tse-Wei Chen
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.18276
• Pdf URL: https://arxiv.org/pdf/2412.18276
• Copy Paste: [[2412.18276]] UNet--: Memory-Efficient and Feature-Enhanced Network Architecture based on U-Net with Reduced Skip-Connections(https://arxiv.org/abs/2412.18276)
• Keywords: restoration
• Abstract: U-Net models with encoder, decoder, and skip-connections components have demonstrated effectiveness in a variety of vision tasks. The skip-connections transmit fine-grained information from the encoder to the decoder. It is necessary to maintain the feature maps used by the skip-connections in memory before the decoding stage. Therefore, they are not friendly to devices with limited resource. In this paper, we propose a universal method and architecture to reduce the memory consumption and meanwhile generate enhanced feature maps to improve network performance. To this end, we design a simple but effective Multi-Scale Information Aggregation Module (MSIAM) in the encoder and an Information Enhancement Module (IEM) in the decoder. The MSIAM aggregates multi-scale feature maps into single-scale with less memory. After that, the aggregated feature maps can be expanded and enhanced to multi-scale feature maps by the IEM. By applying the proposed method on NAFNet, a SOTA model in the field of image restoration, we design a memory-efficient and feature-enhanced network architecture, UNet--. The memory demand by the skip-connections in the UNet-- is reduced by 93.3%, while the performance is improved compared to NAFNet. Furthermore, we show that our proposed method can be generalized to multiple visual tasks, with consistent improvements in both memory consumption and network accuracy compared to the existing efficient architectures.
• 摘要：具有编码器、解码器和跳过连接组件的 U-Net 模型已在各种视觉任务中证明是有效的。跳过连接将细粒度信息从编码器传输到解码器。在解码阶段之前，需要在内存中维护跳过连接使用的特征图。因此，它们对资源有限的设备不友好。在本文中，我们提出了一种通用的方法和架构来减少内存消耗，同时生成增强的特征图来提高网络性能。为此，我们在编码器中设计了一个简单但有效的多尺度信息聚合模块 (MSIAM)，在解码器中设计了一个信息增强模块 (IEM)。MSIAM 以较少的内存将多尺度特征图聚合为单尺度。之后，聚合的特征图可以通过 IEM 扩展和增强为多尺度特征图。通过将所提出的方法应用于图像恢复领域的 SOTA 模型 NAFNet，我们设计了一个内存高效且特征增强的网络架构 UNet--。 UNet-- 中跳过连接的内存需求减少了 93.3%，而性能与 NAFNet 相比有所提高。此外，我们表明，我们提出的方法可以推广到多个视觉任务，与现有的高效架构相比，内存消耗和网络准确性都有持续的改进。

Title: Improved Feature Generating Framework for Transductive Zero-shot Learning

• Authors: Zihan Ye, Xinyuan Ru, Shiming Chen, Yaochu Jin, Kaizhu Huang, Xiaobo Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18282
• Pdf URL: https://arxiv.org/pdf/2412.18282
• Copy Paste: [[2412.18282]] Improved Feature Generating Framework for Transductive Zero-shot Learning(https://arxiv.org/abs/2412.18282)
• Keywords: generation, generative
• Abstract: Feature Generative Adversarial Networks have emerged as powerful generative models in producing high-quality representations of unseen classes within the scope of Zero-shot Learning (ZSL). This paper delves into the pivotal influence of unseen class priors within the framework of transductive ZSL (TZSL) and illuminates the finding that even a marginal prior bias can result in substantial accuracy declines. Our extensive analysis uncovers that this inefficacy fundamentally stems from the utilization of an unconditional unseen discriminator - a core component in existing TZSL. We further establish that the detrimental effects of this component are inevitable unless the generator perfectly fits class-specific distributions. Building on these insights, we introduce our Improved Feature Generation Framework, termed I-VAEGAN, which incorporates two novel components: Pseudo-conditional Feature Adversarial (PFA) learning and Variational Embedding Regression (VER). PFA circumvents the need for prior estimation by explicitly injecting the predicted semantics as pseudo conditions for unseen classes premised by precise semantic regression. Meanwhile, VER utilizes reconstructive pre-training to learn class statistics, obtaining better semantic regression. Our I-VAEGAN achieves state-of-the-art TZSL accuracy across various benchmarks and priors. Our code would be released upon acceptance.
• 摘要：特征生成对抗网络已成为零样本学习 (ZSL) 领域中生成高质量未知类别表示的强大生成模型。本文深入探讨了未知类别先验在传导式零样本学习 (TZSL) 框架中的关键影响，并阐明了即使是微小的先验偏差也会导致准确度大幅下降这一发现。我们广泛的分析发现，这种无效性从根本上源于使用无条件未知判别器 - 现有 TZSL 的核心组件。我们进一步确定，除非生成器完全适合特定类别的分布，否则该组件的不利影响是不可避免的。基于这些见解，我们推出了改进的特征生成框架，称为 I-VAEGAN，它包含两个新组件：伪条件特征对抗 (PFA) 学习和变分嵌入回归 (VER)。 PFA 通过将预测语义明确注入为以精确语义回归为前提的未见类的伪条件，从而避免了先验估计的需要。同时，VER 利用重构预训练来学习类别统计数据，从而获得更好的语义回归。我们的 I-VAEGAN 在各种基准和先验中实现了最先进的 TZSL 准确率。我们的代码将在接受后发布。

Title: FameBias: Embedding Manipulation Bias Attack in Text-to-Image Models

• Authors: Jaechul Roh, Andrew Yuan, Jinsong Mao
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18302
• Pdf URL: https://arxiv.org/pdf/2412.18302
• Copy Paste: [[2412.18302]] FameBias: Embedding Manipulation Bias Attack in Text-to-Image Models(https://arxiv.org/abs/2412.18302)
• Keywords: generation
• Abstract: Text-to-Image (T2I) diffusion models have rapidly advanced, enabling the generation of high-quality images that align closely with textual descriptions. However, this progress has also raised concerns about their misuse for propaganda and other malicious activities. Recent studies reveal that attackers can embed biases into these models through simple fine-tuning, causing them to generate targeted imagery when triggered by specific phrases. This underscores the potential for T2I models to act as tools for disseminating propaganda, producing images aligned with an attacker's objective for end-users. Building on this concept, we introduce FameBias, a T2I biasing attack that manipulates the embeddings of input prompts to generate images featuring specific public figures. Unlike prior methods, Famebias operates solely on the input embedding vectors without requiring additional model training. We evaluate FameBias comprehensively using Stable Diffusion V2, generating a large corpus of images based on various trigger nouns and target public figures. Our experiments demonstrate that FameBias achieves a high attack success rate while preserving the semantic context of the original prompts across multiple trigger-target pairs.
• 摘要：文本到图像 (T2I) 传播模型发展迅速，能够生成与文本描述紧密吻合的高质量图像。然而，这一进展也引发了人们对其被滥用于宣传和其他恶意活动的担忧。最近的研究表明，攻击者可以通过简单的微调将偏见嵌入这些模型中，导致它们在被特定短语触发时生成有针对性的图像。这凸显了 T2I 模型作为传播宣传工具的潜力，为最终用户生成与攻击者目标一致的图像。基于这一概念，我们引入了 FameBias，这是一种 T2I 偏见攻击，它操纵输入提示的嵌入以生成具有特定公众人物的图像。与之前的方法不同，Famebias 仅在输入嵌入向量上运行，而无需额外的模型训练。我们使用 Stable Diffusion V2 全面评估 FameBias，根据各种触发名词和目标公众人物生成大量图像。我们的实验表明，FameBias 实现了较高的攻击成功率，同时在多个触发目标对中保留了原始提示的语义上下文。

Title: RDPM: Solve Diffusion Probabilistic Models via Recurrent Token Prediction

• Authors: Wu Xiaoping, Hu Jie, Wei Xiaoming
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.18390
• Pdf URL: https://arxiv.org/pdf/2412.18390
• Copy Paste: [[2412.18390]] RDPM: Solve Diffusion Probabilistic Models via Recurrent Token Prediction(https://arxiv.org/abs/2412.18390)
• Keywords: generation, generative
• Abstract: Diffusion Probabilistic Models (DPMs) have emerged as the de facto approach for high-fidelity image synthesis, operating diffusion processes on continuous VAE latent, which significantly differ from the text generation methods employed by Large Language Models (LLMs). In this paper, we introduce a novel generative framework, the Recurrent Diffusion Probabilistic Model (RDPM), which enhances the diffusion process through a recurrent token prediction mechanism, thereby pioneering the field of Discrete Diffusion. By progressively introducing Gaussian noise into the latent representations of images and encoding them into vector-quantized tokens in a recurrent manner, RDPM facilitates a unique diffusion process on discrete-value domains. This process iteratively predicts the token codes for subsequent timesteps, transforming the initial standard Gaussian noise into the source data distribution, aligning with GPT-style models in terms of the loss function. RDPM demonstrates superior performance while benefiting from the speed advantage of requiring only a few inference steps. This model not only leverages the diffusion process to ensure high-quality generation but also converts continuous signals into a series of high-fidelity discrete tokens, thereby maintaining a unified optimization strategy with other discrete tokens, such as text. We anticipate that this work will contribute to the development of a unified model for multimodal generation, specifically by integrating continuous signal domains such as images, videos, and audio with text. We will release the code and model weights to the open-source community.
• 摘要：扩散概率模型 (DPM) 已成为高保真图像合成的事实​​上的方法，它在连续 VAE 潜在上操作扩散过程，这与大型语言模型 (LLM) 采用的文本生成方法有显著不同。在本文中，我们介绍了一种新颖的生成框架，即循环扩散概率模型 (RDPM)，它通过循环标记预测机制增强了扩散过程，从而开创了离散扩散领域。通过逐步将高斯噪声引入图像的潜在表示并以循环方式将其编码为矢量量化标记，RDPM 促进了离散值域上的独特扩散过程。此过程迭代地预测后续时间步的标记代码，将初始标准高斯噪声转换为源数据分布，在损失函数方面与 GPT 样式模型保持一致。RDPM 表现出卓越的性能，同时受益于仅需几个推理步骤的速度优势。该模型不仅利用扩散过程来确保高质量的生成，还将连续信号转换为一系列高保真离散标记，从而与其他离散标记（例如文本）保持统一的优化策略。我们预计这项工作将有助于开发用于多模态生成的统一模型，特别是通过将图像、视频和音频等连续信号域与文本相结合。我们将向开源社区发布代码和模型权重。

Title: Extract Free Dense Misalignment from CLIP

• Authors: JeongYeon Nam, Jinbae Im, Wonjae Kim, Taeho Kil
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18404
• Pdf URL: https://arxiv.org/pdf/2412.18404
• Copy Paste: [[2412.18404]] Extract Free Dense Misalignment from CLIP(https://arxiv.org/abs/2412.18404)
• Keywords: generation
• Abstract: Recent vision-language foundation models still frequently produce outputs misaligned with their inputs, evidenced by object hallucination in captioning and prompt misalignment in the text-to-image generation model. Recent studies have explored methods for identifying misaligned elements, aiming not only to enhance interpretability but also to improve model performance. However, current approaches primarily rely on large foundation models in a zero-shot manner or fine-tuned models with human annotations, which limits scalability due to significant computational costs. This work proposes a novel approach, dubbed CLIP4DM, for detecting dense misalignments from pre-trained CLIP, specifically focusing on pinpointing misaligned words between image and text. We carefully revamp the gradient-based attribution computation method, enabling negative gradient of individual text tokens to indicate misalignment. We also propose F-CLIPScore, which aggregates misaligned attributions with a global alignment score. We evaluate our method on various dense misalignment detection benchmarks, covering various image and text domains and misalignment types. Our method demonstrates state-of-the-art performance among zero-shot models and competitive performance with fine-tuned models while maintaining superior efficiency. Our qualitative examples show that our method has a unique strength to detect entity-level objects, intangible objects, and attributes that can not be easily detected for existing works. We conduct ablation studies and analyses to highlight the strengths and limitations of our approach. Our code is publicly available at this https URL.
• 摘要：最近的视觉语言基础模型仍然经常产生与输入不一致的输出，这表现为字幕中的对象幻觉和文本到图像生成模型中的提示错位。最近的研究探索了识别错位元素的方法，不仅旨在增强可解释性，而且旨在提高模型性能。然而，目前的方法主要依赖于零样本方式的大型基础模型或带有人工注释的微调模型，这由于计算成本高而限制了可扩展性。这项工作提出了一种称为 CLIP4DM 的新方法，用于从预训练的 CLIP 中检测密集错位，特别侧重于精确定位图像和文本之间错位的单词。我们仔细改进了基于梯度的归因计算方法，使单个文本标记的负梯度能够指示错位。我们还提出了 F-CLIPScore，它将错位归因与全局对齐分数聚合在一起。我们在各种密集错位检测基准上评估了我们的方法，涵盖了各种图像和文本域以及错位类型。我们的方法在零样本模型中表现出最佳性能，并且在保持卓越效率的同时与微调模型具有竞争力。我们的定性示例表明，我们的方法具有独特的优势，可以检测实体级对象、无形对象和现有作品无法轻易检测到的属性。我们进行消融研究和分析，以突出我们方法的优势和局限性。我们的代码在此 https URL 上公开提供。

Title: Fashionability-Enhancing Outfit Image Editing with Conditional Diffusion Models

• Authors: Qice Qin, Yuki Hirakawa, Ryotaro Shimizu, Takuya Furusawa, Edgar Simo-Serra
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18421
• Pdf URL: https://arxiv.org/pdf/2412.18421
• Copy Paste: [[2412.18421]] Fashionability-Enhancing Outfit Image Editing with Conditional Diffusion Models(https://arxiv.org/abs/2412.18421)
• Keywords: generation
• Abstract: Image generation in the fashion domain has predominantly focused on preserving body characteristics or following input prompts, but little attention has been paid to improving the inherent fashionability of the output images. This paper presents a novel diffusion model-based approach that generates fashion images with improved fashionability while maintaining control over key attributes. Key components of our method include: 1) fashionability enhancement, which ensures that the generated images are more fashionable than the input; 2) preservation of body characteristics, encouraging the generated images to maintain the original shape and proportions of the input; and 3) automatic fashion optimization, which does not rely on manual input or external prompts. We also employ two methods to collect training data for guidance while generating and evaluating the images. In particular, we rate outfit images using fashionability scores annotated by multiple fashion experts through OpenSkill-based and five critical aspect-based pairwise comparisons. These methods provide complementary perspectives for assessing and improving the fashionability of the generated images. The experimental results show that our approach outperforms the baseline Fashion++ in generating images with superior fashionability, demonstrating its effectiveness in producing more stylish and appealing fashion images.
• 摘要：时尚领域的图像生成主要侧重于保留身体特征或遵循输入提示，但很少关注提高输出图像的固有时尚性。本文提出了一种基于扩散模型的新型方法，该方法可生成时尚性更高的时尚图像，同时保持对关键属性的控制。我们方法的关键组成部分包括：1）时尚性增强，确保生成的图像比输入的图像更时尚；2）保留身体特征，鼓励生成的图像保持输入的原始形状和比例；3）自动时尚优化，不依赖于手动输入或外部提示。我们还采用两种方法收集训练数据，以便在生成和评估图像时提供指导。具体来说，我们使用由多位时尚专家通过基于 OpenSkill 和五个关键方面的成对比较注释的时尚性分数对服装图像进行评分。这些方法为评估和提高生成图像的时尚性提供了互补的视角。实验结果表明，我们的方法在生成具有卓越时尚性的图像方面优于基线 Fashion++，证明了其在生成更时尚、更有吸引力的时尚图像方面的有效性。

Title: Underwater Image Restoration via Polymorphic Large Kernel CNNs

• Authors: Xiaojiao Guo, Yihang Dong, Xuhang Chen, Weiwen Chen, Zimeng Li, FuChen Zheng, Chi-Man Pun
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.18459
• Pdf URL: https://arxiv.org/pdf/2412.18459
• Copy Paste: [[2412.18459]] Underwater Image Restoration via Polymorphic Large Kernel CNNs(https://arxiv.org/abs/2412.18459)
• Keywords: restoration
• Abstract: Underwater Image Restoration (UIR) remains a challenging task in computer vision due to the complex degradation of images in underwater environments. While recent approaches have leveraged various deep learning techniques, including Transformers and complex, parameter-heavy models to achieve significant improvements in restoration effects, we demonstrate that pure CNN architectures with lightweight parameters can achieve comparable results. In this paper, we introduce UIR-PolyKernel, a novel method for underwater image restoration that leverages Polymorphic Large Kernel CNNs. Our approach uniquely combines large kernel convolutions of diverse sizes and shapes to effectively capture long-range dependencies within underwater imagery. Additionally, we introduce a Hybrid Domain Attention module that integrates frequency and spatial domain attention mechanisms to enhance feature importance. By leveraging the frequency domain, we can capture hidden features that may not be perceptible to humans but are crucial for identifying patterns in both underwater and on-air images. This approach enhances the generalization and robustness of our UIR model. Extensive experiments on benchmark datasets demonstrate that UIR-PolyKernel achieves state-of-the-art performance in underwater image restoration tasks, both quantitatively and qualitatively. Our results show that well-designed pure CNN architectures can effectively compete with more complex models, offering a balance between performance and computational efficiency. This work provides new insights into the potential of CNN-based approaches for challenging image restoration tasks in underwater environments. The code is available at \href{this https URL}{this https URL}.
• 摘要：由于水下环境中图像的复杂退化，水下图像恢复 (UIR) 仍然是计算机视觉领域的一项艰巨任务。虽然最近的方法利用了各种深度学习技术，包括 Transformers 和复杂的参数密集型模型，以显著改善恢复效果，但我们证明，具有轻量级参数的纯 CNN 架构可以实现类似的结果。在本文中，我们介绍了 UIR-PolyKernel，这是一种利用多态大核 CNN 的水下图像恢复新方法。我们的方法独特地结合了不同大小和形状的大核卷积，以有效捕获水下图像中的长距离依赖关系。此外，我们引入了一个混合域注意模块，该模块集成了频域和空域注意机制以增强特征重要性。通过利用频域，我们可以捕获人类可能无法感知但对于识别水下和空中图像中的模式至关重要的隐藏特征。这种方法增强了我们的 UIR 模型的泛化和鲁棒性。在基准数据集上进行的大量实验表明，UIR-PolyKernel 在水下图像恢复任务中实现了最佳性能，无论是在数量上还是质量上。我们的结果表明，设计良好的纯 CNN 架构可以有效地与更复杂的模型竞争，在性能和计算效率之间取得平衡。这项工作为基于 CNN 的方法在水下环境中完成具有挑战性的图像恢复任务的潜力提供了新的见解。代码可在 \href{this https URL}{this https URL} 上找到。

Title: GeFL: Model-Agnostic Federated Learning with Generative Models

• Authors: Honggu Kang, Seohyeon Cha, Joonhyuk Kang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18460
• Pdf URL: https://arxiv.org/pdf/2412.18460
• Copy Paste: [[2412.18460]] GeFL: Model-Agnostic Federated Learning with Generative Models(https://arxiv.org/abs/2412.18460)
• Keywords: generative
• Abstract: Federated learning (FL) is a promising paradigm in distributed learning while preserving the privacy of users. However, the increasing size of recent models makes it unaffordable for a few users to encompass the model. It leads the users to adopt heterogeneous models based on their diverse computing capabilities and network bandwidth. Correspondingly, FL with heterogeneous models should be addressed, given that FL typically involves training a single global model. In this paper, we propose Generative Model-Aided Federated Learning (GeFL), incorporating a generative model that aggregates global knowledge across users of heterogeneous models. Our experiments on various classification tasks demonstrate notable performance improvements of GeFL compared to baselines, as well as limitations in terms of privacy and scalability. To tackle these concerns, we introduce a novel framework, GeFL-F. It trains target networks aided by feature-generative models. We empirically demonstrate the consistent performance gains of GeFL-F, while demonstrating better privacy preservation and robustness to a large number of clients. Codes are available at [1].
• 摘要：联邦学习 (FL) 是一种很有前途的分布式学习范例，同时可以保护用户的隐私。然而，最近模型的规模不断扩大，使得少数用户无法负担该模型的费用。这导致用户根据其不同的计算能力和网络带宽采用异构模型。相应地，应该解决具有异构模型的 FL，因为 FL 通常涉及训练单个全局模型。在本文中，我们提出了生成模型辅助联邦学习 (GeFL)，它结合了一个跨异构模型用户聚合全局知识的生成模型。我们在各种分类任务上的实验表明，与基线相比，GeFL 的性能显着提高，但在隐私和可扩展性方面存在局限性。为了解决这些问题，我们引入了一个新框架 GeFL-F。它在特征生成模型的帮助下训练目标网络。我们通过经验证明了 GeFL-F 的持续性能提升，同时向大量客户展示了更好的隐私保护和鲁棒性。代码可在 [1] 中找到。

Title: HTR-JAND: Handwritten Text Recognition with Joint Attention Network and Knowledge Distillation

• Authors: Mohammed Hamdan, Abderrahmane Rahiche, Mohamed Cheriet
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18524
• Pdf URL: https://arxiv.org/pdf/2412.18524
• Copy Paste: [[2412.18524]] HTR-JAND: Handwritten Text Recognition with Joint Attention Network and Knowledge Distillation(https://arxiv.org/abs/2412.18524)
• Keywords: generation
• Abstract: Despite significant advances in deep learning, current Handwritten Text Recognition (HTR) systems struggle with the inherent complexity of historical documents, including diverse writing styles, degraded text quality, and computational efficiency requirements across multiple languages and time periods. This paper introduces HTR-JAND (HTR-JAND: Handwritten Text Recognition with Joint Attention Network and Knowledge Distillation), an efficient HTR framework that combines advanced feature extraction with knowledge distillation. Our architecture incorporates three key components: (1) a CNN architecture integrating FullGatedConv2d layers with Squeeze-and-Excitation blocks for adaptive feature extraction, (2) a Combined Attention mechanism fusing Multi-Head Self-Attention with Proxima Attention for robust sequence modeling, and (3) a Knowledge Distillation framework enabling efficient model compression while preserving accuracy through curriculum-based training. The HTR-JAND framework implements a multi-stage training approach combining curriculum learning, synthetic data generation, and multi-task learning for cross-dataset knowledge transfer. We enhance recognition accuracy through context-aware T5 post-processing, particularly effective for historical documents. Comprehensive evaluations demonstrate HTR-JAND's effectiveness, achieving state-of-the-art Character Error Rates (CER) of 1.23\%, 1.02\%, and 2.02\% on IAM, RIMES, and Bentham datasets respectively. Our Student model achieves a 48\% parameter reduction (0.75M versus 1.5M parameters) while maintaining competitive performance through efficient knowledge transfer. Source code and pre-trained models are available at \href{this https URL}{Github}.
• 摘要：尽管深度学习取得了重大进展，但当前的手写文本识别 (HTR) 系统仍难以应对历史文档固有的复杂性，包括多种书写风格、文本质量下降以及跨多种语言和时间段的计算效率要求。本文介绍了 HTR-JAND（HTR-JAND：使用联合注意力网络和知识蒸馏的手写文本识别），这是一种将高级特征提取与知识蒸馏相结合的高效 HTR 框架。我们的架构包含三个关键组件：(1) 集成 FullGatedConv2d 层和 Squeeze-and-Excitation 块的 CNN 架构，用于自适应特征提取；(2) 融合多头自注意力和近端注意力的组合注意力机制，用于稳健的序列建模；(3) 知识蒸馏框架，通过基于课程的培训实现高效的模型压缩，同时保持准确性。HTR-JAND 框架实施了一种多阶段训练方法，结合了课程学习、合成数据生成和多任务学习，用于跨数据集知识转移。我们通过上下文感知的 T5 后处理提高了识别准确率，这对历史文档尤其有效。综合评估证明了 HTR-JAND 的有效性，在 IAM、RIMES 和 Bentham 数据集上分别实现了 1.23\%、1.02\% 和 2.02\% 的最先进的字符错误率 (CER)。我们的学生模型实现了 48\% 的参数减少（0.75M 对 1.5M 参数），同时通过高效的知识转移保持了竞争性能。源代码和预训练模型可在 \href{此 https URL}{Github} 上找到。

Title: Characterizations of Language Generation With Breadth

• Authors: Alkis Kalavasis, Anay Mehrotra, Grigoris Velegkas
• Subjects: cs.LG, cs.AI, cs.CL, cs.DS, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.18530
• Pdf URL: https://arxiv.org/pdf/2412.18530
• Copy Paste: [[2412.18530]] Characterizations of Language Generation With Breadth(https://arxiv.org/abs/2412.18530)
• Keywords: generation
• Abstract: We study language generation in the limit, introduced by Kleinberg and Mullainathan [KM24], building on classical works of Gold [Gol67] and Angluin [Ang79]. [KM24] proposed an algorithm that generates strings from any countable language collection in the limit. While their algorithm eventually outputs strings from the target language $K$, it sacrifices breadth, i.e., the ability to generate all strings in $K$. A key open question in [KM24] is whether this trade-off between consistency and breadth is inherrent. Recent works proposed different notions of consistent generation with breadth. Kalavasis, Mehrotra, and Velegkas [KVM24] introduced three definitions: generation with exact breadth, approximate breadth, and unambiguous generation. Concurrently and independently, Charikar and Pabbaraju [CP24a] proposed exhaustive generation. Both works examined when generation with these notions of breadth is possible. Building on [CP24a, KVM24], we fully characterize language generation for these notions and their natural combinations. For exact breadth, we provide an unconditional lower bound, removing a technical condition from [KVM24] and extending the result of [CP24a] that holds for specific collections of languages. We show that generation with exact breadth is characterized by Angluin's condition for identification. We further introduce a weaker version of Angluin's condition that tightly characterizes both approximate breadth and exhaustive generation, proving their equivalence. Additionally, we show that unambiguous generation is also characterized by Angluin's condition as a special case of a broader result. Finally, we strengthen [KVM24] by giving unconditional lower bounds for stable generators, showing that Angluin's condition characterizes the previous breadth notions for stable generators. This shows a separation between stable and unstable generation with approximate breadth.
• 摘要：我们研究了 Kleinberg 和 Mullainathan [KM24] 引入的极限语言生成，该算法以 Gold [Gol67] 和 Angluin [Ang79] 的经典著作为基础。[KM24] 提出了一种算法，该算法可以从极限中的任何可数语言集合生成字符串。虽然他们的算法最终会从目标语言 $K$ 中输出字符串，但它牺牲了广度，即生成 $K$ 中所有字符串的能力。[KM24] 中的一个关键悬而未决的问题是一致性和广度之间的这种权衡是否是固有的。最近的研究提出了具有广度的一致生成的不同概念。Kalavasis、Mehrotra 和 Velegkas [KVM24] 引入了三个定义：具有精确广度的生成、近似广度和无歧义生成。同时且独立地，Charikar 和 Pabbaraju [CP24a] 提出了详尽的生成。这两部作品都研究了何时可以使用这些广度概念进行生成。在 [CP24a, KVM24] 的基础上，我们充分描述了这些概念及其自然组合的语言生成。对于精确广度，我们提供了一个无条件下界，从 [KVM24] 中删除了一个技术条件，并扩展了 [CP24a] 的结果，该结果适用于特定的语言集合。我们表明，具有精确广度的生成可以用 Angluin 的识别条件来描述。我们进一步引入了 Angluin 条件的弱化版本，该版本可以严格描述近似广度和详尽生成，从而证明它们的等价性。此外，我们表明，无歧义生成也可以用 Angluin 条件来描述，这是更广泛结果的一个特例。最后，我们通过给出稳定生成器的无条件下界来加强 [KVM24]，表明 Angluin 的条件可以描述稳定生成器的先前广度概念。这表明了具有近似广度的稳定生成和不稳定生成之间的区别。

Title: Efficient Aircraft Design Optimization Using Multi-Fidelity Models and Multi-fidelity Physics Informed Neural Networks

• Authors: Apurba Sarker
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2412.18564
• Pdf URL: https://arxiv.org/pdf/2412.18564
• Copy Paste: [[2412.18564]] Efficient Aircraft Design Optimization Using Multi-Fidelity Models and Multi-fidelity Physics Informed Neural Networks(https://arxiv.org/abs/2412.18564)
• Keywords: generative
• Abstract: Aircraft design optimization traditionally relies on computationally expensive simulation techniques such as Finite Element Method (FEM) and Finite Volume Method (FVM), which, while accurate, can significantly slow down the design iteration process. The challenge lies in reducing the computational complexity while maintaining high accuracy for quick evaluations of multiple design alternatives. This research explores advanced methods, including surrogate models, reduced-order models (ROM), and multi-fidelity machine learning techniques, to achieve more efficient aircraft design evaluations. Specifically, the study investigates the application of Multi-fidelity Physics-Informed Neural Networks (MPINN) and autoencoders for manifold alignment, alongside the potential of Generative Adversarial Networks (GANs) for refining design geometries. Through a proof-of-concept task, the research demonstrates the ability to predict high-fidelity results from low-fidelity simulations, offering a path toward faster and more cost effective aircraft design iterations.
• 摘要：飞机设计优化传统上依赖于计算量巨大的模拟技术，例如有限元法 (FEM) 和有限体积法 (FVM)，这些技术虽然准确，但会显著减慢设计迭代过程。挑战在于降低计算复杂度，同时保持高精度，以便快速评估多种设计方案。本研究探索了先进的方法，包括代理模型、降阶模型 (ROM) 和多保真机器学习技术，以实现更高效的飞机设计评估。具体来说，该研究调查了多保真物理信息神经网络 (MPINN) 和自动编码器在流形对齐中的应用，以及生成对抗网络 (GAN) 在细化设计几何方面的潜力。通过概念验证任务，该研究展示了从低保真模拟中预测高保真结果的能力，为更快、更具成本效益的飞机设计迭代提供了一条途径。

Title: 3DEnhancer: Consistent Multi-View Diffusion for 3D Enhancement

• Authors: Yihang Luo, Shangchen Zhou, Yushi Lan, Xingang Pan, Chen Change Loy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18565
• Pdf URL: https://arxiv.org/pdf/2412.18565
• Copy Paste: [[2412.18565]] 3DEnhancer: Consistent Multi-View Diffusion for 3D Enhancement(https://arxiv.org/abs/2412.18565)
• Keywords: generation
• Abstract: Despite advances in neural rendering, due to the scarcity of high-quality 3D datasets and the inherent limitations of multi-view diffusion models, view synthesis and 3D model generation are restricted to low resolutions with suboptimal multi-view consistency. In this study, we present a novel 3D enhancement pipeline, dubbed 3DEnhancer, which employs a multi-view latent diffusion model to enhance coarse 3D inputs while preserving multi-view consistency. Our method includes a pose-aware encoder and a diffusion-based denoiser to refine low-quality multi-view images, along with data augmentation and a multi-view attention module with epipolar aggregation to maintain consistent, high-quality 3D outputs across views. Unlike existing video-based approaches, our model supports seamless multi-view enhancement with improved coherence across diverse viewing angles. Extensive evaluations show that 3DEnhancer significantly outperforms existing methods, boosting both multi-view enhancement and per-instance 3D optimization tasks.
• 摘要：尽管神经渲染取得了进展，但由于高质量 3D 数据集的稀缺性和多视图扩散模型的固有局限性，视图合成和 3D 模型生成仅限于低分辨率且多视图一致性欠佳。在本研究中，我们提出了一种新型 3D 增强管道，称为 3DEnhancer，它采用多视图潜在扩散模型来增强粗 3D 输入，同时保持多视图一致性。我们的方法包括一个姿势感知编码器和一个基于扩散的去噪器，以优化低质量的多视图图像，以及数据增强和具有极线聚合的多视图注意模块，以在各个视图之间保持一致的高质量 3D 输出。与现有的基于视频的方法不同，我们的模型支持无缝多视图增强，并在不同的视角之间提高了连贯性。广泛的评估表明，3DEnhancer 明显优于现有方法，增强了多视图增强和每个实例的 3D 优化任务。

Title: DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation

• Authors: Minghong Cai, Xiaodong Cun, Xiaoyu Li, Wenze Liu, Zhaoyang Zhang, Yong Zhang, Ying Shan, Xiangyu Yue
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.18597
• Pdf URL: https://arxiv.org/pdf/2412.18597
• Copy Paste: [[2412.18597]] DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation(https://arxiv.org/abs/2412.18597)
• Keywords: generation
• Abstract: Sora-like video generation models have achieved remarkable progress with a Multi-Modal Diffusion Transformer MM-DiT architecture. However, the current video generation models predominantly focus on single-prompt, struggling to generate coherent scenes with multiple sequential prompts that better reflect real-world dynamic scenarios. While some pioneering works have explored multi-prompt video generation, they face significant challenges including strict training data requirements, weak prompt following, and unnatural transitions. To address these problems, we propose DiTCtrl, a training-free multi-prompt video generation method under MM-DiT architectures for the first time. Our key idea is to take the multi-prompt video generation task as temporal video editing with smooth transitions. To achieve this goal, we first analyze MM-DiT's attention mechanism, finding that the 3D full attention behaves similarly to that of the cross/self-attention blocks in the UNet-like diffusion models, enabling mask-guided precise semantic control across different prompts with attention sharing for multi-prompt video generation. Based on our careful design, the video generated by DiTCtrl achieves smooth transitions and consistent object motion given multiple sequential prompts without additional training. Besides, we also present MPVBench, a new benchmark specially designed for multi-prompt video generation to evaluate the performance of multi-prompt generation. Extensive experiments demonstrate that our method achieves state-of-the-art performance without additional training.
• 摘要：类似 Sora 的视频生成模型通过多模态扩散变换器 MM-DiT 架构取得了显著进展。然而，当前的视频生成模型主要关注单提示，难以生成具有多个连续提示的连贯场景，以更好地反映现实世界的动态场景。虽然一些先驱作品已经探索了多提示视频生成，但它们面临着重大挑战，包括严格的训练数据要求、提示跟随性弱和过渡不自然。为了解决这些问题，我们首次提出了 DiTCtrl，这是 MM-DiT 架构下的一种无需训练的多提示视频生成方法。我们的主要思想是将多提示视频生成任务视为具有平滑过渡的时间视频编辑。为了实现这一目标，我们首先分析了 MM-DiT 的注意力机制，发现 3D 全注意力的行为类似于 UNet 类扩散模型中的交叉/自注意力块，从而实现了跨不同提示的掩码引导的精确语义控制，并实现了多提示视频生成的注意力共享。基于我们精心的设计，DiTCtrl 生成的视频无需额外训练即可在多个连续提示下实现平滑过渡和一致的物体运动。此外，我们还提出了 MPVBench，这是一个专为多提示视频生成设计的新基准，用于评估多提示生成的性能。大量实验表明，我们的方法无需额外训练即可实现最先进的性能。

Title: ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation

• Authors: Hongjie Li, Hong-Xing Yu, Jiaman Li, Jiajun Wu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.18600
• Pdf URL: https://arxiv.org/pdf/2412.18600
• Copy Paste: [[2412.18600]] ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation(https://arxiv.org/abs/2412.18600)
• Keywords: generation
• Abstract: Human-scene interaction (HSI) generation is crucial for applications in embodied AI, virtual reality, and robotics. While existing methods can synthesize realistic human motions in 3D scenes and generate plausible human-object interactions, they heavily rely on datasets containing paired 3D scene and motion capture data, which are expensive and time-consuming to collect across diverse environments and interactions. We present ZeroHSI, a novel approach that enables zero-shot 4D human-scene interaction synthesis by integrating video generation and neural human rendering. Our key insight is to leverage the rich motion priors learned by state-of-the-art video generation models, which have been trained on vast amounts of natural human movements and interactions, and use differentiable rendering to reconstruct human-scene interactions. ZeroHSI can synthesize realistic human motions in both static scenes and environments with dynamic objects, without requiring any ground-truth motion data. We evaluate ZeroHSI on a curated dataset of different types of various indoor and outdoor scenes with different interaction prompts, demonstrating its ability to generate diverse and contextually appropriate human-scene interactions.
• 摘要：人景交互 (HSI) 生成对于具身 AI、虚拟现实和机器人技术的应用至关重要。虽然现有方法可以在 3D 场景中合成逼真的人体运动并生成合理的人物交互，但它们严重依赖包含成对的 3D 场景和运动捕捉数据的数据集，而这些数据集在不同环境和交互中收集起来成本高昂且耗时。我们提出了 ZeroHSI，这是一种新颖的方法，通过集成视频生成和神经人体渲染来实现零镜头 4D 人景交互合成。我们的主要见解是利用最先进的视频生成模型学习到的丰富运动先验，这些模型已经过大量自然人体运动和交互的训练，并使用可微分渲染来重建人景交互。ZeroHSI 可以在静态场景和具有动态物体的环境中合成逼真的人体运动，而无需任何地面真实运动数据。我们在具有不同交互提示的各种类型的室内和室外场景的精选数据集上对 ZeroHSI 进行评估，展示了其生成多样化且适合情境的人景交互的能力。

Title: Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models

• Authors: Zehan Wang, Ziang Zhang, Tianyu Pang, Chao Du, Hengshuang Zhao, Zhou Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18605
• Pdf URL: https://arxiv.org/pdf/2412.18605
• Copy Paste: [[2412.18605]] Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models(https://arxiv.org/abs/2412.18605)
• Keywords: generation
• Abstract: Orientation is a key attribute of objects, crucial for understanding their spatial pose and arrangement in images. However, practical solutions for accurate orientation estimation from a single image remain underexplored. In this work, we introduce Orient Anything, the first expert and foundational model designed to estimate object orientation in a single- and free-view image. Due to the scarcity of labeled data, we propose extracting knowledge from the 3D world. By developing a pipeline to annotate the front face of 3D objects and render images from random views, we collect 2M images with precise orientation annotations. To fully leverage the dataset, we design a robust training objective that models the 3D orientation as probability distributions of three angles and predicts the object orientation by fitting these distributions. Besides, we employ several strategies to improve synthetic-to-real transfer. Our model achieves state-of-the-art orientation estimation accuracy in both rendered and real images and exhibits impressive zero-shot ability in various scenarios. More importantly, our model enhances many applications, such as comprehension and generation of complex spatial concepts and 3D object pose adjustment.
• 摘要：方向是物体的一个关键属性，对于理解它们在图像中的空间姿势和排列至关重要。然而，从单个图像进行精确方向估计的实用解决方案仍未得到充分探索。在这项工作中，我们介绍了 Orient Anything，这是第一个旨在估计单视图和自由视图图像中物体方向的专家和基础模型。由于标记数据的稀缺性，我们建议从 3D 世界中提取知识。通过开发一个管道来注释 3D 物体的正面并从随机视图渲染图像，我们收集了 200 万张带有精确方向注释的图像。为了充分利用数据集，我们设计了一个强大的训练目标，将 3D 方向建模为三个角度的概率分布，并通过拟合这些分布来预测物体方向。此外，我们采用了几种策略来改进从合成到真实的迁移。我们的模型在渲染图像和真实图像中都实现了最先进的方向估计精度，并在各种场景中表现出令人印象深刻的零样本能力。更重要的是，我们的模型增强了许多应用，例如复杂空间概念的理解和生成以及 3D 物体姿势调整。

Title: DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers

• Authors: Yuntao Chen, Yuqi Wang, Zhaoxiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18607
• Pdf URL: https://arxiv.org/pdf/2412.18607
• Copy Paste: [[2412.18607]] DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers(https://arxiv.org/abs/2412.18607)
• Keywords: generation
• Abstract: World model-based searching and planning are widely recognized as a promising path toward human-level physical intelligence. However, current driving world models primarily rely on video diffusion models, which specialize in visual generation but lack the flexibility to incorporate other modalities like action. In contrast, autoregressive transformers have demonstrated exceptional capability in modeling multimodal data. Our work aims to unify both driving model simulation and trajectory planning into a single sequence modeling problem. We introduce a multimodal driving language based on interleaved image and action tokens, and develop DrivingGPT to learn joint world modeling and planning through standard next-token prediction. Our DrivingGPT demonstrates strong performance in both action-conditioned video generation and end-to-end planning, outperforming strong baselines on large-scale nuPlan and NAVSIM benchmarks.
• 摘要：基于世界模型的搜索和规划被广泛认为是实现人类水平物理智能的一条有希望的道路。然而，目前的驾驶世界模型主要依赖于视频扩散模型，这些模型专注于视觉生成，但缺乏纳入其他模态（如动作）的灵活性。相比之下，自回归变换器在多模态数据建模方面表现出色。我们的工作旨在将驾驶模型模拟和轨迹规划统一为一个序列建模问题。我们引入了一种基于交错图像和动作标记的多模态驾驶语言，并开发了 DrivingGPT，通过标准的下一个标记预测来学习联合世界建模和规划。我们的 DrivingGPT 在动作条件视频生成和端到端规划方面都表现出色，在大型 nuPlan 和 NAVSIM 基准测试中表现优于强大的基线。

Title: PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models

• Authors: Minghao Chen, Roman Shapovalov, Iro Laina, Tom Monnier, Jianyuan Wang, David Novotny, Andrea Vedaldi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18608
• Pdf URL: https://arxiv.org/pdf/2412.18608
• Copy Paste: [[2412.18608]] PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models(https://arxiv.org/abs/2412.18608)
• Keywords: generation, generative
• Abstract: Text- or image-to-3D generators and 3D scanners can now produce 3D assets with high-quality shapes and textures. These assets typically consist of a single, fused representation, like an implicit neural field, a Gaussian mixture, or a mesh, without any useful structure. However, most applications and creative workflows require assets to be made of several meaningful parts that can be manipulated independently. To address this gap, we introduce PartGen, a novel approach that generates 3D objects composed of meaningful parts starting from text, an image, or an unstructured 3D object. First, given multiple views of a 3D object, generated or rendered, a multi-view diffusion model extracts a set of plausible and view-consistent part segmentations, dividing the object into parts. Then, a second multi-view diffusion model takes each part separately, fills in the occlusions, and uses those completed views for 3D reconstruction by feeding them to a 3D reconstruction network. This completion process considers the context of the entire object to ensure that the parts integrate cohesively. The generative completion model can make up for the information missing due to occlusions; in extreme cases, it can hallucinate entirely invisible parts based on the input 3D asset. We evaluate our method on generated and real 3D assets and show that it outperforms segmentation and part-extraction baselines by a large margin. We also showcase downstream applications such as 3D part editing.
• 摘要：文本或图像到 3D 生成器和 3D 扫描仪现在可以生成具有高质量形状和纹理的 3D 资产。这些资产通常由单一的融合表示组成，如隐式神经场、高斯混合或网格，没有任何有用的结构。然而，大多数应用程序和创意工作流程都要求资产由几个可以独立操作的有意义的部分组成。为了解决这一差距，我们引入了 PartGen，这是一种新颖的方法，它从文本、图像或非结构化 3D 对象开始生成由有意义的部分组成的 3D 对象。首先，给定 3D 对象的多个视图（生成或渲染），多视图扩散模型提取一组合理且视图一致的部分分割，将对象分成多个部分。然后，第二个多视图扩散模型分别获取每个部分，填充遮挡，并通过将这些完成的视图馈送到 3D 重建网络来使用这些完成的视图进行 3D 重建。这个完成过程考虑了整个对象的上下文，以确保各部分紧密结合。生成式补全模型可以弥补因遮挡而缺失的信息；在极端情况下，它可以根据输入的 3D 资产产生完全不可见的幻觉。我们在生成的和真实的 3D 资产上评估了我们的方法，并表明它的表现远远优于分割和零件提取基线。我们还展示了 3D 零件编辑等下游应用。