2025-01-08

Title: Method of data forward generation with partial differential equations for machine learning modeling in fluid mechanics

• Authors: Ruilin Chen, Xiaowei Jin, Nikolaus A. Adams, Hui Li
• Subjects: cs.LG, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2501.03300
• Pdf URL: https://arxiv.org/pdf/2501.03300
• Copy Paste: [[2501.03300]] Method of data forward generation with partial differential equations for machine learning modeling in fluid mechanics(https://arxiv.org/abs/2501.03300)
• Keywords: generation
• Abstract: Artificial intelligence (AI) for fluid mechanics has become attractive topic. High-fidelity data is one of most critical issues for the successful applications of AI in fluid mechanics, however, it is expensively obtained or even inaccessible. This study proposes a high-efficient data forward generation method from the partial differential equations (PDEs). Specifically, the solutions of the PDEs are first generated either following a random field (e.g. Gaussian random field, GRF, computational complexity O(NlogN), N is the number of spatial points) or physical laws (e.g. a kind of spectra, computational complexity O(NM), M is the number of modes), then the source terms, boundary conditions and initial conditions are computed to satisfy PDEs. Thus, the data pairs of source terms, boundary conditions and initial conditions with corresponding solutions of PDEs can be constructed. A Poisson neural network (Poisson-NN) embedded in projection method and a wavelet transform convolutional neuro network (WTCNN) embedded in multigrid numerical simulation for solving incompressible Navier-Stokes equations is respectively proposed. The feasibility of generated data for training Poisson-NN and WTCNN is validated. The results indicate that even without any DNS data, the generated data can train these two models with excellent generalization and accuracy. The data following physical laws can significantly improve the convergence rate, generalization and accuracy than that generated following GRF.
• 摘要：人工智能在流体力学中的应用已成为一个备受关注的课题。高保真数据是人工智能在流体力学中成功应用的关键问题之一，然而，高保真数据获取成本高昂，甚至难以获取。本研究提出了一种高效的偏微分方程数据正向生成方法。具体而言，首先根据随机场（例如高斯随机场，GRF，计算复杂度为O(NlogN)，N是空间点的数量）或物理定律（例如一种谱，计算复杂度为O(NM)，M是模式数）生成偏微分方程的解，然后计算满足偏微分方程的源项、边界条件和初始条件。由此，可以构建源项、边界条件和初始条件以及相应偏微分方程解的数据对。分别提出了一种嵌入投影法的泊松神经网络（Poisson-NN）和嵌入多重网格数值模拟的小波变换卷积神经网络（WTCNN），用于求解不可压缩Navier-Stokes方程。验证了生成数据用于训练Poisson-NN和WTCNN的可行性。结果表明，即使没有任何DNS数据，生成数据也可以训练这两个模型，并具有良好的泛化能力和精度。遵循物理定律的数据比遵循GRF生成的数据可以显著提高收敛速度、泛化能力和精度。

Title: License Plate Images Generation with Diffusion Models

• Authors: Mariia Shpir, Nadiya Shvai, Amir Nakib
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03374
• Pdf URL: https://arxiv.org/pdf/2501.03374
• Copy Paste: [[2501.03374]] License Plate Images Generation with Diffusion Models(https://arxiv.org/abs/2501.03374)
• Keywords: generation
• Abstract: Despite the evident practical importance of license plate recognition (LPR), corresponding research is limited by the volume of publicly available datasets due to privacy regulations such as the General Data Protection Regulation (GDPR). To address this challenge, synthetic data generation has emerged as a promising approach. In this paper, we propose to synthesize realistic license plates (LPs) using diffusion models, inspired by recent advances in image and video generation. In our experiments a diffusion model was successfully trained on a Ukrainian LP dataset, and 1000 synthetic images were generated for detailed analysis. Through manual classification and annotation of the generated images, we performed a thorough study of the model output, such as success rate, character distributions, and type of failures. Our contributions include experimental validation of the efficacy of diffusion models for LP synthesis, along with insights into the characteristics of the generated data. Furthermore, we have prepared a synthetic dataset consisting of 10,000 LP images, publicly available at this https URL. Conducted experiments empirically confirm the usefulness of synthetic data for the LPR task. Despite the initial performance gap between the model trained with real and synthetic data, the expansion of the training data set with pseudolabeled synthetic data leads to an improvement in LPR accuracy by 3% compared to baseline.
• 摘要：尽管车牌识别 (LPR) 具有明显的实际重要性，但由于《通用数据保护条例》(GDPR) 等隐私法规，相应的研究受到公开可用数据集数量的限制。为了应对这一挑战，合成数据生成已成为一种有前途的方法。在本文中，我们提出使用扩散模型合成逼真的车牌 (LP)，灵感来自图像和视频生成方面的最新进展。在我们的实验中，扩散模型成功地在乌克兰 LP 数据集上进行了训练，并生成了 1000 张合成图像以供详细分析。通过对生成的图像进行手动分类和注释，我们对模型输出进行了彻底的研究，例如成功率、字符分布和故障类型。我们的贡献包括对扩散模型用于 LP 合成的有效性的实验验证，以及对生成数据特征的洞察。此外，我们还准备了一个由 10,000 张 LP 图像组成的合成数据集，可在此 https URL 上公开获取。进行的实验从经验上证实了合成数据对 LPR 任务的有用性。尽管使用真实数据和合成数据训练的模型在初始性能上存在差距，但使用伪标记合成数据扩展训练数据集后，LPR 准确率与基线相比提高了 3%。

Title: DoubleDiffusion: Combining Heat Diffusion with Denoising Diffusion for Generative Learning on 3D Meshes

• Authors: Xuyang Wang, Ziang Cheng, Zhenyu Li, Jiayu Yang, Haorui Ji, Pan Ji, Mehrtash Harandi, Richard Hartley, Hongdong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03397
• Pdf URL: https://arxiv.org/pdf/2501.03397
• Copy Paste: [[2501.03397]] DoubleDiffusion: Combining Heat Diffusion with Denoising Diffusion for Generative Learning on 3D Meshes(https://arxiv.org/abs/2501.03397)
• Keywords: generation, generative
• Abstract: This paper proposes DoubleDiffusion, a novel framework that combines heat dissipation diffusion and denoising diffusion for direct generative learning on 3D mesh surfaces. Our approach addresses the challenges of generating continuous signal distributions residing on a curve manifold surface. Unlike previous methods that rely on unrolling 3D meshes into 2D or adopting field representations, DoubleDiffusion leverages the Laplacian-Beltrami operator to process features respecting the mesh structure. This combination enables effective geometry-aware signal diffusion across the underlying geometry. As shown in Fig.~\ref{fig:teaser}, we demonstrate that DoubleDiffusion has the ability to generate RGB signal distributions on complex 3D mesh surfaces and achieves per-category shape-conditioned texture generation across different shape geometry. Our work contributes a new direction in diffusion-based generative modeling on 3D surfaces, with potential applications in the field of 3D asset generation.
• 摘要：本文提出了 DoubleDiffusion，这是一种结合散热扩散和去噪扩散的新型框架，用于在 3D 网格表面上进行直接生成学习。我们的方法解决了在曲线流形表面上生成连续信号分布的挑战。与以前依赖于将 3D 网格展开为 2D 或采用场表示的方法不同，DoubleDiffusion 利用拉普拉斯-贝尔特拉米算子来处理尊重网格结构的特征。这种组合可以在底层几何体上实现有效的几何感知信号扩散。如图 ~\ref{fig:teaser} 所示，我们证明了 DoubleDiffusion 能够在复杂的 3D 网格表面上生成 RGB 信号分布，并实现在不同形状几何体上按类别生成形状条件纹理。我们的工作为基于扩散的 3D 表面生成建模开辟了新方向，并有可能应用于 3D 资产生成领域。

Title: Physics-Constrained Generative Artificial Intelligence for Rapid Takeoff Trajectory Design

• Authors: Samuel Sisk, Xiaosong Du
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03445
• Pdf URL: https://arxiv.org/pdf/2501.03445
• Copy Paste: [[2501.03445]] Physics-Constrained Generative Artificial Intelligence for Rapid Takeoff Trajectory Design(https://arxiv.org/abs/2501.03445)
• Keywords: generative
• Abstract: To aid urban air mobility (UAM), electric vertical takeoff and landing (eVTOL) aircraft are being targeted. Conventional multidisciplinary analysis and optimization (MDAO) can be expensive, while surrogate-based optimization can struggle with challenging physical constraints. This work proposes physics-constrained generative adversarial networks (physicsGAN), to intelligently parameterize the takeoff control profiles of an eVTOL aircraft and to transform the original design space to a feasible space. Specifically, the transformed feasible space refers to a space where all designs directly satisfy all design constraints. The physicsGAN-enabled surrogate-based takeoff trajectory design framework was demonstrated on the Airbus A3 Vahana. The physicsGAN generated only feasible control profiles of power and wing angle in the feasible space with around 98.9% of designs satisfying all constraints. The proposed design framework obtained 99.6% accuracy compared with simulation-based optimal design and took only 2.2 seconds, which reduced the computational time by around 200 times. Meanwhile, data-driven GAN-enabled surrogate-based optimization took 21.9 seconds using a derivative-free optimizer, which was around an order of magnitude slower than the proposed framework. Moreover, the data-driven GAN-based optimization using gradient-based optimizers could not consistently find the optimal design during random trials and got stuck in an infeasible region, which is problematic in real practice. Therefore, the proposed physicsGAN-based design framework outperformed data-driven GAN-based design to the extent of efficiency (2.2 seconds), optimality (99.6% accurate), and feasibility (100% feasible). According to the literature review, this is the first physics-constrained generative artificial intelligence enabled by surrogate models.
• 摘要：为了促进城市空中交通 (UAM)，电动垂直起降 (eVTOL) 飞机成为目标。传统的多学科分析和优化 (MDAO) 成本高昂，而基于代理的优化可能会难以应对具有挑战性的物理约束。这项工作提出了物理约束生成对抗网络 (physicsGAN)，以智能地参数化 eVTOL 飞机的起飞控制轮廓，并将原始设计空间转换为可行空间。具体而言，转换后的可行空间是指所有设计都直接满足所有设计约束的空间。支持 physicsGAN 的基于代理的起飞轨迹设计框架在空客 A3 Vahana 上进行了演示。physicsGAN 仅在可行空间中生成功率和机翼角度的可行控制轮廓，约 98.9% 的设计满足所有约束。与基于模拟的最佳设计相比，所提出的设计框架获得了 99.6% 的准确率，并且仅需 2.2 秒，将计算时间缩短了约 200 倍。同时，使用无导数优化器，数据驱动的基于 GAN 的代理优化耗时 21.9 秒，比所提出的框架慢一个数量级。此外，使用基于梯度的优化器的数据驱动的基于 GAN 的优化在随机试验期间无法始终找到最佳设计，并会卡在不可行区域，这在实际操作中存在问题。因此，所提出的基于物理 GAN 的设计框架在效率（2.2 秒）、最优性（99.6% 准确率）和可行性（100% 可行）方面均优于数据驱动的基于 GAN 的设计。根据文献综述，这是第一个由代理模型实现的物理约束生成人工智能。

Title: SceneBooth: Diffusion-based Framework for Subject-preserved Text-to-Image Generation

• Authors: Shang Chai, Zihang Lin, Min Zhou, Xubin Li, Liansheng Zhuang, Houqiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03490
• Pdf URL: https://arxiv.org/pdf/2501.03490
• Copy Paste: [[2501.03490]] SceneBooth: Diffusion-based Framework for Subject-preserved Text-to-Image Generation(https://arxiv.org/abs/2501.03490)
• Keywords: generation
• Abstract: Due to the demand for personalizing image generation, subject-driven text-to-image generation method, which creates novel renditions of an input subject based on text prompts, has received growing research interest. Existing methods often learn subject representation and incorporate it into the prompt embedding to guide image generation, but they struggle with preserving subject fidelity. To solve this issue, this paper approaches a novel framework named SceneBooth for subject-preserved text-to-image generation, which consumes inputs of a subject image, object phrases and text prompts. Instead of learning the subject representation and generating a subject, our SceneBooth fixes the given subject image and generates its background image guided by the text prompts. To this end, our SceneBooth introduces two key components, i.e., a multimodal layout generation module and a background painting module. The former determines the position and scale of the subject by generating appropriate scene layouts that align with text captions, object phrases, and subject visual information. The latter integrates two adapters (ControlNet and Gated Self-Attention) into the latent diffusion model to generate a background that harmonizes with the subject guided by scene layouts and text descriptions. In this manner, our SceneBooth ensures accurate preservation of the subject's appearance in the output. Quantitative and qualitative experimental results demonstrate that SceneBooth significantly outperforms baseline methods in terms of subject preservation, image harmonization and overall quality.
• 摘要：由于对个性化图像生成的需求，主题驱动的文本到图像生成方法（该方法基于文本提示创建输入主题的新颖演绎）已受到越来越多的研究兴趣。现有方法通常学习主题表示并将其合并到提示嵌入中以指导图像生成，但它们在保持主题保真度方面存在困难。为了解决这个问题，本文提出了一种名为 SceneBooth 的新颖框架，用于保留主题的文本到图像生成，该框架使用主题图像、对象短语和文本提示作为输入。我们的 SceneBooth 不是学习主题表示并生成主题，而是修复给定的主题图像并在文本提示的引导下生成其背景图像。为此，我们的 SceneBooth 引入了两个关键组件，即多模态布局生成模块和背景绘制模块。前者通过生成与文本标题、对象短语和主题视觉信息一致的适当场景布局来确定主题的位置和比例。后者将两个适配器（ControlNet 和 Gated Self-Attention）集成到潜在扩散模型中，以生成由场景布局和文本描述引导的与主体协调的背景。通过这种方式，我们的 SceneBooth 确保在输出中准确保留主体的外观。定量和定性实验结果表明，SceneBooth 在主体保留、图像协调和整体质量方面明显优于基线方法。

Title: Textualize Visual Prompt for Image Editing via Diffusion Bridge

• Authors: Pengcheng Xu, Qingnan Fan, Fei Kou, Shuai Qin, Hong Gu, Ruoyu Zhao, Charles Ling, Boyu Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03495
• Pdf URL: https://arxiv.org/pdf/2501.03495
• Copy Paste: [[2501.03495]] Textualize Visual Prompt for Image Editing via Diffusion Bridge(https://arxiv.org/abs/2501.03495)
• Keywords: generative
• Abstract: Visual prompt, a pair of before-and-after edited images, can convey indescribable imagery transformations and prosper in image editing. However, current visual prompt methods rely on a pretrained text-guided image-to-image generative model that requires a triplet of text, before, and after images for retraining over a text-to-image model. Such crafting triplets and retraining processes limit the scalability and generalization of editing. In this paper, we present a framework based on any single text-to-image model without reliance on the explicit image-to-image model thus enhancing the generalizability and scalability. Specifically, by leveraging the probability-flow ordinary equation, we construct a diffusion bridge to transfer the distribution between before-and-after images under the text guidance. By optimizing the text via the bridge, the framework adaptively textualizes the editing transformation conveyed by visual prompts into text embeddings without other models. Meanwhile, we introduce differential attention control during text optimization, which disentangles the text embedding from the invariance of the before-and-after images and makes it solely capture the delicate transformation and generalize to edit various images. Experiments on real images validate competitive results on the generalization, contextual coherence, and high fidelity for delicate editing with just one image pair as the visual prompt.
• 摘要：视觉提示是一对编辑前后的图像，可以传达难以形容的图像转换并在图像编辑中大放异彩。然而，目前的视觉提示方法依赖于预训练的文本引导的图像到图像生成模型，该模型需要一组文本、编辑前和编辑后图像才能在文本到图像模型上进行再训练。这种制作三元组和再训练过程限制了编辑的可扩展性和泛化。在本文中，我们提出了一个基于任何单一文本到图像模型的框架，而不依赖于显式的图像到图像模型，从而增强了泛化性和可扩展性。具体而言，通过利用概率流常方程，我们构建了一个扩散桥，在文本引导下转移前后图像之间的分布。通过经由该桥优化文本，该框架无需其他模型，就能将视觉提示传达的编辑转换自适应地文本化为文本嵌入。同时，我们在文本优化过程中引入了差异化注意力控制，将文本嵌入与前后图像的不变性分离开来，使其仅捕捉细微的变换并推广到编辑各种图像。在真实图像上进行的实验验证了在泛化、上下文连贯性和精细编辑的高保真度方面具有竞争力的结果，仅使用一个图像对作为视觉提示即可实现。

Title: Can Deep Learning Trigger Alerts from Mobile-Captured Images?

• Authors: Pritisha Sarkar, Duranta Durbaar Vishal Saha, Mousumi Saha
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03499
• Pdf URL: https://arxiv.org/pdf/2501.03499
• Copy Paste: [[2501.03499]] Can Deep Learning Trigger Alerts from Mobile-Captured Images?(https://arxiv.org/abs/2501.03499)
• Keywords: quality assessment
• Abstract: Our research presents a comprehensive approach to leveraging mobile camera image data for real-time air quality assessment and recommendation. We develop a regression-based Convolutional Neural Network model and tailor it explicitly for air quality prediction by exploiting the inherent relationship between output parameters. As a result, the Mean Squared Error of 0.0077 and 0.0112 obtained for 2 and 5 pollutants respectively outperforms existing models. Furthermore, we aim to verify the common practice of augmenting the original dataset with a view to introducing more variation in the training phase. It is one of our most significant contributions that our experimental results demonstrate minimal accuracy differences between the original and augmented datasets. Finally, a real-time, user-friendly dashboard is implemented which dynamically displays the Air Quality Index and pollutant values derived from captured mobile camera images. Users' health conditions are considered to recommend whether a location is suitable based on current air quality metrics. Overall, this research contributes to verification of data augmentation techniques, CNN-based regression modelling for air quality prediction, and user-centric air quality monitoring through mobile technology. The proposed system offers practical solutions for individuals to make informed environmental health and well-being decisions.
• 摘要：我们的研究提出了一种利用移动摄像头图像数据进行实时空气质量评估和推荐的综合方法。我们开发了一个基于回归的卷积神经网络模型，并通过利用输出参数之间的固有关系明确地将其定制为空气质量预测。结果，分别针对 2 种和 5 种污染物获得的 0.0077 和 0.0112 的均方误差优于现有模型。此外，我们旨在验证增强原始数据集的常见做法，以期在训练阶段引入更多变化。我们最重要的贡献之一是，我们的实验结果表明原始数据集和增强数据集之间的准确度差异最小。最后，实现了一个实时、用户友好的仪表板，可动态显示从捕获的移动摄像头图像中得出的空气质量指数和污染物值。考虑用户的健康状况，根据当前空气质量指标推荐某个位置是否合适。总体而言，这项研究有助于验证数据增强技术、基于 CNN 的空气质量预测回归模型以及通过移动技术进行以用户为中心的空气质量监测。所提出的系统为个人做出明智的环境健康和福祉决策提供了实用的解决方案。

Title: Deep Learning within Tabular Data: Foundations, Challenges, Advances and Future Directions

• Authors: Weijieying Ren, Tianxiang Zhao, Yuqing Huang, Vasant Honavar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03540
• Pdf URL: https://arxiv.org/pdf/2501.03540
• Copy Paste: [[2501.03540]] Deep Learning within Tabular Data: Foundations, Challenges, Advances and Future Directions(https://arxiv.org/abs/2501.03540)
• Keywords: generation
• Abstract: Tabular data remains one of the most prevalent data types across a wide range of real-world applications, yet effective representation learning for this domain poses unique challenges due to its irregular patterns, heterogeneous feature distributions, and complex inter-column dependencies. This survey provides a comprehensive review of state-of-the-art techniques in tabular data representation learning, structured around three foundational design elements: training data, neural architectures, and learning objectives. Unlike prior surveys that focus primarily on either architecture design or learning strategies, we adopt a holistic perspective that emphasizes the universality and robustness of representation learning methods across diverse downstream tasks. We examine recent advances in data augmentation and generation, specialized neural network architectures tailored to tabular data, and innovative learning objectives that enhance representation quality. Additionally, we highlight the growing influence of self-supervised learning and the adaptation of transformer-based foundation models for tabular data. Our review is based on a systematic literature search using rigorous inclusion criteria, encompassing 127 papers published since 2020 in top-tier conferences and journals. Through detailed analysis and comparison, we identify emerging trends, critical gaps, and promising directions for future research, aiming to guide the development of more generalizable and effective tabular data representation methods.
• 摘要：表格数据仍然是现实世界中广泛应用的最普遍的数据类型之一，但由于其不规则的模式、异构的特征分布和复杂的列间依赖关系，该领域的有效表示学习带来了独特的挑战。本综述全面回顾了表格数据表示学习的最新技术，围绕三个基础设计元素展开：训练数据、神经架构和学习目标。与之前主要关注架构设计或学习策略的综述不同，我们采用整体视角，强调表示学习方法在不同下游任务中的通用性和稳健性。我们研究了数据增强和生成方面的最新进展、针对表格数据的专用神经网络架构以及提高表示质量的创新学习目标。此外，我们还强调了自监督学习日益增长的影响力以及基于 Transformer 的基础模型对表格数据的适应性。我们的综述基于使用严格纳入标准的系统文献检索，涵盖了自 2020 年以来在顶级会议和期刊上发表的 127 篇论文。通过详细的分析和比较，我们确定了新兴趋势、关键差距和未来研究的有希望的方向，旨在指导开发更具通用性和更有效的表格数据表示方法。

Title: PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models

• Authors: Lingzhi Yuan, Xinfeng Li, Chejian Xu, Guanhong Tao, Xiaojun Jia, Yihao Huang, Wei Dong, Yang Liu, XiaoFeng Wang, Bo Li
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.03544
• Pdf URL: https://arxiv.org/pdf/2501.03544
• Copy Paste: [[2501.03544]] PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models(https://arxiv.org/abs/2501.03544)
• Keywords: generation
• Abstract: Text-to-image (T2I) models have been shown to be vulnerable to misuse, particularly in generating not-safe-for-work (NSFW) content, raising serious ethical concerns. In this work, we present PromptGuard, a novel content moderation technique that draws inspiration from the system prompt mechanism in large language models (LLMs) for safety alignment. Unlike LLMs, T2I models lack a direct interface for enforcing behavioral guidelines. Our key idea is to optimize a safety soft prompt that functions as an implicit system prompt within the T2I model's textual embedding space. This universal soft prompt (P*) directly moderates NSFW inputs, enabling safe yet realistic image generation without altering the inference efficiency or requiring proxy models. Extensive experiments across three datasets demonstrate that PromptGuard effectively mitigates NSFW content generation while preserving high-quality benign outputs. PromptGuard achieves 7.8 times faster than prior content moderation methods, surpassing eight state-of-the-art defenses with an optimal unsafe ratio down to 5.84%.
• 摘要：文本转图像 (T2I) 模型已被证明容易被滥用，特别是在生成不适合工作 (NSFW) 的内容时，这引发了严重的道德问题。在这项工作中，我们提出了 PromptGuard，这是一种新颖的内容审核技术，其灵感来自大型语言模型 (LLM) 中的系统提示机制，以实现安全性。与 LLM 不同，T2I 模型缺乏用于执行行为准则的直接界面。我们的主要思想是优化安全软提示，该提示在 T2I 模型的文本嵌入空间中充当隐式系统提示。这种通用软提示 (P*) 直接调节 NSFW 输入，实现安全而逼真的图像生成，而不会改变推理效率或需要代理模型。在三个数据集上进行的大量实验表明，PromptGuard 可有效缓解 NSFW 内容生成，同时保留高质量的良性输出。 PromptGuard 比之前的内容审核方法快 7.8 倍，超越了八种最先进的防御方法，最佳不安全比率降至 5.84%。

Title: Evaluating Image Caption via Cycle-consistent Text-to-Image Generation

• Authors: Tianyu Cui, Jinbin Bai, Guohua Wang, Qingguo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Ye Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03567
• Pdf URL: https://arxiv.org/pdf/2501.03567
• Copy Paste: [[2501.03567]] Evaluating Image Caption via Cycle-consistent Text-to-Image Generation(https://arxiv.org/abs/2501.03567)
• Keywords: generation
• Abstract: Evaluating image captions typically relies on reference captions, which are costly to obtain and exhibit significant diversity and subjectivity. While reference-free evaluation metrics have been proposed, most focus on cross-modal evaluation between captions and images. Recent research has revealed that the modality gap generally exists in the representation of contrastive learning-based multi-modal systems, undermining the reliability of cross-modality metrics like CLIPScore. In this paper, we propose CAMScore, a cyclic reference-free automatic evaluation metric for image captioning models. To circumvent the aforementioned modality gap, CAMScore utilizes a text-to-image model to generate images from captions and subsequently evaluates these generated images against the original images. Furthermore, to provide fine-grained information for a more comprehensive evaluation, we design a three-level evaluation framework for CAMScore that encompasses pixel-level, semantic-level, and objective-level perspectives. Extensive experiment results across multiple benchmark datasets show that CAMScore achieves a superior correlation with human judgments compared to existing reference-based and reference-free metrics, demonstrating the effectiveness of the framework.
• 摘要：评估图像字幕通常依赖于参考字幕，而参考字幕的获取成本高昂，并且具有很大的多样性和主观性。虽然已经提出了无参考的评估指标，但大多数指标都侧重于字幕和图像之间的跨模态评估。最近的研究表明，模态差距通常存在于基于对比学习的多模态系统的表示中，从而削弱了 CLIPScore 等跨模态指标的可靠性。在本文中，我们提出了 CAMScore，这是一种用于图像字幕模型的循环无参考自动评估指标。为了规避上述模态差距，CAMScore 利用文本到图像模型从字幕生成图像，然后根据原始图像评估这些生成的图像。此外，为了提供更全面评估的细粒度信息，我们为 CAMScore 设计了一个三级评估框架，涵盖像素级、语义级和客观级视角。在多个基准数据集上的大量实验结果表明，与现有的基于参考和无参考的指标相比，CAMScore 与人类判断实现了更高的相关性，证明了该框架的有效性。

Title: Discriminative Representation learning via Attention-Enhanced Contrastive Learning for Short Text Clustering

• Authors: Zhihao Yao
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03584
• Pdf URL: https://arxiv.org/pdf/2501.03584
• Copy Paste: [[2501.03584]] Discriminative Representation learning via Attention-Enhanced Contrastive Learning for Short Text Clustering(https://arxiv.org/abs/2501.03584)
• Keywords: generation
• Abstract: Contrastive learning has gained significant attention in short text clustering, yet it has an inherent drawback of mistakenly identifying samples from the same category as negatives and then separating them in the feature space (false negative separation), which hinders the generation of superior representations. To generate more discriminative representations for efficient clustering, we propose a novel short text clustering method, called Discriminative Representation learning via \textbf{A}ttention-\textbf{E}nhanced \textbf{C}ontrastive \textbf{L}earning for Short Text Clustering (\textbf{AECL}). The \textbf{AECL} consists of two modules which are the pseudo-label generation module and the contrastive learning module. Both modules build a sample-level attention mechanism to capture similarity relationships between samples and aggregate cross-sample features to generate consistent representations. Then, the former module uses the more discriminative consistent representation to produce reliable supervision information for assist clustering, while the latter module explores similarity relationships and consistent representations optimize the construction of positive samples to perform similarity-guided contrastive learning, effectively addressing the false negative separation issue. Experimental results demonstrate that the proposed \textbf{AECL} outperforms state-of-the-art methods. If the paper is accepted, we will open-source the code.
• 摘要：对比学习在短文本聚类中引起了广泛关注，但它有一个固有的缺点，就是会错误地将来自同一类别的样本标识为负数，然后在特征空间中将它们分离（假负数分离），这阻碍了优质表示的生成。为了生成更具判别性的表示以进行高效的聚类，我们提出了一种新颖的短文本聚类方法，称为通过 \textbf{A}ttention-\textbf{E}nhanced \textbf{C}contrastive \textbf{L}learning 进行判别性表示学习的短文本聚类（\textbf{AECL}）。\textbf{AECL} 由两个模块组成，即伪标签生成模块和对比学习模块。这两个模块都构建了样本级注意力机制来捕捉样本之间的相似关系，并聚合跨样本特征以生成一致的表示。然后，前一个模块使用更具判别性的一致性表示来产生可靠的监督信息以辅助聚类，而后一个模块探索相似性关系和一致性表示来优化正样本的构造以进行相似性引导的对比学习，从而有效地解决假阴性分离问题。实验结果表明，所提出的 \textbf{AECL} 优于最先进的方法。如果论文被接受，我们将开源代码。

Title: Deep Learning-based Compression Detection for explainable Face Image Quality Assessment

• Authors: Laurin Jonientz, Johannes Merkle, Christian Rathgeb, Benjamin Tams, Georg Merz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03619
• Pdf URL: https://arxiv.org/pdf/2501.03619
• Copy Paste: [[2501.03619]] Deep Learning-based Compression Detection for explainable Face Image Quality Assessment(https://arxiv.org/abs/2501.03619)
• Keywords: quality assessment
• Abstract: The assessment of face image quality is crucial to ensure reliable face recognition. In order to provide data subjects and operators with explainable and actionable feedback regarding captured face images, relevant quality components have to be measured. Quality components that are known to negatively impact the utility of face images include JPEG and JPEG 2000 compression artefacts, among others. Compression can result in a loss of important image details which may impair the recognition performance. In this work, deep neural networks are trained to detect the compression artefacts in a face images. For this purpose, artefact-free facial images are compressed with the JPEG and JPEG 2000 compression algorithms. Subsequently, the PSNR and SSIM metrics are employed to obtain training labels based on which neural networks are trained using a single network to detect JPEG and JPEG 2000 artefacts, respectively. The evaluation of the proposed method shows promising results: in terms of detection accuracy, error rates of 2-3% are obtained for utilizing PSNR labels during training. In addition, we show that error rates of different open-source and commercial face recognition systems can be significantly reduced by discarding face images exhibiting severe compression artefacts. To minimize resource consumption, EfficientNetV2 serves as basis for the presented algorithm, which is available as part of the OFIQ software.
• 摘要：评估人脸图像质量对于确保可靠的人脸识别至关重要。为了向数据主体和操作员提供有关捕获的人脸图像的可解释和可操作的反馈，必须测量相关的质量组件。已知会对人脸图像的效用产生负面影响的质量组件包括 JPEG 和 JPEG 2000 压缩伪影等。压缩会导致重要图像细节的丢失，从而可能损害识别性能。在这项工作中，训练深度神经网络来检测人脸图像中的压缩伪影。为此，使用 JPEG 和 JPEG 2000 压缩算法压缩无伪影的面部图像。随后，使用 PSNR 和 SSIM 指标来获得训练标签，在此基础上分别使用单个网络训练神经网络以检测 JPEG 和 JPEG 2000 伪影。对所提出方法的评估显示出令人鼓舞的结果：就检测准确性而言，在训练期间使用 PSNR 标签的错误率为 2-3%。此外，我们表明，通过丢弃表现出严重压缩伪影的面部图像，可以显著降低不同开源和商业人脸识别系统的错误率。为了最大限度地减少资源消耗，EfficientNetV2 是所提出算法的基础，该算法是 OFIQ 软件的一部分。

Title: Advancing the Understanding of Fine-Grained 3D Forest Structures using Digital Cousins and Simulation-to-Reality: Methods and Datasets

• Authors: Jing Liu, Duanchu Wang, Haoran Gong, Chongyu Wang, Jihua Zhu, Di Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03637
• Pdf URL: https://arxiv.org/pdf/2501.03637
• Copy Paste: [[2501.03637]] Advancing the Understanding of Fine-Grained 3D Forest Structures using Digital Cousins and Simulation-to-Reality: Methods and Datasets(https://arxiv.org/abs/2501.03637)
• Keywords: generation
• Abstract: Understanding and analyzing the spatial semantics and structure of forests is essential for accurate forest resource monitoring and ecosystem research. However, the lack of large-scale and annotated datasets has limited the widespread use of advanced intelligent techniques in this field. To address this challenge, a fully automated synthetic data generation and processing framework based on the concepts of Digital Cousins and Simulation-to-Reality (Sim2Real) is proposed, offering versatility and scalability to any size and platform. Using this process, we created the Boreal3D, the world's largest forest point cloud dataset. It includes 1000 highly realistic and structurally diverse forest plots across four different platforms, totaling 48,403 trees and over 35.3 billion points. Each point is labeled with semantic, instance, and viewpoint information, while each tree is described with structural parameters such as diameter, crown width, leaf area, and total volume. We designed and conducted extensive experiments to evaluate the potential of Boreal3D in advancing fine-grained 3D forest structure analysis in real-world applications. The results demonstrate that with certain strategies, models pre-trained on synthetic data can significantly improve performance when applied to real forest datasets. Especially, the findings reveal that fine-tuning with only 20% of real-world data enables the model to achieve performance comparable to models trained exclusively on entire real-world data, highlighting the value and potential of our proposed framework. The Boreal3D dataset, and more broadly, the synthetic data augmentation framework, is poised to become a critical resource for advancing research in large-scale 3D forest scene understanding and structural parameter estimation.
• 摘要：理解和分析森林的空间语义和结构对于准确的森林资源监测和生态系统研究至关重要。然而，缺乏大规模和带注释的数据集限制了先进智能技术在该领域的广泛使用。为了应对这一挑战，提出了一种基于数字表亲和模拟到现实 (Sim2Real) 概念的全自动合成数据生成和处理框架，为任何规模和平台提供多功能性和可扩展性。通过这一过程，我们创建了世界上最大的森林点云数据集 Boreal3D。它包括四个不同平台上的 1000 个高度逼真且结构多样的森林地块，共计 48,403 棵树和超过 353 亿个点。每个点都标有语义、实例和视点信息，而每棵树都用直径、冠幅、叶面积和总体积等结构参数描述。我们设计并进行了广泛的实验，以评估 Boreal3D 在实际应用中推进细粒度 3D 森林结构分析的潜力。结果表明，通过某些策略，在合成数据上进行预训练的模型在应用于真实森林数据集时可以显著提高性能。特别是，研究结果表明，仅使用 20% 的真实世界数据进行微调就能使模型实现与完全使用整个真实世界数据进行训练的模型相当的性能，这凸显了我们提出的框架的价值和潜力。Boreal3D 数据集，更广泛地说，合成数据增强框架，有望成为推动大规模 3D 森林场景理解和结构参数估计研究的重要资源。

Title: Action Quality Assessment via Hierarchical Pose-guided Multi-stage Contrastive Regression

• Authors: Mengshi Qi, Hao Ye, Jiaxuan Peng, Huadong Ma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03674
• Pdf URL: https://arxiv.org/pdf/2501.03674
• Copy Paste: [[2501.03674]] Action Quality Assessment via Hierarchical Pose-guided Multi-stage Contrastive Regression(https://arxiv.org/abs/2501.03674)
• Keywords: quality assessment
• Abstract: Action Quality Assessment (AQA), which aims at automatic and fair evaluation of athletic performance, has gained increasing attention in recent years. However, athletes are often in rapid movement and the corresponding visual appearance variances are subtle, making it challenging to capture fine-grained pose differences and leading to poor estimation performance. Furthermore, most common AQA tasks, such as diving in sports, are usually divided into multiple sub-actions, each of which contains different durations. However, existing methods focus on segmenting the video into fixed frames, which disrupts the temporal continuity of sub-actions resulting in unavoidable prediction errors. To address these challenges, we propose a novel action quality assessment method through hierarchically pose-guided multi-stage contrastive regression. Firstly, we introduce a multi-scale dynamic visual-skeleton encoder to capture fine-grained spatio-temporal visual and skeletal features. Then, a procedure segmentation network is introduced to separate different sub-actions and obtain segmented features. Afterwards, the segmented visual and skeletal features are both fed into a multi-modal fusion module as physics structural priors, to guide the model in learning refined activity similarities and variances. Finally, a multi-stage contrastive learning regression approach is employed to learn discriminative representations and output prediction results. In addition, we introduce a newly-annotated FineDiving-Pose Dataset to improve the current low-quality human pose labels. In experiments, the results on FineDiving and MTL-AQA datasets demonstrate the effectiveness and superiority of our proposed approach. Our source code and dataset are available at this https URL.
• 摘要：动作质量评估 (AQA) 旨在自动、公平地评估运动表现，近年来受到越来越多的关注。然而，运动员通常处于快速运动状态，相应的视觉外观变化很细微，因此很难捕捉细粒度的姿势差异并导致估计性能不佳。此外，最常见的 AQA 任务，例如运动中的跳水，通常分为多个子动作，每个子动作包含不同的持续时间。然而，现有方法专注于将视频分割成固定帧，这会破坏子动作的时间连续性，导致不可避免的预测误差。为了应对这些挑战，我们通过分层姿势引导的多阶段对比回归提出了一种新颖的动作质量评估方法。首先，我们引入多尺度动态视觉骨架编码器来捕获细粒度的时空视觉和骨架特征。然后，引入过程分割网络来分离不同的子动作并获得分割特征。之后，将分割后的视觉和骨骼特征作为物理结构先验输入到多模态融合模块中，以指导模型学习精细的活动相似性和差异性。最后，采用多阶段对比学习回归方法来学习判别性表示并输出预测结果。此外，我们引入了一个新注释的 FineDiving-Pose 数据集来改进当前低质量的人体姿势标签。在实验中，FineDiving 和 MTL-AQA 数据集上的结果证明了我们提出的方法的有效性和优越性。我们的源代码和数据集可在此 https URL 上找到。

Title: SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoning

• Authors: Andrew Li, Rahul Thapa, Rahul Chalamala, Qingyang Wu, Kezhen Chen, James Zou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03675
• Pdf URL: https://arxiv.org/pdf/2501.03675
• Copy Paste: [[2501.03675]] SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoning(https://arxiv.org/abs/2501.03675)
• Keywords: generation
• Abstract: Vision-Language Models (VLMs) have shown strong performance in understanding single images, aided by numerous high-quality instruction datasets. However, multi-image reasoning tasks are still under-explored in the open-source community due to two main challenges: (1) scaling datasets with multiple correlated images and complex reasoning instructions is resource-intensive and maintaining quality is difficult, and (2) there is a lack of robust evaluation benchmarks for multi-image tasks. To address these issues, we introduce SMIR, an efficient synthetic data-generation pipeline for multi-image reasoning, and a high-quality dataset generated using this pipeline. Our pipeline efficiently extracts highly correlated images using multimodal embeddings, combining visual and descriptive information and leverages open-source LLMs to generate quality instructions. Using this pipeline, we generated 160K synthetic training samples, offering a cost-effective alternative to expensive closed-source solutions. Additionally, we present SMIR-BENCH, a novel multi-image reasoning evaluation benchmark comprising 200 diverse examples across 7 complex multi-image reasoning tasks. SMIR-BENCH is multi-turn and utilizes a VLM judge to evaluate free-form responses, providing a comprehensive assessment of model expressiveness and reasoning capability across modalities. We demonstrate the effectiveness of SMIR dataset by fine-tuning several open-source VLMs and evaluating their performance on SMIR-BENCH. Our results show that models trained on our dataset outperform baseline models in multi-image reasoning tasks up to 8% with a much more scalable data pipeline.
• 摘要：视觉语言模型 (VLM) 在理解单个图像方面表现出色，这得益于大量高质量的指令数据集。然而，由于两个主要挑战，开源社区对多图像推理任务的探索仍然不足：(1) 扩展具有多个相关图像和复杂推理指令的数据集需要大量资源，并且难以保持质量，(2) 缺乏针对多图像任务的稳健评估基准。为了解决这些问题，我们引入了 SMIR，一种用于多图像推理的高效合成数据生成管道，以及使用此管道生成的高质量数据集。我们的管道使用多模态嵌入高效地提取高度相关的图像，结合视觉和描述性信息，并利用开源 LLM 生成高质量指令。使用此管道，我们生成了 160K 个合成训练样本，为昂贵的闭源解决方案提供了一种经济高效的替代方案。此外，我们还提出了 SMIR-BENCH，这是一种新颖的多图像推理评估基准，包含 7 个复杂多图像推理任务中的 200 个不同示例。 SMIR-BENCH 是多轮的，利用 VLM 评判器来评估自由形式的响应，从而全面评估模型的表现力和跨模态的推理能力。我们通过微调几个开源 VLM 并评估它们在 SMIR-BENCH 上的性能来证明 SMIR 数据集的有效性。我们的结果表明，在我们的数据集上训练的模型在多图像推理任务中的表现比基线模型高出 8%，并且数据管道的可扩展性更高。

Title: Exploring Molecule Generation Using Latent Space Graph Diffusion

• Authors: Prashanth Pombala, Gerrit Grossmann, Verena Wolf
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03696
• Pdf URL: https://arxiv.org/pdf/2501.03696
• Copy Paste: [[2501.03696]] Exploring Molecule Generation Using Latent Space Graph Diffusion(https://arxiv.org/abs/2501.03696)
• Keywords: generation, generative
• Abstract: Generating molecular graphs is a challenging task due to their discrete nature and the competitive objectives involved. Diffusion models have emerged as SOTA approaches in data generation across various modalities. For molecular graphs, graph neural networks (GNNs) as a diffusion backbone have achieved impressive results. Latent space diffusion, where diffusion occurs in a low-dimensional space via an autoencoder, has demonstrated computational efficiency. However, the literature on latent space diffusion for molecular graphs is scarce, and no commonly accepted best practices exist. In this work, we explore different approaches and hyperparameters, contrasting generative flow models (denoising diffusion, flow matching, heat dissipation) and architectures (GNNs and E(3)-equivariant GNNs). Our experiments reveal a high sensitivity to the choice of approach and design decisions. Code is made available at this http URL.
• 摘要：生成分子图是一项具有挑战性的任务，因为它们具有离散性质，并且涉及竞争目标。扩散模型已成为跨各种模态数据生成的 SOTA 方法。对于分子图，图神经网络 (GNN) 作为扩散主干取得了令人印象深刻的结果。潜在空间扩散（其中扩散通过自动编码器在低维空间中发生）已证明具有计算效率。然而，关于分子图潜在空间扩散的文献很少，并且没有普遍接受的最佳实践。在这项工作中，我们探索了不同的方法和超参数，对比了生成流模型（去噪扩散、流匹配、散热）和架构（GNN 和 E(3)-等变 GNN）。我们的实验表明，方法的选择和设计决策具有很高的敏感性。代码可在此 http URL 上获得。

Title: Motion-Aware Generative Frame Interpolation

• Authors: Guozhen Zhang, Yuhan Zhu, Yutao Cui, Xiaotong Zhao, Kai Ma, Limin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03699
• Pdf URL: https://arxiv.org/pdf/2501.03699
• Copy Paste: [[2501.03699]] Motion-Aware Generative Frame Interpolation(https://arxiv.org/abs/2501.03699)
• Keywords: generation, generative
• Abstract: Generative frame interpolation, empowered by large-scale pre-trained video generation models, has demonstrated remarkable advantages in complex scenes. However, existing methods heavily rely on the generative model to independently infer the correspondences between input frames, an ability that is inadequately developed during pre-training. In this work, we propose a novel framework, termed Motion-aware Generative frame interpolation (MoG), to significantly enhance the model's motion awareness by integrating explicit motion guidance. Specifically we investigate two key questions: what can serve as an effective motion guidance, and how we can seamlessly embed this guidance into the generative model. For the first question, we reveal that the intermediate flow from flow-based interpolation models could efficiently provide task-oriented motion guidance. Regarding the second, we first obtain guidance-based representations of intermediate frames by warping input frames' representations using guidance, and then integrate them into the model at both latent and feature levels. To demonstrate the versatility of our method, we train MoG on both real-world and animation datasets. Comprehensive evaluations show that our MoG significantly outperforms the existing methods in both domains, achieving superior video quality and improved fidelity.
• 摘要：生成帧插值由大规模预训练视频生成模型提供支持，在复杂场景中表现出显著的优势。然而，现有的方法严重依赖生成模型独立推断输入帧之间的对应关系，而这种能力在预训练期间尚未得到充分发展。在这项工作中，我们提出了一个新框架，称为运动感知生成帧插值 (MoG)，通过集成显式运动指导显著增强模型的运动感知。具体来说，我们研究两个关键问题：什么可以作为有效的运动指导，以及如何将这种指导无缝嵌入到生成模型中。对于第一个问题，我们揭示了基于流的插值模型中的中间流可以有效地提供面向任务的运动指导。关于第二个问题，我们首先通过使用指导扭曲输入帧的表示来获得中间帧的基于指导的表示，然后将它们集成到潜在和特征级别的模型中。为了证明我们方法的多功能性，我们在现实世界和动画数据集上训练 MoG。综合评估表明，我们的 MoG 在两个领域都明显优于现有方法，实现了卓越的视频质量和更高的保真度。

Title: Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control

• Authors: Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, Wenping Wang, Yuan Liu
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.03847
• Pdf URL: https://arxiv.org/pdf/2501.03847
• Copy Paste: [[2501.03847]] Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control(https://arxiv.org/abs/2501.03847)
• Keywords: generation
• Abstract: Diffusion models have demonstrated impressive performance in generating high-quality videos from text prompts or images. However, precise control over the video generation process, such as camera manipulation or content editing, remains a significant challenge. Existing methods for controlled video generation are typically limited to a single control type, lacking the flexibility to handle diverse control demands. In this paper, we introduce Diffusion as Shader (DaS), a novel approach that supports multiple video control tasks within a unified architecture. Our key insight is that achieving versatile video control necessitates leveraging 3D control signals, as videos are fundamentally 2D renderings of dynamic 3D content. Unlike prior methods limited to 2D control signals, DaS leverages 3D tracking videos as control inputs, making the video diffusion process inherently 3D-aware. This innovation allows DaS to achieve a wide range of video controls by simply manipulating the 3D tracking videos. A further advantage of using 3D tracking videos is their ability to effectively link frames, significantly enhancing the temporal consistency of the generated videos. With just 3 days of fine-tuning on 8 H800 GPUs using less than 10k videos, DaS demonstrates strong control capabilities across diverse tasks, including mesh-to-video generation, camera control, motion transfer, and object manipulation.
• 摘要：扩散模型在从文本提示或图像生成高质量视频方面表现出色。然而，对视频生成过程的精确控制，例如相机操作或内容编辑，仍然是一个重大挑战。现有的受控视频生成方法通常仅限于单一控制类型，缺乏处理各种控制需求的灵活性。在本文中，我们介绍了扩散着色器 (DaS)，这是一种在统一架构中支持多种视频控制任务的新方法。我们的主要见解是，实现多功能视频控制需要利用 3D 控制信号，因为视频从根本上来说是动态 3D 内容的 2D 渲染。与之前仅限于 2D 控制信号的方法不同，DaS 利用 3D 跟踪视频作为控制输入，使视频扩散过程本身具有 3D 感知能力。这项创新使 DaS 能够通过简单地操作 3D 跟踪视频来实现广泛的视频控制。使用 3D 跟踪视频的另一个优势是它们能够有效地链接帧，从而显著增强生成视频的时间一致性。通过在 8 个 H800 GPU 上使用不到 10k 个视频仅进行 3 天的微调，DaS 就展示了在不同任务中的强大控制能力，包括网格到视频生成、摄像头控制、运动传输和对象操作。

Title: Magic Mirror: ID-Preserved Video Generation in Video Diffusion Transformers

• Authors: Yuechen Zhang, Yaoyang Liu, Bin Xia, Bohao Peng, Zexin Yan, Eric Lo, Jiaya Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03931
• Pdf URL: https://arxiv.org/pdf/2501.03931
• Copy Paste: [[2501.03931]] Magic Mirror: ID-Preserved Video Generation in Video Diffusion Transformers(https://arxiv.org/abs/2501.03931)
• Keywords: generation
• Abstract: We present Magic Mirror, a framework for generating identity-preserved videos with cinematic-level quality and dynamic motion. While recent advances in video diffusion models have shown impressive capabilities in text-to-video generation, maintaining consistent identity while producing natural motion remains challenging. Previous methods either require person-specific fine-tuning or struggle to balance identity preservation with motion diversity. Built upon Video Diffusion Transformers, our method introduces three key components: (1) a dual-branch facial feature extractor that captures both identity and structural features, (2) a lightweight cross-modal adapter with Conditioned Adaptive Normalization for efficient identity integration, and (3) a two-stage training strategy combining synthetic identity pairs with video data. Extensive experiments demonstrate that Magic Mirror effectively balances identity consistency with natural motion, outperforming existing methods across multiple metrics while requiring minimal parameters added. The code and model will be made publicly available at: this https URL
• 摘要：我们提出了 Magic Mirror，这是一个用于生成具有电影级质量和动态运动的身份保留视频的框架。虽然视频扩散模型的最新进展在文本到视频生成方面表现出了令人印象深刻的能力，但在产生自然运动的同时保持一致的身份仍然具有挑战性。以前的方法要么需要针对个人进行微调，要么难以平衡身份保留和运动多样性。我们的方法基于视频扩散变换器，引入了三个关键组件：（1）双分支面部特征提取器，可同时捕获身份和结构特征，（2）具有条件自适应规范化的轻量级跨模态适配器，可实现有效的身份集成，以及（3）将合成身份对与视频数据相结合的两阶段训练策略。大量实验表明，Magic Mirror 有效地平衡了身份一致性和自然运动，在多个指标上的表现优于现有方法，同时只需添加最少的参数。代码和模型将在以下网址公开：此 https URL

Title: A precise asymptotic analysis of learning diffusion models: theory and insights

• Authors: Hugo Cui, Cengiz Pehlevan, Yue M. Lu
• Subjects: cs.LG, cond-mat.dis-nn
• Abstract URL: https://arxiv.org/abs/2501.03937
• Pdf URL: https://arxiv.org/pdf/2501.03937
• Copy Paste: [[2501.03937]] A precise asymptotic analysis of learning diffusion models: theory and insights(https://arxiv.org/abs/2501.03937)
• Keywords: generative
• Abstract: In this manuscript, we consider the problem of learning a flow or diffusion-based generative model parametrized by a two-layer auto-encoder, trained with online stochastic gradient descent, on a high-dimensional target density with an underlying low-dimensional manifold structure. We derive a tight asymptotic characterization of low-dimensional projections of the distribution of samples generated by the learned model, ascertaining in particular its dependence on the number of training samples. Building on this analysis, we discuss how mode collapse can arise, and lead to model collapse when the generative model is re-trained on generated synthetic data.
• 摘要：在本文中，我们考虑了学习基于流或扩散的生成模型的问题，该模型由两层自动编码器参数化，使用在线随机梯度下降进行训练，在高维目标密度上具有底层低维流形结构。我们推导出由学习模型生成的样本分布的低维投影的严格渐近特征，特别是确定了它对训练样本数量的依赖性。在此分析的基础上，我们讨论了模式崩溃如何发生，以及当生成模型在生成的合成数据上重新训练时导致模型崩溃。

Title: Visual question answering: from early developments to recent advances -- a survey

• Authors: Ngoc Dung Huynh, Mohamed Reda Bouadjenek, Sunil Aryal, Imran Razzak, Hakim Hacid
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2501.03939
• Pdf URL: https://arxiv.org/pdf/2501.03939
• Copy Paste: [[2501.03939]] Visual question answering: from early developments to recent advances -- a survey(https://arxiv.org/abs/2501.03939)
• Keywords: generation
• Abstract: Visual Question Answering (VQA) is an evolving research field aimed at enabling machines to answer questions about visual content by integrating image and language processing techniques such as feature extraction, object detection, text embedding, natural language understanding, and language generation. With the growth of multimodal data research, VQA has gained significant attention due to its broad applications, including interactive educational tools, medical image diagnosis, customer service, entertainment, and social media captioning. Additionally, VQA plays a vital role in assisting visually impaired individuals by generating descriptive content from images. This survey introduces a taxonomy of VQA architectures, categorizing them based on design choices and key components to facilitate comparative analysis and evaluation. We review major VQA approaches, focusing on deep learning-based methods, and explore the emerging field of Large Visual Language Models (LVLMs) that have demonstrated success in multimodal tasks like VQA. The paper further examines available datasets and evaluation metrics essential for measuring VQA system performance, followed by an exploration of real-world VQA applications. Finally, we highlight ongoing challenges and future directions in VQA research, presenting open questions and potential areas for further development. This survey serves as a comprehensive resource for researchers and practitioners interested in the latest advancements and future
• 摘要：视觉问答 (VQA) 是一个不断发展的研究领域，旨在通过集成图像和语言处理技术（例如特征提取、对象检测、文本嵌入、自然语言理解和语言生成）使机器能够回答有关视觉内容的问题。随着多模态数据研究的发展，VQA 因其广泛的应用而备受关注，包括交互式教育工具、医学图像诊断、客户服务、娱乐和社交媒体字幕。此外，VQA 通过从图像中生成描述性内容，在协助视障人士方面发挥着至关重要的作用。本综述介绍了 VQA 架构的分类法，根据设计选择和关键组件对其进行分类，以方便进行比较分析和评估。我们回顾了主要的 VQA 方法，重点关注基于深度学习的方法，并探索大型视觉语言模型 (LVLM) 这一新兴领域，这些模型在 VQA 等多模态任务中取得了成功。本文进一步研究了衡量 VQA 系统性能所必需的可用数据集和评估指标，然后探索了现实世界的 VQA 应用。最后，我们重点介绍了 VQA 研究的持续挑战和未来方向，提出了未解决的问题和有待进一步发展的领域。本调查为对最新进展和未来感兴趣的研究人员和从业者提供了全面的资源

Title: Synthetic Data Privacy Metrics

• Authors: Amy Steier, Lipika Ramaswamy, Andre Manoel, Alexa Haushalter
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03941
• Pdf URL: https://arxiv.org/pdf/2501.03941
• Copy Paste: [[2501.03941]] Synthetic Data Privacy Metrics(https://arxiv.org/abs/2501.03941)
• Keywords: generative
• Abstract: Recent advancements in generative AI have made it possible to create synthetic datasets that can be as accurate as real-world data for training AI models, powering statistical insights, and fostering collaboration with sensitive datasets while offering strong privacy guarantees. Effectively measuring the empirical privacy of synthetic data is an important step in the process. However, while there is a multitude of new privacy metrics being published every day, there currently is no standardization. In this paper, we review the pros and cons of popular metrics that include simulations of adversarial attacks. We also review current best practices for amending generative models to enhance the privacy of the data they create (e.g. differential privacy).
• 摘要：生成式人工智能的最新进展使得创建与真实世界数据一样准确的合成数据集成为可能，这些数据集可用于训练人工智能模型、提供统计见解以及促进与敏感数据集的协作，同时提供强大的隐私保障。有效地衡量合成数据的经验隐私是这一过程中的重要一步。然而，虽然每天都有大量新的隐私指标发布，但目前还没有标准化。在本文中，我们回顾了包括对抗性攻击模拟在内的流行指标的优缺点。我们还回顾了当前修改生成模型以增强其创建的数据的隐私性的最佳实践（例如差异隐私）。

Title: NeuralSVG: An Implicit Representation for Text-to-Vector Generation

• Authors: Sagi Polaczek, Yuval Alaluf, Elad Richardson, Yael Vinker, Daniel Cohen-Or
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.03992
• Pdf URL: https://arxiv.org/pdf/2501.03992
• Copy Paste: [[2501.03992]] NeuralSVG: An Implicit Representation for Text-to-Vector Generation(https://arxiv.org/abs/2501.03992)
• Keywords: generation
• Abstract: Vector graphics are essential in design, providing artists with a versatile medium for creating resolution-independent and highly editable visual content. Recent advancements in vision-language and diffusion models have fueled interest in text-to-vector graphics generation. However, existing approaches often suffer from over-parameterized outputs or treat the layered structure - a core feature of vector graphics - as a secondary goal, diminishing their practical use. Recognizing the importance of layered SVG representations, we propose NeuralSVG, an implicit neural representation for generating vector graphics from text prompts. Inspired by Neural Radiance Fields (NeRFs), NeuralSVG encodes the entire scene into the weights of a small MLP network, optimized using Score Distillation Sampling (SDS). To encourage a layered structure in the generated SVG, we introduce a dropout-based regularization technique that strengthens the standalone meaning of each shape. We additionally demonstrate that utilizing a neural representation provides an added benefit of inference-time control, enabling users to dynamically adapt the generated SVG based on user-provided inputs, all with a single learned representation. Through extensive qualitative and quantitative evaluations, we demonstrate that NeuralSVG outperforms existing methods in generating structured and flexible SVG.
• 摘要：矢量图形在设计中必不可少，它为艺术家提供了一种多功能媒介，用于创建与分辨率无关且高度可编辑的视觉内容。视觉语言和扩散模型的最新进展激发了人们对文本到矢量图形生成的兴趣。然而，现有方法通常存在过度参数化输出的问题，或者将分层结构（矢量图形的核心特征）视为次要目标，从而降低了它们的实际用途。认识到分层 SVG 表示的重要性，我们提出了 NeuralSVG，这是一种用于从文本提示生成矢量图形的隐式神经表示。受神经辐射场 (NeRF) 的启发，NeuralSVG 将整个场景编码为小型 MLP 网络的权重，并使用分数蒸馏采样 (SDS) 进行优化。为了鼓励生成的 SVG 中的分层结构，我们引入了一种基于 dropout 的正则化技术，以增强每个形状的独立含义。我们还证明了利用神经表征提供了推理时间控制的额外好处，使用户能够根据用户提供的输入动态调整生成的 SVG，所有这些都使用单一学习表征。通过广泛的定性和定量评估，我们证明了 NeuralSVG 在生成结构化和灵活的 SVG 方面优于现有方法。

Title: RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance

• Authors: Matin Mortaheb, Mohammad A. Amir Khojastepour, Srimat T. Chakradhar, Sennur Ulukus
• Subjects: cs.LG, cs.CV, cs.IR, cs.IT
• Abstract URL: https://arxiv.org/abs/2501.03995
• Pdf URL: https://arxiv.org/pdf/2501.03995
• Copy Paste: [[2501.03995]] RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance(https://arxiv.org/abs/2501.03995)
• Keywords: generation
• Abstract: Retrieval-augmented generation (RAG) improves large language models (LLMs) by using external knowledge to guide response generation, reducing hallucinations. However, RAG, particularly multi-modal RAG, can introduce new hallucination sources: (i) the retrieval process may select irrelevant pieces (e.g., documents, images) as raw context from the database, and (ii) retrieved images are processed into text-based context via vision-language models (VLMs) or directly used by multi-modal language models (MLLMs) like GPT-4o, which may hallucinate. To address this, we propose a novel framework to evaluate the reliability of multi-modal RAG using two performance measures: (i) the relevancy score (RS), assessing the relevance of retrieved entries to the query, and (ii) the correctness score (CS), evaluating the accuracy of the generated response. We train RS and CS models using a ChatGPT-derived database and human evaluator samples. Results show that both models achieve ~88% accuracy on test data. Additionally, we construct a 5000-sample human-annotated database evaluating the relevancy of retrieved pieces and the correctness of response statements. Our RS model aligns with human preferences 20% more often than CLIP in retrieval, and our CS model matches human preferences ~91% of the time. Finally, we assess various RAG systems' selection and generation performances using RS and CS.
• 摘要：检索增强生成 (RAG) 通过使用外部知识来指导响应生成，从而减少幻觉，从而改进了大型语言模型 (LLM)。然而，RAG，尤其是多模态 RAG，可能会引入新的幻​​觉来源：(i) 检索过程可能会从数据库中选择不相关的部分（例如文档、图像）作为原始上下文，以及 (ii) 检索到的图像通过视觉语言模型 (VLM) 处理为基于文本的上下文，或直接由 GPT-4o 等多模态语言模型 (MLLM) 使用，这可能会产生幻觉。为了解决这个问题，我们提出了一个新颖的框架，使用两个性能指标来评估多模态 RAG 的可靠性：(i) 相关性得分 (RS)，评估检索到的条目与查询的相关性，以及 (ii) 正确性得分 (CS)，评估生成的响应的准确性。我们使用 ChatGPT 衍生的数据库和人类评估者样本训练 RS 和 CS 模型。结果表明，两种模型在测试数据上的准确率均达到约 88%。此外，我们构建了一个 5000 个样本的人工注释数据库，用于评估检索到的片段的相关性和响应语句的正确性。我们的 RS 模型在检索中与人类偏好的匹配率比 CLIP 高 20%，而我们的 CS 模型与人类偏好的匹配率约为 91%。最后，我们使用 RS 和 CS 评估各种 RAG 系统的选择和生成性能。

Title: LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving

• Authors: Lingdong Kong, Xiang Xu, Youquan Liu, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, Ziwei Liu
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.04005
• Pdf URL: https://arxiv.org/pdf/2501.04005
• Copy Paste: [[2501.04005]] LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving(https://arxiv.org/abs/2501.04005)
• Keywords: generation
• Abstract: Recent advancements in vision foundation models (VFMs) have revolutionized visual perception in 2D, yet their potential for 3D scene understanding, particularly in autonomous driving applications, remains underexplored. In this paper, we introduce LargeAD, a versatile and scalable framework designed for large-scale 3D pretraining across diverse real-world driving datasets. Our framework leverages VFMs to extract semantically rich superpixels from 2D images, which are aligned with LiDAR point clouds to generate high-quality contrastive samples. This alignment facilitates cross-modal representation learning, enhancing the semantic consistency between 2D and 3D data. We introduce several key innovations: i) VFM-driven superpixel generation for detailed semantic representation, ii) a VFM-assisted contrastive learning strategy to align multimodal features, iii) superpoint temporal consistency to maintain stable representations across time, and iv) multi-source data pretraining to generalize across various LiDAR configurations. Our approach delivers significant performance improvements over state-of-the-art methods in both linear probing and fine-tuning tasks for both LiDAR-based segmentation and object detection. Extensive experiments on eleven large-scale multi-modal datasets highlight our superior performance, demonstrating the adaptability, efficiency, and robustness in real-world autonomous driving scenarios.
• 摘要：视觉基础模型 (VFM) 的最新进展彻底改变了 2D 中的视觉感知，但它们在 3D 场景理解方面的潜力，尤其是在自动驾驶应用中的潜力仍未得到充分探索。在本文中，我们介绍了 LargeAD，这是一个多功能且可扩展的框架，旨在对各种现实世界驾驶数据集进行大规模 3D 预训练。我们的框架利用 VFM 从 2D 图像中提取语义丰富的超像素，这些超像素与 LiDAR 点云对齐以生成高质量的对比样本。这种对齐有助于跨模态表示学习，增强了 2D 和 3D 数据之间的语义一致性。我们引入了几项关键创新：i) VFM 驱动的超像素生成以实现详细的语义表示，ii) VFM 辅助的对比学习策略以对齐多模态特征，iii) 超点时间一致性以保持跨时间的稳定表示，以及 iv) 多源数据预训练以跨各种 LiDAR 配置进行推广。我们的方法在基于 LiDAR 的分割和物体检测的线性探测和微调任务中，与最先进的方法相比，性能有了显著的提升。在 11 个大型多模态数据集上进行的大量实验凸显了我们的卓越性能，展示了现实世界自动驾驶场景中的适应性、效率和稳健性。