2024-12-09

Title: Solving High-dimensional Inverse Problems Using Amortized Likelihood-free Inference with Noisy and Incomplete Data

• Authors: Jice Zeng, Yuanzhe Wang, Alexandre M. Tartakovsky, David Barajas-Solano
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.04565
• Pdf URL: https://arxiv.org/pdf/2412.04565
• Copy Paste: [[2412.04565]] Solving High-dimensional Inverse Problems Using Amortized Likelihood-free Inference with Noisy and Incomplete Data(https://arxiv.org/abs/2412.04565)
• Keywords: generative
• Abstract: We present a novel likelihood-free probabilistic inversion method based on normalizing flows for high-dimensional inverse problems. The proposed method is comprised of two complementary networks: a summary network for data compression, and an inference network for parameter estimation. The summary network encodes raw observations into a fixed-size vector of summary statistics, while the inference network generates samples of the approximate posterior distribution of the model parameters based on these summary statistics. The posterior samples are produced in a deep generative fashion by sampling from a latent Gaussian distribution and passing these samples through an invertible transformation. We construct this invertible transformation by sequentially alternating conditional invertible neural network (cINN) and conditional neural spline flow (cNSF) layers. The summary and inference networks are trained simultaneously. We apply the proposed method to an inversion problem in groundwater hydrology to estimate the posterior distribution of the system's log-conductivity field conditioned on spatially sparse time-series observations of the system's hydraulic head responses. The conductivity field is represented with 706 degrees of freedom in the considered problem. The comparison with the likelihood-based iterative ensemble smoother PEST-IES method demonstrates that the proposed method accurately estimates the parameter posterior distribution and the observations' predictive posterior distribution at a fraction of the inference time of PEST-IES.
• 摘要：我们提出了一种基于高维逆问题正则化流的新型无似然概率反演方法。所提出的方法由两个互补的网络组成：用于数据压缩的摘要网络和用于参数估计的推理网络。摘要网络将原始观测值编码为固定大小的摘要统计数据向量，而推理网络则根据这些摘要统计数据生成模型参数近似后验分布的样本。后验样本以深度生成的方式生成，即从潜在高斯分布中采样并对这些样本进行可逆变换。我们通过依次交替条件可逆神经网络 (cINN) 和条件神经样条流 (cNSF) 层来构建这种可逆变换。摘要和推理网络同时进行训练。我们将所提出的方法应用于地下水水文学中的反演问题，以估计系统水头响应的空间稀疏时间序列观测条件下系统对数电导率场的后验分布。在所考虑的问题中，电导率场以 706 个自由度表示。与基于似然的迭代集合平滑器 PEST-IES 方法的比较表明，所提出的方法可以在 PEST-IES 推理时间的一小部分内准确估计参数后验分布和观测值的预测后验分布。

Title: ARTeFACT: Benchmarking Segmentation Models on Diverse Analogue Media Damage

• Authors: Daniela Ivanova, Marco Aversa, Paul Henderson, John Williamson
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.04580
• Pdf URL: https://arxiv.org/pdf/2412.04580
• Copy Paste: [[2412.04580]] ARTeFACT: Benchmarking Segmentation Models on Diverse Analogue Media Damage(https://arxiv.org/abs/2412.04580)
• Keywords: restoration
• Abstract: Accurately detecting and classifying damage in analogue media such as paintings, photographs, textiles, mosaics, and frescoes is essential for cultural heritage preservation. While machine learning models excel in correcting degradation if the damage operator is known a priori, we show that they fail to robustly predict where the damage is even after supervised training; thus, reliable damage detection remains a challenge. Motivated by this, we introduce ARTeFACT, a dataset for damage detection in diverse types analogue media, with over 11,000 annotations covering 15 kinds of damage across various subjects, media, and historical provenance. Furthermore, we contribute human-verified text prompts describing the semantic contents of the images, and derive additional textual descriptions of the annotated damage. We evaluate CNN, Transformer, diffusion-based segmentation models, and foundation vision models in zero-shot, supervised, unsupervised and text-guided settings, revealing their limitations in generalising across media types. Our dataset is available at $\href{this https URL}{this https URL}$ as the first-of-its-kind benchmark for analogue media damage detection and restoration.
• 摘要：准确检测和分类模拟媒体（如绘画、照片、纺织品、马赛克和壁画）中的损坏对于文化遗产保护至关重要。虽然机器学习模型在损伤算子先验已知的情况下擅长纠正退化，但我们表明，即使在监督训练之后，它们也无法稳健地预测损伤的位置；因此，可靠的损伤检测仍然是一个挑战。受此启发，我们推出了 ARTeFACT，这是一个用于检测各种模拟媒体损伤的数据集，拥有超过 11,000 条注释，涵盖了不同主题、媒体和历史出处的 15 种损伤。此外，我们还提供了经过人工验证的文本提示来描述图像的语义内容，并得出了带注释的损伤的额外文本描述。我们在零样本、监督、无监督和文本引导设置中评估了 CNN、Transformer、基于扩散的分割模型和基础视觉模型，揭示了它们在跨媒体类型推广方面的局限性。我们的数据集可在$\href{this https URL}{this https URL}$上获取，这是模拟媒体损坏检测和修复的首个基准。

Title: Using Diffusion Priors for Video Amodal Segmentation

• Authors: Kaihua Chen, Deva Ramanan, Tarasha Khurana
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.04623
• Pdf URL: https://arxiv.org/pdf/2412.04623
• Copy Paste: [[2412.04623]] Using Diffusion Priors for Video Amodal Segmentation(https://arxiv.org/abs/2412.04623)
• Keywords: generation, generative
• Abstract: Object permanence in humans is a fundamental cue that helps in understanding persistence of objects, even when they are fully occluded in the scene. Present day methods in object segmentation do not account for this amodal nature of the world, and only work for segmentation of visible or modal objects. Few amodal methods exist; single-image segmentation methods cannot handle high-levels of occlusions which are better inferred using temporal information, and multi-frame methods have focused solely on segmenting rigid objects. To this end, we propose to tackle video amodal segmentation by formulating it as a conditional generation task, capitalizing on the foundational knowledge in video generative models. Our method is simple; we repurpose these models to condition on a sequence of modal mask frames of an object along with contextual pseudo-depth maps, to learn which object boundary may be occluded and therefore, extended to hallucinate the complete extent of an object. This is followed by a content completion stage which is able to inpaint the occluded regions of an object. We benchmark our approach alongside a wide array of state-of-the-art methods on four datasets and show a dramatic improvement of upto 13% for amodal segmentation in an object's occluded region.
• 摘要：人类的物体永久性是理解物体持久性的基本线索，即使它们在场景中被完全遮挡。当今的物体分割方法没有考虑到世界的这种非模态性质，只适用于可见或模态物体的分割。非模态方法很少；单图像分割方法无法处理高水平的遮挡，而使用时间信息可以更好地推断遮挡，而多帧方法仅专注于分割刚性物体。为此，我们建议通过将视频非模态分割制定为条件生成任务来解决它，利用视频生成模型中的基础知识。我们的方法很简单；我们重新利用这些模型来以对象的模态掩码帧序列以及上下文伪深度图为条件，以了解哪些对象边界可能被遮挡，从而扩展到幻觉对象的完整范围。接下来是内容完成阶段，该阶段能够修复对象的遮挡区域。我们在四个数据集上对我们的方法以及多种最先进的方法进行了基准测试，结果显示，在物体遮挡区域的非模态分割中，高达 13% 的显著改善。

Title: BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks

• Authors: Juan Rodriguez, Xiangru Jian, Siba Smarak Panigrahi, Tianyu Zhang, Aarash Feizi, Abhay Puri, Akshay Kalkunte, François Savard, Ahmed Masry, Shravan Nayak, Rabiul Awal, Mahsa Massoud, Amirhossein Abaskohi, Zichao Li, Suyuchen Wang, Pierre-André Noël, Mats Leon Richter, Saverio Vadacchino, Shubbam Agarwal, Sanket Biswas, Sara Shanian, Ying Zhang, Noah Bolger, Kurt MacDonald, Simon Fauvel, Sathwik Tejaswi, Srinivas Sunkara, Joao Monteiro, Krishnamurthy DJ Dvijotham, Torsten Scholak, Nicolas Chapados, Sepideh Kharagani, Sean Hughes, M. Özsu, Siva Reddy, Marco Pedersoli, Yoshua Bengio, Christopher Pal, Issam Laradji, Spandanna Gella, Perouz Taslakian, David Vazquez, Sai Rajeswar
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.04626
• Pdf URL: https://arxiv.org/pdf/2412.04626
• Copy Paste: [[2412.04626]] BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks(https://arxiv.org/abs/2412.04626)
• Keywords: generation
• Abstract: Multimodal AI has the potential to significantly enhance document-understanding tasks, such as processing receipts, understanding workflows, extracting data from documents, and summarizing reports. Code generation tasks that require long-structured outputs can also be enhanced by multimodality. Despite this, their use in commercial applications is often limited due to limited access to training data and restrictive licensing, which hinders open access. To address these limitations, we introduce BigDocs-7.5M, a high-quality, open-access dataset comprising 7.5 million multimodal documents across 30 tasks. We use an efficient data curation process to ensure our data is high-quality and license-permissive. Our process emphasizes accountability, responsibility, and transparency through filtering rules, traceable metadata, and careful content analysis. Additionally, we introduce BigDocs-Bench, a benchmark suite with 10 novel tasks where we create datasets that reflect real-world use cases involving reasoning over Graphical User Interfaces (GUI) and code generation from images. Our experiments show that training with BigDocs-Bench improves average performance up to 25.8% over closed-source GPT-4o in document reasoning and structured output tasks such as Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a preference for outputs from models trained on BigDocs over GPT-4o. This suggests that BigDocs can help both academics and the open-source community utilize and improve AI tools to enhance multimodal capabilities and document reasoning. The project is hosted at this https URL .
• 摘要：多模态 AI 有可能显著增强文档理解任务，例如处理收据、理解工作流、从文档中提取数据和总结报告。需要长结构化输出的代码生成任务也可以通过多模态得到增强。尽管如此，由于对训练数据的访问有限和限制性许可阻碍了开放访问，它们在商业应用中的使用往往受到限制。为了解决这些限制，我们推出了 BigDocs-7.5M，这是一个高质量的开放访问数据集，包含 30 个任务中的 750 万个多模态文档。我们使用高效的数据管理流程来确保我们的数据是高质量和许可许可的。我们的流程通过过滤规则、可跟踪的元数据和仔细的内容分析强调问责制、责任制和透明度。此外，我们还推出了 BigDocs-Bench，这是一个基准套件，包含 10 个新任务，我们在其中创建数据集，这些数据集反映了涉及图形用户界面 (GUI) 推理和从图像生成代码的实际用例。我们的实验表明，在文档推理和结构化输出任务（例如 Screenshot2HTML 或 Image2Latex 生成）中，使用 BigDocs-Bench 进行训练可将平均性能提高 25.8%，高于闭源 GPT-4o。最后，人类评估显示，人们更喜欢在 BigDocs 上训练的模型的输出，而不是 GPT-4o。这表明 BigDocs 可以帮助学术界和开源社区利用和改进 AI 工具来增强多模式能力和文档推理。该项目托管在此 https URL 上。

Title: One Communication Round is All It Needs for Federated Fine-Tuning Foundation Models

• Authors: Ziyao Wang, Bowei Tian, Yexiao He, Zheyu Shen, Luyang Liu, Ang Li
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.04650
• Pdf URL: https://arxiv.org/pdf/2412.04650
• Copy Paste: [[2412.04650]] One Communication Round is All It Needs for Federated Fine-Tuning Foundation Models(https://arxiv.org/abs/2412.04650)
• Keywords: generation
• Abstract: The recent advancement of large foundation models (FMs) has increased the demand for fine-tuning these models on large-scale and cross-domain datasets. To address this, federated fine-tuning has emerged as a solution, allowing models to be fine-tuned on distributed datasets across multiple devices while ensuring data privacy. However, the substantial parameter size of FMs and the multi-round communication required by traditional federated fine-tuning algorithms result in prohibitively high communication costs, challenging the practicality of federated fine-tuning. In this paper, we are the first to reveal, both theoretically and empirically, that the traditional multi-round aggregation algorithms may not be necessary for federated fine-tuning large FMs. Our experiments reveal that a single round of communication (i.e., one-shot federated fine-tuning) yields a global model performance comparable to that achieved through multiple rounds of communication. Through rigorous mathematical and empirical analyses, we demonstrate that large FMs, due to their extensive parameter sizes and pre-training on general tasks, achieve significantly lower training loss in one-shot federated fine-tuning compared to smaller models. Our extensive experiments show that one-shot federated fine-tuning not only reduces communication costs but also enables asynchronous aggregation, enhances privacy, and maintains performance consistency with multi-round federated fine-tuning for models larger than 1 billion parameters, on text generation and text-to-image generation tasks. Our findings have the potential to revolutionize federated fine-tuning in practice, enhancing efficiency, reducing costs, and expanding accessibility for large-scale models. This breakthrough paves the way for broader adoption and application of federated fine-tuning across various domains.
• 摘要：大型基础模型 (FM) 的最新进展增加了在大型和跨领域数据集上对这些模型进行微调的需求。为了解决这个问题，联邦微调作为一种解决方案应运而生，它允许在确保数据隐私的同时在跨多个设备的分布式数据集上对模型进行微调。然而，FM 的大量参数大小和传统联邦微调算法所需的多轮通信导致通信成本过高，对联邦微调的实用性提出了挑战。在本文中，我们首次从理论和实证上揭示了传统的多轮聚合算法对于联邦微调大型 FM 可能不是必需的。我们的实验表明，单轮通信（即一次性联邦微调）产生的全局模型性能与通过多轮通信实现的性能相当。通过严格的数学和实证分析，我们证明了大型 FM 由于其广泛的参数大小和在一般任务上进行预训练，与较小的模型相比，在一次性联邦微调中实现了显著较低的训练损失。我们大量的实验表明，一次性联邦微调不仅可以降低通信成本，还可以实现异步聚合，增强隐私，并且在文本生成和文本到图像生成任务中，对于超过 10 亿个参数的模型，保持与多轮联邦微调的性能一致性。我们的发现有可能在实践中彻底改变联邦微调，提高效率，降低成本，并扩大大型模型的可访问性。这一突破为联邦微调在各个领域的更广泛采用和应用铺平了道路。

Title: Cross-Self KV Cache Pruning for Efficient Vision-Language Inference

• Authors: Xiaohuan Pei, Tao Huang, Chang Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.04652
• Pdf URL: https://arxiv.org/pdf/2412.04652
• Copy Paste: [[2412.04652]] Cross-Self KV Cache Pruning for Efficient Vision-Language Inference(https://arxiv.org/abs/2412.04652)
• Keywords: generation
• Abstract: KV cache pruning has emerged as a promising technique for reducing memory and computation costs in long-context auto-regressive generation. Existing methods for vision-language models (VLMs) typically rely on self-attention scores from large language models (LLMs) to identify and prune irrelevant tokens. However, these approaches overlook the inherent distributional discrepancies between modalities, often leading to inaccurate token importance estimation and the over-pruning of critical visual tokens. To address this, we propose decomposing attention scores into intra-modality attention (within the same modality) and inter-modality attention (across modalities), enabling more precise KV cache pruning by independently managing these distinct attention types. Additionally, we introduce an n-softmax function to counteract distribution shifts caused by pruning, preserving the original smoothness of attention scores and ensuring stable performance. Our final training-free method, \textbf{C}ross-\textbf{S}elf \textbf{P}runing (CSP), achieves competitive performance compared to models with full KV caches while significantly outperforming previous pruning methods. Extensive evaluations on MileBench, a benchmark encompassing 29 multimodal datasets, demonstrate CSP's effectiveness, achieving up to a 41\% performance improvement on challenging tasks like conversational embodied dialogue while reducing the KV cache budget by 13.6\%. The code is available at this https URL
• 摘要：KV 缓存修剪已成为一种很有前途的技术，可用于减少长上下文自回归生成中的内存和计算成本。现有的视觉语言模型 (VLM) 方法通常依赖大型语言模型 (LLM) 的自注意力分数来识别和修剪不相关的标记。然而，这些方法忽略了模态之间固有的分布差异，常常导致标记重要性估计不准确以及关键视觉标记的过度修剪。为了解决这个问题，我们建议将注意力分数分解为模态内注意力（同一模态内）和模态间注意力（跨模态），通过独立管理这些不同的注意力类型实现更精确的 KV 缓存修剪。此外，我们引入了一个 n-softmax 函数来抵消修剪引起的分布变化，保留注意力分数原有的平滑度并确保稳定的性能。我们最终的免训练方法 \textbf{C}ross-\textbf{S}elf \textbf{P}runing (CSP) 与具有完整 KV 缓存的模型相比实现了具有竞争力的性能，同时显著优于以前的修剪方法。在涵盖 29 个多模态数据集的基准 MileBench 上进行的广泛评估证明了 CSP 的有效性，在对话式具体对话等具有挑战性的任务上实现了高达 41\% 的性能提升，同时将 KV 缓存预算减少了 13.6\%。代码可从此 https URL 获得

Title: Hidden in the Noise: Two-Stage Robust Watermarking for Images

• Authors: Kasra Arabi, Benjamin Feuer, R. Teal Witter, Chinmay Hegde, Niv Cohen
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.04653
• Pdf URL: https://arxiv.org/pdf/2412.04653
• Copy Paste: [[2412.04653]] Hidden in the Noise: Two-Stage Robust Watermarking for Images(https://arxiv.org/abs/2412.04653)
• Keywords: generation
• Abstract: As the quality of image generators continues to improve, deepfakes become a topic of considerable societal debate. Image watermarking allows responsible model owners to detect and label their AI-generated content, which can mitigate the harm. Yet, current state-of-the-art methods in image watermarking remain vulnerable to forgery and removal attacks. This vulnerability occurs in part because watermarks distort the distribution of generated images, unintentionally revealing information about the watermarking techniques. In this work, we first demonstrate a distortion-free watermarking method for images, based on a diffusion model's initial noise. However, detecting the watermark requires comparing the initial noise reconstructed for an image to all previously used initial noises. To mitigate these issues, we propose a two-stage watermarking framework for efficient detection. During generation, we augment the initial noise with generated Fourier patterns to embed information about the group of initial noises we used. For detection, we (i) retrieve the relevant group of noises, and (ii) search within the given group for an initial noise that might match our image. This watermarking approach achieves state-of-the-art robustness to forgery and removal against a large battery of attacks.
• 摘要：随着图像生成器质量的不断提高，深度伪造成为社会争论的话题。图像水印允许负责任的模型所有者检测和标记他们的人工智能生成的内容，从而减轻危害。然而，目前最先进的图像水印方法仍然容易受到伪造和删除攻击。这种脆弱性的部分原因是水印扭曲了生成图像的分布，无意中泄露了有关水印技术的信息。在这项工作中，我们首先演示了一种基于扩散模型初始噪声的无失真图像水印方法。然而，检测水印需要将为图像重建的初始噪声与所有先前使用的初始噪声进行比较。为了缓解这些问题，我们提出了一个两阶段水印框架以实现有效检测。在生成过程中，我们用生成的傅里叶模式增强初始噪声，以嵌入有关我们使用的初始噪声组的信息。为了检测，我们 (i) 检索相关的噪声组，(ii) 在给定组中搜索可能与我们的图像匹配的初始噪声。这种水印方法具有最先进的抗伪造和抗大量攻击的鲁棒性。

Title: Diffusion-Augmented Coreset Expansion for Scalable Dataset Distillation

• Authors: Ali Abbasi, Shima Imani, Chenyang An, Gayathri Mahalingam, Harsh Shrivastava, Maurice Diesendruck, Hamed Pirsiavash, Pramod Sharma, Soheil Kolouri
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.04668
• Pdf URL: https://arxiv.org/pdf/2412.04668
• Copy Paste: [[2412.04668]] Diffusion-Augmented Coreset Expansion for Scalable Dataset Distillation(https://arxiv.org/abs/2412.04668)
• Keywords: generative
• Abstract: With the rapid scaling of neural networks, data storage and communication demands have intensified. Dataset distillation has emerged as a promising solution, condensing information from extensive datasets into a compact set of synthetic samples by solving a bilevel optimization problem. However, current methods face challenges in computational efficiency, particularly with high-resolution data and complex architectures. Recently, knowledge-distillation-based dataset condensation approaches have made this process more computationally feasible. Yet, with the recent developments of generative foundation models, there is now an opportunity to achieve even greater compression, enhance the quality of distilled data, and introduce valuable diversity into the data representation. In this work, we propose a two-stage solution. First, we compress the dataset by selecting only the most informative patches to form a coreset. Next, we leverage a generative foundation model to dynamically expand this compressed set in real-time, enhancing the resolution of these patches and introducing controlled variability to the coreset. Our extensive experiments demonstrate the robustness and efficiency of our approach across a range of dataset distillation benchmarks. We demonstrate a significant improvement of over 10% compared to the state-of-the-art on several large-scale dataset distillation benchmarks. The code will be released soon.
• 摘要：随着神经网络的快速扩展，数据存储和通信需求也日益增加。数据集精炼已成为一种有前途的解决方案，通过解决双层优化问题，将大量数据集中的信息浓缩为一组紧凑的合成样本。然而，当前的方法面临着计算效率方面的挑战，尤其是在高分辨率数据和复杂架构方面。最近，基于知识精炼的数据集浓缩方法使这一过程在计算上更加可行。然而，随着生成基础模型的最新发展，现在有机会实现更大的压缩，提高精炼数据的质量，并在数据表示中引入有价值的多样性。在这项工作中，我们提出了一个两阶段的解决方案。首先，我们通过仅选择最具信息量的补丁来压缩数据集以形成核心集。接下来，我们利用生成基础模型实时动态扩展这个压缩集，提高这些补丁的分辨率并为核心集引入受控的可变性。我们进行了大量的实验，证明了我们的方法在一系列数据集精炼基准中的稳健性和效率。我们在多个大规模数据集蒸馏基准上展示了比最新成果高出 10% 以上的显著改进。代码将很快发布。

Title: Zephyr quantum-assisted hierarchical Calo4pQVAE for particle-calorimeter interactions

• Authors: Ian Lu, Hao Jia, Sebastian Gonzalez, Deniz Sogutlu, J. Quetzalcoatl Toledo-Marin, Sehmimul Hoque, Abhishek Abhishek, Colin Gay, Roger Melko, Eric Paquet, Geoffrey Fox, Maximilian Swiatlowski, Wojciech Fedorko
• Subjects: cs.LG, cs.AI, hep-ph, physics.comp-ph, quant-ph
• Abstract URL: https://arxiv.org/abs/2412.04677
• Pdf URL: https://arxiv.org/pdf/2412.04677
• Copy Paste: [[2412.04677]] Zephyr quantum-assisted hierarchical Calo4pQVAE for particle-calorimeter interactions(https://arxiv.org/abs/2412.04677)
• Keywords: generation, generative
• Abstract: With the approach of the High Luminosity Large Hadron Collider (HL-LHC) era set to begin particle collisions by the end of this decade, it is evident that the computational demands of traditional collision simulation methods are becoming increasingly unsustainable. Existing approaches, which rely heavily on first-principles Monte Carlo simulations for modeling event showers in calorimeters, are projected to require millions of CPU-years annually -- far exceeding current computational capacities. This bottleneck presents an exciting opportunity for advancements in computational physics by integrating deep generative models with quantum simulations. We propose a quantum-assisted hierarchical deep generative surrogate founded on a variational autoencoder (VAE) in combination with an energy conditioned restricted Boltzmann machine (RBM) embedded in the model's latent space as a prior. By mapping the topology of D-Wave's Zephyr quantum annealer (QA) into the nodes and couplings of a 4-partite RBM, we leverage quantum simulation to accelerate our shower generation times significantly. To evaluate our framework, we use Dataset 2 of the CaloChallenge 2022. Through the integration of classical computation and quantum simulation, this hybrid framework paves way for utilizing large-scale quantum simulations as priors in deep generative models.
• 摘要：随着高亮度大型强子对撞机 (HL-LHC) 时代的到来，该时代将于本世纪末开始进行粒子碰撞，显然传统碰撞模拟方法的计算需求正变得越来越难以承受。现有的方法严重依赖第一性原理蒙特卡罗模拟来模拟量热仪中的事件簇射，预计每年需要数百万 CPU 年——远远超过当前的计算能力。通过将深度生成模型与量子模拟相结合，这一瓶颈为计算物理学的进步提供了一个令人兴奋的机会。我们提出了一种基于变分自动编码器 (VAE) 的量子辅助分层深度生成代理，结合嵌入在模型潜在空间中的能量调节受限玻尔兹曼机 (RBM) 作为先验。通过将 D-Wave 的 Zephyr 量子退火器 (QA) 的拓扑映射到 4 部分 RBM 的节点和耦合中，我们利用量子模拟显著加快了我们的簇射生成时间。为了评估我们的框架，我们使用了 CaloChallenge 2022 的数据集 2。通过整合经典计算和量子模拟，这个混合框架为在深度生成模型中利用大规模量子模拟作为先验铺平了道路。

Title: Generative Humanization for Therapeutic Antibodies

• Authors: Cade Gordon, Aniruddh Raghu, Hunter Elliott, Peyton Greenside
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2412.04737
• Pdf URL: https://arxiv.org/pdf/2412.04737
• Copy Paste: [[2412.04737]] Generative Humanization for Therapeutic Antibodies(https://arxiv.org/abs/2412.04737)
• Keywords: generative
• Abstract: Antibody therapies have been employed to address some of today's most challenging diseases, but must meet many criteria during drug development before reaching a patient. Humanization is a sequence optimization strategy that addresses one critical risk called immunogenicity - a patient's immune response to the drug - by making an antibody more "human-like" in the absence of a predictive lab-based test for immunogenicity. However, existing humanization strategies generally yield very few humanized candidates, which may have degraded biophysical properties or decreased drug efficacy. Here, we re-frame humanization as a conditional generative modeling task, where humanizing mutations are sampled from a language model trained on human antibody data. We describe a sampling process that incorporates models of therapeutic attributes, such as antigen binding affinity, to obtain candidate sequences that have both reduced immunogenicity risk and maintained or improved therapeutic properties, allowing this algorithm to be readily embedded into an iterative antibody optimization campaign. We demonstrate in silico and in lab validation that in real therapeutic programs our generative humanization method produces diverse sets of antibodies that are both (1) highly-human and (2) have favorable therapeutic properties, such as improved binding to target antigens.
• 摘要：抗体疗法已被用于治疗当今一些最具挑战性的疾病，但在药物开发过程中必须满足许多标准才能用于患者。人源化是一种序列优化策略，它通过使抗体更“像人类”，在没有预测免疫原性的实验室测试的情况下解决一种称为免疫原性（患者对药物的免疫反应）的关键风险。然而，现有的人源化策略通常产生很少的人源化候选物，这些候选物的生物物理特性可能会降低或药物疗效降低。在这里，我们将人源化重新定义为条件生成建模任务，其中人源化突变是从对人类抗体数据进行训练的语言模型中采样的。我们描述了一种采样过程，该过程结合了治疗属性模型，例如抗原结合亲和力，以获得既具有降低免疫原性风险又保持或改善治疗特性的候选序列，从而使该算法能够轻松嵌入到迭代抗体优化活动中。我们通过计算机模拟和实验室验证证明，在实际治疗项目中，我们的生成人源化方法可以产生多样化的抗体，这些抗体既 (1) 高度人性化，又 (2) 具有良好的治疗特性，例如与靶抗原的结合得到改善。

Title: DPGIIL: Dirichlet Process-Deep Generative Model-Integrated Incremental Learning for Clustering in Transmissibility-based Online Structural Anomaly Detection

• Authors: Lin-Feng Mei, Wang-Ji Yan
• Subjects: cs.LG, physics.data-an, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.04781
• Pdf URL: https://arxiv.org/pdf/2412.04781
• Copy Paste: [[2412.04781]] DPGIIL: Dirichlet Process-Deep Generative Model-Integrated Incremental Learning for Clustering in Transmissibility-based Online Structural Anomaly Detection(https://arxiv.org/abs/2412.04781)
• Keywords: generative
• Abstract: Clustering based on vibration responses, such as transmissibility functions (TFs), is promising in structural anomaly detection, but most existing approaches struggle with determining the optimal cluster number and handling high-dimensional streaming data, while their shallow structures also make them sensitive to manually-engineered feature quality. To bridge this gap, this work proposes the Dirichlet process-deep generative model-integrated incremental learning (DPGIIL) for clustering by combining the advantages of deep generative models (DGMs) in representation learning and the Dirichlet process mixture model (DPMM) in identifying distinct patterns in observed data. By introducing a DPMM prior into the latent space of DGMs, DPGIIL automatically captures dissimilarities in extracted latent representations, enabling both generative modeling and clustering. Within the context of variational Bayesian inference, a lower bound on the log marginal likelihood of DPGIIL, tighter than the evidence lower bound given sufficient training data, is derived analytically, which enables the joint optimization of DGM and DPMM parameters, thereby allowing the DPMM to regularize the DGM's feature extraction process. Additionally, a greedy split-merge scheme-based coordinate ascent variational inference method is devised to accelerate the optimization. The summary statistics of the DPMM, along with the network parameters, are used to retain information about previous data for incremental learning. Notably, this study uses variational autoencoder (VAE) within DPGIIL as an illustrative example, while this framework is adaptable to other DGMs. Two case studies show that the proposed method outperforms some state-of-the-art approaches in structural anomaly detection and clustering, while also dynamically generating new clusters to indicate the emergence of new structural conditions for online monitoring.
• 摘要：基于振动响应（例如传递函数 (TF)）的聚类在结构异常检测中很有前景，但大多数现有方法在确定最佳聚类数和处理高维流数据方面都很困难，同时它们的浅层结构也使它们对人工设计的特征质量很敏感。为了弥补这一差距，这项工作提出了用于聚类的狄利克雷过程-深度生成模型集成增量学习 (DPGIIL)，它结合了深度生成模型 (DGM) 在表示学习方面的优势和狄利克雷过程混合模型 (DPMM) 在识别观察数据中的不同模式方面的优势。通过在 DGM 的潜在空间中引入 DPMM 先验，DPGIIL 会自动捕获提取的潜在表示中的差异，从而实现生成建模和聚类。在变分贝叶斯推理的背景下，通过分析得出 DPGIIL 对数边际似然的下限，该下限比在有足够训练数据的情况下的证据下限更严格，这使得 DGM 和 DPMM 参数能够联合优化，从而使 DPMM 能够规范 DGM 的特征提取过程。此外，还设计了一种基于贪婪分裂合并方案的坐标上升变分推理方法来加速优化。DPMM 的汇总统计数据以及网络参数用于保留有关先前数据的信息以进行增量学习。值得注意的是，本研究使用 DPGIIL 中的变分自动编码器 (VAE) 作为说明性示例，而该框架适用于其他 DGM。两个案例研究表明，所提出的方法在结构异常检测和聚类方面优于一些最先进的方法，同时还动态生成新的聚类以指示在线监控新结构条件的出现。

Title: LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment

• Authors: Yibin Wang, Zhiyu Tan, Junyan Wang, Xiaomeng Yang, Cheng Jin, Hao Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.04814
• Pdf URL: https://arxiv.org/pdf/2412.04814
• Copy Paste: [[2412.04814]] LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment(https://arxiv.org/abs/2412.04814)
• Keywords: generative
• Abstract: Recent advancements in text-to-video (T2V) generative models have shown impressive capabilities. However, these models are still inadequate in aligning synthesized videos with human preferences (e.g., accurately reflecting text descriptions), which is particularly difficult to address, as human preferences are inherently subjective and challenging to formalize as objective functions. Therefore, this paper proposes LiFT, a novel fine-tuning method leveraging human feedback for T2V model alignment. Specifically, we first construct a Human Rating Annotation dataset, LiFT-HRA, consisting of approximately 10k human annotations, each including a score and its corresponding rationale. Based on this, we train a reward model LiFT-Critic to learn reward function effectively, which serves as a proxy for human judgment, measuring the alignment between given videos and human expectations. Lastly, we leverage the learned reward function to align the T2V model by maximizing the reward-weighted likelihood. As a case study, we apply our pipeline to CogVideoX-2B, showing that the fine-tuned model outperforms the CogVideoX-5B across all 16 metrics, highlighting the potential of human feedback in improving the alignment and quality of synthesized videos.
• 摘要：文本到视频 (T2V) 生成模型的最新进展已展现出令人印象深刻的功能。然而，这些模型在将合成视频与人类偏好（例如，准确反映文本描述）对齐方面仍然不足，这尤其难以解决，因为人类偏好本质上是主观的，很难形式化为客观函数。因此，本文提出了 LiFT，这是一种利用人类反馈进行 T2V 模型对齐的新型微调方法。具体来说，我们首先构建一个人工评分注释数据集 LiFT-HRA，其中包含大约 10k 个人工注释，每个注释都包含一个分数及其相应的理由。在此基础上，我们训练了一个奖励模型 LiFT-Critic 来有效地学习奖励函数，它作为人类判断的代理，衡量给定视频与人类期望之间的一致性。最后，我们利用学习到的奖励函数通过最大化奖励加权似然来对齐 T2V 模型。作为案例研究，我们将我们的流程应用于 CogVideoX-2B，结果显示微调模型在所有 16 个指标上均优于 CogVideoX-5B，凸显了人工反馈在改善合成视频的对齐和质量方面的潜力。

Title: DAug: Diffusion-based Channel Augmentation for Radiology Image Retrieval and Classification

• Authors: Ying Jin, Zhuoran Zhou, Haoquan Fang, Jenq-Neng Hwang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.04828
• Pdf URL: https://arxiv.org/pdf/2412.04828
• Copy Paste: [[2412.04828]] DAug: Diffusion-based Channel Augmentation for Radiology Image Retrieval and Classification(https://arxiv.org/abs/2412.04828)
• Keywords: generative
• Abstract: Medical image understanding requires meticulous examination of fine visual details, with particular regions requiring additional attention. While radiologists build such expertise over years of experience, it is challenging for AI models to learn where to look with limited amounts of training data. This limitation results in unsatisfying robustness in medical image understanding. To address this issue, we propose Diffusion-based Feature Augmentation (DAug), a portable method that improves a perception model's performance with a generative model's output. Specifically, we extend a radiology image to multiple channels, with the additional channels being the heatmaps of regions where diseases tend to develop. A diffusion-based image-to-image translation model was used to generate such heatmaps conditioned on selected disease classes. Our method is motivated by the fact that generative models learn the distribution of normal and abnormal images, and such knowledge is complementary to image understanding tasks. In addition, we propose the Image-Text-Class Hybrid Contrastive learning to utilize both text and class labels. With two novel approaches combined, our method surpasses baseline models without changing the model architecture, and achieves state-of-the-art performance on both medical image retrieval and classification tasks.
• 摘要：医学图像理解需要对精细的视觉细节进行细致检查，特定区域需要额外关注。虽然放射科医生通过多年的经验积累了这样的专业知识，但对于 AI 模型来说，在有限的训练数据量下学习应该看哪里是一项挑战。这种限制导致医学图像理解的稳健性不令人满意。为了解决这个问题，我们提出了基于扩散的特征增强 (DAug)，这是一种可移植的方法，它利用生成模型的输出来提高感知模型的性能。具体来说，我们将放射图像扩展到多个通道，额外的通道是疾病容易发展的区域的热图。基于扩散的图像到图像转换模型用于生成以选定疾病类别为条件的热图。我们的方法的动机是生成模型学习正常和异常图像的分布，而这种知识是对图像理解任务的补充。此外，我们提出了图像-文本-类混合对比学习，以利用文本和类标签。通过结合两种新方法，我们的方法在不改变模型架构的情况下超越了基线模型，并在医学图像检索和分类任务上取得了最先进的性能。

Title: Customized Generation Reimagined: Fidelity and Editability Harmonized

• Authors: Jian Jin, Yang Shen, Zhenyong Fu, Jian Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.04831
• Pdf URL: https://arxiv.org/pdf/2412.04831
• Copy Paste: [[2412.04831]] Customized Generation Reimagined: Fidelity and Editability Harmonized(https://arxiv.org/abs/2412.04831)
• Keywords: generation
• Abstract: Customized generation aims to incorporate a novel concept into a pre-trained text-to-image model, enabling new generations of the concept in novel contexts guided by textual prompts. However, customized generation suffers from an inherent trade-off between concept fidelity and editability, i.e., between precisely modeling the concept and faithfully adhering to the prompts. Previous methods reluctantly seek a compromise and struggle to achieve both high concept fidelity and ideal prompt alignment simultaneously. In this paper, we propose a Divide, Conquer, then Integrate (DCI) framework, which performs a surgical adjustment in the early stage of denoising to liberate the fine-tuned model from the fidelity-editability trade-off at inference. The two conflicting components in the trade-off are decoupled and individually conquered by two collaborative branches, which are then selectively integrated to preserve high concept fidelity while achieving faithful prompt adherence. To obtain a better fine-tuned model, we introduce an Image-specific Context Optimization} (ICO) strategy for model customization. ICO replaces manual prompt templates with learnable image-specific contexts, providing an adaptive and precise fine-tuning direction to promote the overall performance. Extensive experiments demonstrate the effectiveness of our method in reconciling the fidelity-editability trade-off.
• 摘要：定制生成旨在将新概念纳入预先训练的文本到图像模型，从而能够在文本提示的指导下在新上下文中生成新概念。然而，定制生成存在概念保真度和可编辑性之间的固有权衡，即在精确建模概念和忠实遵循提示之间。以前的方法不情愿地寻求妥协，难以同时实现高概念保真度和理想的提示对齐。在本文中，我们提出了一个分而治之，然后整合（DCI）框架，它在去噪的早期阶段进行外科手术调整，以将微调模型从推理时的保真度-可编辑性权衡中解放出来。权衡中两个相互冲突的组成部分被解耦并由两个协作分支单独征服，然后选择性地集成以保持高概念保真度，同时实现忠实的提示遵守。为了获得更好的微调模型，我们引入了一种针对图像的上下文优化（ICO）策略来进行模型定制。 ICO 用可学习的图像特定上下文取代手动提示模板，提供自适应和精确的微调方向来提升整体性能。大量实验证明了我们的方法在协调保真度与可编辑性权衡方面的有效性。

Title: Wavelet Diffusion Neural Operator

• Authors: Peiyan Hu, Rui Wang, Xiang Zheng, Tao Zhang, Haodong Feng, Ruiqi Feng, Long Wei, Yue Wang, Zhi-Ming Ma, Tailin Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.04833
• Pdf URL: https://arxiv.org/pdf/2412.04833
• Copy Paste: [[2412.04833]] Wavelet Diffusion Neural Operator(https://arxiv.org/abs/2412.04833)
• Keywords: generative
• Abstract: Simulating and controlling physical systems described by partial differential equations (PDEs) are crucial tasks across science and engineering. Recently, diffusion generative models have emerged as a competitive class of methods for these tasks due to their ability to capture long-term dependencies and model high-dimensional states. However, diffusion models typically struggle with handling system states with abrupt changes and generalizing to higher resolutions. In this work, we propose Wavelet Diffusion Neural Operator (WDNO), a novel PDE simulation and control framework that enhances the handling of these complexities. WDNO comprises two key innovations. Firstly, WDNO performs diffusion-based generative modeling in the wavelet domain for the entire trajectory to handle abrupt changes and long-term dependencies effectively. Secondly, to address the issue of poor generalization across different resolutions, which is one of the fundamental tasks in modeling physical systems, we introduce multi-resolution training. We validate WDNO on five physical systems, including 1D advection equation, three challenging physical systems with abrupt changes (1D Burgers' equation, 1D compressible Navier-Stokes equation and 2D incompressible fluid), and a real-world dataset ERA5, which demonstrates superior performance on both simulation and control tasks over state-of-the-art methods, with significant improvements in long-term and detail prediction accuracy. Remarkably, in the challenging context of the 2D high-dimensional and indirect control task aimed at reducing smoke leakage, WDNO reduces the leakage by 33.2% compared to the second-best baseline.
• 摘要：模拟和控制由偏微分方程 (PDE) 描述的物理系统是科学和工程领域的关键任务。最近，扩散生成模型已成为这些任务的一类具有竞争力的方法，因为它们能够捕捉长期依赖关系并建模高维状态。然而，扩散模型通常难以处理具有突然变化的系统状态并推广到更高的分辨率。在这项工作中，我们提出了小波扩散神经算子 (WDNO)，这是一种新颖的 PDE 模拟和控制框架，可增强对这些复杂性的处理。WDNO 包含两个关键创新。首先，WDNO 在小波域中对整个轨迹执行基于扩散的生成建模，以有效处理突然变化和长期依赖关系。其次，为了解决跨不同分辨率泛化能力差的问题，这是建模物理系统的基本任务之一，我们引入了多分辨率训练。我们在五个物理系统上验证了 WDNO，包括一维平流方程、三个具有突变的具有挑战性的物理系统（一维 Burgers 方程、一维可压缩 Navier-Stokes 方程和二维不可压缩流体）以及真实世界数据集 ERA5，它在模拟和控制任务上都表现出优于最先进方法的性能，长期和细节预测准确性有显著提高。值得注意的是，在旨在减少烟雾泄漏的二维高维间接控制任务这一具有挑战性的背景下，与第二好的基线相比，WDNO 将泄漏减少了 33.2%。

Title: UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving

• Authors: Rui Chen, Zehuan Wu, Yichen Liu, Yuxin Guo, Jingcheng Ni, Haifeng Xia, Siyu Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.04842
• Pdf URL: https://arxiv.org/pdf/2412.04842
• Copy Paste: [[2412.04842]] UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving(https://arxiv.org/abs/2412.04842)
• Keywords: generation
• Abstract: The creation of diverse and realistic driving scenarios has become essential to enhance perception and planning capabilities of the autonomous driving system. However, generating long-duration, surround-view consistent driving videos remains a significant challenge. To address this, we present UniMLVG, a unified framework designed to generate extended street multi-perspective videos under precise control. By integrating single- and multi-view driving videos into the training data, our approach updates cross-frame and cross-view modules across three stages with different training objectives, substantially boosting the diversity and quality of generated visual content. Additionally, we employ the explicit viewpoint modeling in multi-view video generation to effectively improve motion transition consistency. Capable of handling various input reference formats (e.g., text, images, or video), our UniMLVG generates high-quality multi-view videos according to the corresponding condition constraints such as 3D bounding boxes or frame-level text descriptions. Compared to the best models with similar capabilities, our framework achieves improvements of 21.4% in FID and 36.5% in FVD.
• 摘要：创建多样化和逼真的驾驶场景对于增强自动驾驶系统的感知和规划能力至关重要。然而，生成长时间、全景视图一致的驾驶视频仍然是一项重大挑战。为了解决这个问题，我们提出了 UniMLVG，这是一个统一的框架，旨在在精确控制下生成扩展街道多视角视频。通过将单视图和多视图驾驶视频集成到训练数据中，我们的方法在三个阶段以不同的训练目标更新跨帧和跨视图模块，大大提高了生成的视觉内容的多样性和质量。此外，我们在多视图视频生成中采用显式视点建模，以有效提高运动过渡一致性。我们的 UniMLVG 能够处理各种输入参考格式（例如文本、图像或视频），根据相应的条件约束（例如 3D 边界框或帧级文本描述）生成高质量的多视图视频。与具有类似功能的最佳模型相比，我们的框架在 FID 方面实现了 21.4% 的改进，在 FVD 方面实现了 36.5% 的改进。

Title: SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models

• Authors: Zilan Wang, Junfeng Guo, Jiacheng Zhu, Yiming Li, Heng Huang, Muhao Chen, Zhengzhong Tu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.04852
• Pdf URL: https://arxiv.org/pdf/2412.04852
• Copy Paste: [[2412.04852]] SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models(https://arxiv.org/abs/2412.04852)
• Keywords: generation, generative
• Abstract: Recent advances in large-scale text-to-image (T2I) diffusion models have enabled a variety of downstream applications, including style customization, subject-driven personalization, and conditional generation. As T2I models require extensive data and computational resources for training, they constitute highly valued intellectual property (IP) for their legitimate owners, yet making them incentive targets for unauthorized fine-tuning by adversaries seeking to leverage these models for customized, usually profitable applications. Existing IP protection methods for diffusion models generally involve embedding watermark patterns and then verifying ownership through generated outputs examination, or inspecting the model's feature space. However, these techniques are inherently ineffective in practical scenarios when the watermarked model undergoes fine-tuning, and the feature space is inaccessible during verification ((i.e., black-box setting). The model is prone to forgetting the previously learned watermark knowledge when it adapts to a new task. To address this challenge, we propose SleeperMark, a novel framework designed to embed resilient watermarks into T2I diffusion models. SleeperMark explicitly guides the model to disentangle the watermark information from the semantic concepts it learns, allowing the model to retain the embedded watermark while continuing to be fine-tuned to new downstream tasks. Our extensive experiments demonstrate the effectiveness of SleeperMark across various types of diffusion models, including latent diffusion models (e.g., Stable Diffusion) and pixel diffusion models (e.g., DeepFloyd-IF), showing robustness against downstream fine-tuning and various attacks at both the image and model levels, with minimal impact on the model's generative capability. The code is available at this https URL.
• 摘要：大规模文本转图像 (T2I) 传播模型的最新进展已实现各种下游应用，包括风格定制、主题驱动的个性化和条件生成。由于 T2I 模型需要大量数据和计算资源进行训练，因此它们为其合法所有者构成了高价值的知识产权 (IP)，但也使其成为对手未经授权进行微调的激励目标，对手试图利用这些模型进行定制，通常是有利可图的应用。现有的传播模型 IP 保护方法通常涉及嵌入水印模式，然后通过检查生成的输出或检查模型的特征空间来验证所有权。然而，在实际场景中，当带水印的模型经过微调，并且在验证期间无法访问特征空间（即黑盒设置）时，这些技术本质上是无效的。当模型适应新任务时，它很容易忘记以前学习到的水印知识。为了应对这一挑战，我们提出了 SleeperMark，这是一个新颖的框架，旨在将弹性水印嵌入到 T2I 扩散模型中。SleeperMark 明确引导模型将水印信息与其学习到的语义概念分离出来，从而使模型能够保留嵌入的水印，同时继续针对新的下游任务进行微调。我们大量的实验证明了 SleeperMark 在各种类型的扩散模型中的有效性，包括潜在扩散模型（例如，稳定扩散）和像素扩散模型（例如，DeepFloyd-IF），表现出对下游微调和图像和模型级别的各种攻击的鲁棒性，对模型的生成能力影响最小。代码可在此 https URL 上获得。

Title: MSECG: Incorporating Mamba for Robust and Efficient ECG Super-Resolution

• Authors: Jie Lin, I Chiu, Kuan-Chen Wang, Kai-Chun Liu, Hsin-Min Wang, Ping-Cheng Yeh, Yu Tsao
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.04861
• Pdf URL: https://arxiv.org/pdf/2412.04861
• Copy Paste: [[2412.04861]] MSECG: Incorporating Mamba for Robust and Efficient ECG Super-Resolution(https://arxiv.org/abs/2412.04861)
• Keywords: super-resolution
• Abstract: Electrocardiogram (ECG) signals play a crucial role in diagnosing cardiovascular diseases. To reduce power consumption in wearable or portable devices used for long-term ECG monitoring, super-resolution (SR) techniques have been developed, enabling these devices to collect and transmit signals at a lower sampling rate. In this study, we propose MSECG, a compact neural network model designed for ECG SR. MSECG combines the strength of the recurrent Mamba model with convolutional layers to capture both local and global dependencies in ECG waveforms, allowing for the effective reconstruction of high-resolution signals. We also assess the model's performance in real-world noisy conditions by utilizing ECG data from the PTB-XL database and noise data from the MIT-BIH Noise Stress Test Database. Experimental results show that MSECG outperforms two contemporary ECG SR models under both clean and noisy conditions while using fewer parameters, offering a more powerful and robust solution for long-term ECG monitoring applications.
• 摘要：心电图 (ECG) 信号在诊断心血管疾病中起着至关重要的作用。为了降低用于长期 ECG 监测的可穿戴或便携式设备的功耗，超分辨率 (SR) 技术已经开发出来，使这些设备能够以较低的采样率收集和传输信号。在本研究中，我们提出了 MSECG，这是一种专为 ECG SR 设计的紧凑神经网络模型。MSECG 结合了循环 Mamba 模型和卷积层的优势，以捕捉 ECG 波形中的局部和全局依赖关系，从而可以有效重建高分辨率信号。我们还利用来自 PTB-XL 数据库的 ECG 数据和来自 MIT-BIH 噪声压力测试数据库的噪声数据来评估该模型在现实世界噪声条件下的性能。实验结果表明，MSECG 在干净和嘈杂条件下均优于两种当代 ECG SR 模型，同时使用更少的参数，为长期 ECG 监测应用提供了更强大、更稳健的解决方案。

Title: Continuous Video Process: Modeling Videos as Continuous Multi-Dimensional Processes for Video Prediction

• Authors: Gaurav Shrivastava, Abhinav Shrivastava
• Subjects: cs.CV, cs.AI, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.04929
• Pdf URL: https://arxiv.org/pdf/2412.04929
• Copy Paste: [[2412.04929]] Continuous Video Process: Modeling Videos as Continuous Multi-Dimensional Processes for Video Prediction(https://arxiv.org/abs/2412.04929)
• Keywords: generation
• Abstract: Diffusion models have made significant strides in image generation, mastering tasks such as unconditional image synthesis, text-image translation, and image-to-image conversions. However, their capability falls short in the realm of video prediction, mainly because they treat videos as a collection of independent images, relying on external constraints such as temporal attention mechanisms to enforce temporal coherence. In our paper, we introduce a novel model class, that treats video as a continuous multi-dimensional process rather than a series of discrete frames. We also report a reduction of 75\% sampling steps required to sample a new frame thus making our framework more efficient during the inference time. Through extensive experimentation, we establish state-of-the-art performance in video prediction, validated on benchmark datasets including KTH, BAIR, Human3.6M, and UCF101. Navigate to the project page this https URL for video results.}
• 摘要：扩散模型在图像生成方面取得了重大进展，掌握了无条件图像合成、文本图像转换和图像到图像转换等任务。然而，它们在视频预测领域的能力不足，主要是因为它们将视频视为独立图像的集合，依靠外部约束（如时间注意机制）来强制时间连贯性。在我们的论文中，我们引入了一种新颖的模型类，它将视频视为一个连续的多维过程，而不是一系列离散帧。我们还报告了对新帧进行采样所需的采样步骤减少了 75\%，从而使我们的框架在推理时间内更加高效。通过大量实验，我们在视频预测方面建立了最先进的性能，并在包括 KTH、BAIR、Human3.6M 和 UCF101 在内的基准数据集上进行了验证。导航到此 https URL 的项目页面以获取视频结果。}

Title: Bed-Attached Vibration Sensor System: A Machine Learning Approach for Fall Detection in Nursing Homes

• Authors: Thomas Bartz-Beielstein, Axel Wellendorf, Noah Pütz, Jens Brandt, Alexander Hinterleitner, Richard Schulz, Richard Scholz, Olaf Mersmann, Robin Knabe
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.04950
• Pdf URL: https://arxiv.org/pdf/2412.04950
• Copy Paste: [[2412.04950]] Bed-Attached Vibration Sensor System: A Machine Learning Approach for Fall Detection in Nursing Homes(https://arxiv.org/abs/2412.04950)
• Keywords: generation
• Abstract: The increasing shortage of nursing staff and the acute risk of falls in nursing homes pose significant challenges for the healthcare system. This study presents the development of an automated fall detection system integrated into care beds, aimed at enhancing patient safety without compromising privacy through wearables or video monitoring. Mechanical vibrations transmitted through the bed frame are processed using a short-time Fourier transform, enabling robust classification of distinct human fall patterns with a convolutional neural network. Challenges pertaining to the quantity and diversity of the data are addressed, proposing the generation of additional data with a specific emphasis on enhancing variation. While the model shows promising results in distinguishing fall events from noise using lab data, further testing in real-world environments is recommended for validation and improvement. Despite limited available data, the proposed system shows the potential for an accurate and rapid response to falls, mitigating health implications, and addressing the needs of an aging population. This case study was performed as part of the ZIM Project. Further research on sensors enhanced by artificial intelligence will be continued in the ShapeFuture Project.
• 摘要：护理人员的日益短缺和养老院跌倒风险的急剧上升对医疗保健系统构成了重大挑战。本研究介绍了一种集成在护理床中的自动跌倒检测系统的开发，旨在通过可穿戴设备或视频监控在不损害隐私的情况下提高患者安全性。通过床架传输的机械振动使用短时傅里叶变换进行处理，从而能够使用卷积神经网络对不同的人类跌倒模式进行稳健分类。解决了与数据数量和多样性有关的挑战，建议生成更多数据，特别强调增强变化。虽然该模型在使用实验室数据区分跌倒事件和噪声方面显示出有希望的结果，但建议在现实环境中进行进一步测试以进行验证和改进。尽管可用数据有限，但所提出的系统显示出对跌倒做出准确和快速反应的潜力，减轻健康影响并满足老龄化人口的需求。本案例研究是作为 ZIM 项目的一部分进行的。ShapeFuture 项目中将继续对人工智能增强的传感器进行进一步研究。

Title: Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation

• Authors: Xi Zhang, Zaiqiao Meng, Jake Lever, Edmond S. L. Ho
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.04954
• Pdf URL: https://arxiv.org/pdf/2412.04954
• Copy Paste: [[2412.04954]] Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation(https://arxiv.org/abs/2412.04954)
• Keywords: generation
• Abstract: We introduce a radiology-focused visual language model designed to generate radiology reports from chest X-rays. Building on previous findings that large language models (LLMs) can acquire multimodal capabilities when aligned with pretrained vision encoders, we demonstrate similar potential with chest X-ray images. This integration enhances the ability of model to understand and describe chest X-ray images. Our model combines an image encoder with a fine-tuned LLM based on the Vicuna-7B architecture, enabling it to generate different sections of a radiology report with notable accuracy. The training process involves a two-stage approach: (i) initial alignment of chest X-ray features with the LLM (ii) followed by fine-tuning for radiology report generation.
• 摘要：我们引入了一种以放射学为中心的视觉语言模型，旨在根据胸部 X 光片生成放射学报告。基于先前的研究，大型语言模型 (LLM) 在与预训练的视觉编码器对齐时可以获得多模态能力，我们在胸部 X 光片图像方面展示了类似的潜力。这种集成增强了模型理解和描述胸部 X 光片图像的能力。我们的模型将图像编码器与基于 Vicuna-7B 架构的微调 LLM 相结合，使其能够以显著的准确性生成放射学报告的不同部分。训练过程涉及两阶段方法：(i) 初步将胸部 X 光片特征与 LLM 对齐 (ii) 然后进行微调以生成放射学报告。

Title: Noise Matters: Diffusion Model-based Urban Mobility Generation with Collaborative Noise Priors

• Authors: Yuheng Zhang, Yuan Yuan, Jingtao Ding, Jian Yuan, Yong Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05000
• Pdf URL: https://arxiv.org/pdf/2412.05000
• Copy Paste: [[2412.05000]] Noise Matters: Diffusion Model-based Urban Mobility Generation with Collaborative Noise Priors(https://arxiv.org/abs/2412.05000)
• Keywords: generation
• Abstract: With global urbanization, the focus on sustainable cities has largely grown, driving research into equity, resilience, and urban planning, which often relies on mobility data. The rise of web-based apps and mobile devices has provided valuable user data for mobility-related research. However, real-world mobility data is costly and raises privacy concerns. To protect privacy while retaining key features of real-world movement, the demand for synthetic data has steadily increased. Recent advances in diffusion models have shown great potential for mobility trajectory generation due to their ability to model randomness and uncertainty. However, existing approaches often directly apply identically distributed (i.i.d.) noise sampling from image generation techniques, which fail to account for the spatiotemporal correlations and social interactions that shape urban mobility patterns. In this paper, we propose CoDiffMob, a diffusion method for urban mobility generation with collaborative noise priors, we emphasize the critical role of noise in diffusion models for generating mobility data. By leveraging both individual movement characteristics and population-wide dynamics, we construct novel collaborative noise priors that provide richer and more informative guidance throughout the generation process. Extensive experiments demonstrate the superiority of our method, with generated data accurately capturing both individual preferences and collective patterns, achieving an improvement of over 32\%. Furthermore, it can effectively replace web-derived mobility data to better support downstream applications, while safeguarding user privacy and fostering a more secure and ethical web. This highlights its tremendous potential for applications in sustainable city-related research.
• 摘要：随着全球城市化进程的加快，人们对可持续城市的关注度大幅提升，推动了公平、弹性和城市规划的研究，而这些研究通常依赖于移动数据。基于网络的应用程序和移动设备的兴​​起为与移动相关的研究提供了宝贵的用户数据。然而，现实世界的移动数据成本高昂，并引发了隐私问题。为了在保留现实世界运动的关键特征的同时保护隐私，对合成数据的需求稳步增长。扩散模型的最新进展显示出巨大的移动轨迹生成潜力，因为它们能够对随机性和不确定性进行建模。然而，现有的方法通常直接应用来自图像生成技术的同分布 (i.i.d.) 噪声采样，而这些方法无法解释塑造城市移动模式的时空相关性和社会互动。在本文中，我们提出了 CoDiffMob，这是一种具有协作噪声先验的城市移动生成扩散方法，我们强调噪声在生成移动数据的扩散模型中的关键作用。通过利用个人运动特征和全人口动态，我们构建了新颖的协作噪声先验，在整个生成过程中提供更丰富、更具信息量的指导。大量实验证明了我们方法的优越性，生成的数据准确地捕捉了个人偏好和集体模式，实现了超过 32% 的改进。此外，它可以有效地取代网络衍生的移动数据，以更好地支持下游应用程序，同时保护用户隐私并促进更安全、更合乎道德的网络。这凸显了其在可持续城市相关研究应用方面的巨大潜力。

Title: SLayR: Scene Layout Generation with Rectified Flow

• Authors: Cameron Braunstein, Hevra Petekkaya, Jan Eric Lenssen, Mariya Toneva, Eddy Ilg
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05003
• Pdf URL: https://arxiv.org/pdf/2412.05003
• Copy Paste: [[2412.05003]] SLayR: Scene Layout Generation with Rectified Flow(https://arxiv.org/abs/2412.05003)
• Keywords: generation
• Abstract: We introduce SLayR, Scene Layout Generation with Rectified flow. State-of-the-art text-to-image models achieve impressive results. However, they generate images end-to-end, exposing no fine-grained control over the process. SLayR presents a novel transformer-based rectified flow model for layout generation over a token space that can be decoded into bounding boxes and corresponding labels, which can then be transformed into images using existing models. We show that established metrics for generated images are inconclusive for evaluating their underlying scene layout, and introduce a new benchmark suite, including a carefully designed repeatable human-evaluation procedure that assesses the plausibility and variety of generated layouts. In contrast to previous works, which perform well in either high variety or plausibility, we show that our approach performs well on both of these axes at the same time. It is also at least 5x times smaller in the number of parameters and 37% faster than the baselines. Our complete text-to-image pipeline demonstrates the added benefits of an interpretable and editable intermediate representation.
• 摘要：我们引入了 SLayR，即使用整流的场景布局生成。最先进的文本到图像模型取得了令人印象深刻的结果。但是，它们以端到端的方式生成图像，因此无法对过程进行细粒度控制。SLayR 提出了一种基于变换器的新型整流模型，用于在标记空间上生成布局，该标记空间可以解码为边界框和相应的标签，然后可以使用现有模型将其转换为图像。我们表明，已建立的生成图像指标对于评估其底层场景布局并不确定，并引入了一个新的基准套件，其中包括一个精心设计的可重复人工评估程序，用于评估生成布局的合理性和多样性。与之前在高多样性或合理性方面表现良好的作品相比，我们表明我们的方法在这两个方面都表现良好。它的参数数量至少是基线的 5 倍，速度快 37%。我们完整的文本到图像管道展示了可解释和可编辑中间表示的额外好处。

Title: ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration

• Authors: Chi-Wei Hsiao, Yu-Lun Liu, Cheng-Kun Yang, Sheng-Po Kuo, Kevin Jou, Chia-Ping Chen
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.05043
• Pdf URL: https://arxiv.org/pdf/2412.05043
• Copy Paste: [[2412.05043]] ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration(https://arxiv.org/abs/2412.05043)
• Keywords: restoration, generation
• Abstract: While recent works on blind face image restoration have successfully produced impressive high-quality (HQ) images with abundant details from low-quality (LQ) input images, the generated content may not accurately reflect the real appearance of a person. To address this problem, incorporating well-shot personal images as additional reference inputs could be a promising strategy. Inspired by the recent success of the Latent Diffusion Model (LDM), we propose ReF-LDM, an adaptation of LDM designed to generate HQ face images conditioned on one LQ image and multiple HQ reference images. Our model integrates an effective and efficient mechanism, CacheKV, to leverage the reference images during the generation process. Additionally, we design a timestep-scaled identity loss, enabling our LDM-based model to focus on learning the discriminating features of human faces. Lastly, we construct FFHQ-Ref, a dataset consisting of 20,405 high-quality (HQ) face images with corresponding reference images, which can serve as both training and evaluation data for reference-based face restoration models.
• 摘要：虽然最近关于盲人脸部图像恢复的研究已经成功地从低质量 (LQ) 输入图像中生成了令人印象深刻的高质量 (HQ) 图像，并且细节丰富，但生成的内容可能无法准确反映人的真实外貌。为了解决这个问题，将拍摄精良的个人图像作为额外的参考输入可能是一种很有前途的策略。受到潜在扩散模型 (LDM) 最近成功的启发，我们提出了 ReF-LDM，这是 LDM 的一种改编，旨在根据一张 LQ 图像和多张 HQ 参考图像生成 HQ 人脸图像。我们的模型集成了一种有效且高效的机制 CacheKV，可在生成过程中利用参考图像。此外，我们设计了一个时间步长缩放的身份损失，使我们的基于 LDM 的模型能够专注于学习人脸的判别特征。最后，我们构建了 FFHQ-Ref，这是一个由 20,405 张高质量 (HQ) 人脸图像和相应的参考图像组成的数据集，可用作基于参考的人脸恢复模型的训练和评估数据。

Title: BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects

• Authors: Wanyue Zhang, Rishabh Dabral, Vladislav Golyanik, Vasileios Choutas, Eduardo Alvarado, Thabo Beeler, Marc Habermann, Christian Theobalt
• Subjects: cs.CV, cs.GR, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.05066
• Pdf URL: https://arxiv.org/pdf/2412.05066
• Copy Paste: [[2412.05066]] BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects(https://arxiv.org/abs/2412.05066)
• Keywords: generative
• Abstract: We present BimArt, a novel generative approach for synthesizing 3D bimanual hand interactions with articulated objects. Unlike prior works, we do not rely on a reference grasp, a coarse hand trajectory, or separate modes for grasping and articulating. To achieve this, we first generate distance-based contact maps conditioned on the object trajectory with an articulation-aware feature representation, revealing rich bimanual patterns for manipulation. The learned contact prior is then used to guide our hand motion generator, producing diverse and realistic bimanual motions for object movement and articulation. Our work offers key insights into feature representation and contact prior for articulated objects, demonstrating their effectiveness in taming the complex, high-dimensional space of bimanual hand-object interactions. Through comprehensive quantitative experiments, we demonstrate a clear step towards simplified and high-quality hand-object animations that excel over the state-of-the-art in motion quality and diversity.
• 摘要：我们提出了 BimArt，一种用于合成 3D 双手与关节物体交互的新型生成方法。与之前的研究不同，我们不依赖于参考抓握、粗略的手部轨迹或单独的抓握和关节模式。为了实现这一点，我们首先根据物体轨迹生成基于距离的接触图，并使用关节感知特征表示，揭示丰富的双手操作模式。然后使用学习到的接触先验来指导我们的手部运动生成器，为物体移动和关节产生多样化和逼真的双手运动。我们的工作为关节物体的特征表示和接触先验提供了关键见解，证明了它们在驯服复杂的高维双手手物体交互空间方面的有效性。通过全面的定量实验，我们展示了朝着简化和高质量手部物体动画迈出的明确一步，这些动画在运动质量和多样性方面优于最先进的技术。

Title: SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization

• Authors: Xiaofeng Tan, Hongsong Wang, Xin Geng, Pan Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05095
• Pdf URL: https://arxiv.org/pdf/2412.05095
• Copy Paste: [[2412.05095]] SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization(https://arxiv.org/abs/2412.05095)
• Keywords: generation
• Abstract: Text-to-motion generation is essential for advancing the creative industry but often presents challenges in producing consistent, realistic motions. To address this, we focus on fine-tuning text-to-motion models to consistently favor high-quality, human-preferred motions, a critical yet largely unexplored problem. In this work, we theoretically investigate the DPO under both online and offline settings, and reveal their respective limitation: overfitting in offline DPO, and biased sampling in online DPO. Building on our theoretical insights, we introduce Semi-online Preference Optimization (SoPo), a DPO-based method for training text-to-motion models using "semi-online" data pair, consisting of unpreferred motion from online distribution and preferred motion in offline datasets. This method leverages both online and offline DPO, allowing each to compensate for the other's limitations. Extensive experiments demonstrate that SoPo outperforms other preference alignment methods, with an MM-Dist of 3.25% (vs e.g. 0.76% of MoDiPO) on the MLD model, 2.91% (vs e.g. 0.66% of MoDiPO) on MDM model, respectively. Additionally, the MLD model fine-tuned by our SoPo surpasses the SoTA model in terms of R-precision and MM Dist. Visualization results also show the efficacy of our SoPo in preference alignment. Our project page is this https URL.
• 摘要：文本转动作生成对于推动创意产业发展至关重要，但生成一致、逼真的动作往往面临挑战。为了解决这个问题，我们专注于微调文本转动作模型，以始终支持高质量、人类偏好的动作，这是一个关键但尚未探索的问题。在这项工作中，我们从理论上研究了在线和离线设置下的 DPO，并揭示了它们各自的局限性：离线 DPO 中的过度拟合和在线 DPO 中的偏差采样。基于我们的理论见解，我们引入了半在线偏好优化 (SoPo)，这是一种基于 DPO 的方法，用于使用“半在线”数据对训练文本转动作模型，包括在线分布中的非偏好动作和离线数据集中的偏好动作。该方法利用在线和离线 DPO，允许每个 DPO 弥补彼此的局限性。大量实验表明，SoPo 优于其他偏好对齐方法，在 MLD 模型上的 MM-Dist 为 3.25%（例如 MoDiPO 为 0.76%），在 MDM 模型上的 MM-Dist 为 2.91%（例如 MoDiPO 为 0.66%）。此外，经过我们的 SoPo 微调的 MLD 模型在 R 精度和 MM Dist 方面超越了 SoTA 模型。可视化结果也显示了我们的 SoPo 在偏好对齐方面的有效性。我们的项目页面是这个 https URL。

Title: The Silent Prompt: Initial Noise as Implicit Guidance for Goal-Driven Image Generation

• Authors: Ruoyu Wang, Huayang Huang, Ye Zhu, Olga Russakovsky, Yu Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05101
• Pdf URL: https://arxiv.org/pdf/2412.05101
• Copy Paste: [[2412.05101]] The Silent Prompt: Initial Noise as Implicit Guidance for Goal-Driven Image Generation(https://arxiv.org/abs/2412.05101)
• Keywords: generation, generative
• Abstract: Text-to-image synthesis (T2I) has advanced remarkably with the emergence of large-scale diffusion models. In the conventional setup, the text prompt provides explicit, user-defined guidance, directing the generation process by denoising a randomly sampled Gaussian noise. In this work, we reveal that the often-overlooked noise itself encodes inherent generative tendencies, acting as a "silent prompt" that implicitly guides the output. This implicit guidance, embedded in the noise scheduler design of diffusion model formulations and their training stages, generalizes across a wide range of T2I models and backbones. Building on this insight, we introduce NoiseQuery, a novel strategy that selects optimal initial noise from a pre-built noise library to meet diverse user needs. Our approach not only enhances high-level semantic alignment with text prompts, but also allows for nuanced adjustments of low-level visual attributes, such as texture, sharpness, shape, and color, which are typically challenging to control through text alone. Extensive experiments across various models and target attributes demonstrate the strong performance and zero-shot transferability of our approach, requiring no additional optimization.
• 摘要：随着大规模扩散模型的出现，文本到图像合成 (T2I) 取得了显著进展。在传统设置中，文本提示提供明确的、用户定义的指导，通过对随机采样的高斯噪声进行去噪来指导生成过程。在这项工作中，我们揭示了经常被忽视的噪声本身编码了固有的生成趋势，充当了隐式指导输出的“无声提示”。这种隐式指导嵌入在扩散模型公式及其训练阶段的噪声调度器设计中，可推广到广泛的 T2I 模型和主干。基于这一洞察，我们引入了 NoiseQuery，这是一种新颖的策略，它从预建的噪声库中选择最佳初始噪声以满足不同的用户需求。我们的方法不仅通过文本提示增强了高级语义对齐，而且还允许对低级视觉属性（例如纹理、锐度、形状和颜色）进行细微调整，而这些属性通常很难仅通过文本进行控制。针对各种模型和目标属性进行的大量实验证明了我们的方法的强大性能和零样本可转移性，无需额外的优化。

Title: LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generation

• Authors: Donald Shenaj, Ondrej Bohdal, Mete Ozay, Pietro Zanuttigh, Umberto Michieli
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05148
• Pdf URL: https://arxiv.org/pdf/2412.05148
• Copy Paste: [[2412.05148]] LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generation(https://arxiv.org/abs/2412.05148)
• Keywords: generation
• Abstract: Recent advancements in image generation models have enabled personalized image creation with both user-defined subjects (content) and styles. Prior works achieved personalization by merging corresponding low-rank adaptation parameters (LoRAs) through optimization-based methods, which are computationally demanding and unsuitable for real-time use on resource-constrained devices like smartphones. To address this, we introduce this http URL, a method that not only improves image quality but also achieves a remarkable speedup of over $4000\times$ in the merging process. this http URL pre-trains a hypernetwork on a diverse set of content-style LoRA pairs, learning an efficient merging strategy that generalizes to new, unseen content-style pairs, enabling fast, high-quality personalization. Moreover, we identify limitations in existing evaluation metrics for content-style quality and propose a new protocol using multimodal large language models (MLLM) for more accurate assessment. Our method significantly outperforms the current state of the art in both content and style fidelity, as validated by MLLM assessments and human evaluations.
• 摘要：图像生成模型的最新进展使得个性化图像创建成为可能，既有用户定义的主题（内容），也有用户定义的风格。先前的研究通过基于优化的方法合并相应的低秩自适应参数 (LoRA) 来实现个性化，但这种方法计算量大，不适合在智能手机等资源受限的设备上实时使用。为了解决这个问题，我们引入了这个 http URL，这种方法不仅可以提高图像质量，而且在合并过程中实现了超过 $4000\times$ 的显着加速。这个 http URL 在多样化的内容风格 LoRA 对集合上预先训练了一个超网络，学习了一种有效的合并策略，该策略可以推广到新的、看不见的内容风格对，从而实现快速、高质量的个性化。此外，我们确定了现有内容风格质量评估指标的局限性，并提出了一种使用多模态大型语言模型 (MLLM) 的新协议，以便进行更准确的评估。经 MLLM 评估和人工评估验证，我们的方法在内容和风格保真度方面都远远优于当前最先进的技术。

Title: A text-to-tabular approach to generate synthetic patient data using LLMs

• Authors: Margaux Tornqvist, Jean-Daniel Zucker, Tristan Fauvel, Nicolas Lambert, Mathilde Berthelot, Antoine Movschin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05153
• Pdf URL: https://arxiv.org/pdf/2412.05153
• Copy Paste: [[2412.05153]] A text-to-tabular approach to generate synthetic patient data using LLMs(https://arxiv.org/abs/2412.05153)
• Keywords: generation
• Abstract: Access to large-scale high-quality healthcare databases is key to accelerate medical research and make insightful discoveries about diseases. However, access to such data is often limited by patient privacy concerns, data sharing restrictions and high costs. To overcome these limitations, synthetic patient data has emerged as an alternative. However, synthetic data generation (SDG) methods typically rely on machine learning (ML) models trained on original data, leading back to the data scarcity problem. We propose an approach to generate synthetic tabular patient data that does not require access to the original data, but only a description of the desired database. We leverage prior medical knowledge and in-context learning capabilities of large language models (LLMs) to generate realistic patient data, even in a low-resource setting. We quantitatively evaluate our approach against state-of-the-art SDG models, using fidelity, privacy, and utility metrics. Our results show that while LLMs may not match the performance of state-of-the-art models trained on the original data, they effectively generate realistic patient data with well-preserved clinical correlations. An ablation study highlights key elements of our prompt contributing to high-quality synthetic patient data generation. This approach, which is easy to use and does not require original data or advanced ML skills, is particularly valuable for quickly generating custom-designed patient data, supporting project implementation and providing educational resources.
• 摘要：访问大规模高质量医疗保健数据库是加速医学研究和对疾病做出深刻发现的关键。然而，访问此类数据通常受到患者隐私问题、数据共享限制和高成本的限制。为了克服这些限制，合成患者数据已成为一种替代方案。然而，合成数据生成 (SDG) 方法通常依赖于在原始数据上训练的机器学习 (ML) 模型，这又导致了数据稀缺问题。我们提出了一种生成合成表格患者数据的方法，该方法不需要访问原始数据，而只需要所需数据库的描述。我们利用大型语言模型 (LLM) 的先前医学知识和上下文学习能力来生成真实的患者数据，即使在资源匮乏的环境中也是如此。我们使用保真度、隐私和实用性指标，根据最先进的 SDG 模型对我们的方法进行了定量评估。我们的结果表明，虽然 LLM 可能无法与在原始数据上训练的最先进的模型的性能相匹配，但它们可以有效地生成具有良好保留临床相关性的真实患者数据。一项消融研究突出了我们快速生成高质量合成患者数据的关键要素。这种方法易于使用，不需要原始数据或高级 ML 技能，对于快速生成定制设计的患者数据、支持项目实施和提供教育资源特别有价值。

Title: Towards Flexible 3D Perception: Object-Centric Occupancy Completion Augments 3D Object Detection

• Authors: Chaoda Zheng, Feng Wang, Naiyan Wang, Shuguang Cui, Zhen Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05154
• Pdf URL: https://arxiv.org/pdf/2412.05154
• Copy Paste: [[2412.05154]] Towards Flexible 3D Perception: Object-Centric Occupancy Completion Augments 3D Object Detection(https://arxiv.org/abs/2412.05154)
• Keywords: generation
• Abstract: While 3D object bounding box (bbox) representation has been widely used in autonomous driving perception, it lacks the ability to capture the precise details of an object's intrinsic geometry. Recently, occupancy has emerged as a promising alternative for 3D scene perception. However, constructing a high-resolution occupancy map remains infeasible for large scenes due to computational constraints. Recognizing that foreground objects only occupy a small portion of the scene, we introduce object-centric occupancy as a supplement to object bboxes. This representation not only provides intricate details for detected objects but also enables higher voxel resolution in practical applications. We advance the development of object-centric occupancy perception from both data and algorithm perspectives. On the data side, we construct the first object-centric occupancy dataset from scratch using an automated pipeline. From the algorithmic standpoint, we introduce a novel object-centric occupancy completion network equipped with an implicit shape decoder that manages dynamic-size occupancy generation. This network accurately predicts the complete object-centric occupancy volume for inaccurate object proposals by leveraging temporal information from long sequences. Our method demonstrates robust performance in completing object shapes under noisy detection and tracking conditions. Additionally, we show that our occupancy features significantly enhance the detection results of state-of-the-art 3D object detectors, especially for incomplete or distant objects in the Waymo Open Dataset.
• 摘要：虽然 3D 物体边界框 (bbox) 表示已广泛应用于自动驾驶感知，但它缺乏捕捉物体内在几何形状的精确细节的能力。最近，占用率已成为 3D 场景感知的一种有前途的替代方案。然而，由于计算限制，构建高分辨率占用图对于大型场景仍然不可行。认识到前景物体只占据场景的一小部分，我们引入了以物体为中心的占用率作为物体边界框的补充。这种表示不仅为检测到的物体提供了复杂的细节，而且还在实际应用中实现了更高的体素分辨率。我们从数据和算法的角度推进了以物体为中心的占用感知的发展。在数据方面，我们使用自动化管道从头开始构建第一个以物体为中心的占用数据集。从算法的角度来看，我们引入了一个新颖的以物体为中心的占用完成网络，该网络配备了一个隐式形状解码器，可以管理动态尺寸的占用率生成。该网络利用长序列的时间信息，准确预测不准确物体提议的完整物体中心占用体积。我们的方法在嘈杂的检测和跟踪条件下，在完成物体形状方面表现出了强大的性能。此外，我们还表明，我们的占用特征显著增强了最先进的 3D 物体检测器的检测结果，尤其是对于 Waymo 开放数据集中不完整或远处的物体。

Title: DNF: Unconditional 4D Generation with Dictionary-based Neural Fields

• Authors: Xinyi Zhang, Naiqi Li, Angela Dai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05161
• Pdf URL: https://arxiv.org/pdf/2412.05161
• Copy Paste: [[2412.05161]] DNF: Unconditional 4D Generation with Dictionary-based Neural Fields(https://arxiv.org/abs/2412.05161)
• Keywords: generation, generative
• Abstract: While remarkable success has been achieved through diffusion-based 3D generative models for shapes, 4D generative modeling remains challenging due to the complexity of object deformations over time. We propose DNF, a new 4D representation for unconditional generative modeling that efficiently models deformable shapes with disentangled shape and motion while capturing high-fidelity details in the deforming objects. To achieve this, we propose a dictionary learning approach to disentangle 4D motion from shape as neural fields. Both shape and motion are represented as learned latent spaces, where each deformable shape is represented by its shape and motion global latent codes, shape-specific coefficient vectors, and shared dictionary information. This captures both shape-specific detail and global shared information in the learned dictionary. Our dictionary-based representation well balances fidelity, contiguity and compression -- combined with a transformer-based diffusion model, our method is able to generate effective, high-fidelity 4D animations.
• 摘要：尽管基于扩散的 3D 形状生成模型已经取得了显著成功，但由于物体随时间变形的复杂性，4D 生成建模仍然具有挑战性。我们提出了 DNF，一种用于无条件生成建模的新型 4D 表示，它可以高效地对具有解开的形状和运动的可变形形状进行建模，同时捕捉变形物体中的高保真细节。为了实现这一点，我们提出了一种字典学习方法，将 4D 运动从形状中分离出来作为神经场。形状和运动都表示为学习到的潜在空间，其中每个可变形形状都由其形状和运动全局潜在代码、形状特定系数向量和共享字典信息表示。这可以在学习到的字典中捕获形状特定的细节和全局共享信息。我们基于字典的表示很好地平衡了保真度、连续性和压缩性——结合基于变压器的扩散模型，我们的方法能够生成有效的高保真 4D 动画。

Title: Variational Encoder-Decoders for Learning Latent Representations of Physical Systems

• Authors: Subashree Venkatasubramanian, David A. Barajas-Solano
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.05175
• Pdf URL: https://arxiv.org/pdf/2412.05175
• Copy Paste: [[2412.05175]] Variational Encoder-Decoders for Learning Latent Representations of Physical Systems(https://arxiv.org/abs/2412.05175)
• Keywords: generative
• Abstract: We present a deep-learning Variational Encoder-Decoder (VED) framework for learning data-driven low-dimensional representations of the relationship between high-dimensional parameters of a physical system and the system's high-dimensional observable response. The framework consists of two deep learning-based probabilistic transformations: An encoder mapping parameters to latent codes and a decoder mapping latent codes to the observable response. The hyperparameters of these transformations are identified by maximizing a variational lower bound on the log-conditional distribution of the observable response given parameters. To promote the disentanglement of latent codes, we equip this variational loss with a penalty on the off-diagonal entries of the aggregate distribution covariance of codes. This regularization penalty encourages the pushforward of a standard Gaussian distribution of latent codes to approximate the marginal distribution of the observable response. Using the proposed framework we successfully model the hydraulic pressure response at observation wells of a groundwater flow model as a function of its discrete log-hydraulic transmissivity field. Compared to the canonical correlation analysis encoding, the VED model achieves a lower-dimensional latent representation, with as low as $r = 50$ latent dimensions without a significant loss of reconstruction accuracy. We explore the impact of regularization on model performance, finding that KL-divergence and covariance regularization improve feature disentanglement in latent space while maintaining reconstruction accuracy. Furthermore, we evaluate the generative capabilities of the regularized model by decoding random Gaussian noise, revealing that tuning both $\beta$ and $\lambda$ parameters enhances the quality of the generated observable response data.
• 摘要：我们提出了一个深度学习变分编码器-解码器 (VED) 框架，用于学习数据驱动的低维表示，以表示物理系统的高维参数与系统的高维可观测响应之间的关系。该框架由两个基于深度学习的概率转换组成：将参数映射到潜在代码的编码器和将潜在代码映射到可观测响应的解码器。这些转换的超参数是通过在给定参数的情况下最大化可观测响应对数条件分布的变分下限来识别的。为了促进潜在代码的解缠，我们为该变分损失配备了代码总分布协方差的非对角线项的惩罚。这种正则化惩罚鼓励将标准高斯潜在代码分布向前推进以近似可观测响应的边际分布。使用所提出的框架，我们成功地将地下水流模型的观测井的水压响应建模为其离散对数水力渗透率场的函数。与典型相关分析编码相比，VED 模型实现了较低维度的潜在表示，潜在维度低至 $r = 50$，而重建精度没有显著损失。我们探索了正则化对模型性能的影响，发现 KL 散度和协方差正则化在保持重建精度的同时改善了潜在空间中的特征解缠。此外，我们通过解码随机高斯噪声来评估正则化模型的生成能力，结果表明调整 $\beta$ 和 $\lambda$ 参数可以提高生成的可观测响应数据的质量。

Title: Mind the Time: Temporally-Controlled Multi-Event Video Generation

• Authors: Ziyi Wu, Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Yuwei Fang, Varnith Chordia, Igor Gilitschenski, Sergey Tulyakov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05263
• Pdf URL: https://arxiv.org/pdf/2412.05263
• Copy Paste: [[2412.05263]] Mind the Time: Temporally-Controlled Multi-Event Video Generation(https://arxiv.org/abs/2412.05263)
• Keywords: generation
• Abstract: Real-world videos consist of sequences of events. Generating such sequences with precise temporal control is infeasible with existing video generators that rely on a single paragraph of text as input. When tasked with generating multiple events described using a single prompt, such methods often ignore some of the events or fail to arrange them in the correct order. To address this limitation, we present MinT, a multi-event video generator with temporal control. Our key insight is to bind each event to a specific period in the generated video, which allows the model to focus on one event at a time. To enable time-aware interactions between event captions and video tokens, we design a time-based positional encoding method, dubbed ReRoPE. This encoding helps to guide the cross-attention operation. By fine-tuning a pre-trained video diffusion transformer on temporally grounded data, our approach produces coherent videos with smoothly connected events. For the first time in the literature, our model offers control over the timing of events in generated videos. Extensive experiments demonstrate that MinT outperforms existing open-source models by a large margin.
• 摘要：现实世界的视频由一系列事件组成。使用现有的视频生成器生成具有精确时间控制的此类序列是不可行的，因为这些生成器依赖于一段文本作为输入。当使用单个提示生成多个事件时，这些方法通常会忽略某些事件或无法按正确的顺序排列它们。为了解决这一限制，我们提出了 MinT，这是一个具有时间控制的多事件视频生成器。我们的主要见解是将每个事件绑定到生成的视频中的特定时间段，这使模型可以一次关注一个事件。为了实现事件字幕和视频标记之间的时间感知交互，我们设计了一种基于时间的位置编码方法，称为 ReRoPE。这种编码有助于指导交叉注意操作。通过在时间基础数据上微调预先训练的视频扩散变换器，我们的方法可以生成具有平滑连接事件的连贯视频。在文献中，我们的模型首次提供了对生成视频中事件时间的控制。大量实验表明，MinT 的表现远远优于现有的开源模型。

Title: Chimera: Accurate retrosynthesis prediction by ensembling models with diverse inductive biases

• Authors: Krzysztof Maziarz, Guoqing Liu, Hubert Misztela, Aleksei Kornev, Piotr Gaiński, Holger Hoefling, Mike Fortunato, Rishi Gupta, Marwin Segler
• Subjects: cs.LG, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2412.05269
• Pdf URL: https://arxiv.org/pdf/2412.05269
• Copy Paste: [[2412.05269]] Chimera: Accurate retrosynthesis prediction by ensembling models with diverse inductive biases(https://arxiv.org/abs/2412.05269)
• Keywords: generative
• Abstract: Planning and conducting chemical syntheses remains a major bottleneck in the discovery of functional small molecules, and prevents fully leveraging generative AI for molecular inverse design. While early work has shown that ML-based retrosynthesis models can predict reasonable routes, their low accuracy for less frequent, yet important reactions has been pointed out. As multi-step search algorithms are limited to reactions suggested by the underlying model, the applicability of those tools is inherently constrained by the accuracy of retrosynthesis prediction. Inspired by how chemists use different strategies to ideate reactions, we propose Chimera: a framework for building highly accurate reaction models that combine predictions from diverse sources with complementary inductive biases using a learning-based ensembling strategy. We instantiate the framework with two newly developed models, which already by themselves achieve state of the art in their categories. Through experiments across several orders of magnitude in data scale and time-splits, we show Chimera outperforms all major models by a large margin, owing both to the good individual performance of its constituents, but also to the scalability of our ensembling strategy. Moreover, we find that PhD-level organic chemists prefer predictions from Chimera over baselines in terms of quality. Finally, we transfer the largest-scale checkpoint to an internal dataset from a major pharmaceutical company, showing robust generalization under distribution shift. With the new dimension that our framework unlocks, we anticipate further acceleration in the development of even more accurate models.
• 摘要：规划和进行化学合成仍然是发现功能性小分子的主要瓶颈，并且阻碍了生成式 AI 进行分子逆向设计的充分利用。虽然早期的研究表明基于 ML 的逆合成模型可以预测合理的路线，但有人指出，它们对于不常见但重要的反应的准确度较低。由于多步搜索算法仅限于底层模型所建议的反应，因此这些工具的适用性本质上受到逆合成预测准确性的限制。受化学家如何使用不同策略来构思反应的启发，我们提出了 Chimera：一个用于构建高精度反应模型的框架，该框架使用基于学习的集成策略将来自不同来源的预测与互补的归纳偏差相结合。我们用两个新开发的模型实例化该框架，这两个模型本身已经在其类别中达到了最先进的水平。通过跨多个数量级的数据规模和时间分割的实验，我们发现 Chimera 的表现远胜于所有主要模型，这不仅归功于其组成部分的良好性能，还归功于我们的集成策略的可扩展性。此外，我们发现博士级有机化学家在质量方面更喜欢 Chimera 的预测而不是基线。最后，我们将最大规模的检查点转移到一家大型制药公司的内部数据集，显示出在分布变化下的稳健泛化。凭借我们的框架解锁的新维度，我们预计更精确的模型的开发将进一步加速。

Title: MotionFlow: Attention-Driven Motion Transfer in Video Diffusion Models

• Authors: Tuna Han Salih Meral, Hidir Yesiltepe, Connor Dunlop, Pinar Yanardag
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05275
• Pdf URL: https://arxiv.org/pdf/2412.05275
• Copy Paste: [[2412.05275]] MotionFlow: Attention-Driven Motion Transfer in Video Diffusion Models(https://arxiv.org/abs/2412.05275)
• Keywords: generative
• Abstract: Text-to-video models have demonstrated impressive capabilities in producing diverse and captivating video content, showcasing a notable advancement in generative AI. However, these models generally lack fine-grained control over motion patterns, limiting their practical applicability. We introduce MotionFlow, a novel framework designed for motion transfer in video diffusion models. Our method utilizes cross-attention maps to accurately capture and manipulate spatial and temporal dynamics, enabling seamless motion transfers across various contexts. Our approach does not require training and works on test-time by leveraging the inherent capabilities of pre-trained video diffusion models. In contrast to traditional approaches, which struggle with comprehensive scene changes while maintaining consistent motion, MotionFlow successfully handles such complex transformations through its attention-based mechanism. Our qualitative and quantitative experiments demonstrate that MotionFlow significantly outperforms existing models in both fidelity and versatility even during drastic scene alterations.
• 摘要：文本转视频模型在制作多样化且引人入胜的视频内容方面表现出了令人印象深刻的能力，展示了生成式人工智能的显著进步。然而，这些模型通常缺乏对运动模式的细粒度控制，限制了它们的实际适用性。我们引入了 MotionFlow，这是一个专为视频扩散模型中的运动转移而设计的新型框架。我们的方法利用交叉注意力图来准确捕捉和操纵空间和时间动态，实现跨各种环境的无缝运动转移。我们的方法不需要训练，并且通过利用预先训练的视频扩散模型的固有功能在测试时起作用。与传统方法相比，MotionFlow 通过其基于注意力的机制成功处理了这种复杂的转换，而传统方法在保持运动一致的情况下难以应对全面的场景变化。我们的定性和定量实验表明，即使在场景发生剧烈变化时，MotionFlow 在保真度和多功能性方面也明显优于现有模型。

Title: Perturb-and-Revise: Flexible 3D Editing with Generative Trajectories

• Authors: Susung Hong, Johanna Karras, Ricardo Martin-Brualla, Ira Kemelmacher-Shlizerman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05279
• Pdf URL: https://arxiv.org/pdf/2412.05279
• Copy Paste: [[2412.05279]] Perturb-and-Revise: Flexible 3D Editing with Generative Trajectories(https://arxiv.org/abs/2412.05279)
• Keywords: generative
• Abstract: The fields of 3D reconstruction and text-based 3D editing have advanced significantly with the evolution of text-based diffusion models. While existing 3D editing methods excel at modifying color, texture, and style, they struggle with extensive geometric or appearance changes, thus limiting their applications. We propose Perturb-and-Revise, which makes possible a variety of NeRF editing. First, we perturb the NeRF parameters with random initializations to create a versatile initialization. We automatically determine the perturbation magnitude through analysis of the local loss landscape. Then, we revise the edited NeRF via generative trajectories. Combined with the generative process, we impose identity-preserving gradients to refine the edited NeRF. Extensive experiments demonstrate that Perturb-and-Revise facilitates flexible, effective, and consistent editing of color, appearance, and geometry in 3D. For 360° results, please visit our project page: this https URL.
• 摘要：随着基于文本的扩散模型的发展，3D 重建和基于文本的 3D 编辑领域取得了长足的进步。虽然现有的 3D 编辑方法擅长修改颜色、纹理和样式，但它们难以应对广泛的几何或外观变化，从而限制了它们的应用。我们提出了扰动和修改方法，它可以实现多种 NeRF 编辑。首先，我们用随机初始化扰动 NeRF 参数以创建一个通用的初始化。我们通过分析局部损失状况自动确定扰动幅度。然后，我们通过生成轨迹修改编辑后的 ​​NeRF。结合生成过程，我们施加身份保留梯度来优化编辑后的 ​​NeRF。大量实验表明，扰动和修改方法有助于灵活、有效和一致地编辑 3D 中的颜色、外观和几何形状。如需 360° 结果，请访问我们的项目页面：此 https URL。

Title: Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Model

• Authors: Lening Wang, Wenzhao Zheng, Dalong Du, Yunpeng Zhang, Yilong Ren, Han Jiang, Zhiyong Cui, Haiyang Yu, Jie Zhou, Jiwen Lu, Shanghang Zhang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05280
• Pdf URL: https://arxiv.org/pdf/2412.05280
• Copy Paste: [[2412.05280]] Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Model(https://arxiv.org/abs/2412.05280)
• Keywords: generation, generative
• Abstract: 4D driving simulation is essential for developing realistic autonomous driving simulators. Despite advancements in existing methods for generating driving scenes, significant challenges remain in view transformation and spatial-temporal dynamic modeling. To address these limitations, we propose a Spatial-Temporal simulAtion for drivinG (Stag-1) model to reconstruct real-world scenes and design a controllable generative network to achieve 4D simulation. Stag-1 constructs continuous 4D point cloud scenes using surround-view data from autonomous vehicles. It decouples spatial-temporal relationships and produces coherent keyframe videos. Additionally, Stag-1 leverages video generation models to obtain photo-realistic and controllable 4D driving simulation videos from any perspective. To expand the range of view generation, we train vehicle motion videos based on decomposed camera poses, enhancing modeling capabilities for distant scenes. Furthermore, we reconstruct vehicle camera trajectories to integrate 3D points across consecutive views, enabling comprehensive scene understanding along the temporal dimension. Following extensive multi-level scene training, Stag-1 can simulate from any desired viewpoint and achieve a deep understanding of scene evolution under static spatial-temporal conditions. Compared to existing methods, our approach shows promising performance in multi-view scene consistency, background coherence, and accuracy, and contributes to the ongoing advancements in realistic autonomous driving simulation. Code: this https URL.
• 摘要：4D 驾驶模拟对于开发逼真的自动驾驶模拟器至关重要。尽管现有的生成驾驶场景的方法取得了进步，但在视图转换和时空动态建模方面仍然存在重大挑战。为了解决这些限制，我们提出了一种时空驾驶模拟 (Stag-1) 模型来重建真实世界场景，并设计了一个可控的生成网络来实现 4D 模拟。Stag-1 使用来自自动驾驶汽车的环视数据构建连续的 4D 点云场景。它解耦了时空关系并生成连贯的关键帧视频。此外，Stag-1 利用视频生成模型从任何角度获得照片般逼真且可控的 4D 驾驶模拟视频。为了扩大视图生成的范围，我们根据分解的相机姿势训练车辆运动视频，增强远距离场景的建模能力。此外，我们重建车辆摄像机轨迹以整合连续视图中的 3D 点，从而实现沿时间维度的全面场景理解。经过广泛的多级场景训练后，Stag-1 可以从任何所需的视角进行模拟，并在静态时空条件下深入了解场景演变。与现有方法相比，我们的方法在多视角场景一致性、背景连贯性和准确性方面表现出色，并为现实自动驾驶模拟的持续进步做出了贡献。代码：此 https URL。