2025-01-27

Title: Procedural Generation of 3D Maize Plant Architecture from LIDAR Data

• Authors: Mozhgan Hadadi, Mehdi Saraeian, Jackson Godbersen, Talukder Jubery, Yawei Li, Lakshmi Attigala, Aditya Balu, Soumik Sarkar, Patrick S. Schnable, Adarsh Krishnamurthy, Baskar Ganapathysubramanian
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13963
• Pdf URL: https://arxiv.org/pdf/2501.13963
• Copy Paste: [[2501.13963]] Procedural Generation of 3D Maize Plant Architecture from LIDAR Data(https://arxiv.org/abs/2501.13963)
• Keywords: generation
• Abstract: This study introduces a robust framework for generating procedural 3D models of maize (Zea mays) plants from LiDAR point cloud data, offering a scalable alternative to traditional field-based phenotyping. Our framework leverages Non-Uniform Rational B-Spline (NURBS) surfaces to model the leaves of maize plants, combining Particle Swarm Optimization (PSO) for an initial approximation of the surface and a differentiable programming framework for precise refinement of the surface to fit the point cloud data. In the first optimization phase, PSO generates an approximate NURBS surface by optimizing its control points, aligning the surface with the LiDAR data, and providing a reliable starting point for refinement. The second phase uses NURBS-Diff, a differentiable programming framework, to enhance the accuracy of the initial fit by refining the surface geometry and capturing intricate leaf details. Our results demonstrate that, while PSO establishes a robust initial fit, the integration of differentiable NURBS significantly improves the overall quality and fidelity of the reconstructed surface. This hierarchical optimization strategy enables accurate 3D reconstruction of maize leaves across diverse genotypes, facilitating the subsequent extraction of complex traits like phyllotaxy. We demonstrate our approach on diverse genotypes of field-grown maize plants. All our codes are open-source to democratize these phenotyping approaches.
• 摘要：本研究介绍了一种强大的框架，用于从 LiDAR 点云数据生成玉米 (Zea mays) 植物的程序化 3D 模型，为传统的基于田间的表型分析提供了一种可扩展的替代方案。我们的框架利用非均匀有理 B 样条 (NURBS) 曲面来模拟玉米植株的叶子，结合粒子群优化 (PSO) 对曲面进行初始近似，并结合可微分编程框架对曲面进行精确细化以拟合点云数据。在第一个优化阶段，PSO 通过优化其控制点、将曲面与 LiDAR 数据对齐并提供可靠的细化起点来生成近似 NURBS 曲面。第二阶段使用可微分编程框架 NURBS-Diff，通过细化曲面几何形状和捕捉复杂的叶子细节来提高初始拟合的准确性。我们的结果表明，虽然 PSO 建立了稳健的初始拟合，但可微分 NURBS 的集成显著提高了重建表面的整体质量和保真度。这种分层优化策略能够对不同基因型的玉米叶片进行精确的 3D 重建，从而有助于随后提取叶序等复杂性状。我们在田间种植的玉米植株的不同基因型上展示了我们的方法。我们所有的代码都是开源的，以使这些表型分析方法更加大众化。

Title: FedDAG: Federated Domain Adversarial Generation Towards Generalizable Medical Image Analysis

• Authors: Haoxuan Che, Yifei Wu, Haibo Jin, Yong Xia, Hao Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13967
• Pdf URL: https://arxiv.org/pdf/2501.13967
• Copy Paste: [[2501.13967]] FedDAG: Federated Domain Adversarial Generation Towards Generalizable Medical Image Analysis(https://arxiv.org/abs/2501.13967)
• Keywords: generation
• Abstract: Federated domain generalization aims to train a global model from multiple source domains and ensure its generalization ability to unseen target domains. {Due to the target domain being with unknown domain shifts, attempting to approximate these gaps by source domains may be the key to improving model generalization capability.} Existing works mainly focus on sharing and recombining local domain-specific attributes to increase data diversity and simulate potential domain shifts. {However, these methods may be insufficient since only the local attribute recombination can be hard to touch the out-of-distribution of global data.} In this paper, we propose a simple-yet-efficient framework named Federated Domain Adversarial Generation (FedDAG). {It aims to simulate the domain shift and improve the model generalization by adversarially generating novel domains different from local and global source domains.} Specifically, it generates novel-style images by maximizing the instance-level feature discrepancy between original and generated images and trains a generalizable task model by minimizing their feature discrepancy. {Further, we observed that FedDAG could cause different performance improvements for local models. It may be due to inherent data isolation and heterogeneity among clients, exacerbating the imbalance in their generalization contributions to the global model.} {Ignoring this imbalance can lead the global model's generalization ability to be sub-optimal, further limiting the novel domain generation procedure. } Thus, to mitigate this imbalance, FedDAG hierarchically aggregates local models at the within-client and across-client levels by using the sharpness concept to evaluate client model generalization contributions. {Extensive experiments across four medical benchmarks demonstrate FedDAG's ability to enhance generalization in federated medical scenarios.}
• 摘要：联邦域泛化旨在从多个源域训练全局模型并确保其对看不见的目标域的泛化能力。{由于目标域具有未知的域偏移，尝试通过源域来近似这些差距可能是提高模型泛化能力的关键。}现有工作主要集中于共享和重组局部域特定属性，以增加数据多样性并模拟潜在的域偏移。{但是，这些方法可能不够，因为仅靠局部属性重组很难触及全局数据的分布不均。}在本文中，我们提出了一个简单而有效的框架，称为联邦域对抗生成（FedDAG）。{它旨在通过对抗性地生成不同于本地和全局源域的新域来模拟域偏移并提高模型泛化能力。}具体而言，它通过最大化原始图像和生成图像之间的实例级特征差异来生成新颖风格的图像，并通过最小化它们的特征差异来训练可泛化的任务模型。 {此外，我们观察到 FedDAG 可能会为本地模型带来不同的性能改进。这可能是由于客户端之间固有的数据隔离和异构性，加剧了它们对全局模型的泛化贡献的不平衡。} {忽略这种不平衡可能会导致全局模型的泛化能力不理想，从而进一步限制新颖的域生成过程。} 因此，为了缓解这种不平衡，FedDAG 使用锐度概念来评估客户端模型的泛化贡献，在客户端内和客户端间级别分层聚合本地模型。{在四个医疗基准上进行的大量实验证明了 FedDAG 能够在联邦医疗场景中增强泛化能力。}

Title: Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generation

• Authors: Kenta Uesugi, Naoki Saito, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.13968
• Pdf URL: https://arxiv.org/pdf/2501.13968
• Copy Paste: [[2501.13968]] Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generation(https://arxiv.org/abs/2501.13968)
• Keywords: generation
• Abstract: Composed Image Retrieval (CIR) provides an effective way to manage and access large-scale visual data. Construction of the CIR model utilizes triplets that consist of a reference image, modification text describing desired changes, and a target image that reflects these changes. For effectively training CIR models, extensive manual annotation to construct high-quality training datasets, which can be time-consuming and labor-intensive, is required. To deal with this problem, this paper proposes a novel triplet synthesis method by leveraging counterfactual image generation. By controlling visual feature modifications via counterfactual image generation, our approach automatically generates diverse training triplets without any manual intervention. This approach facilitates the creation of larger and more expressive datasets, leading to the improvement of CIR model's performance.
• 摘要：合成图像检索 (CIR) 提供了一种管理和访问大规模视觉数据的有效方法。CIR 模型的构建利用由参考图像、描述所需更改的修改文本和反映这些更改的目标图像组成的三元组。为了有效地训练 CIR 模型，需要大量手动注释来构建高质量的训练数据集，这可能既耗时又费力。为了解决这个问题，本文提出了一种利用反事实图像生成的新型三元组合成方法。通过反事实图像生成控制视觉特征修改，我们的方法可以自动生成不同的训练三元组，而无需任何人工干预。这种方法有助于创建更大、更具表现力的数据集，从而提高 CIR 模型的性能。

Title: InsTex: Indoor Scenes Stylized Texture Synthesis

• Authors: Yunfan Zhang, Zhiwei Xiong, Zhiqi Shen, Guosheng Lin, Hao Wang, Nicolas Vun
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13969
• Pdf URL: https://arxiv.org/pdf/2501.13969
• Copy Paste: [[2501.13969]] InsTex: Indoor Scenes Stylized Texture Synthesis(https://arxiv.org/abs/2501.13969)
• Keywords: generative
• Abstract: Generating high-quality textures for 3D scenes is crucial for applications in interior design, gaming, and augmented/virtual reality (AR/VR). Although recent advancements in 3D generative models have enhanced content creation, significant challenges remain in achieving broad generalization and maintaining style consistency across multiple viewpoints. Current methods, such as 2D diffusion models adapted for 3D texturing, suffer from lengthy processing times and visual artifacts, while approaches driven by 3D data often fail to generalize effectively. To overcome these challenges, we introduce InsTex, a two-stage architecture designed to generate high-quality, style-consistent textures for 3D indoor scenes. InsTex utilizes depth-to-image diffusion priors in a coarse-to-fine pipeline, first generating multi-view images with a pre-trained 2D diffusion model and subsequently refining the textures for consistency. Our method supports both textual and visual prompts, achieving state-of-the-art results in visual quality and quantitative metrics, and demonstrates its effectiveness across various 3D texturing applications.
• 摘要：为 3D 场景生成高质量纹理对于室内设计、游戏和增强/虚拟现实 (AR/VR) 中的应用至关重要。尽管 3D 生成模型的最新进展增强了内容创建，但在实现广泛泛化和保持跨多个视点的风格一致性方面仍然存在重大挑战。当前的方法（例如适用于 3D 纹理的 2D 扩散模型）存在处理时间长和视觉伪影的问题，而由 3D 数据驱动的方法通常无法有效泛化。为了克服这些挑战，我们引入了 InsTex，这是一种两阶段架构，旨在为 3D 室内场景生成高质量、风格一致的纹理。InsTex 在粗到细的管道中使用深度到图像扩散先验，首先使用预先训练的 2D 扩散模型生成多视图图像，然后细化纹理以保持一致性。我们的方法支持文本和视觉提示，在视觉质量和定量指标方面取得了最先进的结果，并证明了其在各种 3D 纹理应用中的有效性。

Title: CGI: Identifying Conditional Generative Models with Example Images

• Authors: Zhi Zhou, Hao-Zhe Tan, Peng-Xiao Song, Lan-Zhe Guo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13991
• Pdf URL: https://arxiv.org/pdf/2501.13991
• Copy Paste: [[2501.13991]] CGI: Identifying Conditional Generative Models with Example Images(https://arxiv.org/abs/2501.13991)
• Keywords: generative
• Abstract: Generative models have achieved remarkable performance recently, and thus model hubs have emerged. Existing model hubs typically assume basic text matching is sufficient to search for models. However, in reality, due to different abstractions and the large number of models in model hubs, it is not easy for users to review model descriptions and example images, choosing which model best meets their needs. Therefore, it is necessary to describe model functionality wisely so that future users can efficiently search for the most suitable model for their needs. Efforts to address this issue remain limited. In this paper, we propose Conditional Generative Model Identification (CGI), which aims to provide an effective way to identify the most suitable model using user-provided example images rather than requiring users to manually review a large number of models with example images. To address this problem, we propose the PromptBased Model Identification (PMI) , which can adequately describe model functionality and precisely match requirements with specifications. To evaluate PMI approach and promote related research, we provide a benchmark comprising 65 models and 9100 identification tasks. Extensive experimental and human evaluation results demonstrate that PMI is effective. For instance, 92% of models are correctly identified with significantly better FID scores when four example images are provided.
• 摘要：生成模型近年来取得了令人瞩目的成绩，因此模型中心应运而生。现有的模型中心通常认为基本的文本匹配足以搜索模型。然而，实际上，由于抽象不同且模型中心中的模型数量众多，用户不易查看模型描述和示例图像，从而选择最符合其需求的模型。因此，有必要明智地描述模型功能，以便未来的用户可以有效地搜索最适合其需求的模型。解决这个问题的努力仍然有限。在本文中，我们提出了条件生成模型识别（CGI），旨在提供一种有效的方法来使用用户提供的示例图像来识别最合适的模型，而不是要求用户手动查看带有示例图像的大量模型。为了解决这个问题，我们提出了基于提示的模型识别（PMI），它可以充分描述模型功能并精确地将需求与规范相匹配。为了评估 PMI 方法并促进相关研究，我们提供了一个包含 65 个模型和 9100 个识别任务的基准。大量实验和人工评估结果表明 PMI 是有效的。例如，当提供四张示例图像时，92% 的模型被正确识别，并且 FID 得分明显更高。

Title: INDIGO+: A Unified INN-Guided Probabilistic Diffusion Algorithm for Blind and Non-Blind Image Restoration

• Authors: Di You, Pier Luigi Dragotti
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.14014
• Pdf URL: https://arxiv.org/pdf/2501.14014
• Copy Paste: [[2501.14014]] INDIGO+: A Unified INN-Guided Probabilistic Diffusion Algorithm for Blind and Non-Blind Image Restoration(https://arxiv.org/abs/2501.14014)
• Keywords: restoration, generative
• Abstract: Generative diffusion models are becoming one of the most popular prior in image restoration (IR) tasks due to their remarkable ability to generate realistic natural images. Despite achieving satisfactory results, IR methods based on diffusion models present several limitations. First of all, most non-blind approaches require an analytical expression of the degradation model to guide the sampling process. Secondly, most existing blind approaches rely on families of pre-defined degradation models for training their deep networks. The above issues limit the flexibility of these approaches and so their ability to handle real-world degradation tasks. In this paper, we propose a novel INN-guided probabilistic diffusion algorithm for non-blind and blind image restoration, namely INDIGO and BlindINDIGO, which combines the merits of the perfect reconstruction property of invertible neural networks (INN) with the strong generative capabilities of pre-trained diffusion models. Specifically, we train the forward process of the INN to simulate an arbitrary degradation process and use the inverse to obtain an intermediate image that we use to guide the reverse diffusion sampling process through a gradient step. We also introduce an initialization strategy, to further improve the performance and inference speed of our algorithm. Experiments demonstrate that our algorithm obtains competitive results compared with recently leading methods both quantitatively and visually on synthetic and real-world low-quality images.
• 摘要：生成扩散模型因其生成逼真自然图像的出色能力而成为图像恢复 (IR) 任务中最受欢迎的先验之一。尽管取得了令人满意的结果，但基于扩散模型的 IR 方法仍存在一些局限性。首先，大多数非盲方法需要退化模型的解析表达式来指导采样过程。其次，大多数现有的盲方法依赖于预定义退化模型系列来训练其深度网络。上述问题限制了这些方法的灵活性，从而限制了它们处理现实世界退化任务的能力。在本文中，我们提出了一种用于非盲和盲图像恢复的新型 INN 引导概率扩散算法，即 INDIGO 和 BlindINDIGO，它结合了可逆神经网络 (INN) 的完美重建特性和预训练扩散模型的强大生成能力。具体而言，我们训练 INN 的正向过程来模拟任意退化过程，并使用逆向过程来获得中间图像，我们使用该中间图像通过梯度步骤来指导反向扩散采样过程。我们还引入了初始化策略，以进一步提高算法的性能和推理速度。实验表明，我们的算法在合成和现实世界的低质量图像上，无论是定量还是视觉上，都与最近领先的方法相比获得了具有竞争力的结果。

Title: LLM-guided Instance-level Image Manipulation with Diffusion U-Net Cross-Attention Maps

• Authors: Andrey Palaev, Adil Khan, Syed M. Ahsan Kazmi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14046
• Pdf URL: https://arxiv.org/pdf/2501.14046
• Copy Paste: [[2501.14046]] LLM-guided Instance-level Image Manipulation with Diffusion U-Net Cross-Attention Maps(https://arxiv.org/abs/2501.14046)
• Keywords: generative
• Abstract: The advancement of text-to-image synthesis has introduced powerful generative models capable of creating realistic images from textual prompts. However, precise control over image attributes remains challenging, especially at the instance level. While existing methods offer some control through fine-tuning or auxiliary information, they often face limitations in flexibility and accuracy. To address these challenges, we propose a pipeline leveraging Large Language Models (LLMs), open-vocabulary detectors, cross-attention maps and intermediate activations of diffusion U-Net for instance-level image manipulation. Our method detects objects mentioned in the prompt and present in the generated image, enabling precise manipulation without extensive training or input masks. By incorporating cross-attention maps, our approach ensures coherence in manipulated images while controlling object positions. Our method enables precise manipulations at the instance level without fine-tuning or auxiliary information such as masks or bounding boxes. Code is available at this https URL
• 摘要：文本到图像合成的进步引入了强大的生成模型，能够根据文本提示创建逼真的图像。然而，对图像属性的精确控制仍然具有挑战性，尤其是在实例级别。虽然现有方法通过微调或辅助信息提供了一些控制，但它们往往面临灵活性和准确性的限制。为了应对这些挑战，我们提出了一种利用大型语言模型 (LLM)、开放词汇检测器、交叉注意力图和扩散 U-Net 的中间激活进行实例级图像处理的管道。我们的方法检测提示中提到的和生成的图像中存在的对象，无需大量训练或输入掩码即可实现精确操作。通过结合交叉注意力图，我们的方法在控制对象位置的同时确保了被操纵图像的连贯性。我们的方法可以在实例级别实现精确操作，而无需微调或辅助信息（例如掩码或边界框）。代码可在此 https URL 上找到

Title: GraphRAG under Fire

• Authors: Jiacheng Liang, Yuhui Wang, Changjiang Li, Rongyi Zhu, Tanqiu Jiang, Neil Gong, Ting Wang
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.14050
• Pdf URL: https://arxiv.org/pdf/2501.14050
• Copy Paste: [[2501.14050]] GraphRAG under Fire(https://arxiv.org/abs/2501.14050)
• Keywords: generation
• Abstract: GraphRAG advances retrieval-augmented generation (RAG) by structuring external knowledge as multi-scale knowledge graphs, enabling language models to integrate both broad context and granular details in their reasoning. While GraphRAG has demonstrated success across domains, its security implications remain largely unexplored. To bridge this gap, this work examines GraphRAG's vulnerability to poisoning attacks, uncovering an intriguing security paradox: compared to conventional RAG, GraphRAG's graph-based indexing and retrieval enhance resilience against simple poisoning attacks; meanwhile, the same features also create new attack surfaces. We present GRAGPoison, a novel attack that exploits shared relations in the knowledge graph to craft poisoning text capable of compromising multiple queries simultaneously. GRAGPoison employs three key strategies: i) relation injection to introduce false knowledge, ii) relation enhancement to amplify poisoning influence, and iii) narrative generation to embed malicious content within coherent text. Empirical evaluation across diverse datasets and models shows that GRAGPoison substantially outperforms existing attacks in terms of effectiveness (up to 98% success rate) and scalability (using less than 68% poisoning text). We also explore potential defensive measures and their limitations, identifying promising directions for future research.
• 摘要：GraphRAG 通过将外部知识构建为多尺度知识图谱来推进检索增强生成 (RAG)，使语言模型能够在推理中整合广泛的背景和细节。虽然 GraphRAG 已在各个领域取得成功，但其安全影响仍未得到充分探索。为了弥补这一差距，这项工作研究了 GraphRAG 对毒药攻击的脆弱性，发现了一个有趣的安全悖论：与传统 RAG 相比，GraphRAG 基于图的索引和检索增强了对简单毒药攻击的抵御能力；同时，相同的功能也创造了新的攻击面。我们提出了 GRAGPoison，这是一种新颖的攻击，它利用知识图中的共享关系来制作能够同时危害多个查询的毒药文本。GRAGPoison 采用三种关键策略：i) 关系注入以引入虚假知识，ii) 关系增强以放大毒药影响，以及 iii) 叙述生成以将恶意内容嵌入连贯文本中。针对各种数据集和模型的实证评估表明，GRAGPoison 在有效性（成功率高达 98%）和可扩展性（使用少于 68% 的中毒文本）方面远远优于现有攻击。我们还探讨了潜在的防御措施及其局限性，确定了未来研究的有希望的方向。

Title: StreamingRAG: Real-time Contextual Retrieval and Generation Framework

• Authors: Murugan Sankaradas, Ravi K.Rajendran, Srimat T.Chakradhar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14101
• Pdf URL: https://arxiv.org/pdf/2501.14101
• Copy Paste: [[2501.14101]] StreamingRAG: Real-time Contextual Retrieval and Generation Framework(https://arxiv.org/abs/2501.14101)
• Keywords: generation
• Abstract: Extracting real-time insights from multi-modal data streams from various domains such as healthcare, intelligent transportation, and satellite remote sensing remains a challenge. High computational demands and limited knowledge scope restrict the applicability of Multi-Modal Large Language Models (MM-LLMs) on these data streams. Traditional Retrieval-Augmented Generation (RAG) systems address knowledge limitations of these models, but suffer from slow preprocessing, making them unsuitable for real-time analysis. We propose StreamingRAG, a novel RAG framework designed for streaming data. StreamingRAG constructs evolving knowledge graphs capturing scene-object-entity relationships in real-time. The knowledge graph achieves temporal-aware scene representations using MM-LLMs and enables timely responses for specific events or user queries. StreamingRAG addresses limitations in existing methods, achieving significant improvements in real-time analysis (5-6x faster throughput), contextual accuracy (through a temporal knowledge graph), and reduced resource consumption (using lightweight models by 2-3x).
• 摘要：从医疗保健、智能交通和卫星遥感等各个领域的多模态数据流中提取实时洞察仍然是一项挑战。高计算需求和有限的知识范围限制了多模态大型语言模型 (MM-LLM) 在这些数据流上的适用性。传统的检索增强生成 (RAG) 系统解决了这些模型的知识限制，但预处理速度慢，使其不适合实时分析。我们提出了 StreamingRAG，这是一个专为流数据设计的新型 RAG 框架。StreamingRAG 构建不断发展的知识图谱，实时捕获场景-对象-实体关系。知识图谱使用 MM-LLM 实现时间感知场景表示，并能够及时响应特定事件或用户查询。StreamingRAG 解决了现有方法的局限性，在实时分析（吞吐量提高 5-6 倍）、上下文准确性（通过时间知识图谱）和资源消耗减少（使用轻量级模型减少 2-3 倍）方面取得了显着的改进。

Title: An Extensive and Methodical Review of Smart Grids for Sustainable Energy Management-Addressing Challenges with AI, Renewable Energy Integration and Leading-edge Technologies

• Authors: Parag Biswas, Abdur Rashid, abdullah al masum, MD Abdullah Al Nasim, A.S.M Anas Ferdous, Kishor Datta Gupta, Angona Biswas
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.14143
• Pdf URL: https://arxiv.org/pdf/2501.14143
• Copy Paste: [[2501.14143]] An Extensive and Methodical Review of Smart Grids for Sustainable Energy Management-Addressing Challenges with AI, Renewable Energy Integration and Leading-edge Technologies(https://arxiv.org/abs/2501.14143)
• Keywords: generation
• Abstract: Energy management decreases energy expenditures and consumption while simultaneously increasing energy efficiency, reducing carbon emissions, and enhancing operational performance. Smart grids are a type of sophisticated energy infrastructure that increase the generation and distribution of electricity's sustainability, dependability, and efficiency by utilizing digital communication technologies. They combine a number of cutting-edge techniques and technology to improve energy resource management. A large amount of research study on the topic of smart grids for energy management has been completed in the last several years. The authors of the present study want to cover a number of topics, including smart grid benefits and components, technical developments, integrating renewable energy sources, using artificial intelligence and data analytics, cybersecurity, and privacy. Smart Grids for Energy Management are an innovative field of study aiming at tackling various difficulties and magnifying the efficiency, dependability, and sustainability of energy systems, including: 1) Renewable sources of power like solar and wind are intermittent and unpredictable 2) Defending smart grid system from various cyber-attacks 3) Incorporating an increasing number of electric vehicles into the system of power grid without overwhelming it. Additionally, it is proposed to use AI and data analytics for better performance on the grid, reliability, and energy management. It also looks into how AI and data analytics can be used to optimize grid performance, enhance reliability, and improve energy management. The authors will explore these significant challenges and ongoing research. Lastly, significant issues in this field are noted, and recommendations for further work are provided.
• 摘要：能源管理可降低能源支出和消耗，同时提高能源效率、减少碳排放和提高运营绩效。智能电网是一种复杂的能源基础设施，它利用数字通信技术提高电力生产和分配的可持续性、可靠性和效率。它们结合了许多尖端技术来改善能源资源管理。过去几年，已经完成了大量关于智能电网能源管理的研究。本研究的作者希望涵盖许多主题，包括智能电网的优势和组成部分、技术发展、整合可再生能源、使用人工智能和数据分析、网络安全和隐私。智能电网能源管理是一个创新的研究领域，旨在解决各种困难并提高能源系统的效率、可靠性和可持续性，包括：1) 太阳能和风能等可再生能源是间歇性的和不可预测的 2) 保护智能电网系统免受各种网络攻击 3) 将越来越多的电动汽车纳入电网系统而不会使其不堪重负。此外，还建议使用人工智能和数据分析来提高电网性能、可靠性和能源管理。它还研究了如何使用人工智能和数据分析来优化电网性能、提高可靠性和改善能源管理。作者将探讨这些重大挑战和正在进行的研究。最后，指出了该领域的重要问题，并提出了进一步工作的建议。

Title: Argos: Agentic Time-Series Anomaly Detection with Autonomous Rule Generation via Large Language Models

• Authors: Yile Gu, Yifan Xiong, Jonathan Mace, Yuting Jiang, Yigong Hu, Baris Kasikci, Peng Cheng
• Subjects: cs.LG, cs.DC, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.14170
• Pdf URL: https://arxiv.org/pdf/2501.14170
• Copy Paste: [[2501.14170]] Argos: Agentic Time-Series Anomaly Detection with Autonomous Rule Generation via Large Language Models(https://arxiv.org/abs/2501.14170)
• Keywords: generation
• Abstract: Observability in cloud infrastructure is critical for service providers, driving the widespread adoption of anomaly detection systems for monitoring metrics. However, existing systems often struggle to simultaneously achieve explainability, reproducibility, and autonomy, which are three indispensable properties for production use. We introduce Argos, an agentic system for detecting time-series anomalies in cloud infrastructure by leveraging large language models (LLMs). Argos proposes to use explainable and reproducible anomaly rules as intermediate representation and employs LLMs to autonomously generate such rules. The system will efficiently train error-free and accuracy-guaranteed anomaly rules through multiple collaborative agents and deploy the trained rules for low-cost online anomaly detection. Through evaluation results, we demonstrate that Argos outperforms state-of-the-art methods, increasing $F_1$ scores by up to $9.5\%$ and $28.3\%$ on public anomaly detection datasets and an internal dataset collected from Microsoft, respectively.
• 摘要：云基础设施中的可观察性对于服务提供商至关重要，推动了异常检测系统广泛用于监控指标。然而，现有系统通常难以同时实现可解释性、可重复性和自主性，这三个特性对于生产使用来说是必不可少的。我们介绍了 Argos，这是一个利用大型语言模型 (LLM) 检测云基础设施中时间序列异常的代理系统。Argos 建议使用可解释和可重复的异常规则作为中间表示，并使用 LLM 自主生成此类规则。该系统将通过多个协作代理有效地训练无错误和准确度保证的异常规则，并部署训练有素的规则以进行低成本的在线异常检测。通过评估结果，我们证明 Argos 优于最先进的方法，在公共异常检测数据集和从 Microsoft 收集的内部数据集上分别将 $F_1$ 分数提高了高达 $9.5\%$ 和 $28.3\%$。

Title: Dreamweaver: Learning Compositional World Representations from Pixels

• Authors: Junyeob Baek, Yi-Fu Wu, Gautam Singh, Sungjin Ahn
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14174
• Pdf URL: https://arxiv.org/pdf/2501.14174
• Copy Paste: [[2501.14174]] Dreamweaver: Learning Compositional World Representations from Pixels(https://arxiv.org/abs/2501.14174)
• Keywords: generation
• Abstract: Humans have an innate ability to decompose their perceptions of the world into objects and their attributes, such as colors, shapes, and movement patterns. This cognitive process enables us to imagine novel futures by recombining familiar concepts. However, replicating this ability in artificial intelligence systems has proven challenging, particularly when it comes to modeling videos into compositional concepts and generating unseen, recomposed futures without relying on auxiliary data, such as text, masks, or bounding boxes. In this paper, we propose Dreamweaver, a neural architecture designed to discover hierarchical and compositional representations from raw videos and generate compositional future simulations. Our approach leverages a novel Recurrent Block-Slot Unit (RBSU) to decompose videos into their constituent objects and attributes. In addition, Dreamweaver uses a multi-future-frame prediction objective to capture disentangled representations for dynamic concepts more effectively as well as static concepts. In experiments, we demonstrate our model outperforms current state-of-the-art baselines for world modeling when evaluated under the DCI framework across multiple datasets. Furthermore, we show how the modularized concept representations of our model enable compositional imagination, allowing the generation of novel videos by recombining attributes from different objects.
• 摘要：人类天生具有将他们对世界的感知分解为物体及其属性（例如颜色、形状和运动模式）的能力。这种认知过程使我们能够通过重新组合熟悉的概念来想象新颖的未来。然而，在人工智能系统中复制这种能力已被证明具有挑战性，特别是在将视频建模为构图概念并生成未见过的重新组合的未来而不依赖辅助数据（例如文本、蒙版或边界框）时。在本文中，我们提出了 Dreamweaver，这是一种神经架构，旨在从原始视频中发现层次结构和构图表示并生成构图未来模拟。我们的方法利用一种新颖的循环块槽单元 (RBSU) 将视频分解为其组成对象和属性。此外，Dreamweaver 使用多未来帧预测目标来更有效地捕获动态概念和静态概念的解开表示。在实验中，我们证明，当在多个数据集的 DCI 框架下进行评估时，我们的模型优于当前最先进的世界建模基线。此外，我们展示了我们的模型的模块化概念表示如何实现组合想象，从而通过重新组合来自不同对象的属性来生成新颖的视频。

Title: ENTER: Event Based Interpretable Reasoning for VideoQA

• Authors: Hammad Ayyubi, Junzhang Liu, Ali Asgarov, Zaber Ibn Abdul Hakim, Najibul Haque Sarker, Zhecan Wang, Chia-Wei Tang, Hani Alomari, Md. Atabuzzaman, Xudong Lin, Naveen Reddy Dyava, Shih-Fu Chang, Chris Thomas
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.14194
• Pdf URL: https://arxiv.org/pdf/2501.14194
• Copy Paste: [[2501.14194]] ENTER: Event Based Interpretable Reasoning for VideoQA(https://arxiv.org/abs/2501.14194)
• Keywords: generation
• Abstract: In this paper, we present ENTER, an interpretable Video Question Answering (VideoQA) system based on event graphs. Event graphs convert videos into graphical representations, where video events form the nodes and event-event relationships (temporal/causal/hierarchical) form the edges. This structured representation offers many benefits: 1) Interpretable VideoQA via generated code that parses event-graph; 2) Incorporation of contextual visual information in the reasoning process (code generation) via event graphs; 3) Robust VideoQA via Hierarchical Iterative Update of the event graphs. Existing interpretable VideoQA systems are often top-down, disregarding low-level visual information in the reasoning plan generation, and are brittle. While bottom-up approaches produce responses from visual data, they lack interpretability. Experimental results on NExT-QA, IntentQA, and EgoSchema demonstrate that not only does our method outperform existing top-down approaches while obtaining competitive performance against bottom-up approaches, but more importantly, offers superior interpretability and explainability in the reasoning process.
• 摘要：在本文中，我们介绍了 ENTER，这是一种基于事件图的可解释视频问答 (VideoQA) 系统。事件图将视频转换为图形表示，其中视频事件形成节点，事件-事件关系（时间/因果/层次）形成边缘。这种结构化表示具有许多优势：1) 通过解析事件图的生成代码实现可解释的 VideoQA；2) 通过事件图在推理过程（代码生成）中整合上下文视觉信息；3) 通过事件图的分层迭代更新实现稳健的 VideoQA。现有的可解释 VideoQA 系统通常是自上而下的，在推理计划生成中忽略了低级视觉信息，并且很脆弱。虽然自下而上的方法可以根据视觉数据产生响应，但它们缺乏可解释性。在 NExT-QA、IntentQA 和 EgoSchema 上的实验结果表明，我们的方法不仅优于现有的自上而下的方法，同时获得与自下而上的方法相比具有竞争力的性能，而且更重要的是，在推理过程中提供了卓越的可解释性和可解释性。

Title: VideoShield: Regulating Diffusion-based Video Generation Models via Watermarking

• Authors: Runyi Hu, Jie Zhang, Yiming Li, Jiwei Li, Qing Guo, Han Qiu, Tianwei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14195
• Pdf URL: https://arxiv.org/pdf/2501.14195
• Copy Paste: [[2501.14195]] VideoShield: Regulating Diffusion-based Video Generation Models via Watermarking(https://arxiv.org/abs/2501.14195)
• Keywords: generation
• Abstract: Artificial Intelligence Generated Content (AIGC) has advanced significantly, particularly with the development of video generation models such as text-to-video (T2V) models and image-to-video (I2V) models. However, like other AIGC types, video generation requires robust content control. A common approach is to embed watermarks, but most research has focused on images, with limited attention given to videos. Traditional methods, which embed watermarks frame-by-frame in a post-processing manner, often degrade video quality. In this paper, we propose VideoShield, a novel watermarking framework specifically designed for popular diffusion-based video generation models. Unlike post-processing methods, VideoShield embeds watermarks directly during video generation, eliminating the need for additional training. To ensure video integrity, we introduce a tamper localization feature that can detect changes both temporally (across frames) and spatially (within individual frames). Our method maps watermark bits to template bits, which are then used to generate watermarked noise during the denoising process. Using DDIM Inversion, we can reverse the video to its original watermarked noise, enabling straightforward watermark extraction. Additionally, template bits allow precise detection for potential temporal and spatial modification. Extensive experiments across various video models (both T2V and I2V models) demonstrate that our method effectively extracts watermarks and detects tamper without compromising video quality. Furthermore, we show that this approach is applicable to image generation models, enabling tamper detection in generated images as well. Codes and models are available at \href{this https URL}{this https URL}.
• 摘要：人工智能生成内容 (AIGC) 取得了长足进步，尤其是随着文本转视频 (T2V) 模型和图像转视频 (I2V) 模型等视频生成模型的发展。然而，与其他 AIGC 类型一样，视频生成需要强大的内容控制。一种常见的方法是嵌入水印，但大多数研究都集中在图像上，对视频的关注有限。传统方法以后期处理的方式逐帧嵌入水印，通常会降低视频质量。在本文中，我们提出了 VideoShield，这是一种专为流行的基于扩散的视频生成模型设计的新型水印框架。与后期处理方法不同，VideoShield 在视频生成过程中直接嵌入水印，无需额外训练。为了确保视频完整性，我们引入了篡改定位功能，可以检测时间（跨帧）和空间（单个帧内）的变化。我们的方法将水印位映射到模板位，然后在去噪过程中使用这些模板位生成带水印的噪声。使用 DDIM 反转，我们可以将视频反转为原始的水印噪声，从而实现直接的水印提取。此外，模板位可以精确检测潜在的时间和空间修改。对各种视频模型（T2V 和 I2V 模型）进行的大量实验表明，我们的方法可以有效提取水印并检测篡改，而不会影响视频质量。此外，我们表明这种方法适用于图像生成模型，也可以在生成的图像中检测篡改。代码和模型可在 \href{this https URL}{this https URL} 上找到。

Title: Dynamic Token Reduction during Generation for Vision Language Models

• Authors: Xiaoyu Liang, Chaofeng Guan, Jiaying Lu, Huiyao Chen, Huan Wang, Haoji Hu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.14204
• Pdf URL: https://arxiv.org/pdf/2501.14204
• Copy Paste: [[2501.14204]] Dynamic Token Reduction during Generation for Vision Language Models(https://arxiv.org/abs/2501.14204)
• Keywords: generation
• Abstract: Vision-Language Models (VLMs) have achieved notable success in multimodal tasks but face practical limitations due to the quadratic complexity of decoder attention mechanisms and autoregressive generation. Existing methods like FASTV and VTW have achieved notable results in reducing redundant visual tokens, but these approaches focus on pruning tokens in a single forward pass without systematically analyzing the redundancy of visual tokens throughout the entire generation process. In this paper, we introduce a dynamic pruning strategy tailored for VLMs, namedDynamic Rate (DyRate), which progressively adjusts the compression rate during generation. Our analysis of the distribution of attention reveals that the importance of visual tokens decreases throughout the generation process, inspiring us to adopt a more aggressive compression rate. By integrating a lightweight predictor based on attention distribution, our approach enables flexible adjustment of pruning rates based on the attention distribution. Our experimental results demonstrate that our method not only reduces computational demands but also maintains the quality of responses.
• 摘要：视觉语言模型 (VLM) 在多模态任务中取得了显著的成功，但由于解码器注意机制和自回归生成的二次复杂性，它面临着实际限制。现有的方法如 FASTV 和 VTW 在减少冗余视觉标记方面取得了显著的成果，但这些方法侧重于在单次前向传递中修剪标记，而没有系统地分析整个生成过程中视觉标记的冗余性。在本文中，我们介绍了一种专为 VLM 量身定制的动态修剪策略，称为动态速率 (DyRate)，它在生成过程中逐步调整压缩率。我们对注意力分布的分析表明，视觉标记的重要性在整个生成过程中不断降低，这启发我们采用更积极的压缩率。通过集成基于注意力分布的轻量级预测器，我们的方法能够根据注意力分布灵活调整修剪率。我们的实验结果表明，我们的方法不仅减少了计算需求，而且还保持了响应质量。

Title: TFG-Flow: Training-free Guidance in Multimodal Generative Flow

• Authors: Haowei Lin, Shanda Li, Haotian Ye, Yiming Yang, Stefano Ermon, Yitao Liang, Jianzhu Ma
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2501.14216
• Pdf URL: https://arxiv.org/pdf/2501.14216
• Copy Paste: [[2501.14216]] TFG-Flow: Training-free Guidance in Multimodal Generative Flow(https://arxiv.org/abs/2501.14216)
• Keywords: generation, generative
• Abstract: Given an unconditional generative model and a predictor for a target property (e.g., a classifier), the goal of training-free guidance is to generate samples with desirable target properties without additional training. As a highly efficient technique for steering generative models toward flexible outcomes, training-free guidance has gained increasing attention in diffusion models. However, existing methods only handle data in continuous spaces, while many scientific applications involve both continuous and discrete data (referred to as multimodality). Another emerging trend is the growing use of the simple and general flow matching framework in building generative foundation models, where guided generation remains under-explored. To address this, we introduce TFG-Flow, a novel training-free guidance method for multimodal generative flow. TFG-Flow addresses the curse-of-dimensionality while maintaining the property of unbiased sampling in guiding discrete variables. We validate TFG-Flow on four molecular design tasks and show that TFG-Flow has great potential in drug design by generating molecules with desired properties.
• 摘要：给定一个无条件生成模型和一个目标属性的预测器（例如分类器），无需训练指导的目标是生成具有理想目标属性的样本，而无需额外的训练。作为一种引导生成模型实现灵活结果的高效技术，无需训练指导在扩散模型中受到越来越多的关注。然而，现有方法仅处理连续空间中的数据，而许多科学应用涉及连续和离散数据（称为多模态）。另一个新兴趋势是，在构建生成基础模型时，简单而通用的流匹配框架的使用越来越多，而引导生成仍未得到充分探索。为了解决这个问题，我们引入了 TFG-Flow，一种用于多模态生成流的新型无需训练指导方法。TFG-Flow 解决了维数灾难问题，同时保持了引导离散变量的无偏采样特性。我们在四个分子设计任务上验证了 TFG-Flow，并表明 TFG-Flow 通过生成具有所需特性的分子在药物设计中具有巨大潜力。

Title: Detection and Classification of Acute Lymphoblastic Leukemia Utilizing Deep Transfer Learning

• Authors: Md. Abu Ahnaf Mollick, Md. Mahfujur Rahman, D.M. Asadujjaman, Abdullah Tamim, Nosin Anjum Dristi, Md. Takbir Hossen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.14228
• Pdf URL: https://arxiv.org/pdf/2501.14228
• Copy Paste: [[2501.14228]] Detection and Classification of Acute Lymphoblastic Leukemia Utilizing Deep Transfer Learning(https://arxiv.org/abs/2501.14228)
• Keywords: generation
• Abstract: A mutation in the DNA of a single cell that compromises its function initiates leukemia,leading to the overproduction of immature white blood cells that encroach upon the space required for the generation of healthy blood this http URL is treatable if identified in its initial stages. However,its diagnosis is both arduous and time consuming. This study proposes a novel approach for diagnosing leukemia across four stages Benign,Early,Pre,and Pro using deep learning this http URL employed two Convolutional Neural Network (CNN) models as MobileNetV2 with an altered head and a custom model. The custom model consists of multiple convolutional layers,each paired with corresponding max pooling this http URL utilized MobileNetV2 with ImageNet weights,adjusting the head to integrate the final this http URL dataset used is the publicly available "Acute Lymphoblastic Leukemia (ALL) Image Dataset", and we applied the Synthetic Minority Oversampling Technique (SMOTE) to augment and balance the training this http URL custom model achieved an accuracy of 98.6%, while MobileNetV2 attained a superior accuracy of 99.69%. The pretrained model showed promising results,indicating an increased likelihood of real-world application.
• 摘要：单细胞 DNA 突变会损害其功能，从而引发白血病，导致未成熟白细胞过量产生，侵占生成健康血液所需的空间。如果在早期阶段发现，此 http URL 是可以治疗的。然而，诊断白血病既困难又耗时。本研究提出了一种使用深度学习诊断四个阶段（良性、早期、前期和晚期）白血病的新方法。此 http URL 采用了两个卷积神经网络 (CNN) 模型，即头部经过修改的 MobileNetV2 和一个自定义模型。自定义模型由多个卷积层组成，每个卷积层都与相应的最大池化配对，此 http URL 使用具有 ImageNet 权重的 MobileNetV2，调整头部以整合最终使用的此 http URL 数据集是公开的“急性淋巴细胞白血病 (ALL) 图像数据集”，我们应用合成少数过采样技术 (SMOTE) 来增强和平衡训练此 http URL 自定义模型实现了 98.6% 的准确率，而 MobileNetV2 实现了 99.69% 的卓越准确率。预训练模型显示出令人鼓舞的结果，表明实际应用的可能性增加。

Title: A Data-driven Dynamic Temporal Correlation Modeling Framework for Renewable Energy Scenario Generation

• Authors: Xiaochong Dong, Yilin Liu, Xuemin Zhang, Shengwei Mei
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14233
• Pdf URL: https://arxiv.org/pdf/2501.14233
• Copy Paste: [[2501.14233]] A Data-driven Dynamic Temporal Correlation Modeling Framework for Renewable Energy Scenario Generation(https://arxiv.org/abs/2501.14233)
• Keywords: generation
• Abstract: Renewable energy power is influenced by the atmospheric system, which exhibits nonlinear and time-varying features. To address this, a dynamic temporal correlation modeling framework is proposed for renewable energy scenario generation. A novel decoupled mapping path is employed for joint probability distribution modeling, formulating regression tasks for both marginal distributions and the correlation structure using proper scoring rules to ensure the rationality of the modeling process. The scenario generation process is divided into two stages. Firstly, the dynamic correlation network models temporal correlations based on a dynamic covariance matrix, capturing the time-varying features of renewable energy while enhancing the interpretability of the black-box model. Secondly, the implicit quantile network models the marginal quantile function in a nonparametric, continuous manner, enabling scenario generation through marginal inverse sampling. Experimental results demonstrate that the proposed dynamic correlation quantile network outperforms state-of-the-art methods in quantifying uncertainty and capturing dynamic correlation for short-term renewable energy scenario generation.
• 摘要：可再生能源发电受大气系统的影响，具有非线性和时变性。针对这一问题，提出了一种用于可再生能源情景生成的动态时间相关性建模框架。采用一种新颖的解耦映射路径进行联合概率分布建模，使用适当的评分规则为边际分布和相关性结构制定回归任务，以确保建模过程的合理性。情景生成过程分为两个阶段。首先，动态相关网络基于动态协方差矩阵对时间相关性进行建模，捕捉可再生能源的时变特征，同时增强黑箱模型的可解释性。其次，隐式分位数网络以非参数、连续的方式对边际分位数函数进行建模，从而通过边际逆采样实现情景生成。实验结果表明，所提出的动态相关分位数网络在量化不确定性和捕捉短期可再生能源情景生成的动态相关性方面优于最先进的方法。

Title: TrajFlow: A Generative Framework for Occupancy Density Estimation Using Normalizing Flows

• Authors: Mitch Kosieradzki, Seongjin Choi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14266
• Pdf URL: https://arxiv.org/pdf/2501.14266
• Copy Paste: [[2501.14266]] TrajFlow: A Generative Framework for Occupancy Density Estimation Using Normalizing Flows(https://arxiv.org/abs/2501.14266)
• Keywords: generative
• Abstract: In transportation systems and autonomous vehicles, intelligent agents must understand the future motion of traffic participants to effectively plan motion trajectories. At the same time, the motion of traffic participants is inherently uncertain. In this paper, we propose TrajFlow, a generative framework for estimating the occupancy density of traffic participants. Our framework utilizes a causal encoder to extract semantically meaningful embeddings of the observed trajectory, as well as a normalizing flow to decode these embeddings and determine the most likely future location of traffic participants at some time point in the future. Our formulation differs from existing approaches because we model the marginal distribution of spatial locations instead of the joint distribution of unobserved trajectories. The advantages of a marginal formulation are numerous. First, we demonstrate that the marginal formulation produces higher accuracy on challenging trajectory forecasting benchmarks. Second, the marginal formulation allows for a fully continuous sampling of future locations. Finally, marginal densities are better suited for downstream tasks as they allow for the computation of per-agent motion trajectories and occupancy grids, the two most commonly used representations for motion forecasting. We present a novel architecture based entirely on neural differential equations as an implementation of this framework and provide ablations to demonstrate the advantages of a continuous implementation over a more traditional discrete neural network based approach. The code is available at this https URL .
• 摘要：在交通系统和自动驾驶汽车中，智能代理必须了解交通参与者的未来运动，才能有效地规划运动轨迹。同时，交通参与者的运动本质上是不确定的。在本文中，我们提出了 TrajFlow，这是一个用于估计交通参与者占用密度的生成框架。我们的框架利用因果编码器来提取观察到的轨迹的语义上有意义的嵌入，以及利用规范化流来解码这些嵌入并确定未来某个时间点交通参与者最可能的未来位置。我们的公式与现有方法不同，因为我们模拟的是空间位置的边际分布，而不是未观察轨迹的联合分布。边际公式的优点有很多。首先，我们证明边际公式在具有挑战性的轨迹预测基准上产生了更高的准确性。其次，边际公式允许对未来位置进行完全连续的采样。最后，边际密度更适合下游任务，因为它们允许计算每个代理的运动轨迹和占用网格，这是运动预测最常用的两种表示形式。我们提出了一种完全基于神经微分方程的新型架构作为该框架的实现，并提供消融来展示连续实现相对于更传统的基于离散神经网络的方法的优势。代码可在此 https URL 上找到。

Title: PAID: A Framework of Product-Centric Advertising Image Design

• Authors: Hongyu Chen, Min Zhou, Jing Jiang, Jiale Chen, Yang Lu, Bo Xiao, Tiezheng Ge, Bo Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14316
• Pdf URL: https://arxiv.org/pdf/2501.14316
• Copy Paste: [[2501.14316]] PAID: A Framework of Product-Centric Advertising Image Design(https://arxiv.org/abs/2501.14316)
• Keywords: generation
• Abstract: In E-commerce platforms, a full advertising image is composed of a background image and marketing taglines. Automatic ad image design reduces human costs and plays a crucial role. For the convenience of users, a novel automatic framework named Product-Centric Advertising Image Design (PAID) is proposed in this work. PAID takes the product foreground image, required taglines, and target size as input and creates an ad image automatically. PAID consists of four sequential stages: prompt generation, layout generation, background image generation, and graphics rendering. Different expert models are trained to conduct these sub-tasks. A visual language model (VLM) based prompt generation model is leveraged to produce a product-matching background prompt. The layout generation model jointly predicts text and image layout according to the background prompt, product, and taglines to achieve the best harmony. An SDXL-based layout-controlled inpainting model is trained to generate an aesthetic background image. Previous ad image design methods take a background image as input and then predict the layout of taglines, which limits the spatial layout due to fixed image content. Innovatively, our PAID adjusts the stages to produce an unrestricted layout. To complete the PAID framework, we created two high-quality datasets, PITA and PIL. Extensive experimental results show that PAID creates more visually pleasing advertising images than previous methods.
• 摘要：在电子商务平台中，完整的广告图像由背景图像和营销标语组成。自动广告图像设计减少了人力成本并发挥了至关重要的作用。为了方便用户，本文提出了一种名为以产品为中心的广告图像设计（PAID）的新型自动框架。PAID 将产品前景图像、所需标语和目标尺寸作为输入，并自动创建广告图像。PAID 包含四个连续的阶段：提示生成、布局生成、背景图像生成和图形渲染。不同的专家模型经过训练来执行这些子任务。利用基于视觉语言模型 (VLM) 的提示生成模型来生成与产品匹配的背景提示。布局生成模型根据背景提示、产品和标语联合预测文本和图像布局，以达到最佳和谐。训练基于 SDXL 的布局控制修复模型以生成美观的背景图像。以前的广告图像设计方法将背景图像作为输入，然后预测标语的布局，由于图像内容固定，这限制了空间布局。我们的 PAID 创新性地调整了各个阶段，以生成不受限制的布局。为了完善 PAID 框架，我们创建了两个高质量的数据集，即 PITA 和 PIL。大量实验结果表明，与以前的方法相比，PAID 可以创建更具视觉吸引力的广告图像。

Title: Nautilus: Locality-aware Autoencoder for Scalable Mesh Generation

• Authors: Yuxuan Wang, Xuanyu Yi, Haohan Weng, Qingshan Xu, Xiaokang Wei, Xianghui Yang, Chunchao Guo, Long Chen, Hanwang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14317
• Pdf URL: https://arxiv.org/pdf/2501.14317
• Copy Paste: [[2501.14317]] Nautilus: Locality-aware Autoencoder for Scalable Mesh Generation(https://arxiv.org/abs/2501.14317)
• Keywords: generation
• Abstract: Triangle meshes are fundamental to 3D applications, enabling efficient modification and rasterization while maintaining compatibility with standard rendering pipelines. However, current automatic mesh generation methods typically rely on intermediate representations that lack the continuous surface quality inherent to meshes. Converting these representations into meshes produces dense, suboptimal outputs. Although recent autoregressive approaches demonstrate promise in directly modeling mesh vertices and faces, they are constrained by the limitation in face count, scalability, and structural fidelity. To address these challenges, we propose Nautilus, a locality-aware autoencoder for artist-like mesh generation that leverages the local properties of manifold meshes to achieve structural fidelity and efficient representation. Our approach introduces a novel tokenization algorithm that preserves face proximity relationships and compresses sequence length through locally shared vertices and edges, enabling the generation of meshes with an unprecedented scale of up to 5,000 faces. Furthermore, we develop a Dual-stream Point Conditioner that provides multi-scale geometric guidance, ensuring global consistency and local structural fidelity by capturing fine-grained geometric features. Extensive experiments demonstrate that Nautilus significantly outperforms state-of-the-art methods in both fidelity and scalability.
• 摘要：三角网格是 3D 应用的基础，可实现高效修改和光栅化，同时保持与标准渲染管道的兼容性。然而，当前的自动网格生成方法通常依赖于中间表示，而这些表示缺乏网格固有的连续表面质量。将这些表示转换为网格会产生密集的次优输出。尽管最近的自回归方法在直接建模网格顶点和面方面表现出前景，但它们受到面数、可扩展性和结构保真度的限制。为了应对这些挑战，我们提出了 Nautilus，这是一种局部感知自动编码器，用于艺术家般的网格生成，它利用流形网格的局部属性来实现结构保真度和高效表示。我们的方法引入了一种新颖的标记化算法，该算法保留了面邻近关系并通过本地共享的顶点和边压缩序列长度，从而能够生成具有多达 5,000 个面的空前规模的网格。此外，我们开发了一种双流点调节器，可提供多尺度几何引导，通过捕获细粒度几何特征来确保全局一致性和局部结构保真度。大量实验表明，Nautilus 在保真度和可扩展性方面均明显优于最先进的方法。

Title: Scalable Benchmarking and Robust Learning for Noise-Free Ego-Motion and 3D Reconstruction from Noisy Video

• Authors: Xiaohao Xu, Tianyi Zhang, Shibo Zhao, Xiang Li, Sibo Wang, Yongqi Chen, Ye Li, Bhiksha Raj, Matthew Johnson-Roberson, Sebastian Scherer, Xiaonan Huang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.14319
• Pdf URL: https://arxiv.org/pdf/2501.14319
• Copy Paste: [[2501.14319]] Scalable Benchmarking and Robust Learning for Noise-Free Ego-Motion and 3D Reconstruction from Noisy Video(https://arxiv.org/abs/2501.14319)
• Keywords: restoration, generation
• Abstract: We aim to redefine robust ego-motion estimation and photorealistic 3D reconstruction by addressing a critical limitation: the reliance on noise-free data in existing models. While such sanitized conditions simplify evaluation, they fail to capture the unpredictable, noisy complexities of real-world environments. Dynamic motion, sensor imperfections, and synchronization perturbations lead to sharp performance declines when these models are deployed in practice, revealing an urgent need for frameworks that embrace and excel under real-world noise. To bridge this gap, we tackle three core challenges: scalable data generation, comprehensive benchmarking, and model robustness enhancement. First, we introduce a scalable noisy data synthesis pipeline that generates diverse datasets simulating complex motion, sensor imperfections, and synchronization errors. Second, we leverage this pipeline to create Robust-Ego3D, a benchmark rigorously designed to expose noise-induced performance degradation, highlighting the limitations of current learning-based methods in ego-motion accuracy and 3D reconstruction quality. Third, we propose Correspondence-guided Gaussian Splatting (CorrGS), a novel test-time adaptation method that progressively refines an internal clean 3D representation by aligning noisy observations with rendered RGB-D frames from clean 3D map, enhancing geometric alignment and appearance restoration through visual correspondence. Extensive experiments on synthetic and real-world data demonstrate that CorrGS consistently outperforms prior state-of-the-art methods, particularly in scenarios involving rapid motion and dynamic illumination.
• 摘要：我们旨在通过解决一个关键限制来重新定义稳健的自我运动估计和逼真的 3D 重建：现有模型对无噪声数据的依赖。虽然这种净化条件简化了评估，但它们无法捕捉到现实世界环境中不可预测的嘈杂复杂性。动态运动、传感器缺陷和同步扰动会导致这些模型在实践中部署时性能急剧下降，这表明迫切需要能够适应现实世界噪声并在其下表现出色的框架。为了弥补这一差距，我们解决了三个核心挑战：可扩展的数据生成、全面的基准测试和模型稳健性增强。首先，我们引入了一个可扩展的噪声数据合成管道，该管道生成模拟复杂运动、传感器缺陷和同步误差的各种数据集。其次，我们利用这个管道创建了 Robust-Ego3D，这是一个严格设计的基准，旨在揭示噪声引起的性能下降，突出了当前基于学习的方法在自我运动准确性和 3D 重建质量方面的局限性。第三，我们提出了对应引导高斯分层 (CorrGS)，这是一种新颖的测试时自适应方法，通过将嘈杂的观测值与干净的 3D 地图中渲染的 RGB-D 帧对齐，逐步完善内部干净的 3D 表示，通过视觉对应增强几何对齐和外观恢复。对合成数据和真实世界数据的大量实验表明，CorrGS 始终优于以前的最先进方法，特别是在涉及快速运动和动态照明的场景中。

Title: Distinguishing Parkinson's Patients Using Voice-Based Feature Extraction and Classification

• Authors: Burak Çelik, Ayhan Akbal
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14390
• Pdf URL: https://arxiv.org/pdf/2501.14390
• Copy Paste: [[2501.14390]] Distinguishing Parkinson's Patients Using Voice-Based Feature Extraction and Classification(https://arxiv.org/abs/2501.14390)
• Keywords: generative
• Abstract: Parkinson's disease (PD) is a progressive neurodegenerative disorder that impacts motor functions and speech characteristics This study focuses on differentiating individuals with Parkinson's disease from healthy controls through the extraction and classification of speech features. Patients were further divided into 2 groups. Med On represents the patient with medication, while Med Off represents the patient without medication. The dataset consisted of patients and healthy individuals who read a predefined text using the H1N Zoom microphone in a suitable recording environment at Fırat University Neurology Department. Speech recordings from PD patients and healthy controls were analyzed, and 19 key features were extracted, including jitter, luminance, zero-crossing rate (ZCR), root mean square (RMS) energy, entropy, skewness, and this http URL features were visualized in graphs and statistically evaluated to identify distinctive patterns in PD patients. Using MATLAB's Classification Learner toolbox, several machine learning classification algorithm models were applied to classify groups and significant accuracy rates were achieved. The accuracy of our 3-layer artificial neural network architecture was also compared with classical machine learning algorithms. This study highlights the potential of noninvasive voice analysis combined with machine learning for early detection and monitoring of PD patients. Future research can improve diagnostic accuracy by optimizing feature selection and exploring advanced classification techniques.
• 摘要：帕金森病 (PD) 是一种影响运动功能和言语特征的进行性神经退行性疾病。本研究重点是通过提取和分类言语特征来区分帕金森病患者和健康对照者。患者进一步分为两组。Med On 代表服用药物的患者，而 Med Off 代表未服用药物的患者。该数据集由患者和健康个体组成，他们在菲拉特大学神经病学系的合适录音环境中使用 H1N Zoom 麦克风阅读预定义文本。分析了 PD 患者和健康对照者的语音记录，提取了 19 个关键特征，包括抖动、亮度、零交叉率 (ZCR)、均方根 (RMS) 能量、熵、偏度，并将此 http URL 特征可视化为图形并进行统计评估，以识别 PD 患者的独特模式。使用 MATLAB 的分类学习器工具箱，应用了几种机器学习分类算法模型对组进行分类，并获得了显着的准确率。我们的 3 层人工神经网络架构的准确性也与经典机器学习算法进行了比较。这项研究强调了非侵入性语音分析与机器学习相结合对 PD 患者进行早期检测和监测的潜力。未来的研究可以通过优化特征选择和探索先进的分类技术来提高诊断准确性。

Title: CENTS: Generating synthetic electricity consumption time series for rare and unseen scenarios

• Authors: Michael Fuest, Alfredo Cuesta, Kalyan Veeramachaneni
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14426
• Pdf URL: https://arxiv.org/pdf/2501.14426
• Copy Paste: [[2501.14426]] CENTS: Generating synthetic electricity consumption time series for rare and unseen scenarios(https://arxiv.org/abs/2501.14426)
• Keywords: generation, generative
• Abstract: Recent breakthroughs in large-scale generative modeling have demonstrated the potential of foundation models in domains such as natural language, computer vision, and protein structure prediction. However, their application in the energy and smart grid sector remains limited due to the scarcity and heterogeneity of high-quality data. In this work, we propose a method for creating high-fidelity electricity consumption time series data for rare and unseen context variables (e.g. location, building type, photovoltaics). Our approach, Context Encoding and Normalizing Time Series Generation, or CENTS, includes three key innovations: (i) A context normalization approach that enables inverse transformation for time series context variables unseen during training, (ii) a novel context encoder to condition any state-of-the-art time-series generator on arbitrary numbers and combinations of context variables, (iii) a framework for training this context encoder jointly with a time-series generator using an auxiliary context classification loss designed to increase expressivity of context embeddings and improve model performance. We further provide a comprehensive overview of different evaluation metrics for generative time series models. Our results highlight the efficacy of the proposed method in generating realistic household-level electricity consumption data, paving the way for training larger foundation models in the energy domain on synthetic as well as real-world data.
• 摘要：大规模生成模型的最新突破证明了基础模型在自然语言、计算机视觉和蛋白质结构预测等领域的潜力。然而，由于高质量数据的稀缺性和异质性，它们在能源和智能电网领域的应用仍然有限。在这项工作中，我们提出了一种为罕见和看不见的上下文变量（例如位置、建筑类型、光伏）创建高保真电力消耗时间序列数据的方法。我们的方法，上下文编码和规范化时间序列生成，或 CENTS，包括三个关键创新：（i）一种上下文规范化方法，可以对训练期间看不见的时间序列上下文变量进行逆变换，（ii）一种新颖的上下文编码器，用于在任意数量和上下文变量组合上调节任何最先进的时间序列生成器，（iii）一个框架，用于使用辅助上下文分类损失与时间序列生成器联合训练此上下文编码器，旨在提高上下文嵌入的表达能力并提高模型性能。我们进一步全面概述了生成时间序列模型的不同评估指标。我们的研究结果强调了所提出方法在生成真实的家庭级电力消耗数据方面的有效性，为在合成数据和现实世界数据上训练能源领域的更大的基础模型铺平了道路。

Title: Pesti-Gen: Unleashing a Generative Molecule Approach for Toxicity Aware Pesticide Design

• Authors: Taehan Kim, Wonduk Seo
• Subjects: cs.LG, cs.AI, q-bio.BM, q-bio.MN
• Abstract URL: https://arxiv.org/abs/2501.14469
• Pdf URL: https://arxiv.org/pdf/2501.14469
• Copy Paste: [[2501.14469]] Pesti-Gen: Unleashing a Generative Molecule Approach for Toxicity Aware Pesticide Design(https://arxiv.org/abs/2501.14469)
• Keywords: generative
• Abstract: Global climate change has reduced crop resilience and pesticide efficacy, making reliance on synthetic pesticides inevitable, even though their widespread use poses significant health and environmental risks. While these pesticides remain a key tool in pest management, previous machine-learning applications in pesticide and agriculture have focused on classification or regression, leaving the fundamental challenge of generating new molecular structures or designing novel candidates unaddressed. In this paper, we propose Pesti-Gen, a novel generative model based on variational auto-encoders, designed to create pesticide candidates with optimized properties for the first time. Specifically, Pesti-Gen leverages a two-stage learning process: an initial pre-training phase that captures a generalized chemical structure representation, followed by a fine-tuning stage that incorporates toxicity-specific information. The model simultaneously optimizes over multiple toxicity metrics, such as (1) livestock toxicity and (2) aqua toxicity to generate environmentally friendly pesticide candidates. Notably, Pesti-Gen achieves approximately 68\% structural validity in generating new molecular structures, demonstrating the model's effectiveness in producing optimized and feasible pesticide candidates, thereby providing a new way for safer and more sustainable pest management solutions.
• 摘要：全球气候变化降低了农作物的适应力和农药的效力，使人们不可避免地依赖合成农药，尽管它们的广泛使用会带来重大的健康和环境风险。虽然这些农药仍然是害虫管理的关键工具，但以前在农药和农业领域的机器学习应用主要集中在分类或回归上，而生成新分子结构或设计新候选物的根本挑战却没有得到解决。在本文中，我们提出了 Pesti-Gen，这是一种基于变分自动编码器的新型生成模型，旨在首次创建具有优化特性的农药候选物。具体来说，Pesti-Gen 利用了一个两阶段的学习过程：初始预训练阶段捕获广义的化学结构表示，然后是微调阶段结合毒性特定信息。该模型同时优化多个毒性指标，例如 (1) 牲畜毒性和 (2) 水生毒性，以生成对环境友好的农药候选物。值得注意的是，Pesti-Gen 在生成新的分子结构时实现了约 68% 的结构有效性，证明了该模型在产生优化和可行的农药候选物方面的有效性，从而为更安全、更可持续的害虫管理解决方案提供了新途径。

Title: Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion 2.*

• Authors: Ludovica Schaerf, Andrea Alfarano, Fabrizio Silvestri, Leonardo Impett
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14524
• Pdf URL: https://arxiv.org/pdf/2501.14524
• Copy Paste: [[2501.14524]] Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion 2.*(https://arxiv.org/abs/2501.14524)
• Keywords: generation
• Abstract: Despite significant recent advances in image generation with diffusion models, their internal latent representations remain poorly understood. Existing works focus on the bottleneck layer (h-space) of Stable Diffusion's U-Net or leverage the cross-attention, self-attention, or decoding layers. Our model, SkipInject takes advantage of U-Net's skip connections. We conduct thorough analyses on the role of the skip connections and find that the residual connections passed by the third encoder block carry most of the spatial information of the reconstructed image, splitting the content from the style. We show that injecting the representations from this block can be used for text-based editing, precise modifications, and style transfer. We compare our methods state-of-the-art style transfer and image editing methods and demonstrate that our method obtains the best content alignment and optimal structural preservation tradeoff.
• 摘要：尽管最近在使用扩散模型生成图像方面取得了重大进展，但对其内部潜在表示仍然了解甚少。现有研究主要关注稳定扩散的 U-Net 的瓶颈层（h 空间）或利用交叉注意、自注意或解码层。我们的模型 SkipInject 利用了 U-Net 的跳过连接。我们对跳过连接的作用进行了彻底的分析，发现第三个编码器块传递的残差连接承载了重建图像的大部分空间信息，将内容与风格分开。我们表明，从该块注入表示可用于基于文本的编辑、精确修改和风格转换。我们将我们的方法与最先进的风格转换和图像编辑方法进行了比较，并证明我们的方法获得了最佳的内容对齐和最佳的结构保留权衡。

Title: Towards Scalable Topological Regularizers

• Authors: Hiu-Tung Wong, Darrick Lee, Hong Yan
• Subjects: cs.LG, math.AT
• Abstract URL: https://arxiv.org/abs/2501.14641
• Pdf URL: https://arxiv.org/pdf/2501.14641
• Copy Paste: [[2501.14641]] Towards Scalable Topological Regularizers(https://arxiv.org/abs/2501.14641)
• Keywords: generation, generative
• Abstract: Latent space matching, which consists of matching distributions of features in latent space, is a crucial component for tasks such as adversarial attacks and defenses, domain adaptation, and generative modelling. Metrics for probability measures, such as Wasserstein and maximum mean discrepancy, are commonly used to quantify the differences between such distributions. However, these are often costly to compute, or do not appropriately take the geometric and topological features of the distributions into consideration. Persistent homology is a tool from topological data analysis which quantifies the multi-scale topological structure of point clouds, and has recently been used as a topological regularizer in learning tasks. However, computation costs preclude larger scale computations, and discontinuities in the gradient lead to unstable training behavior such as in adversarial tasks. We propose the use of principal persistence measures, based on computing the persistent homology of a large number of small subsamples, as a topological regularizer. We provide a parallelized GPU implementation of this regularizer, and prove that gradients are continuous for smooth densities. Furthermore, we demonstrate the efficacy of this regularizer on shape matching, image generation, and semi-supervised learning tasks, opening the door towards a scalable regularizer for topological features.
• 摘要：潜在空间匹配由匹配潜在空间中的特征分布组成，是对抗性攻击和防御、领域自适应和生成建模等任务的关键组成部分。概率度量的指标（例如 Wasserstein 和最大均值差异）通常用于量化此类分布之间的差异。然而，这些指标通常计算成本高昂，或者没有适当地考虑分布的几何和拓扑特征。持久同源性是拓扑数据分析中的一种工具，它量化点云的多尺度拓扑结构，最近被用作学习任务中的拓扑正则化器。然而，计算成本阻碍了更大规模的计算，而梯度的不连续性会导致对抗性任务等不稳定的训练行为。我们建议使用基于计算大量小子样本的持久同源性的主要持久性度量作为拓扑正则化器。我们提供了此正则化的并行 GPU 实现，并证明梯度对于平滑密度是连续的。此外，我们证明了该正则化器在形状匹配、图像生成和半监督学习任务中的有效性，为拓扑特征的可扩展正则化器打开了大门。

Title: SyncAnimation: A Real-Time End-to-End Framework for Audio-Driven Human Pose and Talking Head Animation

• Authors: Yujian Liu, Shidang Xu, Jing Guo, Dingbin Wang, Zairan Wang, Xianfeng Tan, Xiaoli Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14646
• Pdf URL: https://arxiv.org/pdf/2501.14646
• Copy Paste: [[2501.14646]] SyncAnimation: A Real-Time End-to-End Framework for Audio-Driven Human Pose and Talking Head Animation(https://arxiv.org/abs/2501.14646)
• Keywords: generation
• Abstract: Generating talking avatar driven by audio remains a significant challenge. Existing methods typically require high computational costs and often lack sufficient facial detail and realism, making them unsuitable for applications that demand high real-time performance and visual quality. Additionally, while some methods can synchronize lip movement, they still face issues with consistency between facial expressions and upper body movement, particularly during silent periods. In this paper, we introduce SyncAnimation, the first NeRF-based method that achieves audio-driven, stable, and real-time generation of speaking avatar by combining generalized audio-to-pose matching and audio-to-expression synchronization. By integrating AudioPose Syncer and AudioEmotion Syncer, SyncAnimation achieves high-precision poses and expression generation, progressively producing audio-synchronized upper body, head, and lip shapes. Furthermore, the High-Synchronization Human Renderer ensures seamless integration of the head and upper body, and achieves audio-sync lip. The project page can be found at this https URL
• 摘要：生成由音频驱动的会说话的虚拟形象仍然是一项重大挑战。现有的方法通常需要很高的计算成本，而且往往缺乏足够的面部细节和真实感，因此不适合对实时性能和视觉质量有较高要求的应用。此外，虽然有些方法可以同步嘴唇运动，但它们仍然面临面部表情和上半身运动一致性的问题，尤其是在安静时期。在本文中，我们介绍了 SyncAnimation，这是第一种基于 NeRF 的方法，它通过结合广义的音频到姿势匹配和音频到表情同步，实现了音频驱动、稳定和实时的会说话的虚拟形象生成。通过集成 AudioPose Syncer 和 AudioEmotion Syncer，SyncAnimation 实现了高精度的姿势和表情生成，逐步产生音频同步的上半身、头部和唇形。此外，高同步人体渲染器确保头部和上半身的无缝集成，并实现音频同步唇形。项目页面可以在这个 https URL 找到

Title: HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation

• Authors: Xin Zhou, Dingkang Liang, Sifan Tu, Xiwu Chen, Yikang Ding, Dingyuan Zhang, Feiyang Tan, Hengshuang Zhao, Xiang Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14729
• Pdf URL: https://arxiv.org/pdf/2501.14729
• Copy Paste: [[2501.14729]] HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation(https://arxiv.org/abs/2501.14729)
• Keywords: generation
• Abstract: Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which involves interpreting and reasoning about the driving environment. In this paper, we present a unified Driving World Model named HERMES. We seamlessly integrate 3D scene understanding and future scene evolution (generation) through a unified framework in driving scenarios. Specifically, HERMES leverages a Bird's-Eye View (BEV) representation to consolidate multi-view spatial information while preserving geometric relationships and interactions. We also introduce world queries, which incorporate world knowledge into BEV features via causal attention in the Large Language Model (LLM), enabling contextual enrichment for understanding and generation tasks. We conduct comprehensive studies on nuScenes and OmniDrive-nuScenes datasets to validate the effectiveness of our method. HERMES achieves state-of-the-art performance, reducing generation error by 32.4% and improving understanding metrics such as CIDEr by 8.0%. The model and code will be publicly released at this https URL.
• 摘要：驾驶世界模型 (DWM) 已成为自动驾驶的关键，因为它能够预测未来场景。然而，现有的 DWM 仅限于场景生成，无法结合场景理解，这涉及对驾驶环境的解释和推理。在本文中，我们提出了一个名为 HERMES 的统一驾驶世界模型。我们通过驾驶场景中的统一框架无缝集成 3D 场景理解和未来场景演变（生成）。具体来说，HERMES 利用鸟瞰图 (BEV) 表示来整合多视图空间信息，同时保留几何关系和交互。我们还引入了世界查询，它通过大型语言模型 (LLM) 中的因果注意将世界知识整合到 BEV 特征中，从而为理解和生成任务提供上下文丰富。我们对 nuScenes 和 OmniDrive-nuScenes 数据集进行了全面研究，以验证我们方法的有效性。HERMES 实现了最先进的性能，将生成误差降低了 32.4%，并将 CIDEr 等理解指标提高了 8.0%。该模型和代码将在此https URL上公开发布。