2025-06-26

Title: Distillation-Enabled Knowledge Alignment for Generative Semantic Communications in AIGC Provisioning Tasks

• Authors: Jingzhi Hu, Geoffrey Ye Li
• Subjects: cs.LG, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2506.19893
• Pdf URL: https://arxiv.org/pdf/2506.19893
• Copy Paste: [[2506.19893]] Distillation-Enabled Knowledge Alignment for Generative Semantic Communications in AIGC Provisioning Tasks(https://arxiv.org/abs/2506.19893)
• Keywords: generation, generative
• Abstract: Due to the surging amount of AI-generated content (AIGC), its provisioning to edges and mobile users from the cloud incurs substantial traffic on networks. Generative semantic communication (GSC) offers a promising solution by transmitting highly compact information, i.e., prompt text and latent representations, instead of high-dimensional AIGC data. However, GSC relies on the alignment between the knowledge in the cloud generative AI (GAI) and that possessed by the edges and users, and between the knowledge for wireless transmission and that of actual channels, which remains challenging. In this paper, we propose DeKA-g, a distillation-enabled knowledge alignment algorithm for GSC systems. The core idea is to distill the generation knowledge from the cloud-GAI into low-rank matrices, which can be incorporated by the edge and used to adapt the transmission knowledge to diverse wireless channel conditions. DeKA-g comprises two novel methods: metaword-aided knowledge distillation (MAKD) and variable-rate grouped SNR adaptation (VGSA). For MAKD, an optimized metaword is employed to enhance the efficiency of knowledge distillation, while VGSA enables efficient adaptation to diverse compression rates and SNR ranges. From simulation results, DeKA-g improves the alignment between the edge-generated images and the cloud-generated ones by 44%. Moreover, it adapts to compression rates with 116% higher efficiency than the baseline and enhances the performance in low-SNR conditions by 28%.
• 摘要：由于AI生成的内容（AIGC）的幅度飙升，因此云中的边缘和移动用户的配置会导致网络上的大量流量。生成语义通信（GSC）通过传输高度紧凑的信息，即及时的文本和潜在表示，而不是高维AIGC数据，提供了一个有希望的解决方案。但是，GSC依赖于云生成AI（GAI）中的知识与边缘和用户所拥有的知识之间的一致性，无线传输的知识与实际渠道的知识之间的一致性，这仍然具有挑战性。在本文中，我们提出了DEKA-G，这是一种启用GSC系统的蒸馏知识一致性算法。核心思想是将生成知识从云-GAI提炼为低级矩阵，这些矩阵可以通过边缘融合，并用来使传输知识适应各种无线通道条件。 DEKA-G包括两种新方法：元多辅助知识蒸馏（MAKD）和可变速率分组的SNR适应（VGSA）。对于MAKD而言，采用了优化的Metaword来提高知识蒸馏的效率，而VGSA可以有效适应各种压缩率和SNR范围。从仿真结果中，DEKA-G可以提高边缘生成图像和云生成的图像之间的比对，提高44％。此外，它适应了比基线高116％的压缩率，并在低SNR条件下的性能提高了28％。

Title: Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture

• Authors: Shuchen Xue, Tianyu Xie, Tianyang Hu, Zijin Feng, Jiacheng Sun, Kenji Kawaguchi, Zhenguo Li, Zhi-Ming Ma
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2506.19935
• Pdf URL: https://arxiv.org/pdf/2506.19935
• Copy Paste: [[2506.19935]] Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture(https://arxiv.org/abs/2506.19935)
• Keywords: generation
• Abstract: Large language models (LLMs) predominantly use autoregressive (AR) approaches, but masked diffusion models (MDMs) are emerging as viable alternatives. A key challenge in comparing AR and MDM paradigms is their typical architectural difference: AR models are often decoder-only, while MDMs have largely been encoder-only. This practice of changing both the modeling paradigm and architecture simultaneously makes direct comparisons unfair, as it's hard to distinguish whether observed differences stem from the paradigm itself or the architectural shift. This research evaluates MDMs within a decoder-only framework to: (1) equitably compare MDM (as Any-Order AR, or AO-AR) and standard AR paradigms. Our investigation suggests that the standard AO-AR objective, which averages over all token permutations, may benefit from refinement, as many permutations appear less informative compared to the language's inherent left-to-right structure. (2) Investigate architectural influences (decoder-only vs. encoder-only) within MDMs. We demonstrate that while encoder-only MDMs model a simpler conditional probability space, decoder-only MDMs can achieve dramatic generation speedups ($\sim25\times$) and comparable perplexity with temperature annealing despite modeling a vastly larger space, highlighting key trade-offs. This work thus decouples core paradigm differences from architectural influences, offering insights for future model design. Code is available at this https URL.
• 摘要：大型语言模型（LLMS）主要采用自回旋（AR）方法，但是蒙版扩散模型（MDMS）正在成为可行的替代方案。比较AR和MDM范式的关键挑战是它们的典型架构差异：AR模型通常仅是解码器的，而MDM在很大程度上是仅编码的。这种改变建模范式和体系结构同时进行的实践使直接比较不公平，因为很难区分观察到的差异是否来自范式本身还是建筑转移。这项研究将仅解码器框架内的MDM评估为：（1）公平地比较MDM（作为任何阶AR或AO-AR）和标准AR范式。我们的调查表明，标准的AO-AR目标（平均所有令牌排列）可能会受益于改进，因为与该语言固有的左右右结构相比，许多排列似乎不那么信息。 （2）研究MDMS内的建筑影响（仅解码与仅编码）。我们证明，尽管仅编码的MDMS模型一个更简单的条件概率空间，但仅解码器的MDMS可以实现戏剧性的生成加速度（$ \ sim25 \ times $），并且尽管建模了大大较大的空间，但仍可以通过温度退火来实现可比较的困惑。因此，这项工作将核心范式差异与建筑影响相融合，从而为未来的模型设计提供了见解。代码可在此HTTPS URL上找到。

Title: Elucidated Rolling Diffusion Models for Probabilistic Weather Forecasting

• Authors: Salva Rühling Cachay, Miika Aittala, Karsten Kreis, Noah Brenowitz, Arash Vahdat, Morteza Mardani, Rose Yu
• Subjects: cs.LG, cs.AI, physics.ao-ph, stat.ML
• Abstract URL: https://arxiv.org/abs/2506.20024
• Pdf URL: https://arxiv.org/pdf/2506.20024
• Copy Paste: [[2506.20024]] Elucidated Rolling Diffusion Models for Probabilistic Weather Forecasting(https://arxiv.org/abs/2506.20024)
• Keywords: generation
• Abstract: Diffusion models are a powerful tool for probabilistic forecasting, yet most applications in high-dimensional chaotic systems predict future snapshots one-by-one. This common approach struggles to model complex temporal dependencies and fails to explicitly account for the progressive growth of uncertainty inherent to such systems. While rolling diffusion frameworks, which apply increasing noise to forecasts at longer lead times, have been proposed to address this, their integration with state-of-the-art, high-fidelity diffusion techniques remains a significant challenge. We tackle this problem by introducing Elucidated Rolling Diffusion Models (ERDM), the first framework to successfully unify a rolling forecast structure with the principled, performant design of Elucidated Diffusion Models (EDM). To do this, we adapt the core EDM components-its noise schedule, network preconditioning, and Heun sampler-to the rolling forecast setting. The success of this integration is driven by three key contributions: (i) a novel loss weighting scheme that focuses model capacity on the mid-range forecast horizons where determinism gives way to stochasticity; (ii) an efficient initialization strategy using a pre-trained EDM for the initial window; and (iii) a bespoke hybrid sequence architecture for robust spatiotemporal feature extraction under progressive denoising. On 2D Navier-Stokes simulations and ERA5 global weather forecasting at 1.5^\circ resolution, ERDM consistently outperforms key diffusion-based baselines, including conditional autoregressive EDM. ERDM offers a flexible and powerful general framework for tackling diffusion-based sequence generation problems where modeling escalating uncertainty is paramount. Code is available at: this https URL
• 摘要：扩散模型是概率预测的强大工具，但在高维混沌系统中的大多数应用都可以逐一预测未来的快照。这种常见的方法努力为复杂的时间依赖性建模而无法明确说明此类系统固有的不确定性的逐步增长。虽然已经提出了在较长的交货时间内将噪声越来越多的噪声应用于预测的滚动扩散框架来解决这一问题，但它们与最先进的高效率扩散技术的集成仍然是一个重大挑战。我们通过引入详尽的滚动扩散模型（ERDM）来解决这个问题，这是成功统一滚动预测结构的第一个框架。为此，我们适应了核心EDM组件 - ITS噪声时间表，网络预处理以及Heun Sampler-to滚动预测设置。这种整合的成功是由三个关键贡献驱动的：（i）一种新颖的减肥方案，将模型容量集中在中等范围的预测范围上，而确定论使其带来了随机性； （ii）使用预训练的EDM进行初始窗口的有效初始化策略； （iii）定制的混合序列体系结构，用于在渐进式降解下鲁棒时空提取。在2D Navier-Stokes模拟和ERA5全球天气预报以1.5^\ Circ分辨率下进行，ERDM始终优于基于密钥扩散的基准，包括有条件的自动回归EDM。 ERDM提供了一个灵活而强大的通用框架，用于解决基于扩散的序列生成问题，在这种问题中建模不确定性至关重要。代码可用：此HTTPS URL

Title: Automated Generation of Diverse Courses of Actions for Multi-Agent Operations using Binary Optimization and Graph Learning

• Authors: Prithvi Poddar, Ehsan Tarkesh Esfahani, Karthik Dantu, Souma Chowdhury
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.20031
• Pdf URL: https://arxiv.org/pdf/2506.20031
• Copy Paste: [[2506.20031]] Automated Generation of Diverse Courses of Actions for Multi-Agent Operations using Binary Optimization and Graph Learning(https://arxiv.org/abs/2506.20031)
• Keywords: generation
• Abstract: Operations in disaster response, search \& rescue, and military missions that involve multiple agents demand automated processes to support the planning of the courses of action (COA). Moreover, traverse-affecting changes in the environment (rain, snow, blockades, etc.) may impact the expected performance of a COA, making it desirable to have a pool of COAs that are diverse in task distributions across agents. Further, variations in agent capabilities, which could be human crews and/or autonomous systems, present practical opportunities and computational challenges to the planning process. This paper presents a new theoretical formulation and computational framework to generate such diverse pools of COAs for operations with soft variations in agent-task compatibility. Key to the problem formulation is a graph abstraction of the task space and the pool of COAs itself to quantify its diversity. Formulating the COAs as a centralized multi-robot task allocation problem, a genetic algorithm is used for (order-ignoring) allocations of tasks to each agent that jointly maximize diversity within the COA pool and overall compatibility of the agent-task mappings. A graph neural network is trained using a policy gradient approach to then perform single agent task sequencing in each COA, which maximizes completion rates adaptive to task features. Our tests of the COA generation process in a simulated environment demonstrate significant performance gain over a random walk baseline, small optimality gap in task sequencing, and execution time of about 50 minutes to plan up to 20 COAs for 5 agent/100 task operations.
• 摘要：灾难响应，搜查\＆救援的操作以及涉及多个代理商的军事任务要求自动化程序以支持行动方案（COA）的规划。此外，环境的遍历变化（雨，雪，封锁等）可能会影响COA的预期性能，从而使人们希望拥有各种代理商的任务分布不同的COA。此外，代理能力的变化（可能是人类机组人员和/或自治系统）对计划过程提出了实用的机会和计算挑战。本文提出了一种新的理论配方和计算框架，以生成如此多样的COA池，以供具有代理任务兼容性柔和变化的操作。问题表述的关键是任务空间和COA本身库的图形抽象，以量化其多样性。将COAs制定为集中式多机器人任务分配问题，将遗传算法用于（订单签署）任务分配给每个代理，这些算法在COA池中共同最大化多样性以及代理任务量映射的整体兼容性。使用策略梯度方法对图神经网络进行了训练，然后在每个COA中执行单个代理任务测序，这最大化了对任务功能的完成率。在模拟环境中，我们对COA生成过程的测试表明，在随机步行基线，任务测序中的小型最佳差距以及执行时间约为50分钟，可为5代理/100个任务操作计划20个COA。

Title: Verifiable Unlearning on Edge

• Authors: Mohammad M Maheri, Alex Davidson, Hamed Haddadi
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2506.20037
• Pdf URL: https://arxiv.org/pdf/2506.20037
• Copy Paste: [[2506.20037]] Verifiable Unlearning on Edge(https://arxiv.org/abs/2506.20037)
• Keywords: generation
• Abstract: Machine learning providers commonly distribute global models to edge devices, which subsequently personalize these models using local data. However, issues such as copyright infringements, biases, or regulatory requirements may require the verifiable removal of certain data samples across all edge devices. Ensuring that edge devices correctly execute such unlearning operations is critical to maintaining integrity. In this work, we introduce a verification framework leveraging zero-knowledge proofs, specifically zk-SNARKs, to confirm data unlearning on personalized edge-device models without compromising privacy. We have developed algorithms explicitly designed to facilitate unlearning operations that are compatible with efficient zk-SNARK proof generation, ensuring minimal computational and memory overhead suitable for constrained edge environments. Furthermore, our approach carefully preserves personalized enhancements on edge devices, maintaining model performance post-unlearning. Our results affirm the practicality and effectiveness of this verification framework, demonstrating verifiable unlearning with minimal degradation in personalization-induced performance improvements. Our methodology ensures verifiable, privacy-preserving, and effective machine unlearning across edge devices.
• 摘要：机器学习提供商通常将全局模型分发到边缘设备，随后使用本地数据对这些模型进行个性化。但是，诸如版权侵权，偏见或监管要求之类的问题可能需要在所有边缘设备上可验证某些数据样本。确保Edge设备正确执行此类学习操作对于保持完整性至关重要。在这项工作中，我们引入了一个验证框架，利用零知识证明，特别是ZK-SNARKS，以确认在不损害隐私的情况下对个性化边缘设备模型上的数据进行了学习。我们已经开发了明确设计的算法，以促进与有效的ZK-SNARK证明生成兼容的学习操作，从而确保适合于约束边缘环境的最小计算和内存开销。此外，我们的方法仔细地保留了边缘设备上的个性化增强功能，并在未检验后保持模型性能。我们的结果肯定了该验证框架的实用性和有效性，证明了可验证的未学习，而个性化诱导的绩效改善的降解最小。我们的方法可确保跨越边缘设备的可验证，保护和有效的机器。

Title: BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videos

• Authors: Jiahao Lin, Weixuan Peng, Bojia Zi, Yifeng Gao, Xianbiao Qi, Xingjun Ma, Yu-Gang Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.20103
• Pdf URL: https://arxiv.org/pdf/2506.20103
• Copy Paste: [[2506.20103]] BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videos(https://arxiv.org/abs/2506.20103)
• Keywords: generation, generative
• Abstract: Recent advances in deep generative models have led to significant progress in video generation, yet the fidelity of AI-generated videos remains limited. Synthesized content often exhibits visual artifacts such as temporally inconsistent motion, physically implausible trajectories, unnatural object deformations, and local blurring that undermine realism and user trust. Accurate detection and spatial localization of these artifacts are crucial for both automated quality control and for guiding the development of improved generative models. However, the research community currently lacks a comprehensive benchmark specifically designed for artifact localization in AI generated videos. Existing datasets either restrict themselves to video or frame level detection or lack the fine-grained spatial annotations necessary for evaluating localization methods. To address this gap, we introduce BrokenVideos, a benchmark dataset of 3,254 AI-generated videos with meticulously annotated, pixel-level masks highlighting regions of visual corruption. Each annotation is validated through detailed human inspection to ensure high quality ground truth. Our experiments show that training state of the art artifact detection models and multi modal large language models (MLLMs) on BrokenVideos significantly improves their ability to localize corrupted regions. Through extensive evaluation, we demonstrate that BrokenVideos establishes a critical foundation for benchmarking and advancing research on artifact localization in generative video models. The dataset is available at: this https URL.
• 摘要：深层生成模型的最新进展导致了视频生成的重大进展，但是AI生成的视频的保真度仍然有限​​。合成的内容通常表现出视觉伪像，例如时间上不一致的运动，身体上难以置信的轨迹，不自然的对象变形以及局部模糊，这会破坏现实主义和用户信任。这些伪影的准确检测和空间定位对于自动质量控制和指导改进的生成模型的发展至关重要。但是，目前，研究界缺乏专门为AI生成的视频中文物本地化设计的全面基准。现有数据集限于视频级别检测或缺乏评估本地化方法所需的细粒度空间注释。为了解决这一差距，我们介绍了brokenvideos，这是一个由3,254个AI生成的视频的基准数据集，其中带有精心注释的像素级掩码，突出了视觉腐败的区域。每个注释都通过详细的人类检查来验证，以确保高质量的地面真相。我们的实验表明，训练最先进的人工制品检测模型和brokenvideos上的多态大语模型（MLLMS）显着提高了其本地化损坏区域的能力。通过广泛的评估，我们证明了BrokenVideos为生成视频模型中的工件本地化研究建立了重要的基础。该数据集可在以下网址提供：此HTTPS URL。

Title: High-Resolution Live Fuel Moisture Content (LFMC) Maps for Wildfire Risk from Multimodal Earth Observation Data

• Authors: Patrick Alan Johnson, Gabriel Tseng, Yawen Zhang, Heather Heward, Virginia Sjahli, Favyen Bastani, Joseph Redmon, Patrick Beukema
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2506.20132
• Pdf URL: https://arxiv.org/pdf/2506.20132
• Copy Paste: [[2506.20132]] High-Resolution Live Fuel Moisture Content (LFMC) Maps for Wildfire Risk from Multimodal Earth Observation Data(https://arxiv.org/abs/2506.20132)
• Keywords: generation
• Abstract: Wildfires are increasing in intensity and severity at an alarming rate. Recent advances in AI and publicly available satellite data enable monitoring critical wildfire risk factors globally, at high resolution and low latency. Live Fuel Moisture Content (LFMC) is a critical wildfire risk factor and is valuable for both wildfire research and operational response. However, ground-based LFMC samples are both labor intensive and costly to acquire, resulting in sparse and infrequent updates. In this work, we explore the use of a pretrained, highly-multimodal earth-observation model for generating large-scale spatially complete (wall-to-wall) LFMC maps. Our approach achieves significant improvements over previous methods using randomly initialized models (20 reduction in RMSE). We provide an automated pipeline that enables rapid generation of these LFMC maps across the United States, and demonstrate its effectiveness in two regions recently impacted by wildfire (Eaton and Palisades).
• 摘要：野火的强度和严重程度以惊人的速度增加。 AI和公开卫星数据的最新进展使全球关键的野火风险因素在高分辨率和低潜伏期中。活燃料水分含量（LFMC）是关键的野火危险因素，对于野火研究和操作反应而言都是有价值的。但是，基于地面的LFMC样品既劳动量很大又昂贵，从而导致稀疏和不经常更新。在这项工作中，我们探讨了使用经过预定的，高度的地球观察模型来生成大规模的空间完整（壁到壁）LFMC地图。我们的方法使用随机初始化模型（RMSE减少20个），对以前的方法实现了显着改进。我们提供了一条自动化管道，该管道能够在美国迅速生成这些LFMC地图，并在最近受野火影响（Eaton和Palisades）影响的两个地区证明了其有效性。

Title: From 2D to 3D Cognition: A Brief Survey of General World Models

• Authors: Ningwei Xie, Zizi Tian, Lei Yang, Xiao-Ping Zhang, Meng Guo, Jie Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.20134
• Pdf URL: https://arxiv.org/pdf/2506.20134
• Copy Paste: [[2506.20134]] From 2D to 3D Cognition: A Brief Survey of General World Models(https://arxiv.org/abs/2506.20134)
• Keywords: generation, generative
• Abstract: World models have garnered increasing attention in the development of artificial general intelligence (AGI), serving as computational frameworks for learning representations of the external world and forecasting future states. While early efforts focused on 2D visual perception and simulation, recent 3D-aware generative world models have demonstrated the ability to synthesize geometrically consistent, interactive 3D environments, marking a shift toward 3D spatial cognition. Despite rapid progress, the field lacks systematic analysis to categorize emerging techniques and clarify their roles in advancing 3D cognitive world models. This survey addresses this need by introducing a conceptual framework, providing a structured and forward-looking review of world models transitioning from 2D perception to 3D cognition. Within this framework, we highlight two key technological drivers, particularly advances in 3D representations and the incorporation of world knowledge, as fundamental pillars. Building on these, we dissect three core cognitive capabilities that underpin 3D world modeling: 3D physical scene generation, 3D spatial reasoning, and 3D spatial interaction. We further examine the deployment of these capabilities in real-world applications, including embodied AI, autonomous driving, digital twin, and gaming/VR. Finally, we identify challenges across data, modeling, and deployment, and outline future directions for advancing more robust and generalizable 3D world models.
• 摘要：世界模型在人工通用智能（AGI）的发展中引起了越来越多的关注，它是用于学习外部世界和预测未来状态的计算框架。尽管早期的努力集中在2D视觉感知和仿真上，但最近的3D感知生成世界模型表明了能够合成几何一致的交互式3D环境，这标志着向3D空间认知的转变。尽管进步迅速，该领域仍缺乏系统的分析来对新兴技术进行分类，并阐明了它们在增进3D认知世界模型中的作用。这项调查通过引入概念框架来满足这一需求，从2D感知到3D认知过渡到世界模型的结构化和前瞻性的评论。在此框架内，我们重点介绍了两个关键的技术驱动力，尤其是3D表示的进步和将世界知识的结合作为基本支柱。在这些基础上，我们剖析了基于3D世界建模的三个核心认知能力：3D物理场景产生，3D空间推理和3D空间互动。我们进一步研究了这些功能在实际应用中的部署，包括体现的AI，自动驾驶，数字双胞胎和游戏/VR。最后，我们确定了跨数据，建模和部署的挑战，并概述了未来的方向，以推动更强大且可推广的3D世界模型。

Title: EAR: Erasing Concepts from Unified Autoregressive Models

• Authors: Haipeng Fan, Shiyuan Zhang, Baohunesitu, Zihang Guo, Huaiwen Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.20151
• Pdf URL: https://arxiv.org/pdf/2506.20151
• Copy Paste: [[2506.20151]] EAR: Erasing Concepts from Unified Autoregressive Models(https://arxiv.org/abs/2506.20151)
• Keywords: generation
• Abstract: Autoregressive (AR) models have achieved unified and strong performance across both visual understanding and image generation tasks. However, removing undesired concepts from AR models while maintaining overall generation quality remains an open challenge. In this paper, we propose Erasure Autoregressive Model (EAR), a fine-tuning method for effective and utility-preserving concept erasure in AR models. Specifically, we introduce Windowed Gradient Accumulation (WGA) strategy to align patch-level decoding with erasure objectives, and Thresholded Loss Masking (TLM) strategy to protect content unrelated to the target concept during fine-tuning. Furthermore, we propose a novel benchmark, Erase Concept Generator and Visual Filter (ECGVF), aim at provide a more rigorous and comprehensive foundation for evaluating concept erasure in AR models. Specifically, we first employ structured templates across diverse large language models (LLMs) to pre-generate a large-scale corpus of target-replacement concept prompt pairs. Subsequently, we generate images from these prompts and subject them to rigorous filtering via a visual classifier to ensure concept fidelity and alignment. Extensive experimental results conducted on the ECGVF benchmark with the AR model Janus-Pro demonstrate that EAR achieves marked improvements in both erasure effectiveness and model utility preservation. Code is available at: this https URL
• 摘要：在视觉理解和图像生成任务中，自回归（AR）模型都实现了统一和强大的性能。但是，从AR模型中删除不希望的概念，同时保持整体发电质量仍然是一个开放的挑战。在本文中，我们提出了Erasure自回旋模型（EAR），这是AR模型中有效且具有公用事业概念擦除的微调方法。具体而言，我们介绍了窗口梯度积累（WGA）策略，以使补丁级解码与擦除目标和阈值损失蒙版（TLM）策略保持一致，以保护在微调过程中与目标概念无关的内容。此外，我们提出了一种新颖的基准，擦除概念生成器和视觉过滤器（ECGVF），目的是为评估AR模型中的概念擦除提供更严格，更全面的基础。具体而言，我们首先采用跨不同语言模型（LLM）的结构化模板来预先生成大规模的目标替换概念促使对。随后，我们从这些提示中生成图像，并通过视觉分类器进行严格的过滤，以确保概念保真度和对齐方式。用AR模型Janus-Pro在ECGVF基准上进行的广泛实验结果表明，EAR在擦除有效性和模型效用保存方面都取得了明显的改善。代码可用：此HTTPS URL

Title: Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models

• Authors: Zhentao He, Can Zhang, Ziheng Wu, Zhenghao Chen, Yufei Zhan, Yifan Li, Zhao Zhang, Xian Wang, Minghui Qiu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.20168
• Pdf URL: https://arxiv.org/pdf/2506.20168
• Copy Paste: [[2506.20168]] Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models(https://arxiv.org/abs/2506.20168)
• Keywords: generation
• Abstract: Recent advancements in multimodal large language models have enhanced document understanding by integrating textual and visual information. However, existing models exhibit incompleteness within their paradigm in real-world scenarios, particularly under visual degradation. In such conditions, the current response paradigm often fails to adequately perceive visual degradation and ambiguity, leading to overreliance on linguistic priors or misaligned visual-textual reasoning. This difficulty in recognizing uncertainty frequently results in the generation of hallucinatory content, especially when a precise answer is not feasible. To better demonstrate and analyze this phenomenon and problem, we propose KIE-HVQA, the first benchmark dedicated to evaluating OCR hallucination in degraded document understanding. This dataset includes test samples spanning identity cards and invoices, with simulated real-world degradations for OCR reliability. This setup allows for evaluating models' capacity, under degraded input, to distinguish reliable visual information and answer accordingly, thereby highlighting the challenge of avoiding hallucination on uncertain data. To achieve vision-faithful reasoning and thereby avoid the aforementioned issues, we further introduce a GRPO-based framework featuring a novel reward mechanism. By incorporating a self-awareness of visual uncertainty and an analysis method that initiates refusal to answer to increase task difficulty within our supervised fine-tuning and reinforcement learning framework, we successfully mitigated hallucinations in ambiguous regions. Experiments on Qwen2.5-VL demonstrate that our 7B-parameter model achieves a 22\% absolute improvement in hallucination-free accuracy over GPT-4o on KIE-HVQA and there is no significant performance drop in standard tasks, highlighting both effectiveness and robustness.
• 摘要：多模式大语言模型的最新进展通过整合文本和视觉信息来增强文档的理解。但是，现有模型在实际情况下，尤其是在视觉降解下表现出其范式中的不完整。在这种情况下，当前的响应范式通常无法充分感知视觉下降和模棱两可，从而导致对语言先验或未对象的视觉文本推理过度依赖。识别不确定性的困难通常会导致幻觉含量的产生，尤其是在确切的答案不可行的情况下。为了更好地证明和分析这种现象和问题，我们提出了Kie-HVQA，这是第一个专门用于评估文档理解中OCR幻觉的基准。该数据集包括涵盖身份证和发票的测试样本，并具有模拟的现实世界降解以实现OCR可靠性。这种设置允许评估模型在退化的输入下的能力，以区分可靠的视觉信息并相应地回答，从而突出了避免在不确定数据上幻觉的挑战。为了实现视觉信仰的推理并避免上述问题，我们进一步介绍了一个基于GRPO的框架，具有新颖的奖励机制。通过纳入对视觉不确定性的自我意识和一种分析方法，该方法启动拒绝回答以增加我们监督的微调和强化学习框架中的任务难度，我们成功地减轻了模棱两可地区的幻觉。关于QWEN2.5-VL的实验表明，我们的7B参数模型可在Kie-HVQA上的GPT-4O幻觉中的无幻觉准确性22 \％，并且标准任务的性能没有显着下降，从而强调了有效性和鲁棒性。

Title: UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation

• Authors: Yanzhe Chen (Yen-chieh Chan), Huasong Zhong, Yan Li, Zhenheng Yang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2506.20214
• Pdf URL: https://arxiv.org/pdf/2506.20214
• Copy Paste: [[2506.20214]] UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation(https://arxiv.org/abs/2506.20214)
• Keywords: generation
• Abstract: Unified multimodal large language models (MLLMs) have shown promise in jointly advancing multimodal understanding and generation, with visual codebooks discretizing images into tokens for autoregressive modeling. Existing codebook-based methods either rely on small vocabularies (~16K entries) that lack fine-grained semantics or naively scale up, resulting in low token utilization and unstable training. We propose UniCode$^2$, a cascaded codebook framework enabling large-scale, semantically aligned, and stable visual tokenization. By clustering millions of SigLIP sequence embeddings, we build a 500K-entry codebook that preserves vision-language alignment while expanding capacity. Stability is ensured via a cascaded design: a frozen codebook anchors the embedding space, and a trainable codebook refines task-specific semantics. This decoupling promotes high utilization and robust learning. Moreover, the alignment of our visual tokens with textual semantics enables seamless integration with pretrained diffusion decoders, supporting high-quality visual synthesis with minimal adaptation. UniCode^2 delivers strong performance across diverse benchmarks, demonstrating the viability of scaling visual token spaces without sacrificing stability, semantics, or modularity.
• 摘要：统一的多模式大型语言模型（MLLM）在共同推进多模式理解和生成方面表现出了希望，视觉代码手册将图像分散到代币中进行自动回归建模。现有的基于密码的方法依赖于缺乏精细语义的小词汇（〜16K条目），或者天真地扩展，从而导致令牌使用率较低和不稳定的培训。我们提出了Unicode $^2 $，这是一个级联的代码簿框架，可实现大规模，语义对齐和稳定的视觉令牌化。通过聚集数百万个siglip序列的嵌入，我们构建了一个500K输入的代码手册，该书籍可以保留视觉统一的同时扩大容量。通过级联设计确保稳定性：冷冻代码簿锚定嵌入空间，而可训练的代码簿可完善特定于任务的语义。这种去耦可促进高利用率和强大的学习。此外，我们的视觉令牌与文本语义的对齐能够与预处理的扩散解码器无缝集成，从而支持高质量的视觉合成，并最少适应。 Unicode^2在不同的基准测试中表现出强大的性能，证明了缩放视觉令牌空间的可行性，而无需牺牲稳定性，语义或模块化。

Title: FedBKD: Distilled Federated Learning to Embrace Gerneralization and Personalization on Non-IID Data

• Authors: Yushan Zhao, Jinyuan He, Donglai Chen, Weijie Luo, Chong Xie, Ri Zhang, Yonghong Chen, Yan Xu
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2506.20245
• Pdf URL: https://arxiv.org/pdf/2506.20245
• Copy Paste: [[2506.20245]] FedBKD: Distilled Federated Learning to Embrace Gerneralization and Personalization on Non-IID Data(https://arxiv.org/abs/2506.20245)
• Keywords: generative
• Abstract: Federated learning (FL) is a decentralized collaborative machine learning (ML) technique. It provides a solution to the issues of isolated data islands and data privacy leakage in industrial ML practices. One major challenge in FL is handling the non-identical and independent distributed (non-IID) data. Current solutions either focus on constructing an all-powerful global model, or customizing personalized local models. Few of them can provide both a well-generalized global model and well-performed local models at the same time. Additionally, many FL solutions to the non-IID problem are benefited from introducing public datasets. However, this will also increase the risk of data leakage. To tackle the problems, we propose a novel data-free distillation framework, Federated Bidirectional Knowledge Distillation (FedBKD). Specifically, we train Generative Adversarial Networks (GAN) for synthetic data. During the GAN training, local models serve as discriminators and their parameters are frozen. The synthetic data is then used for bidirectional distillation between global and local models to achieve knowledge interactions so that performances for both sides are improved. We conduct extensive experiments on 4 benchmarks under different non-IID settings. The results show that FedBKD achieves SOTA performances in every case.
• 摘要：联合学习（FL）是一种分散的协作机器学习（ML）技术。它为工业ML实践中孤立的数据岛和数据隐私泄漏的问题提供了解决方案。 FL中的一个主要挑战是处理非相同和独立的分布式（非IID）数据。当前的解决方案要么着重于构建全能的全球模型，要么定制个性化的本地模型。他们中很少有人能同时提供良好的全球模型和表现出色的本地模型。此外，引入公共数据集从引入公共数据集中受益许多FL解决方案。但是，这也将增加数据泄漏的风险。为了解决问题，我们提出了一个新型的无数据蒸馏框架，联合双向知识蒸馏（FEDBKD）。具体而言，我们训练生成对抗网络（GAN）以获取合成数据。在GAN培训期间，当地模型充当歧视者，其参数被冷冻。然后将合成数据用于全球模型和本地模型之间的双向蒸馏，以实现知识相互作用，以改善双方的性能。我们在不同的非IID设置下对4个基准测试进行了广泛的实验。结果表明，在每种情况下，FEDBKD都会达到SOTA性能。

Title: Time-series surrogates from energy consumers generated by machine learning approaches for long-term forecasting scenarios

• Authors: Ben Gerhards, Nikita Popkov, Annekatrin König, Marcel Arpogaus, Bastian Schäfermeier, Leonie Riedl, Stephan Vogt, Philip Hehlert
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.20253
• Pdf URL: https://arxiv.org/pdf/2506.20253
• Copy Paste: [[2506.20253]] Time-series surrogates from energy consumers generated by machine learning approaches for long-term forecasting scenarios(https://arxiv.org/abs/2506.20253)
• Keywords: generation, generative
• Abstract: Forecasting attracts a lot of research attention in the electricity value chain. However, most studies concentrate on short-term forecasting of generation or consumption with a focus on systems and less on individual consumers. Even more neglected is the topic of long-term forecasting of individual power consumption. Here, we provide an in-depth comparative evaluation of data-driven methods for generating synthetic time series data tailored to energy consumption long-term forecasting. High-fidelity synthetic data is crucial for a wide range of applications, including state estimations in energy systems or power grid planning. In this study, we assess and compare the performance of multiple state-of-the-art but less common techniques: a hybrid Wasserstein Generative Adversarial Network (WGAN), Denoising Diffusion Probabilistic Model (DDPM), Hidden Markov Model (HMM), and Masked Autoregressive Bernstein polynomial normalizing Flows (MABF). We analyze the ability of each method to replicate the temporal dynamics, long-range dependencies, and probabilistic transitions characteristic of individual energy consumption profiles. Our comparative evaluation highlights the strengths and limitations of: WGAN, DDPM, HMM and MABF aiding in selecting the most suitable approach for state estimations and other energy-related tasks. Our generation and analysis framework aims to enhance the accuracy and reliability of synthetic power consumption data while generating data that fulfills criteria like anonymisation - preserving privacy concerns mitigating risks of specific profiling of single customers. This study utilizes an open-source dataset from households in Germany with 15min time resolution. The generated synthetic power profiles can readily be used in applications like state estimations or consumption forecasting.
• 摘要：预测吸引了电力价值链中的大量研究关注。但是，大多数研究都集中于对发电或消费的短期预测，而侧重于系统，而不是个人消费者。更加忽视的是长期预测个人功耗的话题。在这里，我们对数据驱动的方法进行了深入的比较评估，以生成针对能源消耗的长期预测量身定制的合成时间序列数据。高保真综合数据对于广泛的应用程序至关重要，包括能源系统或电网规划的状态估计。在这项研究中，我们评估和比较了多种最先进但不常见技术的性能：混合剂量剂量生成的对抗网络（WGAN），deno string defusion扩散概率模型（DDPM），隐藏的Markov模型（HMM）和掩盖了自动化的自动锻炼的伯恩斯坦伯恩斯坦多种官方官方正常化的正常量（MAB）。我们分析了每种方法复制时间动力学，远程依赖性和概率过渡的能力。我们的比较评估强调了：WGAN，DDPM，HMM和MABF的优势和局限性帮助选择了最合适的状态估计方法和其他与能源相关的任务。我们的一代和分析框架旨在提高合成功耗数据的准确性和可靠性，同时生成满足匿名标准的数据 - 保留隐私涉及​​减轻单个客户特定分析的风险。这项研究利用来自德国家庭的开源数据集，分辨率为15分钟。生成的合成功率轮廓可以轻松地用于状态估计或消费预测等应用中。

Title: From Ideal to Real: Unified and Data-Efficient Dense Prediction for Real-World Scenarios

• Authors: Changliang Xia, Chengyou Jia, Zhuohang Dang, Minnan Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.20279
• Pdf URL: https://arxiv.org/pdf/2506.20279
• Copy Paste: [[2506.20279]] From Ideal to Real: Unified and Data-Efficient Dense Prediction for Real-World Scenarios(https://arxiv.org/abs/2506.20279)
• Keywords: generative
• Abstract: Dense prediction tasks hold significant importance of computer vision, aiming to learn pixel-wise annotated label for an input image. Despite advances in this field, existing methods primarily focus on idealized conditions, with limited generalization to real-world scenarios and facing the challenging scarcity of real-world data. To systematically study this problem, we first introduce DenseWorld, a benchmark spanning a broad set of 25 dense prediction tasks that correspond to urgent real-world applications, featuring unified evaluation across tasks. Then, we propose DenseDiT, which maximally exploits generative models' visual priors to perform diverse real-world dense prediction tasks through a unified strategy. DenseDiT combines a parameter-reuse mechanism and two lightweight branches that adaptively integrate multi-scale context, working with less than 0.1% additional parameters. Evaluations on DenseWorld reveal significant performance drops in existing general and specialized baselines, highlighting their limited real-world generalization. In contrast, DenseDiT achieves superior results using less than 0.01% training data of baselines, underscoring its practical value for real-world deployment. Our data, and checkpoints and codes are available at this https URL
• 摘要：密集的预测任务对计算机视觉的重要性非常重要，旨在学习输入图像的按像素注释的标签。尽管该领域取得了进步，但现有方法主要集中在理想化的条件下，对现实世界情景的概括有限，并且面临着挑战性的现实数据稀缺性。为了系统地研究此问题，我们首先介绍了密集世界，这是一个基准，涵盖了一系列25个密集的预测任务，这些任务与紧急现实世界应用相对应，其中包含跨任务的统一评估。然后，我们提出了密码，它最大程度地利用了生成模型的视觉先验来通过统一的策略执行各种现实世界的密集预测任务。密码结合了一个参数复制机制和两个轻巧的分支，可自适应整合多尺度上下文，与少于0.1％的附加参数一起工作。对密集世界的评估显示，现有的一般和专业基线的表现显着下降，强调了它们的现实概括有限。相比之下，使用基准的训练数据少于0.01％，其实现了较高的结果，从而强调了其现实部署的实用价值。我们的数据以及检查点和代码可在此HTTPS URL上找到

Title: Ctrl-Z Sampling: Diffusion Sampling with Controlled Random Zigzag Explorations

• Authors: Shunqi Mao, Wei Guo, Chaoyi Zhang, Weidong Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.20294
• Pdf URL: https://arxiv.org/pdf/2506.20294
• Copy Paste: [[2506.20294]] Ctrl-Z Sampling: Diffusion Sampling with Controlled Random Zigzag Explorations(https://arxiv.org/abs/2506.20294)
• Keywords: generation
• Abstract: Diffusion models have shown strong performance in conditional generation by progressively denoising Gaussian noise toward a target data distribution. This denoising process can be interpreted as a form of hill climbing in a learned latent space, where the model iteratively refines the sample toward regions of higher probability. However, diffusion models often converge to local optima that are locally visually coherent yet globally inconsistent or conditionally misaligned, due to latent space complexity and suboptimal initialization. Prior efforts attempted to address this by strengthening guidance signals or manipulating the initial noise distribution. We introduce Controlled Random Zigzag Sampling (Ctrl-Z Sampling), a novel sampling strategy designed to detect and escape such local maxima during conditional generation. The method first identifies potential local maxima using a reward model. Upon detection, it injects noise and reverts to a previous, noisier state to escape the current optimization plateau. The reward model then evaluates candidate trajectories, accepting only those that offer improvement, while progressively deeper retreat enables stronger escapes when nearby alternatives fail. This controlled random zigzag process allows dynamic alternation between forward refinement and backward exploration, enhancing both alignment and visual quality in the generated outputs. The proposed Ctrl-Z Sampling is model-agnostic and compatible with existing diffusion frameworks. Experimental results show that Ctrl-Z Sampling substantially improves generation quality with only around 7.6X increase in function evaluations.
• 摘要：扩散模型通过逐步将高斯噪声降低到目标数据分布，在条件生成中表现出强烈的性能。这种脱索过程可以解释为在学习的潜在空间中爬山的一种形式，在该空间中，该模型迭代地将样品划分为较高概率的区域。但是，由于潜在的空间复杂性和次优初始化，扩散模型通常会收敛到本地视觉上相干但全球不一致或有条件地对准的局部优势。先前的努力试图通过加强指导信号或操纵初始噪声分布来解决这一问题。我们引入了受控的随机曲折采样（CTRL-Z采样），这是一种新型的采样策略，旨在检测和逃避条件生成期间这种局部最大值。该方法首先使用奖励模型来标识潜在的本地最大值。在发现后，它会注入噪声并将其恢复到以前的嘈杂状态，以逃避当前的优化高原。然后，奖励模型评估了候选轨迹，仅接受那些提供改进的轨迹，而逐渐更深的撤退可以在附近的替代方案失败时更强大的逃逸。这种受控的随机曲折过程允许向前改进和向后探索之间动态交替，从而增强了生成的输出中的对齐和视觉质量。所提出的CTRL-Z采样是模型的敏锐性，并且与现有的扩散框架兼容。实验结果表明，CTRL-Z采样大大提高了发电质量，功能评估的增加约为7.6倍。

Title: TDiR: Transformer based Diffusion for Image Restoration Tasks

• Authors: Abbas Anwar, Mohammad Shullar, Ali Arshad Nasir, Mudassir Masood, Saeed Anwar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.20302
• Pdf URL: https://arxiv.org/pdf/2506.20302
• Copy Paste: [[2506.20302]] TDiR: Transformer based Diffusion for Image Restoration Tasks(https://arxiv.org/abs/2506.20302)
• Keywords: restoration
• Abstract: Images captured in challenging environments often experience various forms of degradation, including noise, color cast, blur, and light scattering. These effects significantly reduce image quality, hindering their applicability in downstream tasks such as object detection, mapping, and classification. Our transformer-based diffusion model was developed to address image restoration tasks, aiming to improve the quality of degraded images. This model was evaluated against existing deep learning methodologies across multiple quality metrics for underwater image enhancement, denoising, and deraining on publicly available datasets. Our findings demonstrate that the diffusion model, combined with transformers, surpasses current methods in performance. The results of our model highlight the efficacy of diffusion models and transformers in improving the quality of degraded images, consequently expanding their utility in downstream tasks that require high-fidelity visual data.
• 摘要：在具有挑战性的环境中捕获的图像通常会经历各种形式的退化，包括噪声，颜色铸造，模糊和光散射。这些效果大大降低了图像质量，阻碍了它们在对象检测，映射和分类等下游任务中的适用性。我们开发了基于变压器的扩散模型来解决图像恢复任务，旨在提高降级图像的质量。该模型是针对多个质量指标的现有深度学习方法进行评估的，以增强水下图像图像，并在公开可用的数据集中进行了评估。我们的发现表明，扩散模型与变压器相结合，超过了性能中的当前方法。我们模型的结果突出了扩散模型和变压器在改善退化图像质量方面的功效，从而扩展了其在需要高保真视觉数据的下游任务中的效用。

Title: A foundation model with multi-variate parallel attention to generate neuronal activity

• Authors: Francesco Carzaniga, Michael Hersche, Abu Sebastian, Kaspar Schindler, Abbas Rahimi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.20354
• Pdf URL: https://arxiv.org/pdf/2506.20354
• Copy Paste: [[2506.20354]] A foundation model with multi-variate parallel attention to generate neuronal activity(https://arxiv.org/abs/2506.20354)
• Keywords: generative
• Abstract: Learning from multi-variate time-series with heterogeneous channel configurations remains a fundamental challenge for deep neural networks (DNNs), particularly in clinical domains such as intracranial electroencephalography (iEEG), where channel setups vary widely across subjects. In this work, we introduce multi-variate parallel attention (MVPA), a novel self-attention mechanism that disentangles content, temporal, and spatial attention, enabling flexible, generalizable, and efficient modeling of time-series data with varying channel counts and configurations. We use MVPA to build MVPFormer, a generative foundation model for human electrophysiology, trained to predict the evolution of iEEG signals across diverse subjects. To support this and future effort by the community, we release the SWEC iEEG dataset, the largest publicly available iEEG dataset to date, comprising nearly 10,000 hours of recordings from heterogeneous clinical sources. MVPFormer leverages MVPA to achieve strong generalization across subjects, demonstrating expert-level performance in seizure detection and outperforming state-of-the-art Transformer baselines on our SWEC, the MAYO, and the FNUSA dataset. We further validate MVPA on standard time-series forecasting and classification tasks, where it matches or exceeds existing attention-based models. Together, our contributions establish MVPA as a general-purpose attention mechanism for heterogeneous time-series and MVPFormer as the first open-source, open-weights, and open-data iEEG foundation model with state-of-the-art clinical performance. The code is available at this https URL. The SWEC iEEG dataset is available at this https URL.
• 摘要：从多变量的时间序列中学习具有异质通道配置仍然是对深神经网络（DNNS）的基本挑战，尤其是在诸如颅内脑电图（IEEG）之类的临床领域，其中通道设置在各个受试者之间差异很大。在这项工作中，我们引入了多变量平行注意（MVPA），这是一种新型的自我发挥机制，可以消除内容，时间和空间注意力，从而具有不同的通道计数和配置的时间序列数据的灵活，可推广和有效的时间序列建模。我们使用MVPA来构建MVPFormer，这是人类电生理学的生成基础模型，训练有素，可以预测跨不同主题的IEEG信号的演变。为了支持社区的这一工作和将来的工作，我们发布了Swec IEEG数据集，这是迄今为止最大的公开IEEG数据集，包括来自异质临床来源的近10,000小时记录。 MVPFormer利用MVPA在主题之间实现强大的概括，证明在我们的SWEC，Mayo和FNUSA数据集中，在癫痫发作检测方面的专家级别表现优于最先进的变压器基线。我们进一步验证了MVPA在标准时间序列的预测和分类任务上，它匹配或超过了现有的基于注意力的模型。我们的贡献共同将MVPA建立为异质时间序列的通用注意机制，并将MVPFormer作为首个开放源代码，开放式和开放式IEEEG基础模型，具有最先进的临床表现。该代码可在此HTTPS URL上找到。 Swec IEEG数据集可在此HTTPS URL上找到。

Title: A Novel Large Vision Foundation Model (LVFM)-based Approach for Generating High-Resolution Canopy Height Maps in Plantations for Precision Forestry Management

• Authors: Shen Tan, Xin Zhang, Liangxiu Han, Huaguo Huang, Han Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.20388
• Pdf URL: https://arxiv.org/pdf/2506.20388
• Copy Paste: [[2506.20388]] A Novel Large Vision Foundation Model (LVFM)-based Approach for Generating High-Resolution Canopy Height Maps in Plantations for Precision Forestry Management(https://arxiv.org/abs/2506.20388)
• Keywords: generation
• Abstract: Accurate, cost-effective monitoring of plantation aboveground biomass (AGB) is crucial for supporting local livelihoods and carbon sequestration initiatives like the China Certified Emission Reduction (CCER) program. High-resolution canopy height maps (CHMs) are essential for this, but standard lidar-based methods are expensive. While deep learning with RGB imagery offers an alternative, accurately extracting canopy height features remains challenging. To address this, we developed a novel model for high-resolution CHM generation using a Large Vision Foundation Model (LVFM). Our model integrates a feature extractor, a self-supervised feature enhancement module to preserve spatial details, and a height estimator. Tested in Beijing's Fangshan District using 1-meter Google Earth imagery, our model outperformed existing methods, including conventional CNNs. It achieved a mean absolute error of 0.09 m, a root mean square error of 0.24 m, and a correlation of 0.78 against lidar-based CHMs. The resulting CHMs enabled over 90% success in individual tree detection, high accuracy in AGB estimation, and effective tracking of plantation growth, demonstrating strong generalization to non-training areas. This approach presents a promising, scalable tool for evaluating carbon sequestration in both plantations and natural forests.
• 摘要：对种植园（AGB）对种植园（AGB）的准确，具有成本效益的监测对于支持当地生计和碳固存计划至关重要。高分辨率的冠层高度图（CHM）对于这一点至关重要，但是基于标准的激光雷达方法很昂贵。尽管使用RGB图像进行深度学习提供了替代方案，但准确提取冠层高度功能仍然具有挑战性。为了解决这个问题，我们使用大型视觉基础模型（LVFM）开发了一种用于高分辨率CHM生成的新型模型。我们的模型集成了特征提取器，一个自我监管的功能增强模块，以保留空间细节和高度估计器。我们的模型在北京的Fangshan区进行了测试，我们的模型优于包括常规CNN在内的现有方法。它达到了0.09 m的平均绝对误差，均方根误差为0.24 m，基于激光雷达的CHM的相关性为0.78。由此产生的CHM在单个树木检测中获得了超过90％的成功，AGB估计的高精度以及对种植园生长的有效跟踪，证明了对非训练区域的强烈概括。这种方法提出了一种有前途的可扩展工具，用于评估人工林和天然林中的碳固化。

Title: Med-Art: Diffusion Transformer for 2D Medical Text-to-Image Generation

• Authors: Changlu Guo, Anders Nymark Christensen, Morten Rieger Hannemose
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.20449
• Pdf URL: https://arxiv.org/pdf/2506.20449
• Copy Paste: [[2506.20449]] Med-Art: Diffusion Transformer for 2D Medical Text-to-Image Generation(https://arxiv.org/abs/2506.20449)
• Keywords: generation, generative
• Abstract: Text-to-image generative models have achieved remarkable breakthroughs in recent years. However, their application in medical image generation still faces significant challenges, including small dataset sizes, and scarcity of medical textual data. To address these challenges, we propose Med-Art, a framework specifically designed for medical image generation with limited data. Med-Art leverages vision-language models to generate visual descriptions of medical images which overcomes the scarcity of applicable medical textual data. Med-Art adapts a large-scale pre-trained text-to-image model, PixArt-$\alpha$, based on the Diffusion Transformer (DiT), achieving high performance under limited data. Furthermore, we propose an innovative Hybrid-Level Diffusion Fine-tuning (HLDF) method, which enables pixel-level losses, effectively addressing issues such as overly saturated colors. We achieve state-of-the-art performance on two medical image datasets, measured by FID, KID, and downstream classification performance.
• 摘要：近年来，文本到图像生成模型取得了显着的突破。但是，它们在医学图像生成中的应用仍然面临重大挑战，包括小型数据集大小以及医疗文本数据的稀缺性。为了应对这些挑战，我们提出了Med-Art，该框架是专门为医学图像生成有限的数据而设计的框架。 Med-Art利用视觉模型来生成医学图像的视觉描述，从而克服了适用的医学文本数据的稀缺性。 Med-Art根据扩散变压器（DIT）改编大型预训练的文本对图像Pixart-$ \ alpha $，在有限的数据下实现了高性能。此外，我们提出了一种创新的混合级扩散微调（HLDF）方法，该方法可实现像素级损失，有效地解决了诸如过于饱和的颜色之类的问题。我们在FID，KID和下游分类性能衡量的两个医疗图像数据集上实现了最先进的性能。

Title: HiWave: Training-Free High-Resolution Image Generation via Wavelet-Based Diffusion Sampling

• Authors: Tobias Vontobel, Seyedmorteza Sadat, Farnood Salehi, Romann M. Weber
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.20452
• Pdf URL: https://arxiv.org/pdf/2506.20452
• Copy Paste: [[2506.20452]] HiWave: Training-Free High-Resolution Image Generation via Wavelet-Based Diffusion Sampling(https://arxiv.org/abs/2506.20452)
• Keywords: generation
• Abstract: Diffusion models have emerged as the leading approach for image synthesis, demonstrating exceptional photorealism and diversity. However, training diffusion models at high resolutions remains computationally prohibitive, and existing zero-shot generation techniques for synthesizing images beyond training resolutions often produce artifacts, including object duplication and spatial incoherence. In this paper, we introduce HiWave, a training-free, zero-shot approach that substantially enhances visual fidelity and structural coherence in ultra-high-resolution image synthesis using pretrained diffusion models. Our method employs a two-stage pipeline: generating a base image from the pretrained model followed by a patch-wise DDIM inversion step and a novel wavelet-based detail enhancer module. Specifically, we first utilize inversion methods to derive initial noise vectors that preserve global coherence from the base image. Subsequently, during sampling, our wavelet-domain detail enhancer retains low-frequency components from the base image to ensure structural consistency, while selectively guiding high-frequency components to enrich fine details and textures. Extensive evaluations using Stable Diffusion XL demonstrate that HiWave effectively mitigates common visual artifacts seen in prior methods, achieving superior perceptual quality. A user study confirmed HiWave's performance, where it was preferred over the state-of-the-art alternative in more than 80% of comparisons, highlighting its effectiveness for high-quality, ultra-high-resolution image synthesis without requiring retraining or architectural modifications.
• 摘要：扩散模型已成为图像合成的主要方法，证明了异常的光真实和多样性。然而，高分辨率下的训练扩散模型仍然在计算上令人难以置信，并且现有的零发产生技术用于合成图像以外的训练分辨率以外的图像通常会产生伪影，包括对象重复和空间不一致。在本文中，我们介绍了HiWave，这是一种无训练的零射击方法，可实质上提高了使用验证的扩散模型在超高分辨率图像合成中的视觉保真度和结构相干性。我们的方法采用了两阶段的管道：从预验证的模型中生成基本图像，然后是斑块的DDIM倒置步骤和基于新型小波的细节增强器模块。具体而言，我们首先使用反转方法来得出从基本图像中保留全局相干性的初始噪声向量。随后，在采样期间，我们的小波域细节增强子保留了基本图像中的低频组件，以确保结构一致性，同时选择性指导高频组件以丰富细节和纹理。使用稳定扩散XL进行的广泛评估表明，HiWave有效地减轻了先前方法中看到的常见视觉伪像，从而达到了卓越的感知质量。一项用户研究证实了HiWave的性能，在超过80％的比较中，它比最先进的替代方案更喜欢它，强调了其对高质量，超高分辨率图像合成的有效性而无需重新训练或建筑修改。

Title: Pay Less Attention to Deceptive Artifacts: Robust Detection of Compressed Deepfakes on Online Social Networks

• Authors: Manyi Li, Renshuai Tao, Yufan Liu, Chuangchuang Tan, Haotong Qin, Bing Li, Yunchao Wei, Yao Zhao
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2506.20548
• Pdf URL: https://arxiv.org/pdf/2506.20548
• Copy Paste: [[2506.20548]] Pay Less Attention to Deceptive Artifacts: Robust Detection of Compressed Deepfakes on Online Social Networks(https://arxiv.org/abs/2506.20548)
• Keywords: generative
• Abstract: With the rapid advancement of deep learning, particularly through generative adversarial networks (GANs) and diffusion models (DMs), AI-generated images, or ``deepfakes", have become nearly indistinguishable from real ones. These images are widely shared across Online Social Networks (OSNs), raising concerns about their misuse. Existing deepfake detection methods overlook the ``block effects" introduced by compression in OSNs, which obscure deepfake artifacts, and primarily focus on raw images, rarely encountered in real-world scenarios. To address these challenges, we propose PLADA (Pay Less Attention to Deceptive Artifacts), a novel framework designed to tackle the lack of paired data and the ineffective use of compressed images. PLADA consists of two core modules: Block Effect Eraser (B2E), which uses a dual-stage attention mechanism to handle block effects, and Open Data Aggregation (ODA), which processes both paired and unpaired data to improve detection. Extensive experiments across 26 datasets demonstrate that PLADA achieves a remarkable balance in deepfake detection, outperforming SoTA methods in detecting deepfakes on OSNs, even with limited paired data and compression. More importantly, this work introduces the ``block effect" as a critical factor in deepfake detection, providing a robust solution for open-world scenarios. Our code is available at this https URL.
• 摘要：随着深度学习的快速发展，尤其是通过生成性的对抗网络（GAN）和扩散模型（DMS），AI生成的图像或````深层''几乎可以与真实图像没有区别。这些图像在在线社交网络（OSN）之间被广泛共享，从而使他们的滥用范围造成了对现有的效果的关注。在现实世界中很少遇到的深层伪像，主要关注原始图像。为了应对这些挑战，我们提出了PLADA（较少注意欺骗性人工制品），这是一个新颖的框架，旨在解决缺乏配对数据和无效使用压缩图像的框架。 PLADA由两个核心模块组成：块效应橡皮擦（B2E），该模块使用双阶段注意机制来处理块效应和开放数据聚合（ODA），该数据均采用配对和未配对的数据来改善检测。在26个数据集上进行的广泛实验表明，PLADA在深层检测中达到了显着的平衡，即使在有限的配对数据和压缩的情况下，在检测OSN上的深击中的表现都优于SOTA方法。更重要的是，这项工作将``块效应''引入了DeepFake检测的关键因素，为开放世界方案提供了强大的解决方案。我们的代码可在此HTTPS URL上获得。

Title: Causal Representation Learning with Observational Grouping for CXR Classification

• Authors: Rajat Rasal, Avinash Kori, Ben Glocker
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.20582
• Pdf URL: https://arxiv.org/pdf/2506.20582
• Copy Paste: [[2506.20582]] Causal Representation Learning with Observational Grouping for CXR Classification(https://arxiv.org/abs/2506.20582)
• Keywords: generation
• Abstract: Identifiable causal representation learning seeks to uncover the true causal relationships underlying a data generation process. In medical imaging, this presents opportunities to improve the generalisability and robustness of task-specific latent features. This work introduces the concept of grouping observations to learn identifiable representations for disease classification in chest X-rays via an end-to-end framework. Our experiments demonstrate that these causal representations improve generalisability and robustness across multiple classification tasks when grouping is used to enforce invariance w.r.t race, sex, and imaging views.
• 摘要：可识别的因果表示学习旨在揭示数据生成过程的真正因果关系。在医学成像中，这为提高特定于任务的潜在特征的普遍性和鲁棒性提供了机会。这项工作介绍了通过端到端框架进行分组观察以学习胸部X射线疾病分类的可识别表示的概念。我们的实验表明，当使用分组来执行不变性W.R.T种族，性别和成像视图时，这些因果表示可以改善多个分类任务的普遍性和鲁棒性。

Title: WonderFree: Enhancing Novel View Quality and Cross-View Consistency for 3D Scene Exploration

• Authors: Chaojun Ni, Jie Li, Haoyun Li, Hengyu Liu, Xiaofeng Wang, Zheng Zhu, Guosheng Zhao, Boyuan Wang, Chenxin Li, Guan Huang, Wenjun Mei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.20590
• Pdf URL: https://arxiv.org/pdf/2506.20590
• Copy Paste: [[2506.20590]] WonderFree: Enhancing Novel View Quality and Cross-View Consistency for 3D Scene Exploration(https://arxiv.org/abs/2506.20590)
• Keywords: restoration, generation
• Abstract: Interactive 3D scene generation from a single image has gained significant attention due to its potential to create immersive virtual worlds. However, a key challenge in current 3D generation methods is the limited explorability, which cannot render high-quality images during larger maneuvers beyond the original viewpoint, particularly when attempting to move forward into unseen areas. To address this challenge, we propose WonderFree, the first model that enables users to interactively generate 3D worlds with the freedom to explore from arbitrary angles and directions. Specifically, we decouple this challenge into two key subproblems: novel view quality, which addresses visual artifacts and floating issues in novel views, and cross-view consistency, which ensures spatial consistency across different viewpoints. To enhance rendering quality in novel views, we introduce WorldRestorer, a data-driven video restoration model designed to eliminate floaters and artifacts. In addition, a data collection pipeline is presented to automatically gather training data for WorldRestorer, ensuring it can handle scenes with varying styles needed for 3D scene generation. Furthermore, to improve cross-view consistency, we propose ConsistView, a multi-view joint restoration mechanism that simultaneously restores multiple perspectives while maintaining spatiotemporal coherence. Experimental results demonstrate that WonderFree not only enhances rendering quality across diverse viewpoints but also significantly improves global coherence and consistency. These improvements are confirmed by CLIP-based metrics and a user study showing a 77.20% preference for WonderFree over WonderWorld enabling a seamless and immersive 3D exploration experience. The code, model, and data will be publicly available.
• 摘要：从单个图像中产生的交互式3D场景产生由于创造了身临其境虚拟世界的潜力而引起了重大关注。但是，当前3D生成方法中的一个主要挑战是有限的可探索性，在较大的操作中，它无法在最初的观点之外呈现高质量的图像，尤其是在尝试前进到看不见的区域时。为了应对这一挑战，我们提出了Wonderfree，这是第一个使用户可以自由地从任意角度和方向探索的3D世界的模型。具体来说，我们将这一挑战分解为两个关键的子问题：新颖的视图质量，它解决了新型视图中的视觉伪像和浮动问题，以及跨视图的一致性，确保了跨不同观点的空间一致性。为了提高新颖观点的渲染质量，我们介绍了WorldRestorer，这是一个数据驱动的视频恢复模型，旨在消除浮动器和人工制品。此外，还提出了数据收集管道，以自动为世界人的培训数据收集培训数据，以确保它可以通过3D场景生成所需的不同样式处理场景。此外，为了提高跨视图的一致性，我们提出了一种多视图的关节恢复机制，同时恢复了多个观点，同时保持时空相干性。实验结果表明，Wonderfree不仅可以提高各种观点的渲染质量，而且可以显着提高全球连贯性和一致性。这些改进是通过基于剪辑的指标和一项用户研究证实的，表明对Wonderfree而不是Wonderworld的偏好为77.20％，从而实现了无缝和沉浸式3D探索体验。代码，模型和数据将公开可用。

Title: SFNet: Fusion of Spatial and Frequency-Domain Features for Remote Sensing Image Forgery Detection

• Authors: Ji Qi, Xinchang Zhang, Dingqi Ye, Yongjia Ruan, Xin Guo, Shaowen Wang, Haifeng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.20599
• Pdf URL: https://arxiv.org/pdf/2506.20599
• Copy Paste: [[2506.20599]] SFNet: Fusion of Spatial and Frequency-Domain Features for Remote Sensing Image Forgery Detection(https://arxiv.org/abs/2506.20599)
• Keywords: generative
• Abstract: The rapid advancement of generative artificial intelligence is producing fake remote sensing imagery (RSI) that is increasingly difficult to detect, potentially leading to erroneous intelligence, fake news, and even conspiracy theories. Existing forgery detection methods typically rely on single visual features to capture predefined artifacts, such as spatial-domain cues to detect forged objects like roads or buildings in RSI, or frequency-domain features to identify artifacts from up-sampling operations in adversarial generative networks (GANs). However, the nature of artifacts can significantly differ depending on geographic terrain, land cover types, or specific features within the RSI. Moreover, these complex artifacts evolve as generative models become more sophisticated. In short, over-reliance on a single visual cue makes existing forgery detectors struggle to generalize across diverse remote sensing data. This paper proposed a novel forgery detection framework called SFNet, designed to identify fake images in diverse remote sensing data by leveraging spatial and frequency domain features. Specifically, to obtain rich and comprehensive visual information, SFNet employs two independent feature extractors to capture spatial and frequency domain features from input RSIs. To fully utilize the complementary domain features, the domain feature mapping module and the hybrid domain feature refinement module(CBAM attention) of SFNet are designed to successively align and fuse the multi-domain features while suppressing redundant information. Experiments on three datasets show that SFNet achieves an accuracy improvement of 4%-15.18% over the state-of-the-art RS forgery detection methods and exhibits robust generalization capabilities. The code is available at this https URL.
• 摘要：生成人工智能的迅速发展是生产越来越难以检测的伪造遥感图像（RSI），可能导致错误的情报，假新闻甚至阴谋理论。现有的伪造检测方法通常依靠单个视觉特征来捕获预定义的人工制品，例如空间域提示，以检测RSI中的道路或建筑物等伪造物体，或频率域特征来识别从对抗性生成网络（GANS）中向上采样操作（GANS）中的伪像。但是，根据地理地形，土地覆盖类型或RSI内的特定特征，人工制品的性质可能会显着差异。此外，随着生成模型变得更加复杂，这些复杂的伪像也会发展。简而言之，过度依赖单个视觉提示使现有的伪造探测器努力跨越各种遥感数据。本文提出了一个名为SFNET的新型伪造探测框架，旨在通过利用空间和频域特征来识别各种遥感数据中的假图像。具体而言，为了获得丰富而全面的视觉信息，SFNET采用了两个独立的功能提取器来捕获输入RSIS的空间和频域特征。为了充分利用互补的域特​​征，SFNET的域特征映射模块和SFNET的混合域特征改进模块（CBAM注意）旨在连续对齐和融合多域特征，同时抑制冗余信息。三个数据集的实验表明，SFNET的准确性提高了4％-15.18％的伪造方法，并且具有强大的概括能力。该代码可在此HTTPS URL上找到。

Title: Video Perception Models for 3D Scene Synthesis

• Authors: Rui Huang, Guangyao Zhai, Zuria Bauer, Marc Pollefeys, Federico Tombari, Leonidas Guibas, Gao Huang, Francis Engelmann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.20601
• Pdf URL: https://arxiv.org/pdf/2506.20601
• Copy Paste: [[2506.20601]] Video Perception Models for 3D Scene Synthesis(https://arxiv.org/abs/2506.20601)
• Keywords: generation
• Abstract: Traditionally, 3D scene synthesis requires expert knowledge and significant manual effort. Automating this process could greatly benefit fields such as architectural design, robotics simulation, virtual reality, and gaming. Recent approaches to 3D scene synthesis often rely on the commonsense reasoning of large language models (LLMs) or strong visual priors of modern image generation models. However, current LLMs demonstrate limited 3D spatial reasoning ability, which restricts their ability to generate realistic and coherent 3D scenes. Meanwhile, image generation-based methods often suffer from constraints in viewpoint selection and multi-view inconsistencies. In this work, we present Video Perception models for 3D Scene synthesis (VIPScene), a novel framework that exploits the encoded commonsense knowledge of the 3D physical world in video generation models to ensure coherent scene layouts and consistent object placements across views. VIPScene accepts both text and image prompts and seamlessly integrates video generation, feedforward 3D reconstruction, and open-vocabulary perception models to semantically and geometrically analyze each object in a scene. This enables flexible scene synthesis with high realism and structural consistency. For more precise analysis, we further introduce First-Person View Score (FPVScore) for coherence and plausibility evaluation, utilizing continuous first-person perspective to capitalize on the reasoning ability of multimodal large language models. Extensive experiments show that VIPScene significantly outperforms existing methods and generalizes well across diverse scenarios. The code will be released.
• 摘要：传统上，3D场景综合需要专家知识和大量的手动努力。自动化此过程可能会极大地有益于诸如建筑设计，机器人模拟，虚拟现实和游戏等领域。 3D场景合成的最新方法通常依赖于大语言模型（LLM）的常识性推理或现代图像生成模型的强烈视觉先验。但是，当前的LLM表现出有限的3D空间推理能力，这限制了它们产生现实和连贯的3D场景的能力。同时，基于图像生成的方法在观点选择和多视图不一致之下经常受到限制。在这项工作中，我们介绍了3D场景合成（VIPSCENE）的视频感知模型，该模型是一个新颖的框架，可利用视频生成模型中3D物理世界的编码常识知识，以确保相干场景布局和视图跨视图的一致对象放置。 VIPSCENE同时接受文本和图像提示，并无缝地集成了视频生成，FeedForward 3D重建以及开放式Vocabulary感知模型，以在语义上和几何分析场景中的每个对象。这使得具有高现实主义和结构一致性的灵活场景综合。为了进行更精确的分析，我们进一步介绍了第一人称视图分数（FPVSCORE），以进行连贯性和合理性评估，并利用连续的第一人称视角来利用多模式大语言模型的推理能力。广泛的实验表明，乙烯在各种情况下都显着胜过现有方法，并且可以很好地概括。代码将发布。

Title: Shape2Animal: Creative Animal Generation from Natural Silhouettes

• Authors: Quoc-Duy Tran, Anh-Tuan Vo, Dinh-Khoi Vo, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.20616
• Pdf URL: https://arxiv.org/pdf/2506.20616
• Copy Paste: [[2506.20616]] Shape2Animal: Creative Animal Generation from Natural Silhouettes(https://arxiv.org/abs/2506.20616)
• Keywords: generation
• Abstract: Humans possess a unique ability to perceive meaningful patterns in ambiguous stimuli, a cognitive phenomenon known as pareidolia. This paper introduces Shape2Animal framework to mimics this imaginative capacity by reinterpreting natural object silhouettes, such as clouds, stones, or flames, as plausible animal forms. Our automated framework first performs open-vocabulary segmentation to extract object silhouette and interprets semantically appropriate animal concepts using vision-language models. It then synthesizes an animal image that conforms to the input shape, leveraging text-to-image diffusion model and seamlessly blends it into the original scene to generate visually coherent and spatially consistent compositions. We evaluated Shape2Animal on a diverse set of real-world inputs, demonstrating its robustness and creative potential. Our Shape2Animal can offer new opportunities for visual storytelling, educational content, digital art, and interactive media design. Our project page is here: this https URL
• 摘要：人类具有独特的能力，可以在歧义刺激中感知有意义的模式，这是一种被称为pareidolia的认知现象。本文将Shape2animal框架引入了通过重新诠释自然物体轮廓（例如云，石头或火焰）作为合理的动物形式来模仿这种想象力的能力。我们的自动框架首先执行开放式摄影片分段，以提取对象轮廓并使用视觉模型来解释语义上适当的动物概念。然后，它综合了一个符合输入形状的动物图像，利用文本对图扩散模型，并将其无缝融合到原始场景中，以生成视觉上相干和空间一致的组成。我们评估了Shape2animal对各种现实世界的投入，表明了它的稳健性和创造力。我们的Shape2animal可以为视觉讲故事，教育内容，数字艺术和交互式媒体设计提供新的机会。我们的项目页面在这里：此HTTPS URL

Title: Mastering Multiple-Expert Routing: Realizable $H$-Consistency and Strong Guarantees for Learning to Defer

• Authors: Anqi Mao, Mehryar Mohri, Yutao Zhong
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2506.20650
• Pdf URL: https://arxiv.org/pdf/2506.20650
• Copy Paste: [[2506.20650]] Mastering Multiple-Expert Routing: Realizable $H$-Consistency and Strong Guarantees for Learning to Defer(https://arxiv.org/abs/2506.20650)
• Keywords: generation
• Abstract: The problem of learning to defer with multiple experts consists of optimally assigning input instances to experts, balancing the trade-off between their accuracy and computational cost. This is a critical challenge in natural language generation, but also in other fields such as image processing, and medical diagnostics. Recent studies have proposed surrogate loss functions to optimize deferral, but challenges remain in ensuring their consistency properties. This paper introduces novel surrogate loss functions and efficient algorithms with strong theoretical learning guarantees. We address open questions regarding realizable $H$-consistency, $H$-consistency bounds, and Bayes-consistency for both single-stage (jointly learning predictor and deferral function) and two-stage (learning only the deferral function with a fixed expert) learning scenarios. For single-stage deferral, we introduce a family of new realizable $H$-consistent surrogate losses and further prove $H$-consistency for a selected member. For two-stage deferral, we derive new surrogate losses that achieve realizable $H$-consistency, $H$-consistency bounds, and Bayes-consistency for the two-expert scenario and, under natural assumptions, multiple-expert scenario. Additionally, we provide enhanced theoretical guarantees under low-noise assumptions for both scenarios. Finally, we report the results of experiments using our proposed surrogate losses, comparing their performance against existing baselines.
• 摘要：学会推迟多位专家的问题包括将输入实例分配给专家，并在其准确性和计算成本之间取舍。这是自然语言生成的关键挑战，也是图像处理和医学诊断等其他领域。最近的研究提出了替代损失功能以优化延期，但在确保其一致性特性方面仍然存在挑战。本文介绍了具有强大理论学习保证的新型替代损失功能和有效的算法。我们解决了有关可实现的$ h $ - 一致性，$ h $ - 固定范围以及单阶段（共同学习预测指标和延期功能）和两个阶段（仅学习固定专家的延期功能）学习方案的两个阶段（仅学习延期功能）学习方案。对于单阶段的延期，我们介绍了一个新的可实现的$ h $ b $ cosistent替代损失的家族，并进一步证明了$ h $ and-h $ - 一致性。对于两阶段的延期，我们得出了新的替代损失，这些损失可实现可实现的$ h $ - 一致性，$ h $ - 一致性的界限以及贝叶斯的两者持续性，以实现两种表现的场景，并且在自然假设下，多重杂制的情况。此外，我们在低噪声假设下为两种情况提供了增强的理论保证。最后，我们报告了使用我们提出的替代损失的实验结果，并将其绩效与现有基准进行比较。

Title: MMSearch-R1: Incentivizing LMMs to Search

• Authors: Jinming Wu, Zihao Deng, Wei Li, Yiding Liu, Bo You, Bo Li, Zejun Ma, Ziwei Liu
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2506.20670
• Pdf URL: https://arxiv.org/pdf/2506.20670
• Copy Paste: [[2506.20670]] MMSearch-R1: Incentivizing LMMs to Search(https://arxiv.org/abs/2506.20670)
• Keywords: generation
• Abstract: Robust deployment of large multimodal models (LMMs) in real-world scenarios requires access to external knowledge sources, given the complexity and dynamic nature of real-world information. Existing approaches such as retrieval-augmented generation (RAG) and prompt engineered search agents rely on rigid pipelines, often leading to inefficient or excessive search behaviors. We present MMSearch-R1, the first end-to-end reinforcement learning framework that enables LMMs to perform on-demand, multi-turn search in real-world Internet environments. Our framework integrates both image and text search tools, allowing the model to reason about when and how to invoke them guided by an outcome-based reward with a search penalty. To support training, We collect a multimodal search VQA dataset through a semi-automated pipeline that covers diverse visual and textual knowledge needs and curate a search-balanced subset with both search-required and search-free samples, which proves essential for shaping efficient and on-demand search behavior. Extensive experiments on knowledge-intensive and info-seeking VQA tasks show that our model not only outperforms RAG-based baselines of the same model size, but also matches the performance of a larger RAG-based model while reducing search calls by over 30%. We further analyze key empirical findings to offer actionable insights for advancing research in multimodal search.
• 摘要：考虑到现实世界中信息的复杂性和动态性质，在现实世界情景中，大型多模型（LMM）的强大部署需要访问外部知识来源。现有的方法，例如检索功能的生成（RAG）和迅速设计的搜索剂，依赖于严格的管道，通常导致搜索行为效率低下或过度。我们提出了MMSEarch-R1，这是第一个端到端的增强学习框架，使LMM可以在现实世界中的Internet环境中执行按需进行多转弯搜索。我们的框架同时集成了图像和文本搜索工具，允许模型推理何时以及如何以基于结果的奖励和搜索惩罚引导它们。为了支持培训，我们通过半自动化的管道收集多模式搜索VQA数据集，该管道涵盖了各种视觉和文本知识需求，并使用搜索平衡的子集使用搜索要求和搜索的示例来策划搜索平衡的子集，这对于塑造有效和按需搜索行为而言，这对于塑造有效的和按需搜索行为至关重要。关于知识密集型和寻求信息的VQA任务的广泛实验表明，我们的模型不仅超过了基于抹布的基线相同型号大小，而且还与较大的基于抹布的模型的性能相匹配，同时将搜索调用降低30％以上。我们进一步分析了关键的经验发现，以提供可行的见解，以推进多模式搜索的研究。