2026-01-23

Title: Empowering LLMs for Structure-Based Drug Design via Exploration-Augmented Latent Inference

• Authors: Xuanning Hu, Anchen Li, Qianli Xing, Jinglong Ji, Hao Tuo, Bo Yang
• Subjects: cs.LG, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2601.15333
• Pdf URL: https://arxiv.org/pdf/2601.15333
• Copy Paste: [[2601.15333]] Empowering LLMs for Structure-Based Drug Design via Exploration-Augmented Latent Inference(https://arxiv.org/abs/2601.15333)
• Keywords: generation
• Abstract: Large Language Models (LLMs) possess strong representation and reasoning capabilities, but their application to structure-based drug design (SBDD) is limited by insufficient understanding of protein structures and unpredictable molecular generation. To address these challenges, we propose Exploration-Augmented Latent Inference for LLMs (ELILLM), a framework that reinterprets the LLM generation process as an encoding, latent space exploration, and decoding workflow. ELILLM explicitly explores portions of the design problem beyond the model's current knowledge while using a decoding module to handle familiar regions, generating chemically valid and synthetically reasonable molecules. In our implementation, Bayesian optimization guides the systematic exploration of latent embeddings, and a position-aware surrogate model efficiently predicts binding affinity distributions to inform the search. Knowledge-guided decoding further reduces randomness and effectively imposes chemical validity constraints. We demonstrate ELILLM on the CrossDocked2020 benchmark, showing strong controlled exploration and high binding affinity scores compared with seven baseline methods. These results demonstrate that ELILLM can effectively enhance LLMs capabilities for SBDD.
• 摘要：大型语言模型（LLM）具有强大的表示和推理能力，但其在基于结构的药物设计（SBDD）中的应用受到对蛋白质结构理解不足和不可预测的分子生成的限制。为了应对这些挑战，我们提出了 LLM 探索增强潜在推理 (ELILLM)，这是一个将 LLM 生成过程重新解释为编码、潜在空间探索和解码工作流程的框架。 ELILLM 明确探索了模型当前知识之外的部分设计问题，同时使用解码模块处理熟悉的区域，生成化学上有效且合成上合理的分子。在我们的实现中，贝叶斯优化指导对潜在嵌入的系统探索，并且位置感知代理模型有效地预测结合亲和力分布以通知搜索。知识引导解码进一步减少随机性并有效地施加化学有效性约束。我们在 CrossDocked2020 基准测试上展示了 ELILLM，与七种基线方法相比，显示出强大的受控探索和高结合亲和力得分。这些结果表明，ELILLM 可以有效增强法学硕士的 SBDD 能力。

Title: FedUMM: A General Framework for Federated Learning with Unified Multimodal Models

• Authors: Zhaolong Su, Leheng Zhao, Xiaoying Wu, Ziyue Xu, Jindong Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2601.15390
• Pdf URL: https://arxiv.org/pdf/2601.15390
• Copy Paste: [[2601.15390]] FedUMM: A General Framework for Federated Learning with Unified Multimodal Models(https://arxiv.org/abs/2601.15390)
• Keywords: generation
• Abstract: Unified multimodal models (UMMs) are emerging as strong foundation models that can do both generation and understanding tasks in a single architecture. However, they are typically trained in centralized settings where all training and downstream datasets are gathered in a central server, limiting the deployment in privacy-sensitive and geographically distributed scenarios. In this paper, we present FedUMM, a general federated learning framework for UMMs under non-IID multimodal data with low communication cost. Built on NVIDIA FLARE, FedUMM instantiates federation for a BLIP3o backbone via parameter-efficient fine-tuning: clients train lightweight LoRA adapters while freezing the foundation models, and the server aggregates only adapter updates. We evaluate on VQA v2 and the GenEval compositional generation benchmarks under Dirichlet-controlled heterogeneity with up to 16 clients. Results show slight degradation as client count and heterogeneity increase, while remaining competitive with centralized training. We further analyze computation--communication trade-offs and demonstrate that adapter-only federation reduces per-round communication by over an order of magnitude compared to full fine-tuning, enabling practical federated UMM training. This work provides empirical experience for future research on privacy-preserving federated unified multimodal models.
• 摘要：统一多模态模型 (UMM) 正在成为强大的基础模型，可以在单一架构中完成生成和理解任务。然而，它们通常在集中式设置中进行训练，其中所有训练和下游数据集都收集在中央服务器中，从而限制了在隐私敏感和地理分布式场景中的部署。在本文中，我们提出了 FedUMM，一种用于非 IID 多模态数据下 UMM 的通用联邦学习框架，具有低通信成本。 FedUMM 基于 NVIDIA FLARE 构建，通过参数高效的微调实例化 BLIP3o 主干网的联合：客户端在冻结基础模型的同时训练轻量级 LoRA 适配器，而服务器仅聚合适配器更新。我们在 Dirichlet 控制的异质性下（最多 16 个客户端）对 VQA v2 和 GenEval 组合生成基准进行评估。结果显示，随着客户数量和异质性的增加，效果略有下降，同时与集中培训保持竞争力。我们进一步分析计算-通信权衡，并证明与完全微调相比，仅适配器联合将每轮通信减少了一个数量级以上，从而实现了实用的联合 UMM 训练。这项工作为未来隐私保护联合统一多模态模型的研究提供了经验经验。

Title: CURE: Curriculum-guided Multi-task Training for Reliable Anatomy Grounded Report Generation

• Authors: Pablo Messina, Andrés Villa, Juan León Alcázar, Karen Sánchez, Carlos Hinojosa, Denis Parra, Álvaro Soto, Bernard Ghanem
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2601.15408
• Pdf URL: https://arxiv.org/pdf/2601.15408
• Copy Paste: [[2601.15408]] CURE: Curriculum-guided Multi-task Training for Reliable Anatomy Grounded Report Generation(https://arxiv.org/abs/2601.15408)
• Keywords: generation
• Abstract: Medical vision-language models can automate the generation of radiology reports but struggle with accurate visual grounding and factual consistency. Existing models often misalign textual findings with visual evidence, leading to unreliable or weakly grounded predictions. We present CURE, an error-aware curriculum learning framework that improves grounding and report quality without any additional data. CURE fine-tunes a multimodal instructional model on phrase grounding, grounded report generation, and anatomy-grounded report generation using public datasets. The method dynamically adjusts sampling based on model performance, emphasizing harder samples to improve spatial and textual alignment. CURE improves grounding accuracy by +0.37 IoU, boosts report quality by +0.188 CXRFEScore, and reduces hallucinations by 18.6%. CURE is a data-efficient framework that enhances both grounding accuracy and report reliability. Code is available at this https URL and model weights at this https URL
• 摘要：医学视觉语言模型可以自动生成放射学报告，但难以实现准确的视觉基础和事实一致性。现有模型经常将文本发现与视觉证据不一致，导致预测不可靠或缺乏依据。我们推出了 CURE，一种错误感知课程学习框架，无需任何额外数据即可提高基础知识和报告质量。 CURE 使用公共数据集对短语基础、基础报告生成和解剖学基础报告生成的多模式教学模型进行微调。该方法根据模型性能动态调整采样，强调较难的样本以改善空间和文本对齐。 CURE 将接地精度提高了 +0.37 IoU，将报告质量提高了 +0.188 CXRFEScore，并将幻觉减少了 18.6%。 CURE 是一个数据高效的框架，可提高接地准确性和报告可靠性。代码可在此 https URL 获取，模型权重可在此 https URL 获取

Title: Ambient Dataloops: Generative Models for Dataset Refinement

• Authors: Adrián Rodríguez-Muñoz, William Daspit, Adam Klivans, Antonio Torralba, Constantinos Daskalakis, Giannis Daras
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2601.15417
• Pdf URL: https://arxiv.org/pdf/2601.15417
• Copy Paste: [[2601.15417]] Ambient Dataloops: Generative Models for Dataset Refinement(https://arxiv.org/abs/2601.15417)
• Keywords: generation, generative
• Abstract: We propose Ambient Dataloops, an iterative framework for refining datasets that makes it easier for diffusion models to learn the underlying data distribution. Modern datasets contain samples of highly varying quality, and training directly on such heterogeneous data often yields suboptimal models. We propose a dataset-model co-evolution process; at each iteration of our method, the dataset becomes progressively higher quality, and the model improves accordingly. To avoid destructive self-consuming loops, at each generation, we treat the synthetically improved samples as noisy, but at a slightly lower noisy level than the previous iteration, and we use Ambient Diffusion techniques for learning under corruption. Empirically, Ambient Dataloops achieve state-of-the-art performance in unconditional and text-conditional image generation and de novo protein design. We further provide a theoretical justification for the proposed framework that captures the benefits of the data looping procedure.
• 摘要：我们提出了 Ambient Dataloops，这是一种用于细化数据集的迭代框架，使扩散模型更容易学习底层数据分布。现代数据集包含质量差异很大的样本，直接对此类异构数据进行训练通常会产生次优模型。我们提出了数据集模型协同进化过程；在我们方法的每次迭代中，数据集的质量逐渐提高，模型也相应改进。为了避免破坏性的自消耗循环，在每一代，我们将综合改进的样本视为有噪声的，但噪声水平比上一次迭代略低，并且我们使用环境扩散技术在腐败下进行学习。根据经验，Ambient Dataloops 在无条件和文本条件图像生成以及从头蛋白质设计方面实现了最先进的性能。我们进一步为所提出的框架提供了理论依据，该框架体现了数据循环过程的好处。

Title: Learning from Synthetic Data: Limitations of ERM

• Authors: Kareem Amin, Alex Bie, Weiwei Kong, Umar Syed, Sergei Vassilvitskii
• Subjects: cs.LG, cs.DS, stat.ML
• Abstract URL: https://arxiv.org/abs/2601.15468
• Pdf URL: https://arxiv.org/pdf/2601.15468
• Copy Paste: [[2601.15468]] Learning from Synthetic Data: Limitations of ERM(https://arxiv.org/abs/2601.15468)
• Keywords: generation
• Abstract: The prevalence and low cost of LLMs have led to a rise of synthetic content. From review sites to court documents, ``natural'' content has been contaminated by data points that appear similar to natural data, but are in fact LLM-generated. In this work we revisit fundamental learning theory questions in this, now ubiquitous, setting. We model this scenario as a sequence of learning tasks where the input is a mix of natural and synthetic data, and the learning algorithms are oblivious to the origin of any individual example. We study the possibilities and limitations of ERM in this setting. For the problem of estimating the mean of an arbitrary $d$-dimensional distribution, we find that while ERM converges to the true mean, it is outperformed by an algorithm that assigns non-uniform weights to examples from different generations of data. For the PAC learning setting, the disparity is even more stark. We find that ERM does not always converge to the true concept, echoing the model collapse literature. However, we show there are algorithms capable of learning the correct hypothesis for arbitrary VC classes and arbitrary amounts of contamination.
• 摘要：法学硕士的普及和低成本导致了合成内容的增加。从评论网站到法庭文件，“自然”内容已被看似与自然数据相似但实际上是法学硕士生成的数据点所污染。在这项工作中，我们在这个现在无处不在的环境中重新审视基本的学习理论问题。我们将此场景建模为一系列学习任务，其中输入是自然数据和合成数据的混合，并且学习算法不考虑任何单个示例的起源。我们研究了 ERM 在这种情况下的可能性和局限性。对于估计任意 $d$ 维分布的均值问题，我们发现虽然 ERM 收敛于真实均值，但为来自不同代数据的示例分配非均匀权重的算法优于它。对于 PAC 学习环境来说，差距更加明显。我们发现 ERM 并不总是收敛于真正的概念，这与模型崩溃文献相呼应。然而，我们证明有一些算法能够学习任意 VC 类和任意污染量的正确假设。

Title: Martingale Foresight Sampling: A Principled Approach to Inference-Time LLM Decoding

• Authors: Huayu Li, ZhengXiao He, Siyuan Tian, Jinghao Wen, Ao Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2601.15482
• Pdf URL: https://arxiv.org/pdf/2601.15482
• Copy Paste: [[2601.15482]] Martingale Foresight Sampling: A Principled Approach to Inference-Time LLM Decoding(https://arxiv.org/abs/2601.15482)
• Keywords: generation
• Abstract: Standard autoregressive decoding in large language models (LLMs) is inherently short-sighted, often failing to find globally optimal reasoning paths due to its token-by-token generation process. While inference-time strategies like foresight sampling attempt to mitigate this by simulating future steps, they typically rely on ad-hoc heuristics for valuing paths and pruning the search space. This paper introduces Martingale Foresight Sampling (MFS), a principled framework that reformulates LLM decoding as a problem of identifying an optimal stochastic process. By modeling the quality of a reasoning path as a stochastic process, we leverage Martingale theory to design a theoretically-grounded algorithm. Our approach replaces heuristic mechanisms with principles from probability theory: step valuation is derived from the Doob Decomposition Theorem to measure a path's predictable advantage, path selection uses Optional Stopping Theory for principled pruning of suboptimal candidates, and an adaptive stopping rule based on the Martingale Convergence Theorem terminates exploration once a path's quality has provably converged. Experiments on six reasoning benchmarks demonstrate that MFS surpasses state-of-the-art methods in accuracy while significantly improving computational efficiency. Code will be released at this https URL.
• 摘要：大型语言模型（LLM）中的标准自回归解码本质上是短视的，由于其逐个令牌的生成过程，通常无法找到全局最佳推理路径。虽然诸如前瞻采样之类的推理时间策略试图通过模拟未来步骤来缓解这种情况，但它们通常依赖于临时启发式来评估路径和修剪搜索空间。本文介绍了 Martingale Foresight Sampling (MFS)，这是一个原则框架，它将 LLM 解码重新表述为识别最佳随机过程的问题。通过将推理路径的质量建模为随机过程，我们利用 Martingale 理论来设计基于理论的算法。我们的方法用概率论原理取代启发式机制：步长评估源自杜布分解定理，用于衡量路径的可预测优势，路径选择使用可选停止理论对次优候选进行有原则的修剪，一旦路径质量可证明收敛，基于鞅收敛定理的自适应停止规则就会终止探索。六个推理基准的实验表明，MFS 在准确性方面超越了最先进的方法，同时显着提高了计算效率。代码将在此 https URL 发布。

Title: MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification

• Authors: Jingwei Song, Xinyu Wang, Hanbin Wang, Xiaoxuan Lei, Bill Shi, Shixin Han, Eric Yang, Xiao-Wen Chang, Lynn Ai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2601.15498
• Pdf URL: https://arxiv.org/pdf/2601.15498
• Copy Paste: [[2601.15498]] MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification(https://arxiv.org/abs/2601.15498)
• Keywords: generation
• Abstract: Speculative Decoding (SD) accelerates autoregressive large language model (LLM) inference by decoupling generation and verification. While recent methods improve draft quality by tightly coupling the drafter with the target model, the verification mechanism itself remains largely unchanged, relying on strict token-level rejection sampling. In practice, modern LLMs frequently operate in low-margin regimes where the target model exhibits weak preference among top candidates. In such cases, rejecting plausible runner-up tokens yields negligible information gain while incurring substantial rollback cost, leading to a fundamental inefficiency in verification. We propose Margin-Aware Speculative Verification, a training-free and domain-agnostic verification strategy that adapts to the target model's local decisiveness. Our method conditions verification on decision stability measured directly from the target logits and relaxes rejection only when strict verification provides minimal benefit. Importantly, the approach modifies only the verification rule and is fully compatible with existing target-coupled speculative decoding frameworks. Extensive experiments across model scales ranging from 8B to 235B demonstrate that our method delivers consistent and significant inference speedups over state-of-the-art baselines while preserving generation quality across diverse benchmarks.
• 摘要：推测解码 (SD) 通过解耦生成和验证来加速自回归大语言模型 (LLM) 推理。虽然最近的方法通过将起草者与目标模型紧密耦合来提高草案质量，但验证机制本身基本上保持不变，依赖于严格的令牌级拒绝采样。在实践中，现代法学硕士经常在低利润的情况下运作，其中目标模型在顶级候选人中表现出较弱的偏好。在这种情况下，拒绝看似合理的亚军代币所产生的信息增益可以忽略不计，同时会产生大量的回滚成本，从而导致验证效率低下。我们提出了 Margin-Aware Speculative Verification，这是一种无需训练且与领域无关的验证策略，可以适应目标模型的局部决定性。我们的方法对直接从目标逻辑测量的决策稳定性进行验证，并且仅当严格验证提供最小收益时才放宽拒绝。重要的是，该方法仅修改验证规则，并且与现有的目标耦合推测解码框架完全兼容。从 8B 到 235B 的模型规模的广泛实验表明，我们的方法在最先进的基线上提供了一致且显着的推理加速，同时在不同的基准上保持了生成质量。

Title: Controllable Layered Image Generation for Real-World Editing

• Authors: Jinrui Yang, Qing Liu, Yijun Li, Mengwei Ren, Letian Zhang, Zhe Lin, Cihang Xie, Yuyin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.15507
• Pdf URL: https://arxiv.org/pdf/2601.15507
• Copy Paste: [[2601.15507]] Controllable Layered Image Generation for Real-World Editing(https://arxiv.org/abs/2601.15507)
• Keywords: generation
• Abstract: Recent image generation models have shown impressive progress, yet they often struggle to yield controllable and consistent results when users attempt to edit specific elements within an existing image. Layered representations enable flexible, user-driven content creation, but existing approaches often fail to produce layers with coherent compositing relationships, and their object layers typically lack realistic visual effects such as shadows and reflections. To overcome these limitations, we propose LASAGNA, a novel, unified framework that generates an image jointly with its composing layers--a photorealistic background and a high-quality transparent foreground with compelling visual effects. Unlike prior work, LASAGNA efficiently learns correct image composition from a wide range of conditioning inputs--text prompts, foreground, background, and location masks--offering greater controllability for real-world applications. To enable this, we introduce LASAGNA-48K, a new dataset composed of clean backgrounds and RGBA foregrounds with physically grounded visual effects. We also propose LASAGNABENCH, the first benchmark for layer editing. We demonstrate that LASAGNA excels in generating highly consistent and coherent results across multiple image layers simultaneously, enabling diverse post-editing applications that accurately preserve identity and visual effects. LASAGNA-48K and LASAGNABENCH will be publicly released to foster open research in the community. The project page is this https URL.
• 摘要：最近的图像生成模型已经显示出令人印象深刻的进步，但当用户尝试编辑现有图像中的特定元素时，它们通常难以产生可控且一致的结果。分层表示可以实现灵活的、用户驱动的内容创建，但现有方法通常无法生成具有连贯合成关系的图层，并且它们的对象图层通常缺乏真实的视觉效果，例如阴影和反射。为了克服这些限制，我们提出了 LASAGNA，这是一种新颖的、统一的框架，可以与其合成层联合生成图像——逼真的背景和具有引人注目的视觉效果的高质量透明前景。与之前的工作不同，LASAGNA 可以从各种条件输入（文本提示、前景、背景和位置蒙版）中有效地学习正确的图像构成，为现实世界的应用提供更大的可控性。为了实现这一点，我们引入了 LASAGNA-48K，这是一个由干净背景和 RGBA 前景组成的新数据集，具有物理基础的视觉效果。我们还提出了 LASAGNABENCH，这是图层编辑的第一个基准。我们证明，LASAGNA 擅长同时跨多个图像层生成高度一致和连贯的结果，从而支持准确保留身份和视觉效果的各种后期编辑应用程序。 LASAGNA-48K 和 LASAGNABENCH 将公开发布，以促进社区的开放研究。项目页面就是这个 https URL。

Title: Learning Neural Operators from Partial Observations via Latent Autoregressive Modeling

• Authors: Jingren Hou, Hong Wang, Pengyu Xu, Chang Gao, Huafeng Liu, Liping Jing
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2601.15547
• Pdf URL: https://arxiv.org/pdf/2601.15547
• Copy Paste: [[2601.15547]] Learning Neural Operators from Partial Observations via Latent Autoregressive Modeling(https://arxiv.org/abs/2601.15547)
• Keywords: generation
• Abstract: Real-world scientific applications frequently encounter incomplete observational data due to sensor limitations, geographic constraints, or measurement costs. Although neural operators significantly advanced PDE solving in terms of computational efficiency and accuracy, their underlying assumption of fully-observed spatial inputs severely restricts applicability in real-world applications. We introduce the first systematic framework for learning neural operators from partial observation. We identify and formalize two fundamental obstacles: (i) the supervision gap in unobserved regions that prevents effective learning of physical correlations, and (ii) the dynamic spatial mismatch between incomplete inputs and complete solution fields. Specifically, our proposed Latent Autoregressive Neural Operator~(\ours) introduces two novel components designed explicitly to address the core difficulties of partial observations: (i) a mask-to-predict training strategy that creates artificial supervision by strategically masking observed regions, and (ii) a Physics-Aware Latent Propagator that reconstructs solutions through boundary-first autoregressive generation in latent space. Additionally, we develop POBench-PDE, a dedicated and comprehensive benchmark designed specifically for evaluating neural operators under partial observation conditions across three PDE-governed tasks. \ours achieves state-of-the-art performance with 18--69$\%$ relative L2 error reduction across all benchmarks under patch-wise missingness with less than 50$\%$ missing rate, including real-world climate prediction. Our approach effectively addresses practical scenarios involving up to 75$\%$ missing rate, to some extent bridging the existing gap between idealized research settings and the complexities of real-world scientific computing.
• 摘要：由于传感器限制、地理限制或测量成本，现实世界的科学应用经常遇到不完整的观测数据。尽管神经算子在计算效率和准确性方面显着提高了偏微分方程求解的性能，但它们对完全观测的空间输入的基本假设严重限制了其在实际应用中的适用性。我们介绍了第一个从部分观察中学习神经算子的系统框架。我们识别并形式化了两个基本障碍：（i）未观察区域中的监督差距阻碍了物理相关性的有效学习，以及（ii）不完整输入和完整解域之间的动态空间不匹配。具体来说，我们提出的潜在自回归神经算子〜（\ours）引入了两个专门设计的新颖组件，以解决部分观察的核心困难：（i）掩模预测训练策略，通过策略性地屏蔽观察区域来创建人工监督，以及（ii）物理感知潜在传播器，通过潜在空间中的边界优先自回归生成来重建解决方案。此外，我们还开发了 POBench-PDE，这是一个专用且全面的基准测试，专门用于在三个 PDE 管理任务的部分观察条件下评​​估神经算子。我们的系统在补丁缺失率下的所有基准上实现了最先进的性能，相对 L2 误差减少了 18--69$\%$，缺失率低于 50$\%$，包括现实世界的气候预测。我们的方法有效地解决了涉及高达 75$\%$ 缺失率的实际场景，在某种程度上弥补了理想化研究环境与现实世界科学计算复杂性之间的现有差距。

Title: Relative Classification Accuracy: A Calibrated Metric for Identity Consistency in Fine-Grained K-pop Face Generation

• Authors: Sylvey Lin, Eranki Vasistha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.15560
• Pdf URL: https://arxiv.org/pdf/2601.15560
• Copy Paste: [[2601.15560]] Relative Classification Accuracy: A Calibrated Metric for Identity Consistency in Fine-Grained K-pop Face Generation(https://arxiv.org/abs/2601.15560)
• Keywords: generation, generative
• Abstract: Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in high-fidelity image generation. However, evaluating their semantic controllability-specifically for fine-grained, single-domain tasks-remains challenging. Standard metrics like FID and Inception Score (IS) often fail to detect identity misalignment in such specialized contexts. In this work, we investigate Class-Conditional DDPMs for K-pop idol face generation (32x32), a domain characterized by high inter-class similarity. We propose a calibrated metric, Relative Classification Accuracy (RCA), which normalizes generative performance against an oracle classifier's baseline. Our evaluation reveals a critical trade-off: while the model achieves high visual quality (FID 8.93), it suffers from severe semantic mode collapse (RCA 0.27), particularly for visually ambiguous identities. We analyze these failure modes through confusion matrices and attribute them to resolution constraints and intra-gender ambiguity. Our framework provides a rigorous standard for verifying identity consistency in conditional generative models.
• 摘要：去噪扩散概率模型（DDPM）在高保真图像生成方面取得了显着的成功。然而，评估它们的语义可控性（特别是针对细粒度、单域任务）仍然具有挑战性。 FID 和初始分数 (IS) 等标准指标通常无法检测此类专门环境中的身份错位。在这项工作中，我们研究了用于 K-pop 偶像脸部生成 (32x32) 的类条件 DDPM，这是一个以类间高度相似性为特征的领域。我们提出了一个校准指标，即相对分类精度（RCA），它根据预言机分类器的基线标准化生成性能。我们的评估揭示了一个关键的权衡：虽然该模型实现了高视觉质量（FID 8.93），但它遭受了严重的语义模式崩溃（RCA 0.27），特别是对于视觉上模糊的身份。我们通过混淆矩阵分析这些失败模式，并将其归因于分辨率限制和性别内部模糊性。我们的框架为验证条件生成模型中的身份一致性提供了严格的标准。

Title: Explainable Deepfake Detection with RL Enhanced Self-Blended Images

• Authors: Ning Jiang, Dingheng Zeng, Yanhong Liu, Haiyang Yi, Shijie Yu, Minghe Weng, Haifeng Shen, Ying Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.15624
• Pdf URL: https://arxiv.org/pdf/2601.15624
• Copy Paste: [[2601.15624]] Explainable Deepfake Detection with RL Enhanced Self-Blended Images(https://arxiv.org/abs/2601.15624)
• Keywords: generation
• Abstract: Most prior deepfake detection methods lack explainable outputs. With the growing interest in multimodal large language models (MLLMs), researchers have started exploring their use in interpretable deepfake detection. However, a major obstacle in applying MLLMs to this task is the scarcity of high-quality datasets with detailed forgery attribution annotations, as textual annotation is both costly and challenging - particularly for high-fidelity forged images or videos. Moreover, multiple studies have shown that reinforcement learning (RL) can substantially enhance performance in visual tasks, especially in improving cross-domain generalization. To facilitate the adoption of mainstream MLLM frameworks in deepfake detection with reduced annotation cost, and to investigate the potential of RL in this context, we propose an automated Chain-of-Thought (CoT) data generation framework based on Self-Blended Images, along with an RL-enhanced deepfake detection framework. Extensive experiments validate the effectiveness of our CoT data construction pipeline, tailored reward mechanism, and feedback-driven synthetic data generation approach. Our method achieves performance competitive with state-of-the-art (SOTA) approaches across multiple cross-dataset benchmarks. Implementation details are available at this https URL.
• 摘要：大多数现有的深度伪造检测方法缺乏可解释的输出。随着人们对多模态大语言模型（MLLM）的兴趣日益浓厚，研究人员开始探索它们在可解释的深度伪造检测中的应用。然而，将 MLLM 应用于此任务的一个主要障碍是缺乏具有详细伪造属性注释的高质量数据集，因为文本注释既昂贵又具有挑战性 - 特别是对于高保真伪造图像或视频。此外，多项研究表明强化学习（RL）可以显着提高视觉任务的表现，特别是在提高跨领域泛化能力方面。为了促进在 Deepfake 检测中采用主流 MLLM 框架并降低注释成本，并研究 RL 在这方面的潜力，我们提出了一种基于自混合图像的自动化思想链（CoT）数据生成框架，以及 RL 增强的 Deepfake 检测框架。大量的实验验证了我们的 CoT 数据构建管道、定制奖励机制和反馈驱动的合成数据生成方法的有效性。我们的方法在多个跨数据集基准测试中实现了与最先进 (SOTA) 方法相媲美的性能。实现细节可在此 https URL 获取。

Title: Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams

• Authors: Zhenghui Guo, Yuanbin Man, Junyuan Sheng, Bowen Lin, Ahmed Ahmed, Bo Jiang, Boyuan Zhang, Miao Yin, Sian Jin, Omprakash Gnawal, Chengming Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2601.15655
• Pdf URL: https://arxiv.org/pdf/2601.15655
• Copy Paste: [[2601.15655]] Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams(https://arxiv.org/abs/2601.15655)
• Keywords: generation
• Abstract: Real-time understanding of long video streams remains challenging for multimodal large language models (VLMs) due to redundant frame processing and rapid forgetting of past context. Existing streaming systems rely on fixed-interval decoding or cache pruning, which either produce repetitive outputs or discard crucial temporal information. We introduce Event-VStream, an event-aware framework that represents continuous video as a sequence of discrete, semantically coherent events. Our system detects meaningful state transitions by integrating motion, semantic, and predictive cues, and triggers language generation only at those boundaries. Each event embedding is consolidated into a persistent memory bank, enabling long-horizon reasoning while maintaining low latency. Across OVOBench-Realtime, and long-form Ego4D evaluations, Event-VStream achieves competitive performance. It improves over a VideoLLM-Online-8B baseline by +10.4 points on OVOBench-Realtime, achieves performance close to Flash-VStream-7B despite using only a general-purpose LLaMA-3-8B text backbone, and maintains around 70% GPT-5 win rate on 2-hour Ego4D streams.
• 摘要：由于冗余帧处理和过去上下文的快速遗忘，对多模态大语言模型（VLM）的长视频流的实时理解仍然具有挑战性。现有的流系统依赖于固定间隔解码或缓存修剪，这要么产生重复的输出，要么丢弃关键的时间信息。我们引入了 Event-VStream，这是一个事件感知框架，它将连续视频表示为一系列离散的、语义一致的事件。我们的系统通过集成运动、语义和预测线索来检测有意义的状态转换，并仅在这些边界处触发语言生成。每个事件嵌入都被整合到一个持久性内存库中，从而在保持低延迟的同时实现长视野推理。在 OVOBench-Realtime 和长格式 Ego4D 评估中，Event-VStream 实现了具有竞争力的性能。它在 OVOBench-Realtime 上比 VideoLLM-Online-8B 基线提高了 10.4 个点，尽管仅使用通用 LLaMA-3-8B 文本主干，但其性能仍接近 Flash-VStream-7B，并且在 2 小时 Ego4D 流上保持了大约 70% 的 GPT-5 获胜率。

Title: Skywork UniPic 3.0: Unified Multi-Image Composition via Sequence Modeling

• Authors: Hongyang Wei, Hongbo Liu, Zidong Wang, Yi Peng, Baixin Xu, Size Wu, Xuying Zhang, Xianglong He, Zexiang Liu, Peiyu Wang, Xuchen Song, Yangguang Li, Yang Liu, Yahui Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2601.15664
• Pdf URL: https://arxiv.org/pdf/2601.15664
• Copy Paste: [[2601.15664]] Skywork UniPic 3.0: Unified Multi-Image Composition via Sequence Modeling(https://arxiv.org/abs/2601.15664)
• Keywords: generation
• Abstract: The recent surge in popularity of Nano-Banana and Seedream 4.0 underscores the community's strong interest in multi-image composition tasks. Compared to single-image editing, multi-image composition presents significantly greater challenges in terms of consistency and quality, yet existing models have not disclosed specific methodological details for achieving high-quality fusion. Through statistical analysis, we identify Human-Object Interaction (HOI) as the most sought-after category by the community. We therefore systematically analyze and implement a state-of-the-art solution for multi-image composition with a primary focus on HOI-centric tasks. We present Skywork UniPic 3.0, a unified multimodal framework that integrates single-image editing and multi-image composition. Our model supports an arbitrary (1~6) number and resolution of input images, as well as arbitrary output resolutions (within a total pixel budget of 1024x1024). To address the challenges of multi-image composition, we design a comprehensive data collection, filtering, and synthesis pipeline, achieving strong performance with only 700K high-quality training samples. Furthermore, we introduce a novel training paradigm that formulates multi-image composition as a sequence-modeling problem, transforming conditional generation into unified sequence synthesis. To accelerate inference, we integrate trajectory mapping and distribution matching into the post-training stage, enabling the model to produce high-fidelity samples in just 8 steps and achieve a 12.5x speedup over standard synthesis sampling. Skywork UniPic 3.0 achieves state-of-the-art performance on single-image editing benchmark and surpasses both Nano-Banana and Seedream 4.0 on multi-image composition benchmark, thereby validating the effectiveness of our data pipeline and training paradigm. Code, models and dataset are publicly available.
• 摘要：Nano-Banana 和 Seedream 4.0 最近的流行度突显了社区对多图像合成任务的浓厚兴趣。与单图像编辑相比，多图像合成在一致性和质量方面提出了更大的挑战，但现有模型尚未公开实现高质量融合的具体方法细节。通过统计分析，我们将人机交互（HOI）确定为社区最抢手的类别。因此，我们系统地分析和实施了最先进的多图像合成解决方案，主要关注以 HOI 为中心的任务。我们推出了 Skywork UniPic 3.0，这是一个集成了单图像编辑和多图像合成的统一多模式框架。我们的模型支持任意 (1~6) 数量和分辨率的输入图像，以及任意输出分辨率（在 1024x1024 的总像素预算内）。为了解决多图像合成的挑战，我们设计了全面的数据收集、过滤和合成管道，仅用 70 万个高质量训练样本即可实现强大的性能。此外，我们引入了一种新颖的训练范例，将多图像合成表述为序列建模问题，将条件生成转化为统一的序列合成。为了加速推理，我们将轨迹映射和分布匹配集成到训练后阶段，使模型只需 8 个步骤即可生成高保真样本，并比标准合成采样实现 12.5 倍的加速。 Skywork UniPic 3.0 在单图像编辑基准上实现了最先进的性能，并在多图像合成基准上超越了 Nano-Banana 和 Seedream 4.0，从而验证了我们的数据管道和训练范例的有效性。代码、模型和数据集是公开的。

Title: Consistency-Regularized GAN for Few-Shot SAR Target Recognition

• Authors: Yikui Zhai, Shikuang Liu, Wenlve Zhou, Hongsheng Zhang, Zhiheng Zhou, Xiaolin Tian, C. L. Philip Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.15681
• Pdf URL: https://arxiv.org/pdf/2601.15681
• Copy Paste: [[2601.15681]] Consistency-Regularized GAN for Few-Shot SAR Target Recognition(https://arxiv.org/abs/2601.15681)
• Keywords: generative
• Abstract: Few-shot recognition in synthetic aperture radar (SAR) imagery remains a critical bottleneck for real-world applications due to extreme data scarcity. A promising strategy involves synthesizing a large dataset with a generative adversarial network (GAN), pre-training a model via self-supervised learning (SSL), and then fine-tuning on the few labeled samples. However, this approach faces a fundamental paradox: conventional GANs themselves require abundant data for stable training, contradicting the premise of few-shot learning. To resolve this, we propose the consistency-regularized generative adversarial network (Cr-GAN), a novel framework designed to synthesize diverse, high-fidelity samples even when trained under these severe data limitations. Cr-GAN introduces a dual-branch discriminator that decouples adversarial training from representation learning. This architecture enables a channel-wise feature interpolation strategy to create novel latent features, complemented by a dual-domain cycle consistency mechanism that ensures semantic integrity. Our Cr-GAN framework is adaptable to various GAN architectures, and its synthesized data effectively boosts multiple SSL algorithms. Extensive experiments on the MSTAR and SRSDD datasets validate our approach, with Cr-GAN achieving a highly competitive accuracy of 71.21% and 51.64%, respectively, in the 8-shot setting, significantly outperforming leading baselines, while requiring only ~5 of the parameters of state-of-the-art diffusion models. Code is available at: this https URL.
• 摘要：由于数据极度匮乏，合成孔径雷达 (SAR) 图像中的少镜头识别仍然是实际应用的关键瓶颈。一种有前途的策略包括使用生成对抗网络（GAN）合成大型数据集，通过自监督学习（SSL）预训练模型，然后对少数标记样本进行微调。然而，这种方法面临着一个根本性的悖论：传统的 GAN 本身需要大量的数据来进行稳定的训练，这与小样本学习的前提相矛盾。为了解决这个问题，我们提出了一致性正则化生成对抗网络（Cr-GAN），这是一种新颖的框架，旨在合成多样化的高保真样本，即使在这些严格的数据限制下进行训练也是如此。 Cr-GAN 引入了双分支判别器，将对抗性训练与表示学习解耦。该架构支持通道方式的特征插值策略来创建新颖的潜在特征，并辅以确保语义完整性的双域循环一致性机制。我们的 Cr-GAN 框架适用于各种 GAN 架构，其合成数据有效增强了多种 SSL 算法。 MSTAR 和 SRSDD 数据集上的大量实验验证了我们的方法，Cr-GAN 在 8 次设置中分别实现了 71.21% 和 51.64% 的极具竞争力的准确度，显着优于领先的基线，同时仅需要最先进扩散模型的约 5 个参数。代码可在以下位置获得：此 https URL。

Title: Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputs

• Authors: Mingyu Yu, Lana Liu, Zhehao Zhao, Wei Wang, Sujuan Qin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2601.15698
• Pdf URL: https://arxiv.org/pdf/2601.15698
• Copy Paste: [[2601.15698]] Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputs(https://arxiv.org/abs/2601.15698)
• Keywords: generation
• Abstract: The rapid advancement of Multimodal Large Language Models (MLLMs) has introduced complex security challenges, particularly at the intersection of textual and visual safety. While existing schemes have explored the security vulnerabilities of MLLMs, the investigation into their visual safety boundaries remains insufficient. In this paper, we propose Beyond Visual Safety (BVS), a novel image-text pair jailbreaking framework specifically designed to probe the visual safety boundaries of MLLMs. BVS employs a "reconstruction-then-generation" strategy, leveraging neutralized visual splicing and inductive recomposition to decouple malicious intent from raw inputs, thereby leading MLLMs to be induced into generating harmful images. Experimental results demonstrate that BVS achieves a remarkable jailbreak success rate of 98.21\% against GPT-5 (12 January 2026 release). Our findings expose critical vulnerabilities in the visual safety alignment of current MLLMs.
• 摘要：多模态大型语言模型 (MLLM) 的快速发展带来了复杂的安全挑战，特别是在文本和视觉安全的交叉点。虽然现有方案已经探索了 MLLM 的安全漏洞，但对其视觉安全边界的调查仍然不足。在本文中，我们提出了超越视觉安全（BVS），这是一种新颖的图像文本对越狱框架，专门用于探测 MLLM 的视觉安全边界。 BVS 采用“重建然后生成”策略，利用中和视觉拼接和归纳重组将恶意意图与原始输入分离，从而导致 MLLM 被诱导生成有害图像。实验结果表明，BVS 对 GPT-5（2026 年 1 月 12 日发布）的越狱成功率达到了 98.21%。我们的研究结果暴露了当前 MLLM 视觉安全调整中的关键漏洞。

Title: Communication-efficient Federated Graph Classification via Generative Diffusion Modeling

• Authors: Xiuling Wang, Xin Huang, Haibo Hu, Jianliang Xu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2601.15722
• Pdf URL: https://arxiv.org/pdf/2601.15722
• Copy Paste: [[2601.15722]] Communication-efficient Federated Graph Classification via Generative Diffusion Modeling(https://arxiv.org/abs/2601.15722)
• Keywords: generative
• Abstract: Graph Neural Networks (GNNs) unlock new ways of learning from graph-structured data, proving highly effective in capturing complex relationships and patterns. Federated GNNs (FGNNs) have emerged as a prominent distributed learning paradigm for training GNNs over decentralized data. However, FGNNs face two significant challenges: high communication overhead from multiple rounds of parameter exchanges and non-IID data characteristics across clients. To address these issues, we introduce CeFGC, a novel FGNN paradigm that facilitates efficient GNN training over non-IID data by limiting communication between the server and clients to three rounds only. The core idea of CeFGC is to leverage generative diffusion models to minimize direct client-server communication. Each client trains a generative diffusion model that captures its local graph distribution and shares this model with the server, which then redistributes it back to all clients. Using these generative models, clients generate synthetic graphs combined with their local graphs to train local GNN models. Finally, clients upload their model weights to the server for aggregation into a global GNN model. We theoretically analyze the I/O complexity of communication volume to show that CeFGC reduces to a constant of three communication rounds only. Extensive experiments on several real graph datasets demonstrate the effectiveness and efficiency of CeFGC against state-of-the-art competitors, reflecting our superior performance on non-IID graphs by aligning local and global model objectives and enriching the training set with diverse graphs.
• 摘要：图神经网络 (GNN) 开启了从图结构数据中学习的新方法，在捕获复杂关系和模式方面被证明非常有效。联合 GNN（FGNN）已成为一种重要的分布式学习范式，用于在去中心化数据上训练 GNN。然而，FGNN 面临两个重大挑战：多轮参数交换带来的高通信开销以及客户端之间的非 IID 数据特征。为了解决这些问题，我们引入了 CeFGC，这是一种新颖的 FGNN 范式，它通过将服务器和客户端之间的通信限制为仅三轮​​，促进非 IID 数据的高效 GNN 训练。 CeFGC 的核心思想是利用生成扩散模型来最大限度地减少直接的客户端-服务器通信。每个客户端都会训练一个生成扩散模型，该模型捕获其本地图分布并与服务器共享该模型，然后服务器将其重新分配回所有客户端。使用这些生成模型，客户可以生成与其本地图相结合的合成图，以训练本地 GNN 模型。最后，客户端将模型权重上传到服务器以聚合成全局 GNN 模型。我们从理论上分析了通信量的 I/O 复杂度，表明 CeFGC 减少到只有 3 轮通信的常数。对几个真实图数据集的大量实验证明了 CeFGC 相对于最先进的竞争对手的有效性和效率，反映了我们通过调整本地和全局模型目标并用不同的图丰富训练集在非 IID 图上的卓越性能。

Title: Towards Automated Kernel Generation in the Era of LLMs

• Authors: Yang Yu, Peiyu Zang, Chi Hsu Tsai, Haiming Wu, Yixin Shen, Jialing Zhang, Haoyu Wang, Zhiyou Xiao, Jingze Shi, Yuyu Luo, Wentao Zhang, Chunlei Men, Guang Liu, Yonghua Lin
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2601.15727
• Pdf URL: https://arxiv.org/pdf/2601.15727
• Copy Paste: [[2601.15727]] Towards Automated Kernel Generation in the Era of LLMs(https://arxiv.org/abs/2601.15727)
• Keywords: generation
• Abstract: The performance of modern AI systems is fundamentally constrained by the quality of their underlying kernels, which translate high-level algorithmic semantics into low-level hardware operations. Achieving near-optimal kernels requires expert-level understanding of hardware architectures and programming models, making kernel engineering a critical but notoriously time-consuming and non-scalable process. Recent advances in large language models (LLMs) and LLM-based agents have opened new possibilities for automating kernel generation and optimization. LLMs are well-suited to compress expert-level kernel knowledge that is difficult to formalize, while agentic systems further enable scalable optimization by casting kernel development as an iterative, feedback-driven loop. Rapid progress has been made in this area. However, the field remains fragmented, lacking a systematic perspective for LLM-driven kernel generation. This survey addresses this gap by providing a structured overview of existing approaches, spanning LLM-based approaches and agentic optimization workflows, and systematically compiling the datasets and benchmarks that underpin learning and evaluation in this domain. Moreover, key open challenges and future research directions are further outlined, aiming to establish a comprehensive reference for the next generation of automated kernel optimization. To keep track of this field, we maintain an open-source GitHub repository at this https URL.
• 摘要：现代人工智能系统的性能从根本上受到其底层内核质量的限制，底层内核将高级算法语义转化为低级硬件操作。实现接近最优的内核需要对硬件架构和编程模型有专家级的理解，这使得内核工程成为一个关键但众所周知的耗时且不可扩展的过程。大语言模型 (LLM) 和基于 LLM 的代理的最新进展为自动化内核生成和优化开辟了新的可能性。 LLM 非常适合压缩难以形式化的专家级内核知识，而代理系统通过将内核开发转换为迭代、反馈驱动的循环，进一步实现可扩展的优化。该领域已取得快速进展。然而，该领域仍然分散，缺乏 LLM 驱动的内核生成的系统视角。这项调查通过提供对现有方法的结构化概述、涵盖基于法学硕士的方法和代理优化工作流程以及系统地编译支持该领域学习和评估的数据集和基准来解决这一差距。此外，还进一步概述了关键的开放挑战和未来的研究方向，旨在为下一代自动化内核优化建立全面的参考。为了跟踪该字段，我们在此 https URL 维护一个开源 GitHub 存储库。

Title: Breaking the Resolution Barrier: Arbitrary-resolution Deep Image Steganography Framework

• Authors: Xinjue Hu, Chi Wang, Boyu Wang, Xiang Zhang, Zhenshan Tan, Zhangjie Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.15739
• Pdf URL: https://arxiv.org/pdf/2601.15739
• Copy Paste: [[2601.15739]] Breaking the Resolution Barrier: Arbitrary-resolution Deep Image Steganography Framework(https://arxiv.org/abs/2601.15739)
• Keywords: restoration
• Abstract: Deep image steganography (DIS) has achieved significant results in capacity and invisibility. However, current paradigms enforce the secret image to maintain the same resolution as the cover image during hiding and revealing. This leads to two challenges: secret images with inconsistent resolutions must undergo resampling beforehand which results in detail loss during recovery, and the secret image cannot be recovered to its original resolution when the resolution value is unknown. To address these, we propose ARDIS, the first Arbitrary Resolution DIS framework, which shifts the paradigm from discrete mapping to reference-guided continuous signal reconstruction. Specifically, to minimize the detail loss caused by resolution mismatch, we first design a Frequency Decoupling Architecture in hiding stage. It disentangles the secret into a resolution-aligned global basis and a resolution-agnostic high-frequency latent to hide in a fixed-resolution cover. Second, for recovery, we propose a Latent-Guided Implicit Reconstructor to perform deterministic restoration. The recovered detail latent code modulates a continuous implicit function to accurately query and render high-frequency residuals onto the recovered global basis, ensuring faithful restoration of original details. Furthermore, to achieve blind recovery, we introduce an Implicit Resolution Coding strategy. By transforming discrete resolution values into dense feature maps and hiding them in the redundant space of the feature domain, the reconstructor can correctly decode the secret's resolution directly from the steganographic representation. Experimental results demonstrate that ARDIS significantly outperforms state-of-the-art methods in both invisibility and cross-resolution recovery fidelity.
• 摘要：深度图像隐写术（DIS）在容量和隐秘性方面取得了显着的成果。然而，当前的范例强制秘密图像在隐藏和显示期间保持与封面图像相同的分辨率。这带来了两个挑战：分辨率不一致的秘密图像必须事先进行重采样，这会导致恢复过程中细节丢失；并且当分辨率值未知时，秘密图像无法恢复到原始分辨率。为了解决这些问题，我们提出了 ARDIS，这是第一个任意分辨率 DIS 框架，它将范式从离散映射转变为参考引导的连续信号重建。具体来说，为了最大限度地减少分辨率不匹配造成的细节损失，我们首先在隐藏阶段设计了频率解耦架构。它将秘密分解为与分辨率一致的全局基础和隐藏在固定分辨率覆盖物中的与分辨率无关的高频潜力。其次，对于恢复，我们提出了一个潜在引导隐式重建器来执行确定性恢复。恢复的细节潜在代码调制连续隐式函数，以准确查询高频残差并将其渲染到恢复的全局基础上，确保忠实恢复原始细节。此外，为了实现盲恢复，我们引入了隐式解析编码策略。通过将离散分辨率值转换为密集特征图并将其隐藏在特征域的冗余空间中，重建器可以直接从隐写表示中正确解码秘密的分辨率。实验结果表明，ARDIS 在隐形性和跨分辨率恢复保真度方面均显着优于最先进的方法。

Title: LL-GaussianMap: Zero-shot Low-Light Image Enhancement via 2D Gaussian Splatting Guided Gain Maps

• Authors: Yuhan Chen, Ying Fang, Guofa Li, Wenxuan Yu, Yicui Shi, Jingrui Zhang, Kefei Qian, Wenbo Chu, Keqiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.15766
• Pdf URL: https://arxiv.org/pdf/2601.15766
• Copy Paste: [[2601.15766]] LL-GaussianMap: Zero-shot Low-Light Image Enhancement via 2D Gaussian Splatting Guided Gain Maps(https://arxiv.org/abs/2601.15766)
• Keywords: generation
• Abstract: Significant progress has been made in low-light image enhancement with respect to visual quality. However, most existing methods primarily operate in the pixel domain or rely on implicit feature representations. As a result, the intrinsic geometric structural priors of images are often neglected. 2D Gaussian Splatting (2DGS) has emerged as a prominent explicit scene representation technique characterized by superior structural fitting capabilities and high rendering efficiency. Despite these advantages, the utilization of 2DGS in low-level vision tasks remains unexplored. To bridge this gap, LL-GaussianMap is proposed as the first unsupervised framework incorporating 2DGS into low-light image enhancement. Distinct from conventional methodologies, the enhancement task is formulated as a gain map generation process guided by 2DGS primitives. The proposed method comprises two primary stages. First, high-fidelity structural reconstruction is executed utilizing 2DGS. Then, data-driven enhancement dictionary coefficients are rendered via the rasterization mechanism of Gaussian splatting through an innovative unified enhancement module. This design effectively incorporates the structural perception capabilities of 2DGS into gain map generation, thereby preserving edges and suppressing artifacts during enhancement. Additionally, the reliance on paired data is circumvented through unsupervised learning. Experimental results demonstrate that LL-GaussianMap achieves superior enhancement performance with an extremely low storage footprint, highlighting the effectiveness of explicit Gaussian representations for image enhancement.
• 摘要：低光图像增强在视觉质量方面取得了重大进展。然而，大多数现有方法主要在像素域中操作或依赖于隐式特征表示。因此，图像的内在几何结构先验常常被忽视。二维高斯分布（2DGS）已成为一种突出的显式场景表示技术，其特点是卓越的结构拟合能力和高渲染效率。尽管有这些优点，2DGS 在低级视觉任务中的应用仍未得到探索。为了弥补这一差距，LL-GaussianMap 被提出作为第一个将 2DGS 纳入低光图像增强的无监督框架。与传统方法不同，增强任务被制定为由 2DGS 原语引导的增益图生成过程。所提出的方法包括两个主要阶段。首先，利用 2DGS 执行高保真结构重建。然后，通过创新的统一增强模块，通过高斯泼溅的光栅化机制渲染数据驱动的增强字典系数。该设计有效地将 2DGS 的结构感知能力融入增益图生成中，从而在增强过程中保留边缘并抑制伪影。此外，通过无监督学习避免了对配对数据的依赖。实验结果表明，LL-GaussianMap 以极低的存储占用实现了卓越的增强性能，凸显了显式高斯表示对于图像增强的有效性。

Title: Next Generation Active Learning: Mixture of LLMs in the Loop

• Authors: Yuanyuan Qi, Xiaohao Yang, Jueqing Lu, Guoxiang Guo, Joanne Enticott, Gang Liu, Lan Du
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2601.15773
• Pdf URL: https://arxiv.org/pdf/2601.15773
• Copy Paste: [[2601.15773]] Next Generation Active Learning: Mixture of LLMs in the Loop(https://arxiv.org/abs/2601.15773)
• Keywords: generation
• Abstract: With the rapid advancement and strong generalization capabilities of large language models (LLMs), they have been increasingly incorporated into the active learning pipelines as annotators to reduce annotation costs. However, considering the annotation quality, labels generated by LLMs often fall short of real-world applicability. To address this, we propose a novel active learning framework, Mixture of LLMs in the Loop Active Learning, replacing human annotators with labels generated through a Mixture-of-LLMs-based annotation model, aimed at enhancing LLM-based annotation robustness by aggregating the strengths of multiple LLMs. To further mitigate the impact of the noisy labels, we introduce annotation discrepancy and negative learning to identify the unreliable annotations and enhance learning effectiveness. Extensive experiments demonstrate that our framework achieves performance comparable to human annotation and consistently outperforms single-LLM baselines and other LLM-ensemble-based approaches. Moreover, our framework is built on lightweight LLMs, enabling it to operate fully on local machines in real-world applications.
• 摘要：随着大型语言模型（LLM）的快速发展和强大的泛化能力，它们越来越多地作为注释器纳入主动学习流程中，以降低注释成本。然而，考虑到注释质量，法学硕士生成的标签往往达不到现实世界的适用性。为了解决这个问题，我们提出了一种新颖的主动学习框架，即 Loop Active Learning 中的 LLM 混合，用基于 LLM 混合的注释模型生成的标签替换人类注释者，旨在通过聚合多个 LLM 的优势来增强基于 LLM 的注释的鲁棒性。为了进一步减轻噪声标签的影响，我们引入了注释差异和负学习来识别不可靠的注释并提高学习效率。大量实验表明，我们的框架实现了与人类注释相当的性能，并且始终优于单一 LLM 基线和其他基于 LLM 集成的方法。此外，我们的框架建立在轻量级法学硕士的基础上，使其能够在实际应用程序中的本地计算机上完全运行。

Title: An IoT-Based Smart Plant Monitoring and Irrigation System with Real-Time Environmental Sensing, Automated Alerts, and Cloud Analytics

• Authors: Abdul Hasib, A. S. M. Ahsanul Sarkar Akib
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.15830
• Pdf URL: https://arxiv.org/pdf/2601.15830
• Copy Paste: [[2601.15830]] An IoT-Based Smart Plant Monitoring and Irrigation System with Real-Time Environmental Sensing, Automated Alerts, and Cloud Analytics(https://arxiv.org/abs/2601.15830)
• Keywords: generation
• Abstract: The increasing global demand for sustainable agriculture necessitates intelligent monitoring systems that optimize resource utilization and plant health management. Traditional farming methods rely on manual observation and periodic watering, often leading to water wastage, inconsistent plant growth, and delayed response to environmental changes. This paper presents a comprehensive IoT-based smart plant monitoring system that integrates multiple environmental sensors with automated irrigation and cloud analytics. The proposed system utilizes an ESP32 microcontroller to collect real-time data from DHT22 (temperature/humidity), HC-SR04 (water level), and soil moisture sensors, with visual feedback through an OLED display and auditory alerts via a buzzer. All sensor data is wirelessly transmitted to the ThingSpeak cloud platform for remote monitoring, historical analysis, and automated alert generation. Experimental results demonstrate the system's effectiveness in maintaining optimal soil moisture levels (with 92\% accuracy), providing real-time environmental monitoring, and reducing water consumption by approximately 40\% compared to conventional irrigation methods. The integrated web dashboard offers comprehensive visualization of plant health parameters, making it suitable for both small-scale gardening and commercial agriculture applications. With a total implementation cost of \$45.20, this system provides an affordable, scalable solution for precision agriculture and smart farming.
• 摘要：全球对可持续农业的需求不断增长，需要智能监测系统来优化资源利用和植物健康管理。传统的耕作方法依靠人工观察和定期浇水，常常导致水资源浪费、植物生长不一致以及对环境变化的延迟响应。本文提出了一种基于物联网的综合智能植物监控系统，该系统将多个环境传感器与自动灌溉和云分析集成在一起。该系统利用 ESP32 微控制器从 DHT22（温度/湿度）、HC-SR04（水位）和土壤湿度传感器收集实时数据，并通过 OLED 显示屏提供视觉反馈并通过蜂鸣器发出听觉警报。所有传感器数据均无线传输至 ThingSpeak 云平台，用于远程监控、历史分析和自动警报生成。实验结果表明，该系统可有效维持最佳土壤湿度（准确度为 92%），提供实时环境监测，与传统灌溉方法相比，可减少约 40% 的用水量。集成的网络仪表板提供植物健康参数的全面可视化，使其适用于小型园艺和商业农业应用。该系统的总实施成本为 45.20 美元，为精准农业和智能农业提供了经济实惠、可扩展的解决方案。

Title: TinySense: Effective CSI Compression for Scalable and Accurate Wi-Fi Sensing

• Authors: Toan Gian, Dung T. Tran, Viet Quoc Pham, Francesco Restuccia, Van-Dinh Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.15838
• Pdf URL: https://arxiv.org/pdf/2601.15838
• Copy Paste: [[2601.15838]] TinySense: Effective CSI Compression for Scalable and Accurate Wi-Fi Sensing(https://arxiv.org/abs/2601.15838)
• Keywords: generative
• Abstract: With the growing demand for device-free and privacy-preserving sensing solutions, Wi-Fi sensing has emerged as a promising approach for human pose estimation (HPE). However, existing methods often process vast amounts of channel state information (CSI) data directly, ultimately straining networking resources. This paper introduces TinySense, an efficient compression framework that enhances the scalability of Wi-Fi-based human sensing. Our approach is based on a new vector quantization-based generative adversarial network (VQGAN). Specifically, by leveraging a VQGAN-learned codebook, TinySense significantly reduces CSI data while maintaining the accuracy required for reliable HPE. To optimize compression, we employ the K-means algorithm to dynamically adjust compression bitrates to cluster a large-scale pre-trained codebook into smaller subsets. Furthermore, a Transformer model is incorporated to mitigate bitrate loss, enhancing robustness in unreliable networking conditions. We prototype TinySense on an experimental testbed using Jetson Nano and Raspberry Pi to measure latency and network resource use. Extensive results demonstrate that TinySense significantly outperforms state-of-the-art compression schemes, achieving up to 1.5x higher HPE accuracy score (PCK20) under the same compression rate. It also reduces latency and networking overhead, respectively, by up to 5x and 2.5x. The code repository is available online at here.
• 摘要：随着对无设备和隐私保护传感解决方案的需求不断增长，Wi-Fi 传感已成为人体姿态估计 (HPE) 的一种有前途的方法。然而，现有方法通常直接处理大量的信道状态信息（CSI）数据，最终导致网络资源紧张。本文介绍了 TinySense，这是一种高效的压缩框架，可增强基于 Wi-Fi 的人体感应的可扩展性。我们的方法基于一种新的基于矢量量化的生成对抗网络（VQGAN）。具体来说，通过利用 VQGAN 学习的码本，TinySense 显着减少了 CSI 数据，同时保持可靠 HPE 所需的准确性。为了优化压缩，我们采用 K-means 算法动态调整压缩比特率，将大规模预训练码本聚类成更小的子集。此外，还采用了 Transformer 模型来减轻比特率损失，增强不可靠网络条件下的鲁棒性。我们使用 Jetson Nano 和 Raspberry Pi 在实验测试台上制作 TinySense 原型，以测量延迟和网络资源使用情况。大量结果表明，TinySense 的性能显着优于最先进的压缩方案，在相同的压缩率下，HPE 准确度得分 (PCK20) 提高了 1.5 倍。它还将延迟和网络开销分别减少了高达 5 倍和 2.5 倍。代码存储库可在此处在线获取。

Title: Uncertainty-guided Generation of Dark-field Radiographs

• Authors: Lina Felsner, Henriette Bast, Tina Dorosti, Florian Schaff, Franz Pfeiffer, Daniela Pfeiffer, Julia Schnabel
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2601.15859
• Pdf URL: https://arxiv.org/pdf/2601.15859
• Copy Paste: [[2601.15859]] Uncertainty-guided Generation of Dark-field Radiographs(https://arxiv.org/abs/2601.15859)
• Keywords: generation, generative
• Abstract: X-ray dark-field radiography provides complementary diagnostic information to conventional attenuation imaging by visualizing microstructural tissue changes through small-angle scattering. However, the limited availability of such data poses challenges for developing robust deep learning models. In this work, we present the first framework for generating dark-field images directly from standard attenuation chest X-rays using an Uncertainty-Guided Progressive Generative Adversarial Network. The model incorporates both aleatoric and epistemic uncertainty to improve interpretability and reliability. Experiments demonstrate high structural fidelity of the generated images, with consistent improvement of quantitative metrics across stages. Furthermore, out-of-distribution evaluation confirms that the proposed model generalizes well. Our results indicate that uncertainty-guided generative modeling enables realistic dark-field image synthesis and provides a reliable foundation for future clinical applications.
• 摘要：X 射线暗场放射成像通过小角度散射可视化微观结构组织变化，为传统衰减成像提供补充诊断信息。然而，此类数据的可用性有限，给开发强大的深度学习模型带来了挑战。在这项工作中，我们提出了第一个使用不确定性引导渐进生成对抗网络直接从标准衰减胸部 X 射线生成暗场图像的框架。该模型结合了任意和认知不确定性，以提高可解释性和可靠性。实验证明生成的图像具有高结构保真度，并且各个阶段的定量指标得到了一致的改进。此外，分布外评估证实所提出的模型具有良好的泛化性。我们的结果表明，不确定性引导的生成模型可以实现逼真的暗场图像合成，并为未来的临床应用提供可靠的基础。

Title: RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture

• Authors: Anas Anwarul Haq Khan, Mariam Husain, Kshitij Jadhav
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.15891
• Pdf URL: https://arxiv.org/pdf/2601.15891
• Copy Paste: [[2601.15891]] RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture(https://arxiv.org/abs/2601.15891)
• Keywords: generation
• Abstract: Recent advances in medical vision language models guide the learning of visual representations; however, this form of supervision is constrained by the availability of paired image text data, raising the question of whether robust radiology encoders can be learned without relying on language supervision. In this work, we introduce RadJEPA, a self-supervised framework built on a Joint Embedding Predictive Architecture that learns without language supervision. Pre-trained solely on unlabeled chest X-ray images, the model learns to predict latent representations of masked image regions. This predictive objective differs fundamentally from both image text pre-training and DINO-style self-distillation: rather than aligning global representations across views or modalities, RadJEPA explicitly models latent-space prediction. We evaluate the learned encoder on disease classification, semantic segmentation, and report generation tasks. Across benchmarks, RadJEPA achieves performance exceeding state-of-the-art approaches, including Rad-DINO.
• 摘要：医学视觉语言模型的最新进展指导了视觉表征的学习；然而，这种形式的监督受到配对图像文本数据的可用性的限制，这就提出了一个问题：是否可以在不依赖语言监督的情况下学习鲁棒的放射学编码器。在这项工作中，我们介绍了 RadJEPA，这是一个基于联合嵌入预测架构构建的自监督框架，无需语言监督即可学习。该模型仅针对未标记的胸部 X 射线图像进行预训练，学习预测屏蔽图像区域的潜在表示。这种预测目标与图像文本预训练和 DINO 式自蒸馏都有根本的不同：​​RadJEPA 不是跨视图或模态对齐全局表示，而是显式地对潜在空间预测进行建模。我们在疾病分类、语义分割和报告生成任务上评估学习的编码器。在各个基准测试中，RadJEPA 的性能超过了最先进的方法，包括 Rad-DINO。

Title: Opening the Black Box: Preliminary Insights into Affective Modeling in Multimodal Foundation Models

• Authors: Zhen Zhang, Runhao Zeng, Sicheng Zhao, Xiping Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.15906
• Pdf URL: https://arxiv.org/pdf/2601.15906
• Copy Paste: [[2601.15906]] Opening the Black Box: Preliminary Insights into Affective Modeling in Multimodal Foundation Models(https://arxiv.org/abs/2601.15906)
• Keywords: generation
• Abstract: Understanding where and how emotions are represented in large-scale foundation models remains an open problem, particularly in multimodal affective settings. Despite the strong empirical performance of recent affective models, the internal architectural mechanisms that support affective understanding and generation are still poorly understood. In this work, we present a systematic mechanistic study of affective modeling in multimodal foundation models. Across multiple architectures, training strategies, and affective tasks, we analyze how emotion-oriented supervision reshapes internal model parameters. Our results consistently reveal a clear and robust pattern: affective adaptation does not primarily focus on the attention module, but instead localizes to the feed-forward gating projection (\texttt{gate\_proj}). Through controlled module transfer, targeted single-module adaptation, and destructive ablation, we further demonstrate that \texttt{gate\_proj} is sufficient, efficient, and necessary for affective understanding and generation. Notably, by tuning only approximately 24.5\% of the parameters tuned by AffectGPT, our approach achieves 96.6\% of its average performance across eight affective tasks, highlighting substantial parameter efficiency. Together, these findings provide empirical evidence that affective capabilities in foundation models are structurally mediated by feed-forward gating mechanisms and identify \texttt{gate\_proj} as a central architectural locus of affective modeling.
• 摘要：了解大规模基础模型中情感的表现形式和方式仍然是一个悬而未决的问题，特别是在多模态情感环境中。尽管最近的情感模型具有很强的实证表现，但支持情感理解和生成的内部架构机制仍然知之甚少。在这项工作中，我们提出了多模态基础模型中情感建模的系统机制研究。跨越多种架构、训练策略和情感任务，我们分析了面向情感的监督如何重塑内部模型参数。我们的结果始终揭示了一个清晰而稳健的模式：情感适应并不主要关注注意力模块，而是定位于前馈门控投影（\texttt{gate\_proj}）。通过受控模块传输、有针对性的单模块适应和破坏性消融，我们进一步证明 \texttt{gate\_proj} 对于情感理解和生成来说是充分、有效和必要的。值得注意的是，通过仅调整 AffectGPT 调整的大约 24.5% 的参数，我们的方法在八个情感任务中实现了 96.6% 的平均性能，突出了显着的参数效率。总之，这些发现提供了经验证据，表明基础模型中的情感能力在结构上是由前馈门控机制调节的，并将 \texttt{gate\_proj} 识别为情感建模的中心架构轨迹。

Title: HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Models

• Authors: Xin Xie, Jiaxian Guo, Dong Gong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.15968
• Pdf URL: https://arxiv.org/pdf/2601.15968
• Copy Paste: [[2601.15968]] HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Models(https://arxiv.org/abs/2601.15968)
• Keywords: generation, generative
• Abstract: Diffusion models achieve state-of-the-art performance but often fail to generate outputs that align with human preferences and intentions, resulting in images with poor aesthetic quality and semantic inconsistencies. Existing alignment methods present a difficult trade-off: fine-tuning approaches suffer from loss of diversity with reward over-optimization, while test-time scaling methods introduce significant computational overhead and tend to under-optimize. To address these limitations, we propose HyperAlign, a novel framework that trains a hypernetwork for efficient and effective test-time alignment. Instead of modifying latent states, HyperAlign dynamically generates low-rank adaptation weights to modulate the diffusion model's generation operators. This allows the denoising trajectory to be adaptively adjusted based on input latents, timesteps and prompts for reward-conditioned alignment. We introduce multiple variants of HyperAlign that differ in how frequently the hypernetwork is applied, balancing between performance and efficiency. Furthermore, we optimize the hypernetwork using a reward score objective regularized with preference data to reduce reward hacking. We evaluate HyperAlign on multiple extended generative paradigms, including Stable Diffusion and FLUX. It significantly outperforms existing fine-tuning and test-time scaling baselines in enhancing semantic consistency and visual appeal.
• 摘要：扩散模型实现了最先进的性能，但通常无法生成符合人类偏好和意图的输出，导致图像的审美质量较差和语义不一致。现有的对齐方法存在一个困难的权衡：微调方法会因奖励过度优化而损失多样性，而测试时间缩放方法会引入大量的计算开销，并且往往会优化不足。为了解决这些限制，我们提出了 HyperAlign，这是一种新颖的框架，可以训练超网络以实现高效且有效的测试时间对齐。 HyperAlign 不是修改潜在状态，而是动态生成低秩适应权重来调节扩散模型的生成算子。这使得去噪轨迹能够根据输入潜伏、时间步长和奖励条件对齐的提示进行自适应调整。我们引入了 HyperAlign 的多个变体，这些变体的不同之处在于超网络的应用频率，以平衡性能和效率。此外，我们使用偏好数据规范化的奖励分数目标来优化超网络，以减少奖励黑客行为。我们在多种扩展生成范式上评估 HyperAlign，包括稳定扩散和通量。它在增强语义一致性和视觉吸引力方面显着优于现有的微调和测试时间缩放基线。

Title: PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models

• Authors: Chak-Wing Mak, Guanyu Zhu, Boyi Zhang, Hongji Li, Xiaowei Chi, Kevin Zhang, Yichen Wu, Yangfan He, Chun-Kai Fan, Wentao Lu, Kuangzhi Ge, Xinyu Fang, Hongyang He, Kuan Lu, Tianxiang Xu, Li Zhang, Yongxin Ni, Youhua Li, Shanghang Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2601.16007
• Pdf URL: https://arxiv.org/pdf/2601.16007
• Copy Paste: [[2601.16007]] PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models(https://arxiv.org/abs/2601.16007)
• Keywords: generation
• Abstract: Modern foundational Multimodal Large Language Models (MLLMs) and video world models have advanced significantly in mathematical, common-sense, and visual reasoning, but their grasp of the underlying physics remains underexplored. Existing benchmarks attempting to measure this matter rely on synthetic, Visual Question Answer templates or focus on perceptual video quality that is tangential to measuring how well the video abides by physical laws. To address this fragmentation, we introduce PhysicsMind, a unified benchmark with both real and simulation environments that evaluates law-consistent reasoning and generation over three canonical principles: Center of Mass, Lever Equilibrium, and Newton's First Law. PhysicsMind comprises two main tasks: i) VQA tasks, testing whether models can reason and determine physical quantities and values from images or short videos, and ii) Video Generation(VG) tasks, evaluating if predicted motion trajectories obey the same center-of-mass, torque, and inertial constraints as the ground truth. A broad range of recent models and video generation models is evaluated on PhysicsMind and found to rely on appearance heuristics while often violating basic mechanics. These gaps indicate that current scaling and training are still insufficient for robust physical understanding, underscoring PhysicsMind as a focused testbed for physics-aware multimodal models. Our data will be released upon acceptance.
• 摘要：现代基础多模态大语言模型 (MLLM) 和视频世界模型在数学、常识和视觉推理方面取得了显着进步，但它们对底层物理的掌握仍然有待探索。试图衡量这个问题的现有基准依赖于合成的视觉问答模板或关注感知视频质量，这与衡量视频遵守物理定律的程度无关。为了解决这种碎片化问题，我们引入了PhysicsMind，这是一个具有真实和模拟环境的统一基准，可根据三个规范原理（质心、杠杆平衡和牛顿第一定律）评估规律一致的推理和生成。 PhysicsMind 包括两个主要任务：i）VQA 任务，测试模型是否可以推理并确定图像或短视频中的物理量和值；ii）视频生成（VG）任务，评估预测的运动轨迹是否遵循与地面真实情况相同的质心、扭矩和惯性约束。在PhysicsMind 上评估了一系列最新的模型和视频生成模型，发现它们依赖于外观启发法，同时经常违反基本力学。这些差距表明，当前的扩展和训练仍然不足以实现强大的物理理解，这凸显了PhysicsMind作为物理感知多模态模型的重点测试平台。我们的数据将在接受后发布。

Title: PAINT: Pathology-Aware Integrated Next-Scale Transformation for Virtual Immunohistochemistry

• Authors: Rongze Ma, Mengkang Lu, Zhenyu Xiang, Yongsheng Pan, Yicheng Wu, Qingjie Zeng, Yong Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.16024
• Pdf URL: https://arxiv.org/pdf/2601.16024
• Copy Paste: [[2601.16024]] PAINT: Pathology-Aware Integrated Next-Scale Transformation for Virtual Immunohistochemistry(https://arxiv.org/abs/2601.16024)
• Keywords: generation
• Abstract: Virtual immunohistochemistry (IHC) aims to computationally synthesize molecular staining patterns from routine Hematoxylin and Eosin (H\&E) images, offering a cost-effective and tissue-efficient alternative to traditional physical staining. However, this task is particularly challenging: H\&E morphology provides ambiguous cues about protein expression, and similar tissue structures may correspond to distinct molecular states. Most existing methods focus on direct appearance synthesis to implicitly achieve cross-modal generation, often resulting in semantic inconsistencies due to insufficient structural priors. In this paper, we propose Pathology-Aware Integrated Next-Scale Transformation (PAINT), a visual autoregressive framework that reformulates the synthesis process as a structure-first conditional generation task. Unlike direct image translation, PAINT enforces a causal order by resolving molecular details conditioned on a global structural layout. Central to this approach is the introduction of a Spatial Structural Start Map (3S-Map), which grounds the autoregressive initialization in observed morphology, ensuring deterministic, spatially aligned synthesis. Experiments on the IHC4BC and MIST datasets demonstrate that PAINT outperforms state-of-the-art methods in structural fidelity and clinical downstream tasks, validating the potential of structure-guided autoregressive modeling.
• 摘要：虚拟免疫组织化学 (IHC) 旨在通过常规苏木精和曙红 (H\&E) 图像通过计算合成分子染色模式，为传统物理染色提供一种经济有效且组织高效的替代方案。然而，这项任务特别具有挑战性：H\&E 形态提供了有关蛋白质表达的模糊线索，而相似的组织结构可能对应于不同的分子状态。大多数现有方法侧重于直接外观合成以隐式实现跨模态生成，通常由于结构先验不足而导致语义不一致。在本文中，我们提出了病理学感知集成下一尺度转换（PAINT），这是一种视觉自回归框架，它将合成过程重新表述为结构优先的条件生成任务。与直接图像翻译不同，PAINT 通过解析以全局结构布局为条件的分子细节来强制执行因果顺序。该方法的核心是引入空间结构起始图（3S-Map），它将自回归初始化基于观察到的形态，确保确定性、空间对齐的合成。 IHC4BC 和 MIST 数据集上的实验表明，PAINT 在结构保真度和临床下游任务方面优于最先进的方法，验证了结构引导自回归建模的潜力。

Title: ProGiDiff: Prompt-Guided Diffusion-Based Medical Image Segmentation

• Authors: Yuan Lin, Murong Xu, Marc Hölle, Chinmay Prabhakar, Andreas Maier, Vasileios Belagiannis, Bjoern Menze, Suprosanna Shit
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.16060
• Pdf URL: https://arxiv.org/pdf/2601.16060
• Copy Paste: [[2601.16060]] ProGiDiff: Prompt-Guided Diffusion-Based Medical Image Segmentation(https://arxiv.org/abs/2601.16060)
• Keywords: generation
• Abstract: Widely adopted medical image segmentation methods, although efficient, are primarily deterministic and remain poorly amenable to natural language prompts. Thus, they lack the capability to estimate multiple proposals, human interaction, and cross-modality adaptation. Recently, text-to-image diffusion models have shown potential to bridge the gap. However, training them from scratch requires a large dataset-a limitation for medical image segmentation. Furthermore, they are often limited to binary segmentation and cannot be conditioned on a natural language prompt. To this end, we propose a novel framework called ProGiDiff that leverages existing image generation models for medical image segmentation purposes. Specifically, we propose a ControlNet-style conditioning mechanism with a custom encoder, suitable for image conditioning, to steer a pre-trained diffusion model to output segmentation masks. It naturally extends to a multi-class setting simply by prompting the target organ. Our experiment on organ segmentation from CT images demonstrates strong performance compared to previous methods and could greatly benefit from an expert-in-the-loop setting to leverage multiple proposals. Importantly, we demonstrate that the learned conditioning mechanism can be easily transferred through low-rank, few-shot adaptation to segment MR images.
• 摘要：广泛采用的医学图像分割方法虽然有效，但主要是确定性的，并且仍然很难适应自然语言提示。因此，它们缺乏估计多个提议、人类交互和跨模态适应的能力。最近，文本到图像的扩散模型已显示出弥补这一差距的潜力。然而，从头开始训练它们需要一个大的数据集——这是医学图像分割的限制。此外，它们通常仅限于二进制分割，并且不能以自然语言提示为条件。为此，我们提出了一种名为 ProGiDiff 的新颖框架，该框架利用现有的图像生成模型进行医学图像分割。具体来说，我们提出了一种带有自定义编码器的 ControlNet 式调节机制，适用于图像调节，以引导预训练的扩散模型输出分割掩模。只需通过提示目标器官，它就可以自然地扩展到多类别设置。与以前的方法相比，我们对 CT 图像器官分割的实验展示了强大的性能，并且可以极大地受益于专家在环设置中利用多个建议。重要的是，我们证明了学习到的调节机制可以通过低秩、少样本适应轻松转移以分割 MR 图像。

Title: DTP: A Simple yet Effective Distracting Token Pruning Framework for Vision-Language Action Models

• Authors: Chenyang Li, Jieyuan Liu, Bin Li, Bo Gao, Yilin Yuan, Yangfan He, Yuchen Li, Jingqun Tang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2601.16065
• Pdf URL: https://arxiv.org/pdf/2601.16065
• Copy Paste: [[2601.16065]] DTP: A Simple yet Effective Distracting Token Pruning Framework for Vision-Language Action Models(https://arxiv.org/abs/2601.16065)
• Keywords: generation
• Abstract: Vision-Language Action (VLA) models have shown remarkable progress in robotic manipulation by leveraging the powerful perception abilities of Vision-Language Models (VLMs) to understand environments and directly output actions. However, by default, VLA models may overly attend to image tokens in the task-irrelevant region, which we describe as 'distracting tokens'. This behavior can disturb the model from the generation of the desired action tokens in each step, affecting the success rate of tasks. In this paper, we introduce a simple yet effective plug-and-play Distracting Token Pruning (DTP) framework, which dynamically detects and prunes these distracting image tokens. By correcting the model's visual attention patterns, we aim to improve the task success rate, as well as exploring the performance upper boundaries of the model without altering its original architecture or adding additional inputs. Experiments on the SIMPLER Benchmark (Li et al., 2024) show that our method consistently achieving relative improvements in task success rates across different types of novel VLA models, demonstrating generalizability to transformer-based VLAs. Further analysis reveals a negative correlation between the task success rate and the amount of attentions in the task-irrelevant region for all models tested, highlighting a common phenomenon of VLA models that could guide future research. We also publish our code at: this https URL.
• 摘要：视觉语言动作（VLA）模型利用视觉语言模型（VLM）强大的感知能力来理解环境并直接输出动作，在机器人操作方面取得了显着的进展。然而，默认情况下，VLA 模型可能会过度关注与任务无关区域中的图像标记，我们将其描述为“分散注意力的标记”。这种行为可能会干扰模型在每个步骤中生成所需操作标记，从而影响任务的成功率。在本文中，我们介绍了一种简单而有效的即插即用分散注意力标记修剪（DTP）框架，该框架可以动态检测和修剪这些分散注意力的图像标记。通过纠正模型的视觉注意模式，我们的目标是提高任务成功率，并在不改变其原始架构或添加额外输入的情况下探索模型的性能上限。 SIMPLER Benchmark（Li 等人，2024）上的实验表明，我们的方法在不同类型的新型 VLA 模型中始终实现了任务成功率的相对提高，证明了基于 Transformer 的 VLA 的通用性。进一步的分析揭示了所有测试模型的任务成功率与任务无关区域的关注量之间存在负相关，这凸显了 VLA 模型的普遍现象，可以指导未来的研究。我们还将我们的代码发布到：此 https URL。

Title: Masked Modeling for Human Motion Recovery Under Occlusions

• Authors: Zhiyin Qian, Siwei Zhang, Bharat Lal Bhatnagar, Federica Bogo, Siyu Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.16079
• Pdf URL: https://arxiv.org/pdf/2601.16079
• Copy Paste: [[2601.16079]] Masked Modeling for Human Motion Recovery Under Occlusions(https://arxiv.org/abs/2601.16079)
• Keywords: generative
• Abstract: Human motion reconstruction from monocular videos is a fundamental challenge in computer vision, with broad applications in AR/VR, robotics, and digital content creation, but remains challenging under frequent occlusions in real-world this http URL regression-based methods are efficient but fragile to missing observations, while optimization- and diffusion-based approaches improve robustness at the cost of slow inference speed and heavy preprocessing steps. To address these limitations, we leverage recent advances in generative masked modeling and present MoRo: Masked Modeling for human motion Recovery under Occlusions. MoRo is an occlusion-robust, end-to-end generative framework that formulates motion reconstruction as a video-conditioned task, and efficiently recover human motion in a consistent global coordinate system from RGB videos. By masked modeling, MoRo naturally handles occlusions while enabling efficient, end-to-end inference. To overcome the scarcity of paired video-motion data, we design a cross-modality learning scheme that learns multi-modal priors from a set of heterogeneous datasets: (i) a trajectory-aware motion prior trained on MoCap datasets, (ii) an image-conditioned pose prior trained on image-pose datasets, capturing diverse per-frame poses, and (iii) a video-conditioned masked transformer that fuses motion and pose priors, finetuned on video-motion datasets to integrate visual cues with motion dynamics for robust inference. Extensive experiments on EgoBody and RICH demonstrate that MoRo substantially outperforms state-of-the-art methods in accuracy and motion realism under occlusions, while performing on-par in non-occluded scenarios. MoRo achieves real-time inference at 70 FPS on a single H200 GPU.
• 摘要：单目视频中的人体运动重建是计算机视觉中的一项基本挑战，在 AR/VR、机器人技术和数字内容创建中具有广泛的应用，但在现实世界中频繁遮挡的情况下仍然具有挑战性，这种基于 http URL 回归的方法非常有效，但容易丢失观察结果，而基于优化和扩散的方法以缓慢的推理速度和繁重的预处理步骤为代价提高了鲁棒性。为了解决这些限制，我们利用生成掩模建模的最新进展，并提出了 MoRo：遮挡下人体运动恢复的掩模建模。 MoRo 是一种遮挡稳健的端到端生成框架，它将运动重建制定为视频条件任务，并在一致的全局坐标系中从 RGB 视频中有效地恢复人体运动。通过屏蔽建模，MoRo 自然地处理遮挡，同时实现高效的端到端推理。为了克服成对视频运动数据的稀缺性，我们设计了一种跨模态学习方案，从一组异构数据集中学习多模态先验：（i）在MoCap数据集上训练的轨迹感知运动，（ii）在图像姿势数据集上训练的图像条件姿势，捕获不同的每帧姿势，以及（iii）融合运动和姿势先验的视频条件掩模变换器，在视频运动数据集上进行微调，以将视觉线索与运动动力学相结合为了稳健的推理。在 EgoBody 和 RICH 上进行的大量实验表明，MoRo 在遮挡情况下的准确性和运动真实感方面远远优于最先进的方法，而在非遮挡场景中的表现则相当。 MoRo 在单个 H200 GPU 上实现了 70 FPS 的实时推理。

Title: SAMTok: Representing Any Mask with Two Words

• Authors: Yikang Zhou, Tao Zhang, Dengxian Gong, Yuanzheng Wu, Ye Tian, Haochen Wang, Haobo Yuan, Jiacong Wang, Lu Qi, Hao Fei, Anran Wang, Zhuochen Wang, Yujing Wang, Cheng Chen, Shunping Ji, Xiangtai Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.16093
• Pdf URL: https://arxiv.org/pdf/2601.16093
• Copy Paste: [[2601.16093]] SAMTok: Representing Any Mask with Two Words(https://arxiv.org/abs/2601.16093)
• Keywords: generation
• Abstract: Pixel-wise capabilities are essential for building interactive intelligent systems. However, pixel-wise multi-modal LLMs (MLLMs) remain difficult to scale due to complex region-level encoders, specialized segmentation decoders, and incompatible training objectives. To address these challenges, we present SAMTok, a discrete mask tokenizer that converts any region mask into two special tokens and reconstructs the mask using these tokens with high fidelity. By treating masks as new language tokens, SAMTok enables base MLLMs (such as the QwenVL series) to learn pixel-wise capabilities through standard next-token prediction and simple reinforcement learning, without architectural modifications and specialized loss design. SAMTok builds on SAM2 and is trained on 209M diverse masks using a mask encoder and residual vector quantizer to produce discrete, compact, and information-rich tokens. With 5M SAMTok-formatted mask understanding and generation data samples, QwenVL-SAMTok attains state-of-the-art or comparable results on region captioning, region VQA, grounded conversation, referring segmentation, scene graph parsing, and multi-round interactive segmentation. We further introduce a textual answer-matching reward that enables efficient reinforcement learning for mask generation, delivering substantial improvements on GRES and GCG benchmarks. Our results demonstrate a scalable and straightforward paradigm for equipping MLLMs with strong pixel-wise capabilities. Our code and models are available.
• 摘要：像素级功能对于构建交互式智能系统至关重要。然而，由于复杂的区域级编码器、专门的分割解码器和不兼容的训练目标，像素级多模态 LLM (MLLM) 仍然难以扩展。为了解决这些挑战，我们提出了 SAMTok，这是一种离散掩码标记器，可将任何区域掩码转换为两个特殊标记，并使用这些标记以高保真度重建掩码。通过将掩码视为新的语言标记，SAMTok 使基础 MLLM（例如 QwenVL 系列）能够通过标准的下一个标记预测和简单的强化学习来学习像素级功能，而无需架构修改和专门的损失设计。 SAMTok 基于 SAM2 构建，并使用掩码编码器和残差矢量量化器在 209M 个不同掩码上进行训练，以生成离散、紧凑且信息丰富的标记。借助 5M SAMTok 格式的掩模理解和生成数据样本，QwenVL-SAMTok 在区域字幕、区域 VQA、基础对话、引用分割、场景图解析和多轮交互式分割方面获得了最先进的或可比的结果。我们进一步引入了文本答案匹配奖励，可以实现掩码生成的高效强化学习，从而对 GRES 和 GCG 基准进行重大改进。我们的结果展示了一种可扩展且简单的范例，可以为 MLLM 配备强大的像素级功能。我们的代码和型号可供使用。

Title: Learning to Watermark in the Latent Space of Generative Models

• Authors: Sylvestre-Alvise Rebuffi, Tuan Tran, Valeriu Lacatusu, Pierre Fernandez, Tomáš Souček, Nikola Jovanović, Tom Sander, Hady Elsahar, Alexandre Mourachko
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2601.16140
• Pdf URL: https://arxiv.org/pdf/2601.16140
• Copy Paste: [[2601.16140]] Learning to Watermark in the Latent Space of Generative Models(https://arxiv.org/abs/2601.16140)
• Keywords: generative
• Abstract: Existing approaches for watermarking AI-generated images often rely on post-hoc methods applied in pixel space, introducing computational overhead and potential visual artifacts. In this work, we explore latent space watermarking and introduce DistSeal, a unified approach for latent watermarking that works across both diffusion and autoregressive models. Our approach works by training post-hoc watermarking models in the latent space of generative models. We demonstrate that these latent watermarkers can be effectively distilled either into the generative model itself or into the latent decoder, enabling in-model watermarking. The resulting latent watermarks achieve competitive robustness while offering similar imperceptibility and up to 20x speedup compared to pixel-space baselines. Our experiments further reveal that distilling latent watermarkers outperforms distilling pixel-space ones, providing a solution that is both more efficient and more robust.
• 摘要：现有的人工智能生成图像水印方法通常依赖于像素空间中应用的事后方法，从而引入计算开销和潜在的视觉伪影。在这项工作中，我们探索了潜在空间水印并引入了 DistSeal，这是一种适用于扩散模型和自回归模型的统一潜在水印方法。我们的方法通过在生成模型的潜在空间中训练事后水印模型来工作。我们证明这些潜在水印可以有效地提取到生成模型本身或潜在解码器中，从而实现模型内水印。由此产生的潜在水印实现了具有竞争力的鲁棒性，同时提供了类似的不易察觉性，并且与像素空间基线相比，速度提高了 20 倍。我们的实验进一步表明，提取潜在水印优于提取像素空间水印，提供了一种更高效、更稳健的解决方案。

Title: ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion

• Authors: Remy Sabathier, David Novotny, Niloy J. Mitra, Tom Monnier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.16148
• Pdf URL: https://arxiv.org/pdf/2601.16148
• Copy Paste: [[2601.16148]] ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion(https://arxiv.org/abs/2601.16148)
• Keywords: generation, generative
• Abstract: Generating animated 3D objects is at the heart of many applications, yet most advanced works are typically difficult to apply in practice because of their limited setup, their long runtime, or their limited quality. We introduce ActionMesh, a generative model that predicts production-ready 3D meshes "in action" in a feed-forward manner. Drawing inspiration from early video models, our key insight is to modify existing 3D diffusion models to include a temporal axis, resulting in a framework we dubbed "temporal 3D diffusion". Specifically, we first adapt the 3D diffusion stage to generate a sequence of synchronized latents representing time-varying and independent 3D shapes. Second, we design a temporal 3D autoencoder that translates a sequence of independent shapes into the corresponding deformations of a pre-defined reference shape, allowing us to build an animation. Combining these two components, ActionMesh generates animated 3D meshes from different inputs like a monocular video, a text description, or even a 3D mesh with a text prompt describing its animation. Besides, compared to previous approaches, our method is fast and produces results that are rig-free and topology consistent, hence enabling rapid iteration and seamless applications like texturing and retargeting. We evaluate our model on standard video-to-4D benchmarks (Consistent4D, Objaverse) and report state-of-the-art performances on both geometric accuracy and temporal consistency, demonstrating that our model can deliver animated 3D meshes with unprecedented speed and quality.
• 摘要：生成动画 3D 对象是许多应用程序的核心，但大多数先进的作品通常难以在实践中应用，因为它们的设置有限、运行时间长或质量有限。我们引入了 ActionMesh，这是一种生成模型，可以前馈方式预测“正在运行”的可用于生产的 3D 网格。从早期视频模型中汲取灵感，我们的主要见解是修改现有的 3D 扩散模型以包含时间轴，从而形成我们称为“时间 3D 扩散”的框架。具体来说，我们首先调整 3D 扩散阶段来生成一系列代表时变且独立的 3D 形状的同步潜在变量。其次，我们设计了一个时间 3D 自动编码器，它将一系列独立形状转换为预定义参考形状的相应变形，从而使我们能够构建动画。将这两个组件结合起来，ActionMesh 可根据不同的输入生成动画 3D 网格，例如单目视频、文本描述，甚至带有描述其动画的文本提示的 3D 网格。此外，与以前的方法相比，我们的方法速度很快，并且产生的结果是无装备且拓扑一致的，因此可以实现快速迭代和无缝应用，例如纹理和重定向。我们在标准视频到 4D 基准（Consistent4D、Objaverse）上评估我们的模型，并报告几何精度和时间一致性方面的最先进性能，证明我们的模型可以以前所未有的速度和质量提供动画 3D 网格。

Title: 360Anything: Geometry-Free Lifting of Images and Videos to 360°

• Authors: Ziyi Wu, Daniel Watson, Andrea Tagliasacchi, David J. Fleet, Marcus A. Brubaker, Saurabh Saxena
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.16192
• Pdf URL: https://arxiv.org/pdf/2601.16192
• Copy Paste: [[2601.16192]] 360Anything: Geometry-Free Lifting of Images and Videos to 360°(https://arxiv.org/abs/2601.16192)
• Keywords: generation
• Abstract: Lifting perspective images and videos to 360° panoramas enables immersive 3D world generation. Existing approaches often rely on explicit geometric alignment between the perspective and the equirectangular projection (ERP) space. Yet, this requires known camera metadata, obscuring the application to in-the-wild data where such calibration is typically absent or noisy. We propose 360Anything, a geometry-free framework built upon pre-trained diffusion transformers. By treating the perspective input and the panorama target simply as token sequences, 360Anything learns the perspective-to-equirectangular mapping in a purely data-driven way, eliminating the need for camera information. Our approach achieves state-of-the-art performance on both image and video perspective-to-360° generation, outperforming prior works that use ground-truth camera information. We also trace the root cause of the seam artifacts at ERP boundaries to zero-padding in the VAE encoder, and introduce Circular Latent Encoding to facilitate seamless generation. Finally, we show competitive results in zero-shot camera FoV and orientation estimation benchmarks, demonstrating 360Anything's deep geometric understanding and broader utility in computer vision tasks. Additional results are available at this https URL.
• 摘要：将透视图像和视频提升至 360° 全景，从而生成身临其境的 3D 世界。现有方法通常依赖于透视和等距柱状投影 (ERP) 空间之间的显式几何对齐。然而，这需要已知的相机元数据，从而模糊了对野外数据的应用，而这种校准通常不存在或有噪音。我们提出了 360Anything，这是一个基于预先训练的扩散变压器构建的无几何框架。通过将透视输入和全景目标简单地视为标记序列，360Anything 以纯粹数据驱动的方式学习透视到等距矩形的映射，从而无需相机信息。我们的方法在图像和视频 360° 透视生成方面均实现了最先进的性能，优于使用地面实况相机信息的先前作品。我们还将 ERP 边界处接缝伪影的根本原因追溯到 VAE 编码器中的零填充，并引入循环潜在编码以促进无缝生成。最后，我们在零镜头相机 FoV 和方向估计基准中展示了具有竞争力的结果，展示了 360Anything 深刻的几何理解和在计算机视觉任务中更广泛的实用性。其他结果可通过此 https URL 获得。

Title: Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders

• Authors: Shengbang Tong, Boyang Zheng, Ziteng Wang, Bingda Tang, Nanye Ma, Ellis Brown, Jihan Yang, Rob Fergus, Yann LeCun, Saining Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2601.16208
• Pdf URL: https://arxiv.org/pdf/2601.16208
• Copy Paste: [[2601.16208]] Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders(https://arxiv.org/abs/2601.16208)
• Keywords: generation
• Abstract: Representation Autoencoders (RAEs) have shown distinct advantages in diffusion modeling on ImageNet by training in high-dimensional semantic latent spaces. In this work, we investigate whether this framework can scale to large-scale, freeform text-to-image (T2I) generation. We first scale RAE decoders on the frozen representation encoder (SigLIP-2) beyond ImageNet by training on web, synthetic, and text-rendering data, finding that while scale improves general fidelity, targeted data composition is essential for specific domains like text. We then rigorously stress-test the RAE design choices originally proposed for ImageNet. Our analysis reveals that scaling simplifies the framework: while dimension-dependent noise scheduling remains critical, architectural complexities such as wide diffusion heads and noise-augmented decoding offer negligible benefits at scale Building on this simplified framework, we conduct a controlled comparison of RAE against the state-of-the-art FLUX VAE across diffusion transformer scales from 0.5B to 9.8B parameters. RAEs consistently outperform VAEs during pretraining across all model scales. Further, during finetuning on high-quality datasets, VAE-based models catastrophically overfit after 64 epochs, while RAE models remain stable through 256 epochs and achieve consistently better performance. Across all experiments, RAE-based diffusion models demonstrate faster convergence and better generation quality, establishing RAEs as a simpler and stronger foundation than VAEs for large-scale T2I generation. Additionally, because both visual understanding and generation can operate in a shared representation space, the multimodal model can directly reason over generated latents, opening new possibilities for unified models.
• 摘要：通过在高维语义潜在空间中进行训练，表示自动编码器 (RAE) 在 ImageNet 上的扩散建模中显示出明显的优势。在这项工作中，我们研究了该框架是否可以扩展到大规模、自由格式的文本到图像（T2I）生成。我们首先通过对网络、合成和文本渲染数据进行训练，将冻结表示编码器 (SigLIP-2) 上的 RAE 解码器扩展到 ImageNet 之外，发现虽然规模提高了总体保真度，但有针对性的数据组合对于文本等特定领域至关重要。然后，我们对最初为 ImageNet 提出的 RAE 设计选择进行严格的压力测试。我们的分析表明，扩展简化了框架：虽然与维度相关的噪声调度仍然至关重要，但诸如宽扩散头和噪声增强解码之类的架构复杂性在规模上提供的优势可以忽略不计。在这个简化的框架上构建，我们对扩散变压器从 0.5B 到 9.8B 参数范围内的 RAE 与最先进的 FLUX VAE 进行了受控比较。在所有模型规模的预训练过程中，RAE 的表现始终优于 VAE。此外，在对高质量数据集进行微调期间，基于 VAE 的模型在 64 个 epoch 后出现灾难性的过拟合，而 RAE 模型在 256 个 epoch 中保持稳定，并始终获得更好的性能。在所有实验中，基于 RAE 的扩散模型表现出更快的收敛速度和更好的生成质量，使 RAE 成为比 VAE 更简单、更强大的基础，可用于大规模 T2I 生成。此外，由于视觉理解和生成都可以在共享表示空间中运行，因此多模态模型可以直接对生成的潜在变量进行推理，为统一模型开辟了新的可能性。

Title: PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation

• Authors: Onkar Susladkar, Tushar Prakash, Adheesh Juvekar, Kiet A. Nguyen, Dong-Hwan Jang, Inderjit S Dhillon, Ismini Lourentzou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2601.16210
• Pdf URL: https://arxiv.org/pdf/2601.16210
• Copy Paste: [[2601.16210]] PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation(https://arxiv.org/abs/2601.16210)
• Keywords: generation
• Abstract: Discrete video VAEs underpin modern text-to-video generation and video understanding systems, yet existing tokenizers typically learn visual codebooks at a single scale with limited vocabularies and shallow language supervision, leading to poor cross-modal alignment and zero-shot transfer. We introduce PyraTok, a language-aligned pyramidal tokenizer that learns semantically structured discrete latents across multiple spatiotemporal resolutions. PyraTok builds on a pretrained video VAE and a novel Language aligned Pyramidal Quantization (LaPQ) module that discretizes encoder features at several depths using a shared large binary codebook, yielding compact yet expressive video token sequences. To tightly couple visual tokens with language, PyraTok jointly optimizes multi-scale text-guided quantization and a global autoregressive objective over the token hierarchy. Across ten benchmarks, PyraTok delivers state-of-the-art (SOTA) video reconstruction, consistently improves text-to-video quality, and sets new SOTA zero-shot performance on video segmentation, temporal action localization, and video understanding, scaling robustly to up to 4K/8K resolutions.
• 摘要：离散视频 VAE 是现代文本到视频生成和视频理解系统的基础，但现有的分词器通常在单一尺度上学习视觉码本，词汇量有限，语言监督浅层，导致跨模态对齐和零样本传输较差。我们引入了 PyraTok，一种与语言对齐的金字塔分词器，可以跨多个时空分辨率学习语义结构的离散潜在变量。 PyraTok 建立在预训练视频 VAE 和新颖的语言对齐金字塔量化 (LaPQ) 模块的基础上，该模块使用共享的大型二进制码本在多个深度上离散化编码器特征，从而产生紧凑但富有表现力的视频标记序列。为了将视觉标记与语言紧密结合，PyraTok 联合优化了标记层次结构上的多尺度文本引导量化和全局自回归目标。在十个基准测试中，PyraTok 提供了最先进的 (SOTA) 视频重建，持续提高文本到视频的质量，并在视频分割、时间动作定位和视频理解方面设置了新的 SOTA 零镜头性能，稳健地扩展到高达 4K/8K 分辨率。