2025-06-11

Title: KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache

• Authors: Fei Li, Song Liu, Weiguo Wu, Shiqiang Nie, Jinyu Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.08018
• Pdf URL: https://arxiv.org/pdf/2506.08018
• Copy Paste: [[2506.08018]] KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache(https://arxiv.org/abs/2506.08018)
• Keywords: generation
• Abstract: The high memory demands of the Key-Value (KV) Cache during the inference of Large Language Models (LLMs) severely restrict their deployment in resource-constrained platforms. Quantization can effectively alleviate the memory pressure caused by KV Cache. However, existing methods either rely on static one-size-fits-all precision allocation or fail to dynamically prioritize critical KV in long-context tasks, forcing memory-accuracy-throughput tradeoffs. In this work, we propose a novel mixed-precision quantization method for KV Cache named KVmix. KVmix leverages gradient-based importance analysis to evaluate how individual Key and Value projection matrices affect the model loss, enabling layer-specific bit-width allocation for mix-precision quantization. It dynamically prioritizes higher precision for important layers while aggressively quantizing less influential ones, achieving a tunable balance between accuracy and efficiency. KVmix also introduces a dynamic long-context optimization strategy that adaptively keeps full-precision KV pairs for recent pivotal tokens and compresses older ones, achieving high-quality sequence generation with low memory usage. Additionally, KVmix provides efficient low-bit quantization and CUDA kernels to optimize computational overhead. On LLMs such as Llama and Mistral, KVmix achieves near-lossless inference performance with extremely low quantization configuration (Key 2.19bit Value 2.38bit), while delivering a remarkable 4.9x memory compression and a 5.3x speedup in inference throughput.
• 摘要：大型语言模型（LLMS）推断期间，密钥值（KV）缓存的高内存要求严重限制了其在资源约束平台中的部署。量化可以有效缓解KV缓存引起的记忆压力。但是，现有方法要么依赖于静态的单一尺寸 - 所有精度分配，要么无法在长篇文本任务中动态优先考虑关键KV，从而迫使内存 - 准确性 - 直播权衡。在这项工作中，我们为名为KVMIX的KV缓存提出了一种新型的混合精液量化方法。 KVMIX利用基于梯度的重要性分析来评估单个密钥和价值投影矩阵如何影响模型损失，从而实现了特定于层的位宽度分配以进行混合精确量化。它动态优先考虑重要层的更高精度，同时积极地量化影响力较小的层，从而在准确性和效率之间达到可调平衡。 KVMIX还引入了动态的长篇文本优化策略，该策略可自适应地保持全精确的KV对，以用于最近的关键令牌并压缩较旧的代币，从而实现高质量的序列生成，并使用低内存使用。此外，KVMIX提供有效的低位量化和CUDA内核来优化计算开销。在LLMS（例如Llama和Mistral）上，KVMIX以极低的量化配置（密钥2.19位值2.38位）实现了近乎无情的推理性能，同时提供了显着的4.9倍存储压缩和5.3倍加速的推理推理吞吐量。

Title: Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques

• Authors: Asankhaya Sharma
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.08060
• Pdf URL: https://arxiv.org/pdf/2506.08060
• Copy Paste: [[2506.08060]] Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques(https://arxiv.org/abs/2506.08060)
• Keywords: generation
• Abstract: Large language models have transformed natural language processing, yet supervised fine-tuning (SFT) remains computationally intensive. This paper formally proves that capabilities acquired through SFT can be approximated by a base transformer model using inference-time techniques, specifically in-context learning (ICL), without altering model parameters, under idealized assumptions including unbounded computational resources and access to the fine-tuning dataset. We extend these results to practical scenarios with finite context lengths and partial dataset access. For text generation tasks with fixed output length $l$, datasets of size $\mathrm{O}\left( \frac{m V}{\varepsilon^2} \log \frac{m}{\delta} \right)$ or, with bounded context, $\mathrm{O}\left( \frac{l \log V}{\varepsilon^2} \log \frac{1}{\delta} \right)$ suffice to approximate fine-tuned behavior across $m$ contexts within error $\varepsilon$, where $V$ is the vocabulary size and $\delta$ is the failure probability. For linear classification, datasets of size $\mathrm{O}\left( \frac{d}{\varepsilon} \right)$ or, with fixed context, $\mathrm{O}\left( \frac{1}{\varepsilon^2} \log \frac{1}{\delta} \right)$ are sufficient, where $d$ is the input dimension. Grounded in the Turing completeness of transformers, these results provide a theoretical foundation for resource-efficient deployment of large language models, with practical techniques like retrieval-augmented generation bridging theory to real-world applications.
• 摘要：大型语言模型已经改变了自然语言处理，但是受监督的微调（SFT）仍然在计算密集型上。本文正式证明，通过推理时间技术，特别是文本学习（ICL），可以通过基本变压器模型近似通过SFT获得的功能，而无需更改模型参数，在理想化的假设下，包括无界的计算资源和访问精细调整数据集。我们将这些结果扩展到具有有限上下文长度和部分数据集访问的实际情况。对于具有固定输出长度$ l $的文本生成任务，大小$ \ mathrm {o} \ left的数据集（\ frac {m v} {\ varepsilon^2} \ log \ log \ frac {m} {\ delta} {\ delta} {\ delta} \ right） V}{\varepsilon^2} \log \frac{1}{\delta} \right)$ suffice to approximate fine-tuned behavior across $m$ contexts within error $\varepsilon$, where $V$ is the vocabulary size and $\delta$ is the failure probability.对于线性分类，大小$ \ mathrm {o} \ left（\ frac {d} {\ varepsilon} \ right）$的数据集或，$ \ mathrm {o} \ right）$足够，其中$ d $是输入维度。这些结果基于变形金刚的图丁完整性，为大型语言模型的资源有效部署提供了理论基础，并采用了实用的技术，例如检索启动的生成桥接理论，以实现现实世界的应用。

Title: Benchmarking Pre-Trained Time Series Models for Electricity Price Forecasting

• Authors: Timothée Hornek Amir Sartipi, Igor Tchappi, Gilbert Fridgen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.08113
• Pdf URL: https://arxiv.org/pdf/2506.08113
• Copy Paste: [[2506.08113]] Benchmarking Pre-Trained Time Series Models for Electricity Price Forecasting(https://arxiv.org/abs/2506.08113)
• Keywords: generative
• Abstract: Accurate electricity price forecasting (EPF) is crucial for effective decision-making in power trading on the spot market. While recent advances in generative artificial intelligence (GenAI) and pre-trained large language models (LLMs) have inspired the development of numerous time series foundation models (TSFMs) for time series forecasting, their effectiveness in EPF remains uncertain. To address this gap, we benchmark several state-of-the-art pretrained models--Chronos-Bolt, Chronos-T5, TimesFM, Moirai, Time-MoE, and TimeGPT--against established statistical and machine learning (ML) methods for EPF. Using 2024 day-ahead auction (DAA) electricity prices from Germany, France, the Netherlands, Austria, and Belgium, we generate daily forecasts with a one-day horizon. Chronos-Bolt and Time-MoE emerge as the strongest among the TSFMs, performing on par with traditional models. However, the biseasonal MSTL model, which captures daily and weekly seasonality, stands out for its consistent performance across countries and evaluation metrics, with no TSFM statistically outperforming it.
• 摘要：准确的电价预测（EPF）对于现货市场上的电力交易有效决策至关重要。尽管最新的生成人工智能（Genai）和预训练的大语言模型（LLM）的进步激发了为时间序列预测的众多时间序列基础模型（TSFM）的发展，但它们在EPF中的有效性仍然不确定。为了解决这一差距，我们基准了几种最先进的预处理模型 - Chronos Bolt，Chronos-T5，TimesFM，Moirai，Time-MoE和TimeGpt-against against against建立的EPF的统计和机器学习（ML）方法。我们使用2024年的日前拍卖（DAA）电价，来自德国，法国，荷兰，奥地利和比利时，我们每天都会以一日的视野产生预测。 Chronos螺栓和Time-MoE成为TSFM中最强的，与传统模型相当。但是，捕获每日和每周季节性的双季度MSTL模型在各国和评估指标之间的稳定表现而脱颖而出，而TSFM统计上没有表现优于它。

Title: BLUR: A Bi-Level Optimization Approach for LLM Unlearning

• Authors: Hadi Reisizadeh, Jinghan Jia, Zhiqi Bu, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, Sijia Liu, Mingyi Hong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2506.08164
• Pdf URL: https://arxiv.org/pdf/2506.08164
• Copy Paste: [[2506.08164]] BLUR: A Bi-Level Optimization Approach for LLM Unlearning(https://arxiv.org/abs/2506.08164)
• Keywords: generative
• Abstract: Enabling large language models (LLMs) to unlearn knowledge and capabilities acquired during training has proven vital for ensuring compliance with data regulations and promoting ethical practices in generative AI. Although there are growing interests in developing various unlearning algorithms, it remains unclear how to best formulate the unlearning problem. The most popular formulation uses a weighted sum of forget and retain loss, but it often leads to performance degradation due to the inherent trade-off between forget and retain losses. In this work, we argue that it is important to model the hierarchical structure of the unlearning problem, where the forget problem (which \textit{unlearns} certain knowledge and/or capabilities) takes priority over the retain problem (which preserves model utility). This hierarchical structure naturally leads to a bi-level optimization formulation where the lower-level objective focuses on minimizing the forget loss, while the upper-level objective aims to maintain the model's utility. Based on this new formulation, we propose a novel algorithm, termed Bi-Level UnleaRning (\texttt{BLUR}), which not only possesses strong theoretical guarantees but more importantly, delivers superior performance. In particular, our extensive experiments demonstrate that \texttt{BLUR} consistently outperforms all the state-of-the-art algorithms across various unlearning tasks, models, and metrics. Codes are available at this https URL.
• 摘要：事实证明，使大型语言模型（LLM）能够在培训期间获得知识和能力，这对于确保遵守数据法规并促进生成AI中的道德实践至关重要。尽管在开发各种学习算法方面越来越兴趣，但尚不清楚如何最好地制定未学习问题。最受欢迎的配方使用忘记和保留损失的加权总和，但由于忘记和保留损失之间的固有权衡，它通常会导致性能退化。在这项工作中，我们认为对未学习问题的层次结构进行建模非常重要，在这种情况下，忘记问题（\ textit {uncorearns}某些知识和/或能力）优先于保留问题（保留模型实用程序）。这种层次结构自然会导致双层优化公式，其中低级目标着重于最大程度地减少忘记损失，而高级目标则旨在维持模型的效用。基于这种新的表述，我们提出了一种新型算法，称为双层学习（\ texttt {blur}），该算法不仅具有强大的理论保证，而且更重要的是，还提供了卓越的性能。特别是，我们的广泛实验表明，\ texttt {Blur}始终优于各种未学习任务，模型和指标上所有最新算法。代码可在此HTTPS URL上找到。

Title: Open World Scene Graph Generation using Vision Language Models

• Authors: Amartya Dutta, Kazi Sajeed Mehrab, Medha Sawhney, Abhilash Neog, Mridul Khurana, Sepideh Fatemi, Aanish Pradhan, M. Maruf, Ismini Lourentzou, Arka Daw, Anuj Karpatne
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.08189
• Pdf URL: https://arxiv.org/pdf/2506.08189
• Copy Paste: [[2506.08189]] Open World Scene Graph Generation using Vision Language Models(https://arxiv.org/abs/2506.08189)
• Keywords: generation
• Abstract: Scene-Graph Generation (SGG) seeks to recognize objects in an image and distill their salient pairwise relationships. Most methods depend on dataset-specific supervision to learn the variety of interactions, restricting their usefulness in open-world settings, involving novel objects and/or relations. Even methods that leverage large Vision Language Models (VLMs) typically require benchmark-specific fine-tuning. We introduce Open-World SGG, a training-free, efficient, model-agnostic framework that taps directly into the pretrained knowledge of VLMs to produce scene graphs with zero additional learning. Casting SGG as a zero-shot structured-reasoning problem, our method combines multimodal prompting, embedding alignment, and a lightweight pair-refinement strategy, enabling inference over unseen object vocabularies and relation sets. To assess this setting, we formalize an Open-World evaluation protocol that measures performance when no SGG-specific data have been observed either in terms of objects and relations. Experiments on Visual Genome, Open Images V6, and the Panoptic Scene Graph (PSG) dataset demonstrate the capacity of pretrained VLMs to perform relational understanding without task-level training.
• 摘要：场景图（SGG）试图在图像中识别对象并提炼其显着的成对关系。大多数方法都取决于特定于数据集的监督来学习各种交互，从而限制了它们在开放世界中的有用性，涉及新颖的对象和/或关系。即使是利用大型视觉语言模型（VLM）的方法，通常也需要特定于基准的微调。我们介绍了开放世界的SGG，这是一种无训练，高效，模型不足的框架，它直接挖掘到VLMS验证的知识中，以产生零额外学习的场景图。我们的方法将SGG铸造为一个零射的结构化问题，结合了多模式提示，嵌入对准和轻巧的配对策略，从而对未看到的对象词汇和关系集进行了推断。为了评估此设置，我们将一个开放世界评估协议正式化，该协议在未观察到对象和关系方面没有观察到SGG特定数据时衡量性能。在视觉基因组，开放图像V6和Panoptic场景图（PSG）数据集上进行的实验证明了验证的VLM在没有任务级别训练的情况下执行关系理解的能力。

Title: Generative Learning of Differentiable Object Models for Compositional Interpretation of Complex Scenes

• Authors: Antoni Nowinowski, Krzysztof Krawiec
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.08191
• Pdf URL: https://arxiv.org/pdf/2506.08191
• Copy Paste: [[2506.08191]] Generative Learning of Differentiable Object Models for Compositional Interpretation of Complex Scenes(https://arxiv.org/abs/2506.08191)
• Keywords: generative
• Abstract: This study builds on the architecture of the Disentangler of Visual Priors (DVP), a type of autoencoder that learns to interpret scenes by decomposing the perceived objects into independent visual aspects of shape, size, orientation, and color appearance. These aspects are expressed as latent parameters which control a differentiable renderer that performs image reconstruction, so that the model can be trained end-to-end with gradient using reconstruction loss. In this study, we extend the original DVP so that it can handle multiple objects in a scene. We also exploit the interpretability of its latent by using the decoder to sample additional training examples and devising alternative training modes that rely on loss functions defined not only in the image space, but also in the latent space. This significantly facilitates training, which is otherwise challenging due to the presence of extensive plateaus in the image-space reconstruction loss. To examine the performance of this approach, we propose a new benchmark featuring multiple 2D objects, which subsumes the previously proposed Multi-dSprites dataset while being more parameterizable. We compare the DVP extended in these ways with two baselines (MONet and LIVE) and demonstrate its superiority in terms of reconstruction quality and capacity to decompose overlapping objects. We also analyze the gradients induced by the considered loss functions, explain how they impact the efficacy of training, and discuss the limitations of differentiable rendering in autoencoders and the ways in which they can be addressed.
• 摘要：这项研究建立在视觉先验的脱节器（DVP）的架构上，这是一种自动编码器，通过将感知到的对象分解为形状，大小，方向和颜色外观的独立视觉方面来学会解释场景。这些方面表示为潜在参数，该参数控制执行图像重建的可区分渲染器，因此可以使用重建损失通过梯度端对端训练模型。在这项研究中，我们扩展了原始DVP，以便它可以在场景中处理多个对象。我们还通过使用解码器来采样其他训练示例，并设计替代依赖损失功能的替代训练模式，不仅在图像空间中，而且在潜在空间中定义了损失功能，从而利用其潜在的解释性。这显着促进了训练，否则这是由于图像空间重建损失中存在广泛的高原而具有挑战性的。为了检查这种方法的性能，我们提出了一个具有多个2D对象的新基准测试，该基准包含了先前提出的Multi-Dsprites数据集，同时更具参数化。我们将以这些方式扩展的DVP与两个基线（Monet和Live）进行了比较，并在重建质量和分解重叠对象的能力方面展示了其优越性。我们还分析了由考虑的损失功能引起的梯度，解释它们如何影响训练的功效，并讨论自动编码器中可区分渲染的局限性及其解决方式。

Title: A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation

• Authors: Andrew Z. Wang, Songwei Ge, Tero Karras, Ming-Yu Liu, Yogesh Balaji
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.08210
• Pdf URL: https://arxiv.org/pdf/2506.08210
• Copy Paste: [[2506.08210]] A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation(https://arxiv.org/abs/2506.08210)
• Keywords: generation
• Abstract: Both text-to-image generation and large language models (LLMs) have made significant advancements. However, many text-to-image models still employ the somewhat outdated T5 and CLIP as their text encoders. In this work, we investigate the effectiveness of using modern decoder-only LLMs as text encoders for text-to-image diffusion models. We build a standardized training and evaluation pipeline that allows us to isolate and evaluate the effect of different text embeddings. We train a total of 27 text-to-image models with 12 different text encoders to analyze the critical aspects of LLMs that could impact text-to-image generation, including the approaches to extract embeddings, different LLMs variants, and model sizes. Our experiments reveal that the de facto way of using last-layer embeddings as conditioning leads to inferior performance. Instead, we explore embeddings from various layers and find that using layer-normalized averaging across all layers significantly improves alignment with complex prompts. Most LLMs with this conditioning outperform the baseline T5 model, showing enhanced performance in advanced visio-linguistic reasoning skills.
• 摘要：文本到图像生成和大型语言模型（LLM）都取得了重大进步。但是，许多文本对图像模型仍然采用过时的T5和剪辑作为文本编码。在这项工作中，我们研究了使用仅现代解码器LLM作为文本到图像扩散模型的文本编码的有效性。我们建立了标准化的培训和评估管道，使我们能够隔离和评估不同文本嵌入的效果。我们总共使用12个不同的文本编码器来培训27个文本对图像模型，以分析LLM的关键方面，这些方面可能影响文本到图像生成，包括提取嵌入式，不同的LLMS变体和模型大小的方法。我们的实验表明，将最后一层嵌入作为条件的事实上的方法会导致劣等性能。取而代之的是，我们探索来自各个层的嵌入，发现在所有层中使用层归一化平均化可以显着改善复杂的提示。大多数具有这种条件的LLM的表现都优于基线T5模型，显示出高级视觉语言推理技能的性能增强。

Title: Ensuring Reliability of Curated EHR-Derived Data: The Validation of Accuracy for LLM/ML-Extracted Information and Data (VALID) Framework

• Authors: Melissa Estevez, Nisha Singh, Lauren Dyson, Blythe Adamson, Qianyu Yuan, Megan W. Hildner, Erin Fidyk, Olive Mbah, Farhad Khan, Kathi Seidl-Rathkopf, Aaron B. Cohen
• Subjects: cs.LG, cs.AI, cs.PF
• Abstract URL: https://arxiv.org/abs/2506.08231
• Pdf URL: https://arxiv.org/pdf/2506.08231
• Copy Paste: [[2506.08231]] Ensuring Reliability of Curated EHR-Derived Data: The Validation of Accuracy for LLM/ML-Extracted Information and Data (VALID) Framework(https://arxiv.org/abs/2506.08231)
• Keywords: generation
• Abstract: Large language models (LLMs) are increasingly used to extract clinical data from electronic health records (EHRs), offering significant improvements in scalability and efficiency for real-world data (RWD) curation in oncology. However, the adoption of LLMs introduces new challenges in ensuring the reliability, accuracy, and fairness of extracted data, which are essential for research, regulatory, and clinical applications. Existing quality assurance frameworks for RWD and artificial intelligence do not fully address the unique error modes and complexities associated with LLM-extracted data. In this paper, we propose a comprehensive framework for evaluating the quality of clinical data extracted by LLMs. The framework integrates variable-level performance benchmarking against expert human abstraction, automated verification checks for internal consistency and plausibility, and replication analyses comparing LLM-extracted data to human-abstracted datasets or external standards. This multidimensional approach enables the identification of variables most in need of improvement, systematic detection of latent errors, and confirmation of dataset fitness-for-purpose in real-world research. Additionally, the framework supports bias assessment by stratifying metrics across demographic subgroups. By providing a rigorous and transparent method for assessing LLM-extracted RWD, this framework advances industry standards and supports the trustworthy use of AI-powered evidence generation in oncology research and practice.
• 摘要：大型语言模型（LLM）越来越多地用于从电子健康记录（EHR）中提取临床数据，从而在肿瘤学中可扩展到现实世界数据（RWD）策划的可伸缩性和效率显着提高。但是，LLM的采用引入了确保提取数据的可靠性，准确性和公平性的新挑战，这对于研究，监管和临床应用至关重要。 RWD和人工智能的现有质量保证框架无法完全解决与LLM提取数据相关的唯一错误模式和复杂性。在本文中，我们提出了一个综合框架，用于评估LLMS提取的临床数据的质量。该框架集成了可变级别的性能基准，以针对人类的专家抽象，自动验证检查，以确保内部一致性和合理性，并进行了将LLM提取的数据与人类吸收的数据集或外部标准的复制分析。这种多维方法可以识别最需要改进的变量，潜在错误的系统检测以及在现实世界中研究数据集健身的确认。此外，该框架通过对跨人口亚组进行分层指标来支持偏差评估。通过提供一种严格透明的方法来评估LLM提取的RWD，该框架提高了行业标准，并支持在肿瘤学研究和实践中可信赖地使用AI驱动的证据。

Title: SHIELD: Secure Hypernetworks for Incremental Expansion Learning Defense

• Authors: Patryk Krukowski, Łukasz Gorczyca, Piotr Helm, Kamil Książek, Przemysław Spurek
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.08255
• Pdf URL: https://arxiv.org/pdf/2506.08255
• Copy Paste: [[2506.08255]] SHIELD: Secure Hypernetworks for Incremental Expansion Learning Defense(https://arxiv.org/abs/2506.08255)
• Keywords: generation
• Abstract: Traditional deep neural networks suffer from several limitations, including catastrophic forgetting. When models are adapted to new datasets, they tend to quickly forget previously learned knowledge. Another significant issue is the lack of robustness to even small perturbations in the input data. In practice, we can often easily perform adversarial attacks and change the network's predictions, adding minimal noise to the input. Dedicated architectures and training procedures can solve each of the above problems separately. Unfortunately, currently, no model can simultaneously address both catastrophic forgetting and vulnerability to adversarial attacks. We introduce SHIELD (Secure Hypernetworks for Incremental Expansion and Learning Defense), a novel approach that integrates a hypernetwork-based continual learning approach with interval arithmetic. SHIELD use the hypernetwork to transfer trainable task embedding vectors into the weights of a target model dedicated to specific data. This paradigm allows for the dynamic generation of separate networks for each subtask, while the hypernetwork aggregates and analyzes information across all tasks. The target model takes in the input a data sample with a defined interval range, and by creating a hypercube, produces a prediction for the given range. Therefore, such target models provide strict guarantees against all possible attacks for data samples within the interval range. Our approach enhances security without sacrificing network adaptability, addressing the overlooked challenge of safety in continual learning.
• 摘要：传统的深层神经网络遭受了多种局限性，包括灾难性的遗忘。当模型适应新的数据集时，他们倾向于迅速忘记以前学习的知识。另一个重要的问题是，在输入数据中，甚至对小扰动都缺乏鲁棒性。实际上，我们通常可以轻松地执行对抗性攻击并更改网络的预测，从而为输入增加最小的噪声。专门的体系结构和培训程序可以分别解决上述每个问题。不幸的是，目前，没有任何模型可以同时解决灾难性遗忘和对对抗攻击的脆弱性。我们介绍了Shield（用于增量扩展和学习防御的安全超网络），这是一种新颖的方法，该方法将基于超网络的持续学习方法与间隔算术相结合。盾牌使用超网络将嵌入向量嵌入的可训练任务转移到专门针对特定数据的目标模型的权重中。该范式允许为每个子任务的单独网络动态生成，而HyperNetwork聚集物和分析所有任务中的信息。目标模型以定义的间隔范围为输入数据样本，并通过创建超立方体对给定范围产生预测。因此，此类目标模型可为间隔范围内的数据样本提供严格的保证。我们的方法在不牺牲网络适应性的情况下增强了安全性，解决了持续学习中安全挑战所忽视的挑战。

Title: Highly Compressed Tokenizer Can Generate Without Training

• Authors: L. Lao Beyer, T. Li, X. Chen, S. Karaman, K. He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.08257
• Pdf URL: https://arxiv.org/pdf/2506.08257
• Copy Paste: [[2506.08257]] Highly Compressed Tokenizer Can Generate Without Training(https://arxiv.org/abs/2506.08257)
• Keywords: generation, generative
• Abstract: Commonly used image tokenizers produce a 2D grid of spatially arranged tokens. In contrast, so-called 1D image tokenizers represent images as highly compressed one-dimensional sequences of as few as 32 discrete tokens. We find that the high degree of compression achieved by a 1D tokenizer with vector quantization enables image editing and generative capabilities through heuristic manipulation of tokens, demonstrating that even very crude manipulations -- such as copying and replacing tokens between latent representations of images -- enable fine-grained image editing by transferring appearance and semantic attributes. Motivated by the expressivity of the 1D tokenizer's latent space, we construct an image generation pipeline leveraging gradient-based test-time optimization of tokens with plug-and-play loss functions such as reconstruction or CLIP similarity. Our approach is demonstrated for inpainting and text-guided image editing use cases, and can generate diverse and realistic samples without requiring training of any generative model.
• 摘要：常用的图像令牌产生的2D网格是空间布置的令牌。相反，所谓的1D图像令牌代表图像表示高度压缩的一维序列，该序列的序列少于32个离散令牌。我们发现，具有矢量量化的1D令牌实现的高度压缩能够通过对代币的启发式操纵来实现图像编辑和生成能力，这表明即使是非常粗略的操作，例如复制和替换图像的潜在表示 - 通过传递图像的潜在表示，通过传递良好的图像编辑外观和语义属性和语义属性。由1D令牌的潜在空间的表达性激励，我们构建了一个图像生成管道，利用基于梯度的测试时间优化令牌具有插件损耗功能，例如重建或剪辑相似性。我们的方法是为了介绍和文本指导的图像编辑用例，并且可以生成多样化和现实的样本，而无需培训任何生成模型。

Title: Universal Differential Equations for Scientific Machine Learning of Node-Wise Battery Dynamics in Smart Grids

• Authors: Tarushri N. S.
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2506.08272
• Pdf URL: https://arxiv.org/pdf/2506.08272
• Copy Paste: [[2506.08272]] Universal Differential Equations for Scientific Machine Learning of Node-Wise Battery Dynamics in Smart Grids(https://arxiv.org/abs/2506.08272)
• Keywords: generation
• Abstract: Universal Differential Equations (UDEs), which blend neural networks with physical differential equations, have emerged as a powerful framework for scientific machine learning (SciML), enabling data-efficient, interpretable, and physically consistent modeling. In the context of smart grid systems, modeling node-wise battery dynamics remains a challenge due to the stochasticity of solar input and variability in household load profiles. Traditional approaches often struggle with generalization and fail to capture unmodeled residual dynamics. This work proposes a UDE-based approach to learn node-specific battery evolution by embedding a neural residual into a physically inspired battery ODE. Synthetic yet realistic solar generation and load demand data are used to simulate battery dynamics over time. The neural component learns to model unobserved or stochastic corrections arising from heterogeneity in node demand and environmental conditions. Comprehensive experiments reveal that the trained UDE aligns closely with ground truth battery trajectories, exhibits smooth convergence behavior, and maintains stability in long-term forecasts. These findings affirm the viability of UDE-based SciML approaches for battery modeling in decentralized energy networks and suggest broader implications for real-time control and optimization in renewable-integrated smart grids.
• 摘要：将神经网络与物理微分方程混合在一起的通用微分方程（UDES）已成为科学机器学习（SCIML）的有力框架，从而有助于数据有效，可解释和物理一致的建模。在智能电网系统的背景下，由于太阳能输入的随机性和家庭负载概况的可变性，对节点的电池动力学进行建模仍然是一个挑战。传统方法通常在概括方面遇到困难，并且无法捕获未建模的残差动态。这项工作提出了一种基于UDE的方法，可以通过将神经残留物嵌入到物理启发的电池颂歌中来学习节点特定的电池演变。合成而逼真的太阳能生成和负载需求数据用于模拟电池动力学随着时间的流逝。神经成分学会为在节点需求和环境条件下异质性引起的未观察或随机校正建模。全面的实验表明，受过训练的UDE与地面真相电池轨迹紧密一致，表现出平滑的收敛行为，并在长期预测中保持稳定性。这些发现肯定了基于UDE的SCIML方法在分散能源网络中进行电池建模的可行性，并提出了对可再生综合智能电网中实时控制和优化的更广泛含义。

Title: Seeing Voices: Generating A-Roll Video from Audio with Mirage

• Authors: Aditi Sundararaman, Amogh Adishesha, Andrew Jaegle, Dan Bigioi, Hyoung-Kyu Song, Jon Kyl, Justin Mao, Kevin Lan, Mojtaba Komeili, ShahRukh Athar, Sheila Babayan, Stanislau Beliasau, William Buchwalter
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.08279
• Pdf URL: https://arxiv.org/pdf/2506.08279
• Copy Paste: [[2506.08279]] Seeing Voices: Generating A-Roll Video from Audio with Mirage(https://arxiv.org/abs/2506.08279)
• Keywords: generation
• Abstract: From professional filmmaking to user-generated content, creators and consumers have long recognized that the power of video depends on the harmonious integration of what we hear (the video's audio track) with what we see (the video's image sequence). Current approaches to video generation either ignore sound to focus on general-purpose but silent image sequence generation or address both visual and audio elements but focus on restricted application domains such as re-dubbing. We introduce Mirage, an audio-to-video foundation model that excels at generating realistic, expressive output imagery from scratch given an audio input. When integrated with existing methods for speech synthesis (text-to-speech, or TTS), Mirage results in compelling multimodal video. When trained on audio-video footage of people talking (A-roll) and conditioned on audio containing speech, Mirage generates video of people delivering a believable interpretation of the performance implicit in input audio. Our central technical contribution is a unified method for training self-attention-based audio-to-video generation models, either from scratch or given existing weights. This methodology allows Mirage to retain generality as an approach to audio-to-video generation while producing outputs of superior subjective quality to methods that incorporate audio-specific architectures or loss components specific to people, speech, or details of how images or audio are captured. We encourage readers to watch and listen to the results of Mirage for themselves (see paper and comments for links).
• 摘要：从专业电影制作到用户生成的内容，创作者和消费者长期以来都认识到，视频的力量取决于我们听到的（视频的音轨）与我们所看到的（视频的图像序列）的和谐整合。视频生成的当前方法要么忽略声音，要么专注于通用，但无声的图像序列生成，或者解决视觉和音频元素，而要关注限制的应用域，例如重新调试。我们介绍了Mirage，这是一种音频到视频基础模型，它在给定音频输入的情况下擅长从头开始生成现实的表达性输出图像。当与现有语音合成方法（文本到语音或TTS）集成时，幻影会导致引人注目的多模式视频。当对人说话的人（a-roll）的音频视频录像并在包含语音的音频条件下进行培训时，Mirage生成了人们对输入音频隐含的性能的可信解释的视频。我们的中心技术贡献是一种统一的方法，用于训练基于自发的音频到视频生成模型，无论是从头开始还是给予现有权重。这种方法允许Mirage保留一般性作为音频到视频生成的方法，同时产生优质主观质量的输出，以结合特定于音频的体系结构或特定于人，语音或如何捕获图像或音频的细节的方法。我们鼓励读者自己观看和倾听幻影的结果（请参阅纸张和链接的评论）。

Title: Private Evolution Converges

• Authors: Tomás González, Giulia Fanti, Aaditya Ramdas
• Subjects: cs.LG, cs.CR, cs.DS, math.PR, math.ST
• Abstract URL: https://arxiv.org/abs/2506.08312
• Pdf URL: https://arxiv.org/pdf/2506.08312
• Copy Paste: [[2506.08312]] Private Evolution Converges(https://arxiv.org/abs/2506.08312)
• Keywords: generation
• Abstract: Private Evolution (PE) is a promising training-free method for differentially private (DP) synthetic data generation. While it achieves strong performance in some domains (e.g., images and text), its behavior in others (e.g., tabular data) is less consistent. To date, the only theoretical analysis of the convergence of PE depends on unrealistic assumptions about both the algorithm's behavior and the structure of the sensitive dataset. In this work, we develop a new theoretical framework to explain PE's practical behavior and identify sufficient conditions for its convergence. For $d$-dimensional sensitive datasets with $n$ data points from a bounded domain, we prove that PE produces an $(\epsilon, \delta)$-DP synthetic dataset with expected 1-Wasserstein distance of order $\tilde{O}(d(n\epsilon)^{-1/d})$ from the original, establishing worst-case convergence of the algorithm as $n \to \infty$. Our analysis extends to general Banach spaces as well. We also connect PE to the Private Signed Measure Mechanism, a method for DP synthetic data generation that has thus far not seen much practical adoption. We demonstrate the practical relevance of our theoretical findings in simulations.
• 摘要：私人进化（PE）是一种有希望的无培训方法，用于差异私有（DP）合成数据生成。尽管它在某些域（例如图像和文本）中实现了强劲的性能，但其在其他域中（例如，表格数据）的行为却不一致。迄今为止，对PE收敛性的唯一理论分析取决于对算法行为和敏感数据集结构的不现实假设。在这项工作中，我们开发了一个新的理论框架来解释PE的实际行为，并确定其融合的足够条件。 For $d$-dimensional sensitive datasets with $n$ data points from a bounded domain, we prove that PE produces an $(\epsilon, \delta)$-DP synthetic dataset with expected 1-Wasserstein distance of order $\tilde{O}(d(n\epsilon)^{-1/d})$ from the original, establishing worst-case convergence of the算法为$ n \ to \ infty $。我们的分析也扩展到了Banach一般的空间。我们还将PE连接到私人签名的度量机制，这是DP合成数据生成的一种方法，到目前为止，没有太多实用的采用。我们在模拟中证明了我们的理论发现的实际相关性。

Title: Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusion

• Authors: Alan N. Amin, Nate Gruver, Andrew Gordon Wilson
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2506.08316
• Pdf URL: https://arxiv.org/pdf/2506.08316
• Copy Paste: [[2506.08316]] Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusion(https://arxiv.org/abs/2506.08316)
• Keywords: generation
• Abstract: Discrete diffusion models, like continuous diffusion models, generate high-quality samples by gradually undoing noise applied to datapoints with a Markov process. Gradual generation in theory comes with many conceptual benefits; for example, inductive biases can be incorporated into the noising Markov process, and access to improved sampling algorithms. In practice, however, the consistently best performing discrete diffusion model is, surprisingly, masking diffusion, which does not denoise gradually. Here we explain the superior performance of masking diffusion by noting that it makes use of a fundamental difference between continuous and discrete Markov processes: discrete Markov processes evolve by discontinuous jumps at a fixed rate and, unlike other discrete diffusion models, masking diffusion builds in the known distribution of jump times and only learns where to jump to. We show that we can similarly bake in the known distribution of jump times into any discrete diffusion model. The resulting models - schedule-conditioned discrete diffusion (SCUD) - generalize classical discrete diffusion and masking diffusion. By applying SCUD to models with noising processes that incorporate inductive biases on images, text, and protein data, we build models that outperform masking.
• 摘要：离散扩散模型（例如连续扩散模型）通过逐渐消除使用Markov过程应用于数据点的噪声来生成高质量的样本。理论上逐渐发电带来了许多概念上的好处。例如，可以将电感偏见纳入noising马尔可夫过程中，并访问改进的采样算法。但是，在实践中，始终如一的最佳性能离散扩散模型是掩盖扩散的，它不会逐渐变形。在这里，我们通过指出连续和离散的马尔可夫流程之间的根本差异来解释掩蔽扩散的出色性能：离散的马尔可夫流程通过不连续的跳跃以固定速率跳跃而演变，并且与其他离散扩散模型不同，在已知的跳跃时间中掩盖了扩散时间，并且只需学习在哪里跳跃。我们表明，我们可以类似地将跳跃时间的已知分布烘烤到任何离散扩散模型中。结果模型 - 计划条件的离散扩散（SCUD） - 概括经典的离散扩散和掩盖扩散。通过将SCUD应用于具有尖锐过程的模型，这些模型在图像，文本和蛋白质数据上都包含归纳性偏见，我们构建了效果超过掩盖的模型。

Title: A Simple Analysis of Discretization Error in Diffusion Models

• Authors: Juhyeok Choi, Chenglin Fan
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2506.08337
• Pdf URL: https://arxiv.org/pdf/2506.08337
• Copy Paste: [[2506.08337]] A Simple Analysis of Discretization Error in Diffusion Models(https://arxiv.org/abs/2506.08337)
• Keywords: generative
• Abstract: Diffusion models, formulated as discretizations of stochastic differential equations (SDEs), achieve state-of-the-art generative performance. However, existing analyses of their discretization error often rely on complex probabilistic tools. In this work, we present a simplified theoretical framework for analyzing the Euler--Maruyama discretization of variance-preserving SDEs (VP-SDEs) in Denoising Diffusion Probabilistic Models (DDPMs), where $ T $ denotes the number of denoising steps in the diffusion process. Our approach leverages Grönwall's inequality to derive a convergence rate of $ \mathcal{O}(1/T^{1/2}) $ under Lipschitz assumptions, significantly streamlining prior proofs. Furthermore, we demonstrate that the Gaussian noise in the discretization can be replaced by a discrete random variable (e.g., Rademacher or uniform noise) without sacrificing convergence guarantees-an insight with practical implications for efficient sampling. Experiments validate our theory, showing that (1) the error scales as predicted, (2) discrete noise achieves comparable sample quality to Gaussian noise, and (3) incorrect noise scaling degrades performance. By unifying simplified analysis and discrete noise substitution, our work bridges theoretical rigor with practical efficiency in diffusion-based generative modeling.
• 摘要：扩散模型，被公式为随机微分方程（SDE）的离散化，实现了最新的生成性能。但是，现有对其离散错误的分析通常依赖于复杂的概率工具。在这项工作中，我们提出了一个简化的理论框架，用于分析Euler-Maruyama-Maruyama在DeNo的扩散概率模型（DDPMS）中的方差提供SDE（VP-SDE）（VP-SDE），其中$ t $表示扩散过程中剥离步骤的数量。我们的方法利用Grönwall的不平等来得出$ \ Mathcal {O}（1/T^{1/2}）$在Lipschitz假设下的收敛速率，从而大大简化了先前的证明。此外，我们证明，离散化中的高斯噪声可以用离散的随机变量（例如，rademacher或统一噪声）代替，而无需牺牲收敛保证 - 对有效采样的实践意义。实验验证了我们的理论，表明（1）误差尺度如所预测的，（2）离散的噪声可与高斯噪声达到可比的样品质量，以及（3）不正确的噪声缩放降低性能。通过统一简化的分析和离散的噪声替代，我们的工作在基于扩散的生成建模中以实用的效率桥接了理论上的严谨性。

Title: Dynamical System Optimization

• Authors: Emo Todorov
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2506.08340
• Pdf URL: https://arxiv.org/pdf/2506.08340
• Copy Paste: [[2506.08340]] Dynamical System Optimization(https://arxiv.org/abs/2506.08340)
• Keywords: generative
• Abstract: We develop an optimization framework centered around a core idea: once a (parametric) policy is specified, control authority is transferred to the policy, resulting in an autonomous dynamical system. Thus we should be able to optimize policy parameters without further reference to controls or actions, and without directly using the machinery of approximate Dynamic Programming and Reinforcement Learning. Here we derive simpler algorithms at the autonomous system level, and show that they compute the same quantities as policy gradients and Hessians, natural gradients, proximal methods. Analogs to approximate policy iteration and off-policy learning are also available. Since policy parameters and other system parameters are treated uniformly, the same algorithms apply to behavioral cloning, mechanism design, system identification, learning of state estimators. Tuning of generative AI models is not only possible, but is conceptually closer to the present framework than to Reinforcement Learning.
• 摘要：我们开发了一个以核心思想为中心的优化框架：一旦指定了（参数）策略，控制权限就会转移到策略中，从而产生了自主动力学系统。因此，我们应该能够在不进一步参考控件或操作的情况下优化策略参数，而无需直接使用近似动态编程和强化学习的机制。在这里，我们在自主系统级别中得出了更简单的算法，并表明它们计算了与策略梯度和黑姐，自然梯度，近端方法相同的数量。还提供了近似政策迭代和违反政策学习的类似物。由于策略参数和其他系统参数均匀处理，因此相同的算法适用于行为克隆，机理设计，系统识别，对状态估计器的学习。调整生成的AI模型不仅是可能的，而且在概念上比当前框架更接近强化学习。

Title: How Much To Guide: Revisiting Adaptive Guidance in Classifier-Free Guidance Text-to-Vision Diffusion Models

• Authors: Huixuan Zhang, Junzhe Zhang, Xiaojun Wan
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2506.08351
• Pdf URL: https://arxiv.org/pdf/2506.08351
• Copy Paste: [[2506.08351]] How Much To Guide: Revisiting Adaptive Guidance in Classifier-Free Guidance Text-to-Vision Diffusion Models(https://arxiv.org/abs/2506.08351)
• Keywords: generation
• Abstract: With the rapid development of text-to-vision generation diffusion models, classifier-free guidance has emerged as the most prevalent method for conditioning. However, this approach inherently requires twice as many steps for model forwarding compared to unconditional generation, resulting in significantly higher costs. While previous study has introduced the concept of adaptive guidance, it lacks solid analysis and empirical results, making previous method unable to be applied to general diffusion models. In this work, we present another perspective of applying adaptive guidance and propose Step AG, which is a simple, universally applicable adaptive guidance strategy. Our evaluations focus on both image quality and image-text alignment. whose results indicate that restricting classifier-free guidance to the first several denoising steps is sufficient for generating high-quality, well-conditioned images, achieving an average speedup of 20% to 30%. Such improvement is consistent across different settings such as inference steps, and various models including video generation models, highlighting the superiority of our method.
• 摘要：随着文本到视觉生成扩散模型的快速发展，无分类器的指导已成为最普遍的调节方法。但是，与无条件生成相比，这种方法本质上需要两倍的模型转发步骤，从而导致成本明显更高。尽管先前的研究介绍了自适应指导的概念，但它缺乏可靠的分析和经验结果，因此无法将先前的方法应用于一般扩散模型。在这项工作中，我们提出了应用自适应指导并提出步骤AG的另一种观点，这是一种简单，普遍适用的自适应指导策略。我们的评估侧重于图像质量和图像文本对齐。其结果表明，将无分类器的指导限制为前几个剥离步骤足以产生高质量的，条件良好的图像，平均速度达到20％至30％。这种改进在不同的设置（例如推理步骤）以及包括视频生成模型在内的各种模型，突出了我们方法的优势。

Title: Spatiotemporal deep learning models for detection of rapid intensification in cyclones

• Authors: Vamshika Sutar, Amandeep Singh, Rohitash Chandra
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2506.08397
• Pdf URL: https://arxiv.org/pdf/2506.08397
• Copy Paste: [[2506.08397]] Spatiotemporal deep learning models for detection of rapid intensification in cyclones(https://arxiv.org/abs/2506.08397)
• Keywords: generation
• Abstract: Cyclone rapid intensification is the rapid increase in cyclone wind intensity, exceeding a threshold of 30 knots, within 24 hours. Rapid intensification is considered an extreme event during a cyclone, and its occurrence is relatively rare, contributing to a class imbalance in the dataset. A diverse array of factors influences the likelihood of a cyclone undergoing rapid intensification, further complicating the task for conventional machine learning models. In this paper, we evaluate deep learning, ensemble learning and data augmentation frameworks to detect cyclone rapid intensification based on wind intensity and spatial coordinates. We note that conventional data augmentation methods cannot be utilised for generating spatiotemporal patterns replicating cyclones that undergo rapid intensification. Therefore, our framework employs deep learning models to generate spatial coordinates and wind intensity that replicate cyclones to address the class imbalance problem of rapid intensification. We also use a deep learning model for the classification module within the data augmentation framework to differentiate between rapid and non-rapid intensification events during a cyclone. Our results show that data augmentation improves the results for rapid intensification detection in cyclones, and spatial coordinates play a critical role as input features to the given models. This paves the way for research in synthetic data generation for spatiotemporal data with extreme events.
• 摘要：气旋快速强化是在24小时内旋风强度的快速增加，超过30节的阈值。快速强化被认为是旋风中的极端事件，其发生相对较少，导致数据集中的类不平衡。各种各样的因素会影响旋风正在快速加强的可能性，从而使传统机器学习模型的任务变得更加复杂。在本文中，我们评估了深度学习，集合学习和数据增强框架，以根据风强度和空间坐标来检测旋风的快速强化。我们注意到，常规数据增强方法不能用于生成时空模式复制经历快速强化的旋风。因此，我们的框架采用深度学习模型来产生空间坐标和风强度，以复制旋风分离器以解决快速强化的类不平衡问题。我们还为数据增强框架内的分类模块使用深度学习模型，以区分旋风期间的快速和非比值强化事件。我们的结果表明，数据增强改善了旋风中快速强化检测的结果，并且空间坐标作为给定模型的输入特征起关键作用。这为与极端事件的时空数据合成数据生成研究铺平了道路。

Title: Learning to Hear Broken Motors: Signature-Guided Data Augmentation for Induction-Motor Diagnostics

• Authors: Saraa Ali, Aleksandr Khizhik, Stepan Svirin, Artem Ryzhikov, Denis Derkach
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2506.08412
• Pdf URL: https://arxiv.org/pdf/2506.08412
• Copy Paste: [[2506.08412]] Learning to Hear Broken Motors: Signature-Guided Data Augmentation for Induction-Motor Diagnostics(https://arxiv.org/abs/2506.08412)
• Keywords: generation
• Abstract: The application of machine learning (ML) algorithms in the intelligent diagnosis of three-phase engines has the potential to significantly enhance diagnostic performance and accuracy. Traditional methods largely rely on signature analysis, which, despite being a standard practice, can benefit from the integration of advanced ML techniques. In our study, we innovate by combining ML algorithms with a novel unsupervised anomaly generation methodology that takes into account the engine physics model. We propose Signature-Guided Data Augmentation (SGDA), an unsupervised framework that synthesizes physically plausible faults directly in the frequency domain of healthy current signals. Guided by Motor Current Signature Analysis, SGDA creates diverse and realistic anomalies without resorting to computationally intensive simulations. This hybrid approach leverages the strengths of both supervised ML and unsupervised signature analysis, achieving superior diagnostic accuracy and reliability along with wide industrial application. The findings highlight the potential of our approach to contribute significantly to the field of engine diagnostics, offering a robust and efficient solution for real-world applications.
• 摘要：机器学习（ML）算法在三相发动机的智能诊断中的应用有可能显着提高诊断性能和准确性。传统方法在很大程度上取决于签名分析，尽管是标准实践，但该分析可以从高级ML技术的整合中受益。在我们的研究中，我们通过将ML算法与一种新颖的无监督异常产生方法相结合来创新，该方法考虑了发动机物理模型。我们提出了签名引导的数据增强（SGDA），这是一个无监督的框架，可直接在健康电流信号的频域中综合物理上合理的故障。在电动电流签名分析的指导下，SGDA创建了多种且逼真的异常，而无需诉诸于计算密集的模拟。这种混合方法利用了监督的ML和无监督的签名分析的优势，从而达到了卓越的诊断准确性和可靠性以及广泛的工业应用。这些发现突出了我们方法对发动机诊断领域的重要贡献的潜力，为现实世界应用提供了强大而有效的解决方案。

Title: Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring

• Authors: Mingjie Xu, Andrew Estornell, Hongzheng Yang, Yuzhi Zhao, Zhaowei Zhu, Qi Xuan, Jiaheng Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.08429
• Pdf URL: https://arxiv.org/pdf/2506.08429
• Copy Paste: [[2506.08429]] Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring(https://arxiv.org/abs/2506.08429)
• Keywords: quality assessment
• Abstract: The application of visual instruction tuning and other post-training techniques has significantly enhanced the capabilities of Large Language Models (LLMs) in visual understanding, enriching Vision-Language Models (VLMs) with more comprehensive visual language datasets. However, the effectiveness of VLMs is highly dependent on large-scale, high-quality datasets that ensure precise recognition and accurate reasoning. Two key challenges hinder progress: (1) noisy alignments between images and the corresponding text, which leads to misinterpretation, and (2) ambiguous or misleading text, which obscures visual content. To address these challenges, we propose SCALE (Single modality data quality and Cross modality Alignment Evaluation), a novel quality-driven data selection pipeline for VLM instruction tuning datasets. Specifically, SCALE integrates a cross-modality assessment framework that first assigns each data entry to its appropriate vision-language task, generates general and task-specific captions (covering scenes, objects, style, etc.), and evaluates the alignment, clarity, task rarity, text coherence, and image clarity of each entry based on the generated captions. We reveal that: (1) current unimodal quality assessment methods evaluate one modality while overlooking the rest, which can underestimate samples essential for specific tasks and discard the lower-quality instances that help build model robustness; and (2) appropriately generated image captions provide an efficient way to transfer the image-text multimodal task into a unified text modality.
• 摘要：视觉指导调整和其他训练后技术的应用可显着增强大语模型（LLMS）在视觉理解中的功能，并使用更全面的视觉语言数据集丰富了视觉语言模型（VLM）。但是，VLM的有效性高度取决于确保精确识别和准确推理的大规模高质量数据集。两个关键挑战阻碍了进步：（1）图像和相应文本之间的嘈杂对齐，这导致了误解，以及（2）模棱两可或误导性文本，这掩盖了视觉内容。为了应对这些挑战，我们提出了量表（单形态数据质量和交叉模态对准评估），这是一种用于VLM指令调谐数据集的新型质量驱动的数据选择管道。具体而言，秤集成了一个跨模式评估框架，该框架首先将每个数据输入分配给其适当的视觉语言任务，生成一般和特定于任务的字幕（涵盖场景，对象，样式等），并根据生成的尺寸来评估每个条目的对齐，清晰度，任务稀有性，文本相干性和图像清晰度。我们透露：（1）当前的单峰质量评估方法评估了一种模式，同时忽略其余的方式，这可能低估了针对特定任务必不可少的样本，并丢弃了有助于建立模型稳健性的较低质量的实例； （2）适当生成的图像字幕提供了一种有效的方法，将图像文本多模式任务传输到统一的文本模式中。

Title: Enhancing Motion Dynamics of Image-to-Video Models via Adaptive Low-Pass Guidance

• Authors: June Suk Choi, Kyungmin Lee, Sihyun Yu, Yisol Choi, Jinwoo Shin, Kimin Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.08456
• Pdf URL: https://arxiv.org/pdf/2506.08456
• Copy Paste: [[2506.08456]] Enhancing Motion Dynamics of Image-to-Video Models via Adaptive Low-Pass Guidance(https://arxiv.org/abs/2506.08456)
• Keywords: generation
• Abstract: Recent text-to-video (T2V) models have demonstrated strong capabilities in producing high-quality, dynamic videos. To improve the visual controllability, recent works have considered fine-tuning pre-trained T2V models to support image-to-video (I2V) generation. However, such adaptation frequently suppresses motion dynamics of generated outputs, resulting in more static videos compared to their T2V counterparts. In this work, we analyze this phenomenon and identify that it stems from the premature exposure to high-frequency details in the input image, which biases the sampling process toward a shortcut trajectory that overfits to the static appearance of the reference image. To address this, we propose adaptive low-pass guidance (ALG), a simple fix to the I2V model sampling procedure to generate more dynamic videos without compromising per-frame image quality. Specifically, ALG adaptively modulates the frequency content of the conditioning image by applying low-pass filtering at the early stage of denoising. Extensive experiments demonstrate that ALG significantly improves the temporal dynamics of generated videos, while preserving image fidelity and text alignment. Especially, under VBench-I2V test suite, ALG achieves an average improvement of 36% in dynamic degree without a significant drop in video quality or image fidelity.
• 摘要：最近的文本对视频（T2V）模型表现出在制作高质量，动态视频中的强大功能。为了提高视觉可控性，最近的作品考虑了微调预训练的T2V模型，以支持图像到视频（I2V）生成。但是，这种适应经常抑制生成的输出的运动动力学，与T2V对应物相比，导致更多的静态视频。在这项工作中，我们分析了这一现象，并确定它是由于输入图像中的高频细节的过早暴露，这将采样过程偏向于捷径轨迹，该轨迹过度适应参考图像的静态外观。为了解决这个问题，我们提出了自适应低通指导（ALG），这是对I2V模型采样程序的简单修复程序，可以生成更动态的视频，而不会损害人均图像质量。具体而言，ALG通过在DeNoising的早期阶段应用低通滤波来自适应调节调节图像的频率含量。广泛的实验表明，ALG显着改善了生成的视频的时间动态，同时保留了图像保真度和文本对齐。特别是，在VBENCH-I2V测试套件下，ALG在动态程度上平均提高了36％，而视频质量质量或图像保真度显着下降。

Title: Thermodynamically Consistent Latent Dynamics Identification for Parametric Systems

• Authors: Xiaolong He, Yeonjong Shin, Anthony Gruber, Sohyeon Jung, Kookjin Lee, Youngsoo Choi
• Subjects: cs.LG, cs.CE, math.NA
• Abstract URL: https://arxiv.org/abs/2506.08475
• Pdf URL: https://arxiv.org/pdf/2506.08475
• Copy Paste: [[2506.08475]] Thermodynamically Consistent Latent Dynamics Identification for Parametric Systems(https://arxiv.org/abs/2506.08475)
• Keywords: generation
• Abstract: We propose an efficient thermodynamics-informed latent space dynamics identification (tLaSDI) framework for the reduced-order modeling of parametric nonlinear dynamical systems. This framework integrates autoencoders for dimensionality reduction with newly developed parametric GENERIC formalism-informed neural networks (pGFINNs), which enable efficient learning of parametric latent dynamics while preserving key thermodynamic principles such as free energy conservation and entropy generation across the parameter space. To further enhance model performance, a physics-informed active learning strategy is incorporated, leveraging a greedy, residual-based error indicator to adaptively sample informative training data, outperforming uniform sampling at equivalent computational cost. Numerical experiments on the Burgers' equation and the 1D/1V Vlasov-Poisson equation demonstrate that the proposed method achieves up to 3,528x speed-up with 1-3% relative errors, and significant reduction in training (50-90%) and inference (57-61%) cost. Moreover, the learned latent space dynamics reveal the underlying thermodynamic behavior of the system, offering valuable insights into the physical-space dynamics.
• 摘要：我们为参数非线性动力学系统的减少阶建模提供了有效的热力学潜在空间动力学标识（TLASDI）框架。该框架将自动编码器与新开发的参数通用形式主义神经网络（PGFINNS）集成在一起，从而有效地学习参数潜在动力学，同时保留关键的热力学原理，例如在整个参数空间中进行自由能量保护和熵产生。为了进一步提高模型性能，采用了物理信息的主动学习策略，利用贪婪，基于残余的错误指标来适应性地采样信息培训数据，以等效的计算成本优于统一抽样。关于汉堡方程和1D/1V Vlasov-Poisson方程的数值实验表明，所提出的方法以1-3％的相对误差的速度达到3,528倍，训练（50-90％）和推理（57-61％）的成本显着降低（57-90％）。此外，博学的潜在空间动力学揭示了系统的基本热力学行为，从而为物理空间动力学提供了宝贵的见解。

Title: LiftVSR: Lifting Image Diffusion to Video Super-Resolution via Hybrid Temporal Modeling with Only 4$\times$RTX 4090s

• Authors: Xijun Wang, Xin Li, Bingchen Li, Zhibo Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.08529
• Pdf URL: https://arxiv.org/pdf/2506.08529
• Copy Paste: [[2506.08529]] LiftVSR: Lifting Image Diffusion to Video Super-Resolution via Hybrid Temporal Modeling with Only 4$\times$RTX 4090s(https://arxiv.org/abs/2506.08529)
• Keywords: super-resolution
• Abstract: Diffusion models have significantly advanced video super-resolution (VSR) by enhancing perceptual quality, largely through elaborately designed temporal modeling to ensure inter-frame consistency. However, existing methods usually suffer from limited temporal coherence and prohibitively high computational costs (e.g., typically requiring over 8 NVIDIA A100-80G GPUs), especially for long videos. In this work, we propose LiftVSR, an efficient VSR framework that leverages and elevates the image-wise diffusion prior from PixArt-$\alpha$, achieving state-of-the-art results using only 4$\times$RTX 4090 GPUs. To balance long-term consistency and efficiency, we introduce a hybrid temporal modeling mechanism that decomposes temporal learning into two complementary components: (i) Dynamic Temporal Attention (DTA) for fine-grained temporal modeling within short frame segment ($\textit{i.e.}$, low complexity), and (ii) Attention Memory Cache (AMC) for long-term temporal modeling across segments ($\textit{i.e.}$, consistency). Specifically, DTA identifies multiple token flows across frames within multi-head query and key tokens to warp inter-frame contexts in the value tokens. AMC adaptively aggregates historical segment information via a cache unit, ensuring long-term coherence with minimal overhead. To further stabilize the cache interaction during inference, we introduce an asymmetric sampling strategy that mitigates feature mismatches arising from different diffusion sampling steps. Extensive experiments on several typical VSR benchmarks have demonstrated that LiftVSR achieves impressive performance with significantly lower computational costs.
• 摘要：扩散模型通过提高感知质量（主要是通过精心设计的时间建模，以确保框架间的一致性），具有显着高级的视频超分辨率（VSR）。但是，现有方法通常会遭受时间连贯性有限和高度高的计算成本（例如，通常需要8个NVIDIA A100-80G GPU），尤其是对于长视频而言。在这项工作中，我们提出了LiftVSR，这是一个有效的VSR框架，从PixArt-Pixart-$ \ alpha $中利用和提高图像扩散，仅使用4 $ \ times $ rtx 4090 gpus就可以实现最先进的结果。为了平衡长期的一致性和效率，我们引入了一种混合时间建模机制，将时间学习分解为两个互补组件：（i）在短帧段内进行细粒度的时间建模的动态时间关注（DTA）（$ \ textit {$ \ textit {i.e.e.e.e.e.e.e。} $，低复杂性），以及（ii）注意记忆（II）（ii）的（ii）的（ii）（ii）（ii）的（ii）（ii）（ii）（ii）（ii）（ii）（ii） （$ \ textit {i.e。} $，一致性）。具体而言，DTA识别多头查询和关键令牌内的多个令牌流，以使值代币中的框架间上下文。 AMC通过缓存单元适应了历史段信息，从而确保了长期连贯性与最小的开销。为了进一步稳定推断期间的缓存相互作用，我们引入了一种不对称的采样策略，该策略可减轻由不同的扩散采样步骤引起的不匹配。对几个典型VSR基准测试的广泛实验表明，LiftVSR的计算成本显着降低，实现了令人印象深刻的性能。

Title: Robust Evolutionary Multi-Objective Network Architecture Search for Reinforcement Learning (EMNAS-RL)

• Authors: Nihal Acharya Adde, Alexandra Gianzina, Hanno Gottschalk, Andreas Ebert
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2506.08533
• Pdf URL: https://arxiv.org/pdf/2506.08533
• Copy Paste: [[2506.08533]] Robust Evolutionary Multi-Objective Network Architecture Search for Reinforcement Learning (EMNAS-RL)(https://arxiv.org/abs/2506.08533)
• Keywords: generation
• Abstract: This paper introduces Evolutionary Multi-Objective Network Architecture Search (EMNAS) for the first time to optimize neural network architectures in large-scale Reinforcement Learning (RL) for Autonomous Driving (AD). EMNAS uses genetic algorithms to automate network design, tailored to enhance rewards and reduce model size without compromising performance. Additionally, parallelization techniques are employed to accelerate the search, and teacher-student methodologies are implemented to ensure scalable optimization. This research underscores the potential of transfer learning as a robust framework for optimizing performance across iterative learning processes by effectively leveraging knowledge from earlier generations to enhance learning efficiency and stability in subsequent generations. Experimental results demonstrate that tailored EMNAS outperforms manually designed models, achieving higher rewards with fewer parameters. The findings of these strategies contribute positively to EMNAS for RL in autonomous driving, advancing the field toward better-performing networks suitable for real-world scenarios.
• 摘要：本文首次介绍了进化多目标网络体系结构搜索（EMNA），以优化大规模强化学习（RL）的神经网络体系结构用于自主驾驶（AD）。 EMNA使用遗传算法来自动化网络设计，量身定制，以增强奖励和减少模型大小而不会损害性能。此外，还采用并行化技术来加速搜索，并实施教师研究方法以确保可扩展的优化。这项研究强调了转移学习的潜力，是通过有效利用早期的知识来提高随后几代学习效率和稳定性的知识来优化跨越迭代学习过程的强大框架。实验结果表明，量身定制的EMNA优于手动设计的模型，获得更高的参数奖励。这些策略的发现对自动驾驶中RL的EMNA有效，将领域推进了适合现实情况的更好表现的网络。

Title: TrajFlow: Multi-modal Motion Prediction via Flow Matching

• Authors: Qi Yan, Brian Zhang, Yutong Zhang, Daniel Yang, Joshua White, Di Chen, Jiachao Liu, Langechuan Liu, Binnan Zhuang, Shaoshuai Shi, Renjie Liao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.08541
• Pdf URL: https://arxiv.org/pdf/2506.08541
• Copy Paste: [[2506.08541]] TrajFlow: Multi-modal Motion Prediction via Flow Matching(https://arxiv.org/abs/2506.08541)
• Keywords: generative
• Abstract: Efficient and accurate motion prediction is crucial for ensuring safety and informed decision-making in autonomous driving, particularly under dynamic real-world conditions that necessitate multi-modal forecasts. We introduce TrajFlow, a novel flow matching-based motion prediction framework that addresses the scalability and efficiency challenges of existing generative trajectory prediction methods. Unlike conventional generative approaches that employ i.i.d. sampling and require multiple inference passes to capture diverse outcomes, TrajFlow predicts multiple plausible future trajectories in a single pass, significantly reducing computational overhead while maintaining coherence across predictions. Moreover, we propose a ranking loss based on the Plackett-Luce distribution to improve uncertainty estimation of predicted trajectories. Additionally, we design a self-conditioning training technique that reuses the model's own predictions to construct noisy inputs during a second forward pass, thereby improving generalization and accelerating inference. Extensive experiments on the large-scale Waymo Open Motion Dataset (WOMD) demonstrate that TrajFlow achieves state-of-the-art performance across various key metrics, underscoring its effectiveness for safety-critical autonomous driving applications. The code and other details are available on the project website this https URL.
• 摘要：有效，准确的运动预测对于确保自主驾驶中的安全性和明智的决策至关重要，尤其是在需要多模式预测的动态现实世界中。我们介绍了Trajflow，这是一种基于流动匹配的新型运动预测框架，该框架解决了现有生成轨迹预测方法的可扩展性和效率挑战。与采用I.I.D.的常规生成方法不同抽样并需要多个推理通过以捕获各种结果，Trajflow预测了单个通过中的多个合理的未来轨迹，从而大大降低了计算开销，同时保持了整个预测的连贯性。此外，我们提出了基于种子分布的排名损失，以提高预测轨迹的不确定性估计。此外，我们设计了一种自我调节训练技术，该技术可以重复模型自己的预测，以在第二次前传球期间构建嘈杂的输入，从而改善概括和加速推断。大规模Waymo开放运动数据集（WOMD）进行的广泛实验表明，Trajflow在各种关键指标上实现了最新的性能，从而强调了其对安全至关重要的自主驾驶应用程序的有效性。该代码和其他详细信息可在项目网站上获得此HTTPS URL。

Title: Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations

• Authors: Yibo Cui, Liang Xie, Yu Zhao, Jiawei Sun, Erwei Yin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.08566
• Pdf URL: https://arxiv.org/pdf/2506.08566
• Copy Paste: [[2506.08566]] Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations(https://arxiv.org/abs/2506.08566)
• Keywords: generation, generative
• Abstract: Vision-Language Navigation (VLN) enables intelligent agents to navigate environments by integrating visual perception and natural language instructions, yet faces significant challenges due to the scarcity of fine-grained cross-modal alignment annotations. Existing datasets primarily focus on global instruction-trajectory matching, neglecting sub-instruction-level and entity-level alignments critical for accurate navigation action decision-making. To address this limitation, we propose FCA-NIG, a generative framework that automatically constructs navigation instructions with dual-level fine-grained cross-modal annotations. In this framework, an augmented trajectory is first divided into sub-trajectories, which are then processed through GLIP-based landmark detection, crafted instruction construction, OFA-Speaker based R2R-like instruction generation, and CLIP-powered entity selection, generating sub-instruction-trajectory pairs with entity-landmark annotations. Finally, these sub-pairs are aggregated to form a complete instruction-trajectory pair. The framework generates the FCA-R2R dataset, the first large-scale augmentation dataset featuring precise sub-instruction-sub-trajectory and entity-landmark alignments. Extensive experiments demonstrate that training with FCA-R2R significantly improves the performance of multiple state-of-the-art VLN agents, including SF, EnvDrop, RecBERT, and HAMT. Incorporating sub-instruction-trajectory alignment enhances agents' state awareness and decision accuracy, while entity-landmark alignment further boosts navigation performance and generalization. These results highlight the effectiveness of FCA-NIG in generating high-quality, scalable training data without manual annotation, advancing fine-grained cross-modal learning in complex navigation tasks.
• 摘要：视觉语言导航（VLN）使智能代理可以通过整合视觉感知和自然语言说明来导航环境，但由于缺乏细粒度的跨模式对准注释，面临重大挑战。现有数据集主要集中于全局指导 - 区域匹配，忽略了子指导级和实体级别级别对准确的导航行动决策至关重要。为了解决此限制，我们提出了FCA-NIG，这是一种生成框架，该框架自动构建具有双级细粒跨模式注释的导航指令。在此框架中，首先将增强的轨迹分为子孔径，然后通过基于GLIP的Landmark检测，精心设计的指令构造，基于言论的R2R的R2R式指令生成以及剪贴机式实体选择，生成子建筑 - 特定 - 特殊 - 特定对实体标记的注释。最后，将这些子对聚合以形成一个完整的指令 - 防护对。该框架生成FCA-R2R数据集，这是第一个具有精确子Instruction-Sub-Sub-Trajectory和Entity-Landmark Arignments的大规模增强数据集。广泛的实验表明，FCA-R2R的培训显着提高了包括SF，Envdrop，Recbert和Hamt在内的多种最先进的VLN代理的性能。合并子指导性对准对准可以提高代理人的州意识和决策准确性，而实体 - 地标对准进一步促进了导航性能和概括。这些结果突出了FCA-NIG在不手动注释的情况下生成高质量的可扩展培训数据的有效性，从而在复杂的导航任务中推进了细粒度的跨模式学习。

Title: Diffusion-based Time Series Forecasting for Sewerage Systems

• Authors: Nicholas A. Pearson, Francesca Cairoli, Luca Bortolussi, Davide Russo, Francesca Zanello
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.08577
• Pdf URL: https://arxiv.org/pdf/2506.08577
• Copy Paste: [[2506.08577]] Diffusion-based Time Series Forecasting for Sewerage Systems(https://arxiv.org/abs/2506.08577)
• Keywords: generative
• Abstract: We introduce a novel deep learning approach that harnesses the power of generative artificial intelligence to enhance the accuracy of contextual forecasting in sewerage systems. By developing a diffusion-based model that processes multivariate time series data, our system excels at capturing complex correlations across diverse environmental signals, enabling robust predictions even during extreme weather events. To strengthen the model's reliability, we further calibrate its predictions with a conformal inference technique, tailored for probabilistic time series data, ensuring that the resulting prediction intervals are statistically reliable and cover the true target values with a desired confidence level. Our empirical tests on real sewerage system data confirm the model's exceptional capability to deliver reliable contextual predictions, maintaining accuracy even under severe weather conditions.
• 摘要：我们介绍了一种新颖的深度学习方法，该方法利用了生成人工智能的力量，以增强污水处理系统中情境预测的准确性。通过开发一个基于扩散的模型来处理多元时间序列数据，我们的系统擅长捕获各种环境信号之间的复杂相关性，即使在极端天气事件中也可以实现强大的预测。为了增强模型的可靠性，我们通过共形推理技术进一步校准了其预测，该技术针对概率时间序列数据量身定制，以确保所得的预测间隔在统计上是可靠的，并以所需的置信度覆盖真实的目标值。我们对实际污水处理系统数据的经验测试证实了该模型提供可靠的上下文预测的非凡能力，即使在恶劣天气条件下也保持准确性。

Title: Transformers Meet Hyperspectral Imaging: A Comprehensive Study of Models, Challenges and Open Problems

• Authors: Guyang Zhang, Waleed Abdulla
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.08596
• Pdf URL: https://arxiv.org/pdf/2506.08596
• Copy Paste: [[2506.08596]] Transformers Meet Hyperspectral Imaging: A Comprehensive Study of Models, Challenges and Open Problems(https://arxiv.org/abs/2506.08596)
• Keywords: generation
• Abstract: Transformers have become the architecture of choice for learning long-range dependencies, yet their adoption in hyperspectral imaging (HSI) is still emerging. We reviewed more than 300 papers published up to 2025 and present the first end-to-end survey dedicated to Transformer-based HSI classification. The study categorizes every stage of a typical pipeline-pre-processing, patch or pixel tokenization, positional encoding, spatial-spectral feature extraction, multi-head self-attention variants, skip connections, and loss design-and contrasts alternative design choices with the unique spatial-spectral properties of HSI. We map the field's progress against persistent obstacles: scarce labeled data, extreme spectral dimensionality, computational overhead, and limited model explainability. Finally, we outline a research agenda prioritizing valuable public data sets, lightweight on-edge models, illumination and sensor shifts robustness, and intrinsically interpretable attention mechanisms. Our goal is to guide researchers in selecting, combining, or extending Transformer components that are truly fit for purpose for next-generation HSI applications.
• 摘要：变形金刚已成为学习长期依赖性的首选架构，但是它们在高光谱成像（HSI）中的采用仍在出现。我们审查了截至2025年发表的300多篇论文，并介绍了专门针对基于变压器的HSI分类的第一个端到端调查。该研究对典型的管道预处理，贴片或像素令牌化，位置编码，空间 - 光谱特征提取，多头自我发项式变体，跳过连接以及损失设计和对比的替代设计选择与HSI独特的空间谱性特性。我们将该领域的进度映射到持续的障碍：稀缺标记的数据，极端频谱维度，计算开销和有限的模型解释性。最后，我们概述了一项研究议程，优先考虑有价值的公共数据集，轻量级的边缘模型，照明和传感器转移鲁棒性以及本质上可解释的注意力机制。我们的目标是指导研究人员选择，结合或扩展变压器组件，这些组件真正适合于下一代HSI应用程序。

Title: Flow Matching Meets PDEs: A Unified Framework for Physics-Constrained Generation

• Authors: Giacomo Baldan, Qiang Liu, Alberto Guardone, Nils Thuerey
• Subjects: cs.LG, cs.AI, cs.CE, math.NA
• Abstract URL: https://arxiv.org/abs/2506.08604
• Pdf URL: https://arxiv.org/pdf/2506.08604
• Copy Paste: [[2506.08604]] Flow Matching Meets PDEs: A Unified Framework for Physics-Constrained Generation(https://arxiv.org/abs/2506.08604)
• Keywords: generation, generative
• Abstract: Generative machine learning methods, such as diffusion models and flow matching, have shown great potential in modeling complex system behaviors and building efficient surrogate models. However, these methods typically learn the underlying physics implicitly from data. We propose Physics-Based Flow Matching (PBFM), a novel generative framework that explicitly embeds physical constraints, both PDE residuals and algebraic relations, into the flow matching objective. We also introduce temporal unrolling at training time that improves the accuracy of the final, noise-free sample prediction. Our method jointly minimizes the flow matching loss and the physics-based residual loss without requiring hyperparameter tuning of their relative weights. Additionally, we analyze the role of the minimum noise level, $\sigma_{\min}$, in the context of physical constraints and evaluate a stochastic sampling strategy that helps to reduce physical residuals. Through extensive benchmarks on three representative PDE problems, we show that our approach yields up to an $8\times$ more accurate physical residuals compared to FM, while clearly outperforming existing algorithms in terms of distributional accuracy. PBFM thus provides a principled and efficient framework for surrogate modeling, uncertainty quantification, and accelerated simulation in physics and engineering applications.
• 摘要：生成机器学习方法（例如扩散模型和流量匹配）在建模复杂的系统行为和构建有效的替代模型方面显示出很大的潜力。但是，这些方法通常会隐含地从数据中学习潜在的物理。我们提出了基于物理学的流量匹配（PBFM），这是一种新型生成框架，将PDE残差和代数关系明确嵌入物理约束中。我们还在训练时间引入时间展开，以提高最终无噪声样本预测的准确性。我们的方法共同最大程度地减少了流量匹配损失和基于物理的残余损失，而无需对其相对权重的高参数调整。此外，在物理约束的背景下，我们分析了最小噪声水平的作用，即$ \ sigma _ {\ min} $，并评估一种随机抽样策略，有助于减少物理残差。通过针对三个代表性PDE问题的广泛基准测试，我们表明我们的方法与FM相比，我们的方法高达$ 8 \ times $更准确的物理残留物，同时在分配准确性方面显然超过了现有的算法。因此，PBFM为替代建模，不确定性量化以及物理和工程应用中的仿真提供了一个有原则有效的框架。

Title: RoboSwap: A GAN-driven Video Diffusion Framework For Unsupervised Robot Arm Swapping

• Authors: Yang Bai, Liudi Yang, George Eskandar, Fengyi Shen, Dong Chen, Mohammad Altillawi, Ziyuan Liu, Gitta Kutyniok
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.08632
• Pdf URL: https://arxiv.org/pdf/2506.08632
• Copy Paste: [[2506.08632]] RoboSwap: A GAN-driven Video Diffusion Framework For Unsupervised Robot Arm Swapping(https://arxiv.org/abs/2506.08632)
• Keywords: generative
• Abstract: Recent advancements in generative models have revolutionized video synthesis and editing. However, the scarcity of diverse, high-quality datasets continues to hinder video-conditioned robotic learning, limiting cross-platform generalization. In this work, we address the challenge of swapping a robotic arm in one video with another: a key step for crossembodiment learning. Unlike previous methods that depend on paired video demonstrations in the same environmental settings, our proposed framework, RoboSwap, operates on unpaired data from diverse environments, alleviating the data collection needs. RoboSwap introduces a novel video editing pipeline integrating both GANs and diffusion models, combining their isolated advantages. Specifically, we segment robotic arms from their backgrounds and train an unpaired GAN model to translate one robotic arm to another. The translated arm is blended with the original video background and refined with a diffusion model to enhance coherence, motion realism and object interaction. The GAN and diffusion stages are trained independently. Our experiments demonstrate that RoboSwap outperforms state-of-the-art video and image editing models on three benchmarks in terms of both structural coherence and motion consistency, thereby offering a robust solution for generating reliable, cross-embodiment data in robotic learning.
• 摘要：生成模型的最新进展彻底改变了视频综合和编辑。但是，多样化，高质量数据集的稀缺性继续阻碍视频条件的机器人学习，从而限制了跨平台的概括。在这项工作中，我们解决了在一个视频中与另一个视频中交换机器人手臂的挑战：交流训练学习的关键步骤。与以前取决于在相同环境环境中配对的视频演示的方法不同，我们提出的框架RobosWap在不同环境中的未配对数据进行操作，从而减轻了数据收集需求。 RobosWap介绍了一条新型的视频编辑管道，将gan和扩散模型都集成在一起，结合了它们孤立的优势。具体来说，我们将机器人臂从其背景中分割出来，并训练一个未配对的GAN模型将一个机器人臂转换为另一个机器人手臂。翻译后的臂与原始视频背景混合在一起，并与扩散模型进行了完善，以增强连贯性，运动现实主义和对象相互作用。 GAN和扩散阶段是独立训练的。我们的实验表明，就结构相干性和运动一致性而言，RobosWap在三个基准上的最先进的视频和图像编辑模型都优于最先进的视频和图像编辑模型，从而提供了可靠的解决方案，以生成机器人学习中的可靠的，交叉的数据。

Title: Orientation Matters: Making 3D Generative Models Orientation-Aligned

• Authors: Yichong Lu, Yuzhuo Tian, Zijin Jiang, Yikun Zhao, Yuanbo Yang, Hao Ouyang, Haoji Hu, Huimin Yu, Yujun Shen, Yiyi Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.08640
• Pdf URL: https://arxiv.org/pdf/2506.08640
• Copy Paste: [[2506.08640]] Orientation Matters: Making 3D Generative Models Orientation-Aligned(https://arxiv.org/abs/2506.08640)
• Keywords: generation, generative
• Abstract: Humans intuitively perceive object shape and orientation from a single image, guided by strong priors about canonical poses. However, existing 3D generative models often produce misaligned results due to inconsistent training data, limiting their usability in downstream tasks. To address this gap, we introduce the task of orientation-aligned 3D object generation: producing 3D objects from single images with consistent orientations across categories. To facilitate this, we construct Objaverse-OA, a dataset of 14,832 orientation-aligned 3D models spanning 1,008 categories. Leveraging Objaverse-OA, we fine-tune two representative 3D generative models based on multi-view diffusion and 3D variational autoencoder frameworks to produce aligned objects that generalize well to unseen objects across various categories. Experimental results demonstrate the superiority of our method over post-hoc alignment approaches. Furthermore, we showcase downstream applications enabled by our aligned object generation, including zero-shot object orientation estimation via analysis-by-synthesis and efficient arrow-based object rotation manipulation.
• 摘要：人类直观地感知单个图像的物体形状和方向，并以强大的先验有关规范姿势的指导。但是，由于训练数据不一致，现有的3D生成模型通常会产生未对准的结果，从而限制了其在下游任务中的可用性。为了解决此差距，我们介绍了定向对准的3D对象生成的任务：从单个图像中产生3D对象，该对象跨类别具有一致的方向。为了促进这一点，我们构建了Objaverse-OA，该数据集的数据集为14,832个跨越1,008个类别的3D模型。利用objaverse-oa，我们基于多视图扩散和3D变化自动编码器框架来微调两个代表性的3D生成模型，以产生一个对齐对象，这些对象可以很好地推广到各种类别的看不见的对象。实验结果证明了我们方法比事后比对方法的优越性。此外，我们展示了由对象生成的对象生成启用的下游应用程序，包括通过分析和基于有效的箭头对象旋转操作通过分析和有效的箭头操作进行零摄像对象方向估算。

Title: Breaking the ICE: Exploring promises and challenges of benchmarks for Inference Carbon & Energy estimation for LLMs

• Authors: Samarth Sikand, Rohit Mehra, Priyavanshi Pathania, Nikhil Bamby, Vibhu Saujanya Sharma, Vikrant Kaulgud, Sanjay Podder, Adam P. Burden
• Subjects: cs.LG, cs.AI, cs.CY, cs.SE
• Abstract URL: https://arxiv.org/abs/2506.08727
• Pdf URL: https://arxiv.org/pdf/2506.08727
• Copy Paste: [[2506.08727]] Breaking the ICE: Exploring promises and challenges of benchmarks for Inference Carbon & Energy estimation for LLMs(https://arxiv.org/abs/2506.08727)
• Keywords: generative
• Abstract: While Generative AI stands to be one of the fastest adopted technologies ever, studies have made evident that the usage of Large Language Models (LLMs) puts significant burden on energy grids and our environment. It may prove a hindrance to the Sustainability goals of any organization. A crucial step in any Sustainability strategy is monitoring or estimating the energy consumption of various components. While there exist multiple tools for monitoring energy consumption, there is a dearth of tools/frameworks for estimating the consumption or carbon emissions. Current drawbacks of both monitoring and estimation tools include high input data points, intrusive nature, high error margin, etc. We posit that leveraging emerging LLM benchmarks and related data points can help overcome aforementioned challenges while balancing accuracy of the emission estimations. To that extent, we discuss the challenges of current approaches and present our evolving framework, R-ICE, which estimates prompt level inference carbon emissions by leveraging existing state-of-the-art(SOTA) benchmark. This direction provides a more practical and non-intrusive way to enable emerging use-cases like dynamic LLM routing, carbon accounting, etc. Our promising validation results suggest that benchmark-based modelling holds great potential for inference emission estimation and warrants further exploration from the scientific community.
• 摘要：尽管生成的AI是有史以来最快的采用技术之一，但研究表明，大型语言模型（LLMS）的使用给能源网格和我们的环境带来了巨大负担。它可能证明对任何组织的可持续性目标都有障碍。任何可持续性策略的关键步骤是监视或估计各个组件的能耗。尽管有多种用于监视能源消耗的工具，但缺乏估计消耗量或碳排放的工具/框架。监视和估计工具的当前缺点包括高输入数据点，侵入性性质，高误差率等。我们认为利用新兴的LLM基准和相关数据点可以帮助克服上述挑战，同时平衡排放估算的准确性。在这个程度上，我们讨论了当前方法的挑战，并提出了我们不断发展的框架R-Ice，该框架通过利用现有的最新技术（SOTA）基准来估算推理碳排放及其迅速推理。这个方向提供了一种更实用和非侵入性的方法，可以使新兴用例（如动态LLM路由，碳核算会计等）等新兴用例。我们的有前途的验证结果表明，基于基准的建模具有巨大的推理估算的潜力，并保证了科学界进一步探索。

Title: HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation

• Authors: Ziyao Huang, Zixiang Zhou, Juan Cao, Yifeng Ma, Yi Chen, Zejing Rao, Zhiyong Xu, Hongmei Wang, Qin Lin, Yuan Zhou, Qinglin Lu, Fan Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.08797
• Pdf URL: https://arxiv.org/pdf/2506.08797
• Copy Paste: [[2506.08797]] HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation(https://arxiv.org/abs/2506.08797)
• Keywords: generation
• Abstract: To address key limitations in human-object interaction (HOI) video generation -- specifically the reliance on curated motion data, limited generalization to novel objects/scenarios, and restricted accessibility -- we introduce HunyuanVideo-HOMA, a weakly conditioned multimodal-driven framework. HunyuanVideo-HOMA enhances controllability and reduces dependency on precise inputs through sparse, decoupled motion guidance. It encodes appearance and motion signals into the dual input space of a multimodal diffusion transformer (MMDiT), fusing them within a shared context space to synthesize temporally consistent and physically plausible interactions. To optimize training, we integrate a parameter-space HOI adapter initialized from pretrained MMDiT weights, preserving prior knowledge while enabling efficient adaptation, and a facial cross-attention adapter for anatomically accurate audio-driven lip synchronization. Extensive experiments confirm state-of-the-art performance in interaction naturalness and generalization under weak supervision. Finally, HunyuanVideo-HOMA demonstrates versatility in text-conditioned generation and interactive object manipulation, supported by a user-friendly demo interface. The project page is at this https URL.
• 摘要：为了解决人类对象相互作用（HOI）视频生成的关键局限性 - 特别是对精选运动数据的依赖，对新型对象/场景的概括有限，以及受限的可访问性 - 我们引入了Hunyuanvideo-Homa，这是一个弱条件的多模态驱动框架。 Hunyuanvideo-Homa通过稀疏，脱钩的运动引导可以增强可控性，并降低对精确输入的依赖。它将外观和运动信号编码到多模式扩散变压器（MMDIT）的双输入空间中，将它们融合在共享上下文空间中，以综合时间一致且在物理上可见的相互作用。为了优化训练，我们整合了从预算的MMDIT权重初始化的参数空间HOI适配器，在启用有效适应的同时保留了先验知识，以及一个面部跨注意适配器，用于解剖学上准确的音频驱动的唇部同步。广泛的实验证实了在弱监督下的自然性和概括方面的最新性能。最后，Hunyuanvideo-Homa在文本条件生成和交互式对象操纵中表现出多功能性，并由用户友好的演示接口支持。项目页面位于此HTTPS URL。

Title: CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics

• Authors: Shravan Nayak, Mehar Bhatia, Xiaofeng Zhang, Verena Rieser, Lisa Anne Hendricks, Sjoerd van Steenkiste, Yash Goyal, Karolina Stańczak, Aishwarya Agrawal
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2506.08835
• Pdf URL: https://arxiv.org/pdf/2506.08835
• Copy Paste: [[2506.08835]] CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics(https://arxiv.org/abs/2506.08835)
• Keywords: generation
• Abstract: The increasing ubiquity of text-to-image (T2I) models as tools for visual content generation raises concerns about their ability to accurately represent diverse cultural contexts. In this work, we present the first study to systematically quantify the alignment of T2I models and evaluation metrics with respect to both explicit as well as implicit cultural expectations. To this end, we introduce CulturalFrames, a novel benchmark designed for rigorous human evaluation of cultural representation in visual generations. Spanning 10 countries and 5 socio-cultural domains, CulturalFrames comprises 983 prompts, 3637 corresponding images generated by 4 state-of-the-art T2I models, and over 10k detailed human annotations. We find that T2I models not only fail to meet the more challenging implicit expectations but also the less challenging explicit expectations. Across models and countries, cultural expectations are missed an average of 44% of the time. Among these failures, explicit expectations are missed at a surprisingly high average rate of 68%, while implicit expectation failures are also significant, averaging 49%. Furthermore, we demonstrate that existing T2I evaluation metrics correlate poorly with human judgments of cultural alignment, irrespective of their internal reasoning. Collectively, our findings expose critical gaps, providing actionable directions for developing more culturally informed T2I models and evaluation methodologies.
• 摘要：文本对图像（T2I）模型的普遍性日益普及，作为视觉内容产生的工具引起了人们对它们准确代表各种文化背景的能力的担忧。在这项工作中，我们介绍了第一项研究，以系统地量化T2I模型的对齐方式和评估指标相对于显式和隐式文化期望。为此，我们介绍了文化框架，这是一种新颖的基准，旨在对视觉中的文化代表制度进行严格的人类评估。跨越10个国家和5个社会文化领域，包括983个提示，3637个由4种最先进的T2I模型生成的相应图像，以及超过10K的详细人类注释。我们发现，T2I模型不仅无法满足更具挑战性的隐性期望，而且还无法达到挑战性的明确期望。在整个模型和国家 /地区，平均44％的时间错过了文化期望。在这些失败中，遗失了明显的期望，令人惊讶的是平均68％，而隐性期望失败也很明显，平均为49％。此外，我们证明，现有的T2I评估指标与人类对文化一致性的判断的相关性很差，而不论其内部推理如何。总的来说，我们的发现揭示了关键的差距，为开发更多文化知情的T2I模型和评估方法提供了可行的方向。

Title: IMAGIC-500: IMputation benchmark on A Generative Imaginary Country (500k samples)

• Authors: Siyi Sun, David Antony Selby, Yunchuan Huang, Sebastian Vollmer, Seth Flaxman, Anisoara Calinescu
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2506.08844
• Pdf URL: https://arxiv.org/pdf/2506.08844
• Copy Paste: [[2506.08844]] IMAGIC-500: IMputation benchmark on A Generative Imaginary Country (500k samples)(https://arxiv.org/abs/2506.08844)
• Keywords: generative
• Abstract: Missing data imputation in tabular datasets remains a pivotal challenge in data science and machine learning, particularly within socioeconomic research. However, real-world socioeconomic datasets are typically subject to strict data protection protocols, which often prohibit public sharing, even for synthetic derivatives. This severely limits the reproducibility and accessibility of benchmark studies in such settings. Further, there are very few publicly available synthetic datasets. Thus, there is limited availability of benchmarks for systematic evaluation of imputation methods on socioeconomic datasets, whether real or synthetic. In this study, we utilize the World Bank's publicly available synthetic dataset, Synthetic Data for an Imaginary Country, which closely mimics a real World Bank household survey while being fully public, enabling broad access for methodological research. With this as a starting point, we derived the IMAGIC-500 dataset: we select a subset of 500k individuals across approximately 100k households with 19 socioeconomic features, designed to reflect the hierarchical structure of real-world household surveys. This paper introduces a comprehensive missing data imputation benchmark on IMAGIC-500 under various missing mechanisms (MCAR, MAR, MNAR) and missingness ratios (10\%, 20\%, 30\%, 40\%, 50\%). Our evaluation considers the imputation accuracy for continuous and categorical variables, computational efficiency, and impact on downstream predictive tasks, such as estimating educational attainment at the individual level. The results highlight the strengths and weaknesses of statistical, traditional machine learning, and deep learning imputation techniques, including recent diffusion-based methods. The IMAGIC-500 dataset and benchmark aim to facilitate the development of robust imputation algorithms and foster reproducible social science research.
• 摘要：在表格数据集中缺少数据插值仍然是数据科学和机器学习中的关键挑战，尤其是在社会经济研究中。但是，现实世界中的社会经济数据集通常受到严格的数据保护协议的约束，这些协议通常禁止公众共享，即使对于合成衍生物也是如此。这严重限制了在这种情况下基准研究的可重复性和可访问性。此外，很少有公开可用的合成数据集。因此，基准的可用性有限，用于系统地评估社会经济数据集的插补方法，无论是真实的还是合成的。在这项研究中，我们利用了世界银行公开可用的合成数据集，一个虚构国家的合成数据，这些数据集在完全公开的同时，密切模仿了现实世界银行的家庭调查，从而为方法论研究提供了广泛的访问。以此为起点，我们得出了Imagic-500数据集：我们在大约100,000个家庭中选择了一个具有19个社会经济特征的50万个个体的子集，旨在反映现实世界中家庭调查的层次结构。本文在各种缺失的机制（MCAR，MAR，MNAR）和缺失率（10 \％，20 \％，30 \％，40 \％，40 \％，50 \％）下介绍了Imagic-500的全面数据基准。我们的评估认为连续和分类变量，计算效率以及对下游预测任务的影响（例如估计单个级别的教育程度）的归合精度。结果突出了统计，传统机器学习和深度学习归合技术的优点和劣势，包括最近的基于扩散的方法。 Imagic-500数据集和基准旨在促进强大的归纳算法的开发并促进可重复的社会科学研究。

Title: Adapting Vision-Language Foundation Model for Next Generation Medical Ultrasound Image Analysis

• Authors: Jingguo Qu, Xinyang Han, Tonghuan Xiao, Jia Ai, Juan Wu, Tong Zhao, Jing Qin, Ann Dorothy King, Winnie Chiu-Wing Chu, Jing Cai, Michael Tin-Cheung Yingınst
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.08849
• Pdf URL: https://arxiv.org/pdf/2506.08849
• Copy Paste: [[2506.08849]] Adapting Vision-Language Foundation Model for Next Generation Medical Ultrasound Image Analysis(https://arxiv.org/abs/2506.08849)
• Keywords: generation
• Abstract: Medical ultrasonography is an essential imaging technique for examining superficial organs and tissues, including lymph nodes, breast, and thyroid. It employs high-frequency ultrasound waves to generate detailed images of the internal structures of the human body. However, manually contouring regions of interest in these images is a labor-intensive task that demands expertise and often results in inconsistent interpretations among individuals. Vision-language foundation models, which have excelled in various computer vision applications, present new opportunities for enhancing ultrasound image analysis. Yet, their performance is hindered by the significant differences between natural and medical imaging domains. This research seeks to overcome these challenges by developing domain adaptation methods for vision-language foundation models. In this study, we explore the fine-tuning pipeline for vision-language foundation models by utilizing large language model as text refiner with special-designed adaptation strategies and task-driven heads. Our approach has been extensively evaluated on six ultrasound datasets and two tasks: segmentation and classification. The experimental results show that our method can effectively improve the performance of vision-language foundation models for ultrasound image analysis, and outperform the existing state-of-the-art vision-language and pure foundation models. The source code of this study is available at \href{this https URL}{GitHub}.
• 摘要：医学超声检查是检查浅表器官和组织（包括淋巴结，乳房和甲状腺）的必不可少的成像技术。它采用高频超声波来生成人体内部结构的详细图像。但是，在这些图像中手动构造兴趣的区域是一项劳动密集型的任务，需要专业知识，并且通常会导致个人之间的解释不一致。在各种计算机视觉应用中表现出色的视觉基础模型为增强超声图像分析提供了新的机会。然而，自然和医学成像域之间的显着差异阻碍了它们的性能。这项研究试图通过为视觉基础模型开发域适应方法来克服这些挑战。在这项研究中，我们通过将大型语言模型用作具有特殊设计的适应策略和任务驱动的头部的文本炼油厂来探索视觉基础模型的微调管道。我们的方法已在六个超声数据集和两个任务上进行了广泛的评估：分割和分类。实验结果表明，我们的方法可以有效地提高视觉基础模型的超声图像分析的性能，并优于现有的最新视觉语言和纯粉底模型。该研究的源代码可在\ href {this HTTPS url} {github}上获得。

Title: InfoDPCCA: Information-Theoretic Dynamic Probabilistic Canonical Correlation Analysis

• Authors: Shiqin Tang, Shujian Yu
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2506.08884
• Pdf URL: https://arxiv.org/pdf/2506.08884
• Copy Paste: [[2506.08884]] InfoDPCCA: Information-Theoretic Dynamic Probabilistic Canonical Correlation Analysis(https://arxiv.org/abs/2506.08884)
• Keywords: generative
• Abstract: Extracting meaningful latent representations from high-dimensional sequential data is a crucial challenge in machine learning, with applications spanning natural science and engineering. We introduce InfoDPCCA, a dynamic probabilistic Canonical Correlation Analysis (CCA) framework designed to model two interdependent sequences of observations. InfoDPCCA leverages a novel information-theoretic objective to extract a shared latent representation that captures the mutual structure between the data streams and balances representation compression and predictive sufficiency while also learning separate latent components that encode information specific to each sequence. Unlike prior dynamic CCA models, such as DPCCA, our approach explicitly enforces the shared latent space to encode only the mutual information between the sequences, improving interpretability and robustness. We further introduce a two-step training scheme to bridge the gap between information-theoretic representation learning and generative modeling, along with a residual connection mechanism to enhance training stability. Through experiments on synthetic and medical fMRI data, we demonstrate that InfoDPCCA excels as a tool for representation learning. Code of InfoDPCCA is available at this https URL.
• 摘要：从高维顺序数据中提取有意义的潜在表示是机器学习的至关重要挑战，其应用程序涵盖了自然科学和工程。我们介绍了InfoDPCCA，这是一种动态概率的规范相关分析（CCA）框架，旨在模拟两个相互依存的观测序列。 InfoDPCCA利用一个新型的信息理论目标来提取共享的潜在表示，该图表捕获数据流和平衡表示压缩和预测充足性之间的相互结构，同时还学习了分别编码针对每个序列特定信息的潜在组件。与先前的动态CCA模型（例如DPCCA）不同，我们的方法明确执行共享的潜在空间以仅编码序列之间的相互信息，从而提高可解释性和鲁棒性。我们进一步引入了两步训练计划，以弥合信息理论表示学习与生成建模之间的差距，以及一种残留的连接机制，以增强训练稳定性。通过有关合成和医学功能磁共振成像数据的实验，我们证明InfoDPCCA擅长作为表示学习的工具。 InfoDPCCA代码可在此HTTPS URL上找到。

Title: Product of Experts for Visual Generation

• Authors: Yunzhi Zhang, Carson Murtuza-Lanier, Zizhang Li, Yilun Du, Jiajun Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.08894
• Pdf URL: https://arxiv.org/pdf/2506.08894
• Copy Paste: [[2506.08894]] Product of Experts for Visual Generation(https://arxiv.org/abs/2506.08894)
• Keywords: generation, generative
• Abstract: Modern neural models capture rich priors and have complementary knowledge over shared data domains, e.g., images and videos. Integrating diverse knowledge from multiple sources -- including visual generative models, visual language models, and sources with human-crafted knowledge such as graphics engines and physics simulators -- remains under-explored. We propose a Product of Experts (PoE) framework that performs inference-time knowledge composition from heterogeneous models. This training-free approach samples from the product distribution across experts via Annealed Importance Sampling (AIS). Our framework shows practical benefits in image and video synthesis tasks, yielding better controllability than monolithic methods and additionally providing flexible user interfaces for specifying visual generation goals.
• 摘要：现代神经模型捕获了丰富的先验，并拥有有关共享数据域（例如图像和视频）的互补知识。将来自多种来源的多样化知识（包括视觉生成模型，视觉语言模型以及诸如图形引擎和物理模拟器之类的人工知识）的来源整合在一起。我们提出了专家（POE）框架的产品，该框架可以从异质模型中执行推理时间知识组成。通过退火重要性采样（AIS），这种无培训方法从专家跨产品分布中进行样本。我们的框架在图像和视频综合任务中显示出实际的好处，比单片方法提供了更好的可控性，并为指定视觉生成目标提供了灵活的用户界面。

Title: Intention-Conditioned Flow Occupancy Models

• Authors: Chongyi Zheng, Seohong Park, Sergey Levine, Benjamin Eysenbach
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.08902
• Pdf URL: https://arxiv.org/pdf/2506.08902
• Copy Paste: [[2506.08902]] Intention-Conditioned Flow Occupancy Models(https://arxiv.org/abs/2506.08902)
• Keywords: generative
• Abstract: Large-scale pre-training has fundamentally changed how machine learning research is done today: large foundation models are trained once, and then can be used by anyone in the community (including those without data or compute resources to train a model from scratch) to adapt and fine-tune to specific tasks. Applying this same framework to reinforcement learning (RL) is appealing because it offers compelling avenues for addressing core challenges in RL, including sample efficiency and robustness. However, there remains a fundamental challenge to pre-train large models in the context of RL: actions have long-term dependencies, so training a foundation model that reasons across time is important. Recent advances in generative AI have provided new tools for modeling highly complex distributions. In this paper, we build a probabilistic model to predict which states an agent will visit in the temporally distant future (i.e., an occupancy measure) using flow matching. As large datasets are often constructed by many distinct users performing distinct tasks, we include in our model a latent variable capturing the user intention. This intention increases the expressivity of our model, and enables adaptation with generalized policy improvement. We call our proposed method intention-conditioned flow occupancy models (InFOM). Comparing with alternative methods for pre-training, our experiments on $36$ state-based and $4$ image-based benchmark tasks demonstrate that the proposed method achieves $1.8 \times$ median improvement in returns and increases success rates by $36\%$. Website: this https URL Code: this https URL
• 摘要：大规模的预培训从根本上改变了当今的机器学习研究：大型基础模型经过培训，然后可以被社区中的任何人（包括没有数据或计算资源的人使用来从头开始培训模型的人）以适应并遵守特定任务。将相同的框架应用于加固学习（RL）具有吸引力，因为它为解决RL的核心挑战提供了令人信服的途径，包括样本效率和鲁棒性。但是，在RL的背景下，预先培训的大型模型仍然存在着根本的挑战：行动具有长期的依赖性，因此训练一个基础模型，跨时间的原因很重要。生成AI的最新进展为建模高度复杂的分布提供了新的工具。在本文中，我们构建了一个概率模型，以预测使用流匹配在时间遥远的将来（即，占用度量）将访问的代理商将访问。由于大型数据集通常是由许多执行不同任务的不同用户构建的，因此我们在模型中包括一个潜在变量来捕获用户意图。这种意图增加了我们的模型的表现力，并可以通过一般的政策改进来适应。我们称我们提出的方法意图条件占用模型（Infom）。与预训练的替代方法相比，我们对36美元的$ 36 $和4 $基于图像的基准任务的实验表明，所提出的方法可实现$ 1.8 \ times $ $ $ $的改善，并将成功率提高$ 36 \％\％$。网站：此HTTPS URL代码：此HTTPS URL

Title: SkipVAR: Accelerating Visual Autoregressive Modeling via Adaptive Frequency-Aware Skipping

• Authors: Jiajun Li (1 and 5), Yue Ma (2), Xinyu Zhang (1), Qingyan Wei (3), Songhua Liu (4 and 5), Linfeng Zhang (5) ((1) University of Electronic Science and Technology of China, (2) The Hong Kong University of Science and Technology, (3) Central South University, (4) National University of Singapore, (5) Shanghai Jiaotong University)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.08908
• Pdf URL: https://arxiv.org/pdf/2506.08908
• Copy Paste: [[2506.08908]] SkipVAR: Accelerating Visual Autoregressive Modeling via Adaptive Frequency-Aware Skipping(https://arxiv.org/abs/2506.08908)
• Keywords: generation
• Abstract: Recent studies on Visual Autoregressive (VAR) models have highlighted that high-frequency components, or later steps, in the generation process contribute disproportionately to inference latency. However, the underlying computational redundancy involved in these steps has yet to be thoroughly investigated. In this paper, we conduct an in-depth analysis of the VAR inference process and identify two primary sources of inefficiency: step redundancy and unconditional branch redundancy. To address step redundancy, we propose an automatic step-skipping strategy that selectively omits unnecessary generation steps to improve efficiency. For unconditional branch redundancy, we observe that the information gap between the conditional and unconditional branches is minimal. Leveraging this insight, we introduce unconditional branch replacement, a technique that bypasses the unconditional branch to reduce computational cost. Notably, we observe that the effectiveness of acceleration strategies varies significantly across different samples. Motivated by this, we propose SkipVAR, a sample-adaptive framework that leverages frequency information to dynamically select the most suitable acceleration strategy for each instance. To evaluate the role of high-frequency information, we introduce high-variation benchmark datasets that test model sensitivity to fine details. Extensive experiments show SkipVAR achieves over 0.88 average SSIM with up to 1.81x overall acceleration and 2.62x speedup on the GenEval benchmark, maintaining model quality. These results confirm the effectiveness of frequency-aware, training-free adaptive acceleration for scalable autoregressive image generation. Our code is available at this https URL and has been publicly released.
• 摘要：关于视觉自回旋（VAR）模型的最新研究表明，生成过程中的高频组成部分或更后的步骤对推理潜伏期造成了不成比例的贡献。但是，这些步骤中涉及的潜在计算冗余尚未得到彻底研究。在本文中，我们对VAR推理过程进行了深入的分析，并确定了两个主要效率的主要来源：步骤冗余和无条件分支冗余。为了解决步骤冗余，我们提出了一种自动踩踏策略，该策略有选择地省略了不必要的生成步骤以提高效率。对于无条件的分支冗余，我们观察到条件和无条件分支之间的信息差距很小。利用这种见解，我们引入了无条件的分支更换，该技术绕过无条件分支以降低计算成本。值得注意的是，我们观察到，加速策略的有效性在不同的样本之间有显着变化。在此激励的情况下，我们提出了Skipvar，Skipvar是一种样本自适应框架，利用频率信息来动态选择每个实例的最合适的加速策略。为了评估高频信息的作用，我们介绍了高差异基准数据集，以测试模型对细节的敏感性。广泛的实验表明，Skipvar的平均SSIM超过0.88，总加速度高达1.81倍，而在Geneval基准上达到了2.62倍的速度，从而保持模型质量。这些结果证实了频率吸引，无训练的自适应加速度的有效性，可扩展自回归图像产生。我们的代码可在此HTTPS URL上获得，并已公开发布。

Title: Segment Concealed Objects with Incomplete Supervision

• Authors: Chunming He, Kai Li, Yachao Zhang, Ziyun Yang, Youwei Pang, Longxiang Tang, Chengyu Fang, Yulun Zhang, Linghe Kong, Xiu Li, Sina Farsiu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.08955
• Pdf URL: https://arxiv.org/pdf/2506.08955
• Copy Paste: [[2506.08955]] Segment Concealed Objects with Incomplete Supervision(https://arxiv.org/abs/2506.08955)
• Keywords: generation
• Abstract: Incompletely-Supervised Concealed Object Segmentation (ISCOS) involves segmenting objects that seamlessly blend into their surrounding environments, utilizing incompletely annotated data, such as weak and semi-annotations, for model training. This task remains highly challenging due to (1) the limited supervision provided by the incompletely annotated training data, and (2) the difficulty of distinguishing concealed objects from the background, which arises from the intrinsic similarities in concealed scenarios. In this paper, we introduce the first unified method for ISCOS to address these challenges. To tackle the issue of incomplete supervision, we propose a unified mean-teacher framework, SEE, that leverages the vision foundation model, ``\emph{Segment Anything Model (SAM)}'', to generate pseudo-labels using coarse masks produced by the teacher model as prompts. To mitigate the effect of low-quality segmentation masks, we introduce a series of strategies for pseudo-label generation, storage, and supervision. These strategies aim to produce informative pseudo-labels, store the best pseudo-labels generated, and select the most reliable components to guide the student model, thereby ensuring robust network training. Additionally, to tackle the issue of intrinsic similarity, we design a hybrid-granularity feature grouping module that groups features at different granularities and aggregates these results. By clustering similar features, this module promotes segmentation coherence, facilitating more complete segmentation for both single-object and multiple-object images. We validate the effectiveness of our approach across multiple ISCOS tasks, and experimental results demonstrate that our method achieves state-of-the-art performance. Furthermore, SEE can serve as a plug-and-play solution, enhancing the performance of existing models.
• 摘要：不完全监督的隐藏对象细分（ISCO）涉及将无缝融合到周围环境中的对象进行分割，并利用未完全注释的数据（例如弱和半通量）进行模型培训。由于（1）未完全注释的培训数据提供的有限监督，以及（2）将隐藏物体与背景区分开来，这是由于隐藏场景中的固有相似性引起的。在本文中，我们介绍了第一种统一方法，以解决这些挑战。为了解决不完整的监督问题，我们提出了一个统一的均值老师框架，请参阅，请参阅视觉基础模型，``\ emph {sagment nothing nothing Model（sam）}''，以使用教师模型作为提示来生成伪标签。为了减轻低质量分割口罩的影响，我们引入了一系列伪标签生成，存储和监督的策略。这些策略旨在生产信息丰富的伪标签，存储生成的最佳伪标签，并选择最可靠的组件来指导学生模型，从而确保强大的网络培训。此外，为了解决固有相似性的问题，我们设计了一个杂交特征分组模块，该模块将分组在不同的粒度上特征并汇总这些结果。通过聚集类似的特征，该模块促进了分割的连贯性，从而促进了单对象和多对象图像的更完整的分割。我们在多个ISCOS任务中验证了方法的有效性，实验结果表明我们的方法可以实现最先进的性能。此外，请参见可以用作插件解决方案，从而增强现有模型的性能。

Title: ORIDa: Object-centric Real-world Image Composition Dataset

• Authors: Jinwoo Kim, Sangmin Han, Jinho Jeong, Jiwoo Choi, Dongyoung Kim, Seon Joo Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.08964
• Pdf URL: https://arxiv.org/pdf/2506.08964
• Copy Paste: [[2506.08964]] ORIDa: Object-centric Real-world Image Composition Dataset(https://arxiv.org/abs/2506.08964)
• Keywords: generative
• Abstract: Object compositing, the task of placing and harmonizing objects in images of diverse visual scenes, has become an important task in computer vision with the rise of generative models. However, existing datasets lack the diversity and scale required to comprehensively explore real-world scenarios. We introduce ORIDa (Object-centric Real-world Image Composition Dataset), a large-scale, real-captured dataset containing over 30,000 images featuring 200 unique objects, each of which is presented across varied positions and scenes. ORIDa has two types of data: factual-counterfactual sets and factual-only scenes. The factual-counterfactual sets consist of four factual images showing an object in different positions within a scene and a single counterfactual (or background) image of the scene without the object, resulting in five images per scene. The factual-only scenes include a single image containing an object in a specific context, expanding the variety of environments. To our knowledge, ORIDa is the first publicly available dataset with its scale and complexity for real-world image composition. Extensive analysis and experiments highlight the value of ORIDa as a resource for advancing further research in object compositing.
• 摘要：物体合成，将对象放置在不同视觉场景的图像中的任务已成为计算机视觉中的重要任务，生成模型的兴起。但是，现有数据集缺乏全面探索现实情况所需的多样性和规模。我们介绍了Orida（以对象为中心的现实世界图像构图数据集），这是一个大规模，实际捕获的数据集，其中包含30,000多个图像，其中包含200个独特对象，每个对象都跨不同位置和场景呈现。 Orida有两种类型的数据：事实相互作用集和仅事实的场景。事实相关事实集由四个事实图像组成，显示一个场景中不同位置的对象，而没有对象的场景中的单个反事实（或背景）图像，每个场景都会产生五个图像。仅事实的场景包括一个在特定上下文中包含对象的单个图像，从而扩展了各种环境。据我们所知，Orida是第一个公开可用的数据集，其规模和复杂性用于现实世界图像组成。广泛的分析和实验突出了Orida作为进一步研究对象合成研究的资源的价值。

Title: GFRIEND: Generative Few-shot Reward Inference through EfficieNt DPO

• Authors: Yiyang Zhao, Huiyu Bai, Xuejiao Zhao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.08965
• Pdf URL: https://arxiv.org/pdf/2506.08965
• Copy Paste: [[2506.08965]] GFRIEND: Generative Few-shot Reward Inference through EfficieNt DPO(https://arxiv.org/abs/2506.08965)
• Keywords: generative
• Abstract: The ability to train high-performing reward models with few-shot data is critical for enhancing the efficiency and scalability of Reinforcement Learning from Human Feedback (RLHF). We propose a data augmentation and expansion framework that enables generative reward models trained on small datasets to achieve comparable performance to those trained on large-scale datasets. Traditional methods to train a generative reward model, such as Direct Preference Optimization (DPO), are constrained by inefficiencies in sample pairing and limited data diversity. This work introduces preference refinement, which employs Chain-of-Thought (CoT) sampling to uncover diverse and high-quality preference relationships. It also incorporates a perplexity-based scoring mechanism to assign nuanced preference levels and utilizes Multi-level Direct Preference Optimization (M-DPO) to enable the model to capture finer-grained preference differences between samples. Experimental results demonstrate that the proposed method significantly enhances data efficiency and model performance, enabling reward models trained in a few-shot setting to achieve results on par with those trained on large-scale datasets. This study underscores the potential of data-efficient strategies in advancing reward model optimization, offering a robust solution for low-resource RLHF applications.
• 摘要：使用几乎没有拍摄数据训练高性能奖励模型的能力对于提高从人类反馈（RLHF）学习增强性学习的效率和可扩展性至关重要。我们提出了一个数据增强和扩展框架，该框架可以使在小数据集上训练的生成奖励模型与在大规模数据集中训练的人达到可比的性能。训练生成奖励模型（例如直接偏好优化（DPO））的传统方法受样本配对和有限的数据多样性的效率低下的限制。这项工作介绍了偏好精炼，该偏好是采用了思想链（COT）抽样来揭示各种和高质量的偏好关系。它还结合了基于困惑的评分机制来分配细微的首选项水平，并利用多级直接优先优化（M-DPO）使模型能够捕获样品之间的细粒度偏好差异。实验结果表明，所提出的方法可显着提高数据效率和模型性能，从而使经过几次训练的奖励模型能够与大规模数据集训练的奖励相同。这项研究强调了数据有效策略在推进奖励模型优化方面的潜力，为低资源RLHF应用提供了强大的解决方案。

Title: Do Concept Replacement Techniques Really Erase Unacceptable Concepts?

• Authors: Anudeep Das, Gurjot Singh, Prach Chantasantitam, N. Asokan
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2506.08991
• Pdf URL: https://arxiv.org/pdf/2506.08991
• Copy Paste: [[2506.08991]] Do Concept Replacement Techniques Really Erase Unacceptable Concepts?(https://arxiv.org/abs/2506.08991)
• Keywords: generative
• Abstract: Generative models, particularly diffusion-based text-to-image (T2I) models, have demonstrated astounding success. However, aligning them to avoid generating content with unacceptable concepts (e.g., offensive or copyrighted content, or celebrity likenesses) remains a significant challenge. Concept replacement techniques (CRTs) aim to address this challenge, often by trying to "erase" unacceptable concepts from models. Recently, model providers have started offering image editing services which accept an image and a text prompt as input, to produce an image altered as specified by the prompt. These are known as image-to-image (I2I) models. In this paper, we first use an I2I model to empirically demonstrate that today's state-of-the-art CRTs do not in fact erase unacceptable concepts. Existing CRTs are thus likely to be ineffective in emerging I2I scenarios, despite their proven ability to remove unwanted concepts in T2I pipelines, highlighting the need to understand this discrepancy between T2I and I2I settings. Next, we argue that a good CRT, while replacing unacceptable concepts, should preserve other concepts specified in the inputs to generative models. We call this fidelity. Prior work on CRTs have neglected fidelity in the case of unacceptable concepts. Finally, we propose the use of targeted image-editing techniques to achieve both effectiveness and fidelity. We present such a technique, AntiMirror, and demonstrate its viability.
• 摘要：生成模型，尤其是基于扩散的文本对图像（T2I）模型，已经证明了惊人的成功。但是，使它们保持一致，以避免产生具有不可接受的概念（例如，令人反感或受版权保护的内容或名人相似性）的内容仍然是一个重大挑战。概念替代技术（CRT）旨在解决这一挑战，通常是通过试图从模型中“删除”不可接受的概念。最近，模型提供商已经开始提供图像编辑服务，该服务接受图像和文本提示为输入，以产生按照提示指定的图像。这些被称为图像对图像（I2i）模型。在本文中，我们首先使用I2I模型来证明当今最先进的CRT实际上并没有删除不可接受的概念。因此，现有的CRT可能在新兴的I2i方案中无效，尽管它们可以在T2I管道中删除不需要的概念，从而强调了了解T2i和I2i设置之间的这种差异的必要性。接下来，我们认为，在替换不可接受的概念的同时，应保留在生成模型的输入中指定的其他概念。我们称这种忠诚。在不可接受的概念的情况下，关于CRT的先前工作已忽略了忠诚。最后，我们建议使用有针对性的图像编辑技术来实现有效性和忠诚度。我们提出了这种技术，抗微龙，并证明了它的生存能力。

Title: Branched Schrödinger Bridge Matching

• Authors: Sophia Tang, Yinuo Zhang, Alexander Tong, Pranam Chatterjee
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2506.09007
• Pdf URL: https://arxiv.org/pdf/2506.09007
• Copy Paste: [[2506.09007]] Branched Schrödinger Bridge Matching(https://arxiv.org/abs/2506.09007)
• Keywords: generative
• Abstract: Predicting the intermediate trajectories between an initial and target distribution is a central problem in generative modeling. Existing approaches, such as flow matching and Schrödinger Bridge Matching, effectively learn mappings between two distributions by modeling a single stochastic path. However, these methods are inherently limited to unimodal transitions and cannot capture branched or divergent evolution from a common origin to multiple distinct outcomes. To address this, we introduce Branched Schrödinger Bridge Matching (BranchSBM), a novel framework that learns branched Schrödinger bridges. BranchSBM parameterizes multiple time-dependent velocity fields and growth processes, enabling the representation of population-level divergence into multiple terminal distributions. We show that BranchSBM is not only more expressive but also essential for tasks involving multi-path surface navigation, modeling cell fate bifurcations from homogeneous progenitor states, and simulating diverging cellular responses to perturbations.
• 摘要：预测初始分布和目标分布之间的中间轨迹是生成建模的中心问题。现有方法（例如流匹配和施罗丁桥匹配）通过对单个随机路径进行建模，有效地学习了两个分布之间的映射。但是，这些方法固有地限于单峰过渡，无法捕获从共同起源到多个不同结果的分支或发散的演变。为了解决这个问题，我们介绍了分支的Schrödinger桥匹配（Branchsbm），这是一个学习分支的Schrödinger桥梁的新型框架。 BranchSBM参数化了多个时间依赖性速度场和增长过程，从而使人口级差异表示为多个终端分布。我们表明，BranchSBM不仅表达更具表达性，而且对于涉及多路径表面导航的任务，对均相祖细胞状态的细胞命运分叉进行建模以及模拟细胞对扰动的分歧响应。

Title: Edit Flows: Flow Matching with Edit Operations

• Authors: Marton Havasi, Brian Karrer, Itai Gat, Ricky T. Q. Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.09018
• Pdf URL: https://arxiv.org/pdf/2506.09018
• Copy Paste: [[2506.09018]] Edit Flows: Flow Matching with Edit Operations(https://arxiv.org/abs/2506.09018)
• Keywords: generation, generative
• Abstract: Autoregressive generative models naturally generate variable-length sequences, while non-autoregressive models struggle, often imposing rigid, token-wise structures. We propose Edit Flows, a non-autoregressive model that overcomes these limitations by defining a discrete flow over sequences through edit operations-insertions, deletions, and substitutions. By modeling these operations within a Continuous-time Markov Chain over the sequence space, Edit Flows enable flexible, position-relative generation that aligns more closely with the structure of sequence data. Our training method leverages an expanded state space with auxiliary variables, making the learning process efficient and tractable. Empirical results show that Edit Flows outperforms both autoregressive and mask models on image captioning and significantly outperforms the mask construction in text and code generation.
• 摘要：自回归生成模型自然会生成可变长度序列，而非自动回忆模型通常会施加刚性，刻有代币的结构。我们提出了编辑Flow，这是一种非自动回旋模型，通过通过编辑操作插入，删除和替换来定义序列上的离散流来克服这些局限性。通过在序列空间上的连续时间马尔可夫链中对这些操作进行建模，编辑流使柔性，相关的生成与序列数据的结构更紧密地保持一致。我们的培训方法利用辅助变量利用扩展的状态空间，使学习过程有效且可进行。经验结果表明，在图像字幕上，编辑流量的表现优于自回归和面具模型，并且在文本和代码生成中的表现明显优于掩盖构造。

Title: e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs

• Authors: Amrith Setlur, Matthew Y. R. Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, Aviral Kumar
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2506.09026
• Pdf URL: https://arxiv.org/pdf/2506.09026
• Copy Paste: [[2506.09026]] e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs(https://arxiv.org/abs/2506.09026)
• Keywords: generation
• Abstract: Test-time scaling offers a promising path to improve LLM reasoning by utilizing more compute at inference time; however, the true promise of this paradigm lies in extrapolation (i.e., improvement in performance on hard problems as LLMs keep "thinking" for longer, beyond the maximum token budget they were trained on). Surprisingly, we find that most existing reasoning models do not extrapolate well. We show that one way to enable extrapolation is by training the LLM to perform in-context exploration: training the LLM to effectively spend its test time budget by chaining operations (such as generation, verification, refinement, etc.), or testing multiple hypotheses before it commits to an answer. To enable in-context exploration, we identify three key ingredients as part of our recipe e3: (1) chaining skills that the base LLM has asymmetric competence in, e.g., chaining verification (easy) with generation (hard), as a way to implement in-context search; (2) leveraging "negative" gradients from incorrect traces to amplify exploration during RL, resulting in longer search traces that chains additional asymmetries; and (3) coupling task difficulty with training token budget during training via a specifically-designed curriculum to structure in-context exploration. Our recipe e3 produces the best known 1.7B model according to AIME'25 and HMMT'25 scores, and extrapolates to 2x the training token budget. Our e3-1.7B model not only attains high pass@1 scores, but also improves pass@k over the base model.
• 摘要：测试时间扩展提供了一种有希望的途径来通过在推理时使用更多的计算来改善LLM推理；但是，这种范式的真正承诺在于推断（即，随着LLMS不断“思考”更长的时间，超出他们接受过培训的最大标记预算时，硬性问题的绩效提高了）。令人惊讶的是，我们发现大多数现有的推理模型不能很好地推断。我们表明，启用外推的一种方法是训练LLM以执行语义探索：培训LLM通过链接操作（例如生成，验证，改进等）有效地花费其测试时间预算，或在承诺答案之前测试多个假设。为了启用秘密探索，我们将三种关键成分确定为食谱E3的一部分：（1）基本LLM具有不对称能力的链接技能，例如，具有一代（硬）的链式验证（易于），作为实施封闭式搜索的一种方式； （2）利用“负”梯度从不正确的轨迹来放大RL期间的探索，从而产生较长的搜索痕迹，从而链接出其他不对称性； （3）在培训期间通过专门设计的课程来培训令牌预算的耦合困难，以结构构造中文探索。我们的食谱E3根据Aime'25和HMMT'25分数生产出最著名的1.7b模型，并将其外推至2倍训练令牌预算。我们的E3-1.7B模型不仅可以达到1分的高分，而且还可以改善基本模型的Pass@K。

Title: Diffuse and Disperse: Image Generation with Representation Regularization

• Authors: Runqian Wang, Kaiming He
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.09027
• Pdf URL: https://arxiv.org/pdf/2506.09027
• Copy Paste: [[2506.09027]] Diffuse and Disperse: Image Generation with Representation Regularization(https://arxiv.org/abs/2506.09027)
• Keywords: generation, generative
• Abstract: The development of diffusion-based generative models over the past decade has largely proceeded independently of progress in representation learning. These diffusion models typically rely on regression-based objectives and generally lack explicit regularization. In this work, we propose \textit{Dispersive Loss}, a simple plug-and-play regularizer that effectively improves diffusion-based generative models. Our loss function encourages internal representations to disperse in the hidden space, analogous to contrastive self-supervised learning, with the key distinction that it requires no positive sample pairs and therefore does not interfere with the sampling process used for regression. Compared to the recent method of representation alignment (REPA), our approach is self-contained and minimalist, requiring no pre-training, no additional parameters, and no external data. We evaluate Dispersive Loss on the ImageNet dataset across a range of models and report consistent improvements over widely used and strong baselines. We hope our work will help bridge the gap between generative modeling and representation learning.
• 摘要：在过去的十年中，基于扩散的生成模型的发展在很大程度上是独立于代表学习的进步。这些扩散模型通常依赖于基于回归的目标，并且通常缺乏明确的正则化。在这项工作中，我们提出了\ textit {配色损失}，这是一种简单的插件正常器，可有效改善基于扩散的生成模型。我们的损失函数鼓励内部表示分散在隐藏的空间中，类似于对比度的自学学习，其关键区别不需要正面样本对，因此不会干扰用于回归的采样过程。与最新的表示对准方法（REPA）相比，我们的方法是独立的和极简主义的，不需要预训练，没有其他参数，也没有外部数据。我们评估了一系列模型的Imagenet数据集上的分散损失，并报告对广泛使用和强质基线的一致改进。我们希望我们的工作将有助于弥合生成建模和表示学习之间的差距。

Title: Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better

• Authors: Dianyi Wang, Wei Song, Yikun Wang, Siyuan Wang, Kaicheng Yu, Zhongyu Wei, Jiaqi Wang
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2506.09040
• Pdf URL: https://arxiv.org/pdf/2506.09040
• Copy Paste: [[2506.09040]] Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better(https://arxiv.org/abs/2506.09040)
• Keywords: generation
• Abstract: Typical large vision-language models (LVLMs) apply autoregressive supervision solely to textual sequences, without fully incorporating the visual modality into the learning process. This results in three key limitations: (1) an inability to utilize images without accompanying captions, (2) the risk that captions omit critical visual details, and (3) the challenge that certain vision-centric content cannot be adequately conveyed through text. As a result, current LVLMs often prioritize vision-to-language alignment while potentially overlooking fine-grained visual information. While some prior works have explored autoregressive image generation, effectively leveraging autoregressive visual supervision to enhance image understanding remains an open challenge. In this paper, we introduce Autoregressive Semantic Visual Reconstruction (ASVR), which enables joint learning of visual and textual modalities within a unified autoregressive framework. We show that autoregressively reconstructing the raw visual appearance of images does not enhance and may even impair multimodal understanding. In contrast, autoregressively reconstructing the semantic representation of images consistently improves comprehension. Notably, we find that even when models are given continuous image features as input, they can effectively reconstruct discrete semantic tokens, resulting in stable and consistent improvements across a wide range of multimodal understanding benchmarks. Our approach delivers significant performance gains across varying data scales (556k-2M) and types of LLM bacbones. Specifically, ASVR improves LLaVA-1.5 by 5% in average scores across 14 multimodal benchmarks. The code is available at this https URL.
• 摘要：典型的大型视觉模型（LVLM）仅将自回归监督应用于文本序列，而无需将视觉模态完全融合到学习过程中。这导致了三个关键局限性：（1）无法使用图像而不随附的标题，（2）字幕省略关键视觉细节的风险，以及（3）挑战是无法通过文本充分传达某些以视觉的内容。结果，当前的LVLM通常优先考虑视力到语言对齐，同时有可能忽略细粒度的视觉信息。尽管一些先前的作品探索了自回归图像的产生，但有效利用自回归视觉监督来增强图像理解仍然是一个开放的挑战。在本文中，我们介绍了自回归语义视觉重建（ASVR），从而可以在统一自动回归框架内联合学习视觉和文本方式。我们表明，自动重建图像的原始视觉外观不会增强，甚至可能会损害多模式的理解。相比之下，自动调节图像的语义表示始终提高理解。值得注意的是，我们发现，即使将模型作为输入赋予连续的图像特征，它们也可以有效地重建离散的语义令牌，从而在广泛的多模式理解基准测试基准中稳定且一致地改进。我们的方法可在不同的数据量表（556K-2M）和LLM Bacbones的类型中带来显着的性能增长。具体而言，ASVR在14个多模式基准的平均得分中将LLAVA-1.5提高了5％。该代码可在此HTTPS URL上找到。

Title: Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models

• Authors: Xuanchi Ren, Yifan Lu, Tianshi Cao, Ruiyuan Gao, Shengyu Huang, Amirmojtaba Sabour, Tianchang Shen, Tobias Pfaff, Jay Zhangjie Wu, Runjian Chen, Seung Wook Kim, Jun Gao, Laura Leal-Taixe, Mike Chen, Sanja Fidler, Huan Ling
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.09042
• Pdf URL: https://arxiv.org/pdf/2506.09042
• Copy Paste: [[2506.09042]] Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models(https://arxiv.org/abs/2506.09042)
• Keywords: generation
• Abstract: Collecting and annotating real-world data for safety-critical physical AI systems, such as Autonomous Vehicle (AV), is time-consuming and costly. It is especially challenging to capture rare edge cases, which play a critical role in training and testing of an AV system. To address this challenge, we introduce the Cosmos-Drive-Dreams - a synthetic data generation (SDG) pipeline that aims to generate challenging scenarios to facilitate downstream tasks such as perception and driving policy training. Powering this pipeline is Cosmos-Drive, a suite of models specialized from NVIDIA Cosmos world foundation model for the driving domain and are capable of controllable, high-fidelity, multi-view, and spatiotemporally consistent driving video generation. We showcase the utility of these models by applying Cosmos-Drive-Dreams to scale the quantity and diversity of driving datasets with high-fidelity and challenging scenarios. Experimentally, we demonstrate that our generated data helps in mitigating long-tail distribution problems and enhances generalization in downstream tasks such as 3D lane detection, 3D object detection and driving policy learning. We open source our pipeline toolkit, dataset and model weights through the NVIDIA's Cosmos platform. Project page: this https URL
• 摘要：收集和注释用于安全至关重要的物理AI系统（例如自动驾驶汽车（AV））的现实数据是耗时且昂贵的。捕获稀有边缘案例尤其具有挑战性，稀有边缘案例在AV系统的训练和测试中起着至关重要的作用。为了应对这一挑战，我们介绍了Cosmos-Drive-Dreams-综合数据生成（SDG）管道，旨在生成具有挑战性的场景，以促进下游任务，例如感知和驱动政策培训。为这款管道提供动力的是Cosmos-Drive，这是一套专门的驾驶领域Cosmos World Foundation模型的型号，并且能够具有可控，高保真性，多视图和时空一致的驾驶视频生成。我们通过应用Cosmos-Drive-Dreams来扩展具有高保真性和具有挑战性的情况的驾驶数据集的数量和多样性来展示这些模型的实用性。在实验上，我们证明了我们的生成数据有助于减轻长尾分配问题，并增强下游任务中的概括，例如3D车道检测，3D对象检测和驱动政策学习。我们通过NVIDIA的Cosmos平台开源的管道工具包，数据集和模型权重。项目页面：此HTTPS URL

Title: MagCache: Fast Video Generation with Magnitude-Aware Cache

• Authors: Zehong Ma, Longhui Wei, Feng Wang, Shiliang Zhang, Qi Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.09045
• Pdf URL: https://arxiv.org/pdf/2506.09045
• Copy Paste: [[2506.09045]] MagCache: Fast Video Generation with Magnitude-Aware Cache(https://arxiv.org/abs/2506.09045)
• Keywords: generation
• Abstract: Existing acceleration techniques for video diffusion models often rely on uniform heuristics or time-embedding variants to skip timesteps and reuse cached features. These approaches typically require extensive calibration with curated prompts and risk inconsistent outputs due to prompt-specific overfitting. In this paper, we introduce a novel and robust discovery: a unified magnitude law observed across different models and prompts. Specifically, the magnitude ratio of successive residual outputs decreases monotonically and steadily in most timesteps while rapidly in the last several steps. Leveraging this insight, we introduce a Magnitude-aware Cache (MagCache) that adaptively skips unimportant timesteps using an error modeling mechanism and adaptive caching strategy. Unlike existing methods requiring dozens of curated samples for calibration, MagCache only requires a single sample for calibration. Experimental results show that MagCache achieves 2.1x and 2.68x speedups on Open-Sora and Wan 2.1, respectively, while preserving superior visual fidelity. It significantly outperforms existing methods in LPIPS, SSIM, and PSNR, under comparable computational budgets.
• 摘要：视频扩散模型的现有加速技术通常依赖于统一的启发式方法或时间出现的变体来跳过时间段和重复使用的缓存功能。这些方法通常需要通过策划的提示进行广泛的校准，并且由于特定特定的过度拟合而导致的风险不一致。在本文中，我们介绍了一个新颖而健壮的发现：在不同模型和提示中观察到的统一级别定律。具体而言，在大多数时间步长中，连续残留输出的幅度比单调和稳定地降低，而在最后几个步骤中迅速降低。利用这种见解，我们引入了一个幅度感知的高速缓存（Magcache），该缓存使用错误建模机制和自适应缓存策略自适应地跳过了不重要的时间段。与需要数十个策划样品进行校准的现有方法不同，Magcache仅需要一个单个样品进行校准。实验结果表明，Magcache分别在开放式和WAN 2.1上达到2.1倍和2.68倍的速度，同时保留了出色的视觉保真度。在可比的计算预算下，它大大优于LPIP，SSIM和PSNR中现有的方法。