2025-04-08

Title: Geometric Flow Models over Neural Network Weights

• Authors: Ege Erdogan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.03710
• Pdf URL: https://arxiv.org/pdf/2504.03710
• Copy Paste: [[2504.03710]] Geometric Flow Models over Neural Network Weights(https://arxiv.org/abs/2504.03710)
• Keywords: generative
• Abstract: Deep generative models such as flow and diffusion models have proven to be effective in modeling high-dimensional and complex data types such as videos or proteins, and this has motivated their use in different data modalities, such as neural network weights. A generative model of neural network weights would be useful for a diverse set of applications, such as Bayesian deep learning, learned optimization, and transfer learning. However, the existing work on weight-space generative models often ignores the symmetries of neural network weights, or only takes into account a subset of them. Modeling those symmetries, such as permutation symmetries between subsequent layers in an MLP, the filters in a convolutional network, or scaling symmetries arising with the use of non-linear activations, holds the potential to make weight-space generative modeling more efficient by effectively reducing the dimensionality of the problem. In this light, we aim to design generative models in weight-space that more comprehensively respect the symmetries of neural network weights. We build on recent work on generative modeling with flow matching, and weight-space graph neural networks to design three different weight-space flows. Each of our flows takes a different approach to modeling the geometry of neural network weights, and thus allows us to explore the design space of weight-space flows in a principled way. Our results confirm that modeling the geometry of neural networks more faithfully leads to more effective flow models that can generalize to different tasks and architectures, and we show that while our flows obtain competitive performance with orders of magnitude fewer parameters than previous work, they can be further improved by scaling them up. We conclude by listing potential directions for future work on weight-space generative models.
• 摘要：事实证明，诸如流动和扩散模型之类的深层生成模型可以有效地建模高维和复杂的数据类型，例如视频或蛋白质，这激发了它们在不同数据模式（例如神经网络权重）中的使用。神经网络权重的生成模型对于贝叶斯深度学习，学习优化和转移学习等各种应用程序将很有用。但是，重量空间生成模型的现有工作通常忽略神经网络权重的对称性，或者仅考虑其中的一个子集。建模这些对称性，例如MLP中随后层之间的置换对称性，卷积网络中的过滤器或使用非线性激活产生的对称对称性，具有通过有效地降低问题的维度来使重量空间生成模型更有效地使问题空间产生建模的潜力。从这个角度来看，我们旨在在重量空间中设计生成模型，以更全面地尊重神经网络重量的对称性。我们以流量匹配和权重空间图神经网络为生成建模的最新工作，以设计三种不同的重量空间流。我们的每个流都采用不同的方法来建模神经网络重量的几何形状，因此使我们能够以原则上的方式探索权重空间流的设计空间。我们的结果证实，对神经网络的几何形状进行建模更忠实地导致更有效的流程模型可以推广到不同的任务和体系结构，并且我们表明，尽管我们的流量以比以前的工作少的参数较少的数量级获得竞争性能，但可以通过扩展它们来进一步改善它们。我们通过列出了在重量空间生成模型上的未来工作的潜在方向来结束。

Title: Multi-Objective Quality-Diversity in Unstructured and Unbounded Spaces

• Authors: Hannah Janmohamed, Antoine Cully
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2504.03715
• Pdf URL: https://arxiv.org/pdf/2504.03715
• Copy Paste: [[2504.03715]] Multi-Objective Quality-Diversity in Unstructured and Unbounded Spaces(https://arxiv.org/abs/2504.03715)
• Keywords: generation
• Abstract: Quality-Diversity algorithms are powerful tools for discovering diverse, high-performing solutions. Recently, Multi-Objective Quality-Diversity (MOQD) extends QD to problems with several objectives while preserving solution diversity. MOQD has shown promise in fields such as robotics and materials science, where finding trade-offs between competing objectives like energy efficiency and speed, or material properties is essential. However, existing methods in MOQD rely on tessellating the feature space into a grid structure, which prevents their application in domains where feature spaces are unknown or must be learned, such as complex biological systems or latent exploration tasks. In this work, we introduce Multi-Objective Unstructured Repertoire for Quality-Diversity (MOUR-QD), a MOQD algorithm designed for unstructured and unbounded feature spaces. We evaluate MOUR-QD on five robotic tasks. Importantly, we show that our method excels in tasks where features must be learned, paving the way for applying MOQD to unsupervised domains. We also demonstrate that MOUR-QD is advantageous in domains with unbounded feature spaces, outperforming existing grid-based methods. Finally, we demonstrate that MOUR-QD is competitive with established MOQD methods on existing MOQD tasks and achieves double the MOQD-score in some environments. MOUR-QD opens up new opportunities for MOQD in domains like protein design and image generation.
• 摘要：质量多样性算法是发现多样化，高性能解决方案的强大工具。最近，多目标质量多样性（MOQD）将QD扩展到了几个目标的问题，同时保留了解决方案多样性。 MOQD在机器人技术和材料科学等领域表现出了希望，在这种领域，在竞争目标之间找到权衡的权衡，例如能源效率和速度或材料特性。但是，MOQD中的现有方法依赖于将特征空间插入网格结构，从而阻止其在特征空间未知或必须学习的域中的应用，例如复杂的生物系统或潜在的勘探任务。在这项工作中，我们为质量多样性（MOR-QD）引入了多目标非结构化曲目，这是一种专为非结构和无限制特征空间而设计的MOQD算法。我们在五项机器人任务上评估了MOR-QD。重要的是，我们表明我们的方法在必须学习功能的任务中表现出色，为将MOQD应用于无监督域铺平了道路。我们还证明，在具有无限特征空间的域中，MOR-QD是有利的，表现优于现有的基于网格的方法。最后，我们证明了MOR-QD具有有关现有MOQD任务的已建立的MOQD方法的竞争力，并在某些环境中获得了MOQD得分的两倍。 MOR-QD在蛋白质设计和图像生成等领域中为OOQD打开了新的机会。

Title: Attention in Diffusion Model: A Survey

• Authors: Litao Hua, Fan Liu, Jie Su, Xingyu Miao, Zizhou Ouyang, Zeyu Wang, Runze Hu, Zhenyu Wen, Bing Zhai, Yang Long, Haoran Duan, Yuan Zhou
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2504.03738
• Pdf URL: https://arxiv.org/pdf/2504.03738
• Copy Paste: [[2504.03738]] Attention in Diffusion Model: A Survey(https://arxiv.org/abs/2504.03738)
• Keywords: generative
• Abstract: Attention mechanisms have become a foundational component in diffusion models, significantly influencing their capacity across a wide range of generative and discriminative tasks. This paper presents a comprehensive survey of attention within diffusion models, systematically analysing its roles, design patterns, and operations across different modalities and tasks. We propose a unified taxonomy that categorises attention-related modifications into parts according to the structural components they affect, offering a clear lens through which to understand their functional diversity. In addition to reviewing architectural innovations, we examine how attention mechanisms contribute to performance improvements in diverse applications. We also identify current limitations and underexplored areas, and outline potential directions for future research. Our study provides valuable insights into the evolving landscape of diffusion models, with a particular focus on the integrative and ubiquitous role of attention.
• 摘要：注意机制已成为扩散模型中的基础组成部分，从而在广泛的生成和歧视性任务中显着影响其能力。本文对扩散模型中的关注进行了全面的调查，并系统地分析了其角色，设计模式以及跨不同模式和任务的操作。我们提出了一种统一的分类法，该分类法将与注意力相关的修饰分为部分，根据它们影响的结构成分，提供清晰的镜头，通过该镜头了解它们的功能多样性。除了审查建筑创新外，我们还研究了注意机制如何促进各种应用程序的性能提高。我们还确定了当前的局限性和未充满信心的领域，并概述了未来研究的潜在方向。我们的研究为扩散模型的不断发展的景观提供了宝贵的见解，特别关注注意力的综合和无处不在的作用。

Title: Enhancing Biologically Inspired Hierarchical Temporal Memory with Hardware-Accelerated Reflex Memory

• Authors: Pavia Bera, Sabrina Hassan Moon, Jennifer Adorno, Dayane Alfenas Reis, Sanjukta Bhanja
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.03746
• Pdf URL: https://arxiv.org/pdf/2504.03746
• Copy Paste: [[2504.03746]] Enhancing Biologically Inspired Hierarchical Temporal Memory with Hardware-Accelerated Reflex Memory(https://arxiv.org/abs/2504.03746)
• Keywords: generation
• Abstract: The rapid expansion of the Internet of Things (IoT) generates zettabytes of data that demand efficient unsupervised learning systems. Hierarchical Temporal Memory (HTM), a third-generation unsupervised AI algorithm, models the neocortex of the human brain by simulating columns of neurons to process and predict sequences. These neuron columns can memorize and infer sequences across multiple orders. While multiorder inferences offer robust predictive capabilities, they often come with significant computational overhead. The Sequence Memory (SM) component of HTM, which manages these inferences, encounters bottlenecks primarily due to its extensive programmable interconnects. In many cases, it has been observed that first-order temporal relationships have proven to be sufficient without any significant loss in efficiency. This paper introduces a Reflex Memory (RM) block, inspired by the Spinal Cord's working mechanisms, designed to accelerate the processing of first-order inferences. The RM block performs these inferences significantly faster than the SM. The integration of RM with HTM forms a system called the Accelerated Hierarchical Temporal Memory (AHTM), which processes repetitive information more efficiently than the original HTM while still supporting multiorder inferences. The experimental results demonstrate that the HTM predicts an event in 0.945 s, whereas the AHTM module does so in 0.125 s. Additionally, the hardware implementation of RM in a content-addressable memory (CAM) block, known as Hardware-Accelerated Hierarchical Temporal Memory (H-AHTM), predicts an event in just 0.094 s, significantly improving inference speed. Compared to the original algorithm \cite{bautista2020matlabhtm}, AHTM accelerates inference by up to 7.55x, while H-AHTM further enhances performance with a 10.10x speedup.
• 摘要：物联网（IoT）的快速扩展产生了需要有效无监督学习系统的数据的zettabytes。分层时间内存（HTM）是第三代无监督的AI算法，通过模拟神经元的列来处理和预测序列，对人脑的新皮层进行建模。这些神经元柱可以记住并推断跨多个顺序的序列。尽管多德推断具有强大的预测能力，但它们通常带有大量的计算开销。管理这些推论的HTM的序列内存（SM）组件主要是由于其广泛的可编程互连而遇到的瓶颈。在许多情况下，已经观察到，一阶时间关系已被证明足够了，而没有任何明显的效率损失。本文引入了反射记忆（RM）块，灵感来自脊髓的工作机制，旨在加速一阶推断的处理。 RM块执行这些推论的速度明显快于SM。 RM与HTM的集成形成了一个称为加速层次时间内存（AHTM）的系统，该系统比原始HTM更有效地处理重复信息，同时仍支持多率推断。实验结果表明，HTM预测为0.945 s的事件，而AHTM模块则在0.125 s中进行。此外，RM在可调地理的内存（CAM）块中的硬件实现（称为硬件加速层次的时间内存（H-AHTM））预测事件仅为0.094 s，可显着提高推理速度。与原始算法\ cite {Bautista2020matlabhtm}相比，AHTM的推理最多可加速7.55倍，而H-AHTM则通过10倍加速增强了性能。

Title: Revolutionizing Fractional Calculus with Neural Networks: Voronovskaya-Damasclin Theory for Next-Generation AI Systems

• Authors: Rômulo Damasclin Chaves dos Santos, Jorge Henrique de Oliveira Sales
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.03751
• Pdf URL: https://arxiv.org/pdf/2504.03751
• Copy Paste: [[2504.03751]] Revolutionizing Fractional Calculus with Neural Networks: Voronovskaya-Damasclin Theory for Next-Generation AI Systems(https://arxiv.org/abs/2504.03751)
• Keywords: generation
• Abstract: This work introduces rigorous convergence rates for neural network operators activated by symmetrized and perturbed hyperbolic tangent functions, utilizing novel Voronovskaya-Damasclin asymptotic expansions. We analyze basic, Kantorovich, and quadrature-type operators over infinite domains, extending classical approximation theory to fractional calculus via Caputo derivatives. Key innovations include parameterized activation functions with asymmetry control, symmetrized density operators, and fractional Taylor expansions for error analysis. The main theorem demonstrates that Kantorovich operators achieve \(o(n^{-\beta(N-\varepsilon)})\) convergence rates, while basic operators exhibit \(\mathcal{O}(n^{-\beta N})\) error decay. For deep networks, we prove \(\mathcal{O}(L^{-\beta(N-\varepsilon)})\) approximation bounds. Stability results under parameter perturbations highlight operator robustness. By integrating neural approximation theory with fractional calculus, this work provides foundational mathematical insights and deployable engineering solutions, with potential applications in complex system modeling and signal processing.
• 摘要：这项工作引入了通过对称和扰动双曲线切线功能激活的神经网络运算符的严格合并率，利用了新型的Voronovskaya-Damasclin渐近扩张。我们通过无限域分析了基本，坎托洛维奇和正交型运算符，从而将经典近似理论扩展到了通过Caputo衍生物的分数演算。关键创新包括具有不对称控制的参数化激活函数，对称性密度运算符和分数泰勒的扩展进行误差分析。主要定理证明了Kantorovich运算符实现\（o（n^{ - \ beta（n- \ varepsilon）}））\）\）收敛速率，而基本操作员则展示\（\ Mathcal {o}（n^{ - n^{ - \ beta n}）\）误差。对于深网，我们证明\（\ Mathcal {o}（l^{ - \ beta（n- \ varepsilon）}）\）近似范围。参数扰动下的稳定性结果突出显示操作员的鲁棒性。通过将神经近似理论与分数计算整合在一起，这项工作提供了基本的数学见解和可部署的工程解决方案，并提供了在复杂的系统建模和信号处理中的潜在应用。

Title: Recursive Training Loops in LLMs: How training data properties modulate distribution shift in generated data?

• Authors: Grgur Kovač, Jérémy Perez, Rémy Portelas, Peter Ford Dominey, Pierre-Yves Oudeyer
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2504.03814
• Pdf URL: https://arxiv.org/pdf/2504.03814
• Copy Paste: [[2504.03814]] Recursive Training Loops in LLMs: How training data properties modulate distribution shift in generated data?(https://arxiv.org/abs/2504.03814)
• Keywords: generation
• Abstract: Large language models (LLMs) are increasingly contributing to the creation of content on the Internet. This creates a feedback loop as subsequent generations of models will be trained on this generated, synthetic data. This phenomenon is receiving increasing interest, in particular because previous studies have shown that it may lead to distribution shift - models misrepresent and forget the true underlying distributions of human data they are expected to approximate (e.g. resulting in a drastic loss of quality). In this study, we study the impact of human data properties on distribution shift dynamics in iterated training loops. We first confirm that the distribution shift dynamics greatly vary depending on the human data by comparing four datasets (two based on Twitter and two on Reddit). We then test whether data quality may influence the rate of this shift. We find that it does on the twitter, but not on the Reddit datasets. We then focus on a Reddit dataset and conduct a more exhaustive evaluation of a large set of dataset properties. This experiment associated lexical diversity with larger, and semantic diversity with smaller detrimental shifts, suggesting that incorporating text with high lexical (but limited semantic) diversity could exacerbate the degradation of generated text. We then focus on the evolution of political bias, and find that the type of shift observed (bias reduction, amplification or inversion) depends on the political lean of the human (true) distribution. Overall, our work extends the existing literature on the consequences of recursive fine-tuning by showing that this phenomenon is highly dependent on features of the human data on which training occurs. This suggests that different parts of internet (e.g. GitHub, Reddit) may undergo different types of shift depending on their properties.
• 摘要：大型语言模型（LLM）越来越多地为互联网上的内容创建做出贡献。这会创建一个反馈循环，因为随后的模型将经过此生成的合成数据的培训。这种现象正在接受越来越多的兴趣，特别是因为以前的研究表明，它可能导致分布转移 - 模型歪曲了，并且忘记了他们期望的人类数据的真实潜在分布（例如，导致质量的严重丧失）。在这项研究中，我们研究了人类数据特性对迭代训练循环中分布移动动态的影响。我们首先确认通过比较四个数据集（基于Twitter，两个基于Reddit），根据人类数据，分布变化动态有很大不同。然后，我们测试数据质量是否可能影响此转变的速度。我们发现它可以在Twitter上进行，但在Reddit数据集上不做。然后，我们专注于REDDIT数据集，并对大量数据集属性进行更详尽的评估。该实验将词汇多样性与较大的语义多样性与较小的有害转移相关联，这表明将文本纳入具有高词汇（但有限的语义）多样性的文本可能会加剧生成文本的降级。然后，我们专注于政治偏见的演变，并发现观察到的转变类型（减少，放大或反转）取决于人类（真实）分布的政治倾向。总体而言，我们的工作通过表明这种现象高度依赖于发生训练的人类数据的特征，从而扩展了有关递归微调的后果的现有文献。这表明Internet的不同部分（例如Github，Reddit）可能会根据其属性而进行不同类型的轮班。

Title: A Hybrid Wavelet-Fourier Method for Next-Generation Conditional Diffusion Models

• Authors: Andrew Kiruluta, Andreas Lemos
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2504.03821
• Pdf URL: https://arxiv.org/pdf/2504.03821
• Copy Paste: [[2504.03821]] A Hybrid Wavelet-Fourier Method for Next-Generation Conditional Diffusion Models(https://arxiv.org/abs/2504.03821)
• Keywords: generation, generative
• Abstract: We present a novel generative modeling framework,Wavelet-Fourier-Diffusion, which adapts the diffusion paradigm to hybrid frequency representations in order to synthesize high-quality, high-fidelity images with improved spatial localization. In contrast to conventional diffusion models that rely exclusively on additive noise in pixel space, our approach leverages a multi-transform that combines wavelet sub-band decomposition with partial Fourier steps. This strategy progressively degrades and then reconstructs images in a hybrid spectral domain during the forward and reverse diffusion processes. By supplementing traditional Fourier-based analysis with the spatial localization capabilities of wavelets, our model can capture both global structures and fine-grained features more effectively. We further extend the approach to conditional image generation by integrating embeddings or conditional features via cross-attention. Experimental evaluations on CIFAR-10, CelebA-HQ, and a conditional ImageNet subset illustrate that our method achieves competitive or superior performance relative to baseline diffusion models and state-of-the-art GANs, as measured by Fréchet Inception Distance (FID) and Inception Score (IS). We also show how the hybrid frequency-based representation improves control over global coherence and fine texture synthesis, paving the way for new directions in multi-scale generative modeling.
• 摘要：我们提出了一种新颖的生成建模框架，小波折叠扩散，该框架将扩散范式适应混合频率表示，以便合成具有改进空间定位的高质量，高效率图像。与仅依赖于像素空间中添加噪声的常规扩散模型相反，我们的方法利用了将小波的副频分解与部分傅立叶步骤相结合的多转换。该策略逐渐降解，然后在正向和反向扩散过程中重建在混合光谱域中的图像。通过通过小波的空间定位功能来补充传统的基于傅立叶的分析，我们的模型可以更有效地捕获全球结构和细粒度的特征。我们通过通过交叉注意来整合嵌入或条件特征，进一步扩展了有条件图像生成的方法。对CIFAR-10，Celeba-HQ和条件成像子集的实验评估表明，相对于基线扩散模型和最先进的gan，我们的方法实现了竞争性或卓越的性能，如FréchetInpertion Intection距离（FID）和INCEPTION评分（IS）。我们还展示了基于混合频率的表示如何改善对全局相干性和精细纹理合成的控制，从而为多尺度生成建模的新方向铺平了道路。

Title: Detection Limits and Statistical Separability of Tree Ring Watermarks in Rectified Flow-based Text-to-Image Generation Models

• Authors: Ved Umrajkar, Aakash Kumar Singh
• Subjects: cs.CV, cs.AI, cs.CR, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2504.03850
• Pdf URL: https://arxiv.org/pdf/2504.03850
• Copy Paste: [[2504.03850]] Detection Limits and Statistical Separability of Tree Ring Watermarks in Rectified Flow-based Text-to-Image Generation Models(https://arxiv.org/abs/2504.03850)
• Keywords: generation
• Abstract: Tree-Ring Watermarking is a significant technique for authenticating AI-generated images. However, its effectiveness in rectified flow-based models remains unexplored, particularly given the inherent challenges of these models with noise latent inversion. Through extensive experimentation, we evaluated and compared the detection and separability of watermarks between SD 2.1 and FLUX.1-dev models. By analyzing various text guidance configurations and augmentation attacks, we demonstrate how inversion limitations affect both watermark recovery and the statistical separation between watermarked and unwatermarked images. Our findings provide valuable insights into the current limitations of Tree-Ring Watermarking in the current SOTA models and highlight the critical need for improved inversion methods to achieve reliable watermark detection and separability. The official implementation, dataset release and all experimental results are available at this \href{this https URL}{\textbf{link}}.
• 摘要：树环水印是对AI生成的图像进行身份验证的重要技术。但是，它在基于整流的基于流动的模型中的有效性仍未开发，尤其是考虑到这些模型具有噪声潜在反演的固有挑战。通过广泛的实验，我们评估并比较了SD 2.1和Flux.1-DEV模型之间水印的检测和分离性。通过分析各种文本指导配置和增强攻击，我们演示了反演限制如何影响水印恢复和水印和不标记图像之间的统计分离。我们的发现提供了对当前SOTA模型中树环水印的当前局限性的宝贵见解，并突出了对改进反转方法的关键需求，以实现可靠的水印检测和可分离性。官方实现，数据集发布和所有实验结果均可在此\ href {此https url} {\ textbf {link}}中获得。

Title: Can ChatGPT Learn My Life From a Week of First-Person Video?

• Authors: Keegan Harris
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.03857
• Pdf URL: https://arxiv.org/pdf/2504.03857
• Copy Paste: [[2504.03857]] Can ChatGPT Learn My Life From a Week of First-Person Video?(https://arxiv.org/abs/2504.03857)
• Keywords: generative
• Abstract: Motivated by recent improvements in generative AI and wearable camera devices (e.g. smart glasses and AI-enabled pins), I investigate the ability of foundation models to learn about the wearer's personal life through first-person camera data. To test this, I wore a camera headset for 54 hours over the course of a week, generated summaries of various lengths (e.g. minute-long, hour-long, and day-long summaries), and fine-tuned both GPT-4o and GPT-4o-mini on the resulting summary hierarchy. By querying the fine-tuned models, we are able to learn what the models learned about me. The results are mixed: Both models learned basic information about me (e.g. approximate age, gender). Moreover, GPT-4o correctly deduced that I live in Pittsburgh, am a PhD student at CMU, am right-handed, and have a pet cat. However, both models also suffered from hallucination and would make up names for the individuals present in the video footage of my life.
• 摘要：由于最近改进了生成AI和可穿戴式摄像头设备（例如智能眼镜和AI-ai-aigable Pins）的动机，我研究了基础模型通过第一人称相机数据了解佩戴者的个人生活的能力。为了测试这一点，我在一周的时间内戴了相机耳机54小时，生成了各种长度的摘要（例如，长达一分钟，长达一个小时和一整天的摘要），并在由此产生的摘要层次上进行了微调的GPT-4O和GPT-4O和GPT-4O-Mini。通过查询微调模型，我们可以了解模型对我的了解。结果混合在一起：两种模型都学到了有关我的基本信息（例如大约年龄，性别）。此外，GPT-4O正确地推断出我住在匹兹堡，是CMU的博士生，Am Am Am右撇子，并有一只宠物猫。但是，这两种模型也都遭受了幻觉的困扰，并将为我一生的视频录像中的个人构成名字。

Title: Control Map Distribution using Map Query Bank for Online Map Generation

• Authors: Ziming Liu, Leichen Wang, Ge Yang, Xinrun Li, Xingtao Hu, Hao Sun, Guangyu Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.03868
• Pdf URL: https://arxiv.org/pdf/2504.03868
• Copy Paste: [[2504.03868]] Control Map Distribution using Map Query Bank for Online Map Generation(https://arxiv.org/abs/2504.03868)
• Keywords: generation
• Abstract: Reliable autonomous driving systems require high-definition (HD) map that contains detailed map information for planning and navigation. However, pre-build HD map requires a large cost. Visual-based Online Map Generation (OMG) has become an alternative low-cost solution to build a local HD map. Query-based BEV Transformer has been a base model for this task. This model learns HD map predictions from an initial map queries distribution which is obtained by offline optimization on training set. Besides the quality of BEV feature, the performance of this model also highly relies on the capacity of initial map query distribution. However, this distribution is limited because the limited query number. To make map predictions optimal on each test sample, it is essential to generate a suitable initial distribution for each specific scenario. This paper proposes to decompose the whole HD map distribution into a set of point representations, namely map query bank (MQBank). To build specific map query initial distributions of different scenarios, low-cost standard definition map (SD map) data is introduced as a kind of prior knowledge. Moreover, each layer of map decoder network learns instance-level map query features, which will lose detailed information of each point. However, BEV feature map is a point-level dense feature. It is important to keep point-level information in map queries when interacting with BEV feature map. This can also be solved with map query bank method. Final experiments show a new insight on SD map prior and a new record on OpenLaneV2 benchmark with 40.5%, 45.7% mAP on vehicle lane and pedestrian area.
• 摘要：可靠的自动驾驶系统需要高清（HD）图，其中包含用于规划和导航的详细地图信息。但是，前构建高清图需要大量成本。基于视觉的在线地图生成（OMG）已成为构建本地高清图的替代低成本解决方案。基于查询的BEV变压器一直是该任务的基础模型。该模型从初始地图查询分布中学习高清地图预测，该预测是通过在训练集上的离线优化获得的。除了BEV功能的质量外，该模型的性能还高度依赖于初始地图​​查询分布的能力。但是，此分布受到限制，因为有限的查询编号。为了在每个测试样本上进行最佳的地图预测，对于每个特定方案生成合适的初始分布至关重要。本文建议将整个高清图分布分解为一组点表示，即MAP查询库（MQBANK）。为了构建不同方案的特定地图查询初始分布，将低成本标准定义图（SD地图）数据作为一种先验知识引入。此外，地图解码器网络的每一层都会学习实例级地图查询功能，这将失去每个点的详细信息。但是，BEV功能映射是一个点级密集的功能。与BEV功能映射交互时，将点级信息保留在地图查询中很重要。这也可以通过地图查询银行方法来解决。最终实验显示了有关SD地图先验的新见解，以及OpenLaneV2基准的新记录，在车道和行人区域的地图为40.5％，45.7％的地图。

Title: 3D Scene Understanding Through Local Random Access Sequence Modeling

• Authors: Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh, Jared Watrous, Honglin Chen, Khai Loong Aw, Daniel L. K. Yamins
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.03875
• Pdf URL: https://arxiv.org/pdf/2504.03875
• Copy Paste: [[2504.03875]] 3D Scene Understanding Through Local Random Access Sequence Modeling(https://arxiv.org/abs/2504.03875)
• Keywords: generation, generative
• Abstract: 3D scene understanding from single images is a pivotal problem in computer vision with numerous downstream applications in graphics, augmented reality, and robotics. While diffusion-based modeling approaches have shown promise, they often struggle to maintain object and scene consistency, especially in complex real-world scenarios. To address these limitations, we propose an autoregressive generative approach called Local Random Access Sequence (LRAS) modeling, which uses local patch quantization and randomly ordered sequence generation. By utilizing optical flow as an intermediate representation for 3D scene editing, our experiments demonstrate that LRAS achieves state-of-the-art novel view synthesis and 3D object manipulation capabilities. Furthermore, we show that our framework naturally extends to self-supervised depth estimation through a simple modification of the sequence design. By achieving strong performance on multiple 3D scene understanding tasks, LRAS provides a unified and effective framework for building the next generation of 3D vision models.
• 摘要：来自单个图像的3D场景理解是计算机视觉中的关键问题，具有图形，增强现实和机器人技术的下游应用程序。尽管基于扩散的建模方法已经表现出了希望，但它们通常很难维持对象和场景一致性，尤其是在复杂的现实情况下。为了解决这些局限性，我们提出了一种称为局部随机访问序列（LRAS）建模的自回归生成方法，该方法使用局部补丁量化和随机有序的序列生成。通过利用光流作为3D场景编辑的中间表示，我们的实验表明，LRAS可以实现最新的新型视图合成和3D对象操纵能力。此外，我们表明我们的框架自然地通过简单地修改序列设计来扩展到自我监督的深度估计。通过在多个3D场景理解任务上实现强大的性能，LRAS为建立下一代3D视觉模型提供了一个统一有效的框架。

Title: Concept-based Rubrics Improve LLM Formative Assessment and Data Synthesis

• Authors: Yuchen Wei, Dennis Pearl, Matthew Beckman, Rebecca J. Passonneau
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.03877
• Pdf URL: https://arxiv.org/pdf/2504.03877
• Copy Paste: [[2504.03877]] Concept-based Rubrics Improve LLM Formative Assessment and Data Synthesis(https://arxiv.org/abs/2504.03877)
• Keywords: generative
• Abstract: Formative assessment in STEM topics aims to promote student learning by identifying students' current understanding, thus targeting how to promote further learning. Previous studies suggest that the assessment performance of current generative large language models (LLMs) on constructed responses to open-ended questions is significantly lower than that of supervised classifiers trained on high-quality labeled data. However, we demonstrate that concept-based rubrics can significantly enhance LLM performance, which narrows the gap between LLMs as off-the shelf assessment tools, and smaller supervised models, which need large amounts of training data. For datasets where concept-based rubrics allow LLMs to achieve strong performance, we show that the concept-based rubrics help the same LLMs generate high quality synthetic data for training lightweight, high-performance supervised models. Our experiments span diverse STEM student response datasets with labels of varying quality, including a new real-world dataset that contains some AI-assisted responses, which introduces additional considerations.
• 摘要：STEM主题中的形成性评估旨在通过确定学生当前的理解来促进学生学习，从而针对如何促进进一步的学习。先前的研究表明，对开放式问题的构建回答的当前生成大语言模型（LLM）的评估表现明显低于接受高质量标记数据培训的监督分类器。但是，我们证明了基于概念的标题可以显着提高LLM的性能，从而缩小了LLMS作为货架评估工具的差距和需要大量培训数据的较小监督模型。对于基于概念的标题允许LLMS实现强大性能的数据集，我们表明基于概念的专栏可帮助相同的LLM生成高质量的合成数据，以训练轻巧，高性能监督模型。我们的实验涵盖了不同的STEM学生响应数据集，标签具有不同质量的标签，其中包括一个新的现实世界数据集，其中包含一些AI辅助响应，引入了其他注意事项。

Title: Analysis of Robustness of a Large Game Corpus

• Authors: Mahsa Bazzaz, Seth Cooper
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.03940
• Pdf URL: https://arxiv.org/pdf/2504.03940
• Copy Paste: [[2504.03940]] Analysis of Robustness of a Large Game Corpus(https://arxiv.org/abs/2504.03940)
• Keywords: generation
• Abstract: Procedural content generation via machine learning (PCGML) in games involves using machine learning techniques to create game content such as maps and levels. 2D tile-based game levels have consistently served as a standard dataset for PCGML because they are a simplified version of game levels while maintaining the specific constraints typical of games, such as being solvable. In this work, we highlight the unique characteristics of game levels, including their structured discrete data nature, the local and global constraints inherent in the games, and the sensitivity of the game levels to small changes in input. We define the robustness of data as a measure of sensitivity to small changes in input that cause a change in output, and we use this measure to analyze and compare these levels to state-of-the-art machine learning datasets, showcasing the subtle differences in their nature. We also constructed a large dataset from four games inspired by popular classic tile-based games that showcase these characteristics and address the challenge of sparse data in PCGML by providing a significantly larger dataset than those currently available.
• 摘要：通过机器学习（PCGML）的程序内容生成游戏中，涉及使用机器学习技术来创建游戏内容，例如地图和级别。基于2D图块的游戏水平一直是PCGML的标准数据集，因为它们是游戏级别的简化版本，同时保持了典型的游戏的特定约束，例如可解决的问题。在这项工作中，我们强调了游戏级别的独特特征，包括它们的结构化离散数据性质，游戏中固有的本地和全局约束以及游戏水平对输入小变化的敏感性。我们将数据的鲁棒性定义为对导致输出变化的小变化的敏感性的量度，我们使用此措施来分析和比较这些级别与最新的机器学习数据集，并展示其性质的细微差异。我们还从四个游戏启发的大型数据集中构建了一个大型数据集，该游戏灵感来自经典的基于经典的瓷砖游戏，这些游戏展示了这些特征，并通过提供比当前可用的数据集更大的数据集来解决PCGML中稀疏数据的挑战。

Title: DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusion

• Authors: Maksim Siniukov, Di Chang, Minh Tran, Hongkun Gong, Ashutosh Chaubey, Mohammad Soleymani
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.04010
• Pdf URL: https://arxiv.org/pdf/2504.04010
• Copy Paste: [[2504.04010]] DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusion(https://arxiv.org/abs/2504.04010)
• Keywords: generation
• Abstract: Generating naturalistic and nuanced listener motions for extended interactions remains an open problem. Existing methods often rely on low-dimensional motion codes for facial behavior generation followed by photorealistic rendering, limiting both visual fidelity and expressive richness. To address these challenges, we introduce DiTaiListener, powered by a video diffusion model with multimodal conditions. Our approach first generates short segments of listener responses conditioned on the speaker's speech and facial motions with DiTaiListener-Gen. It then refines the transitional frames via DiTaiListener-Edit for a seamless transition. Specifically, DiTaiListener-Gen adapts a Diffusion Transformer (DiT) for the task of listener head portrait generation by introducing a Causal Temporal Multimodal Adapter (CTM-Adapter) to process speakers' auditory and visual cues. CTM-Adapter integrates speakers' input in a causal manner into the video generation process to ensure temporally coherent listener responses. For long-form video generation, we introduce DiTaiListener-Edit, a transition refinement video-to-video diffusion model. The model fuses video segments into smooth and continuous videos, ensuring temporal consistency in facial expressions and image quality when merging short video segments produced by DiTaiListener-Gen. Quantitatively, DiTaiListener achieves the state-of-the-art performance on benchmark datasets in both photorealism (+73.8% in FID on RealTalk) and motion representation (+6.1% in FD metric on VICO) spaces. User studies confirm the superior performance of DiTaiListener, with the model being the clear preference in terms of feedback, diversity, and smoothness, outperforming competitors by a significant margin.
• 摘要：为扩展互动而产生自然主义和细微的听众动议仍然是一个开放的问题。现有的方法通常依靠低维运动代码来产生面部行为，然后是情感化渲染，从而限制了视觉保真度和表现力丰富。为了应对这些挑战，我们介绍了Ditailistener，并由具有多模式条件的视频扩散模型提供支持。我们的方法首先产生了聆听者的反应的简短片段，该响应是根据ditailister-gen的演讲者的言语和面部动作来调节的。然后，它通过ditailistener-edit来完善过渡帧，以进行无缝过渡。具体而言，ditailister-gen通过引入因果关系多模式适配器（CTM-apapter）来处理扬声器的听觉和视觉提示，从而适应了传播变压器（DIT）来制定侦听器头像生成的任务。 CTM适配器以因果方式将扬声器的输入集成到视频生成过程中，以确保暂时连贯的听众响应。对于长期视频生成，我们介绍了Ditailistener-edit，这是一种过渡细化视频与视频扩散模型。该模型将视频片段融合到平滑而连续的视频中，在合并Ditailisterener-Gen制作的简短视频片段时，确保了面部表情和图像质量的时间一致性。从数量上讲，ditailistener在光真相中（在Realtalk上的FID中+73.8％）和运动表示（VICO上FD指标的+6.1％）中的基准数据集中达到了最先进的性能。用户研究证实了Ditailistener的出色表现，该模型在反馈，多样性和平滑度方面明显偏爱，并以显着的优于竞争对手。

Title: UCS: A Universal Model for Curvilinear Structure Segmentation

• Authors: Dianshuo Li, Li Chen, Yunxiang Cao, Kai Zhu, Jun Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04034
• Pdf URL: https://arxiv.org/pdf/2504.04034
• Copy Paste: [[2504.04034]] UCS: A Universal Model for Curvilinear Structure Segmentation(https://arxiv.org/abs/2504.04034)
• Keywords: generation
• Abstract: Curvilinear structure segmentation (CSS) is vital in various domains, including medical imaging, landscape analysis, industrial surface inspection, and plant analysis. While existing methods achieve high performance within specific domains, their generalizability is limited. On the other hand, large-scale models such as Segment Anything Model (SAM) exhibit strong generalization but are not optimized for curvilinear structures. Existing adaptations of SAM primarily focus on general object segmentation and lack specialized design for CSS tasks. To bridge this gap, we propose the Universal Curvilinear structure Segmentation (\textit{UCS}) model, which adapts SAM to CSS tasks while enhancing its generalization. \textit{UCS} features a novel encoder architecture integrating a pretrained SAM encoder with two innovations: a Sparse Adapter, strategically inserted to inherit the pre-trained SAM encoder's generalization capability while minimizing the number of fine-tuning parameters, and a Prompt Generation module, which leverages Fast Fourier Transform with a high-pass filter to generate curve-specific prompts. Furthermore, the \textit{UCS} incorporates a mask decoder that eliminates reliance on manual interaction through a dual-compression module: a Hierarchical Feature Compression module, which aggregates the outputs of the sampled encoder to enhance detail preservation, and a Guidance Feature Compression module, which extracts and compresses image-driven guidance features. Evaluated on a comprehensive multi-domain dataset, including an in-house dataset covering eight natural curvilinear structures, \textit{UCS} demonstrates state-of-the-art generalization and open-set segmentation performance across medical, engineering, natural, and plant imagery, establishing a new benchmark for universal CSS.
• 摘要：曲线结构分割（CSS）在各种领域至关重要，包括医学成像，景观分析，工业表面检查和植物分析。尽管现有方法在特定领域内实现高性能，但它们的推广性是有限的。另一方面，大型模型（例如段）任何模型（SAM）表现出强烈的概括，但并未针对曲线结构进行优化。 SAM的现有改编主要集中于一般对象细分，并且缺乏针对CSS任务的专业设计。为了弥合这一差距，我们提出了通用曲线结构分割（\ textIt {ucs}）模型，该模型在增强其概括的同时，将SAM适应CSS任务。 \ textIt {ucs}具有一个新颖的编码器架构，该体系结构集成了一个预测的SAM编码器，具有两个创新：一个稀疏的适配器，从策略上插入以继承预先训练的SAM编码能力，同时最大程度地减少迅速生成的模块，并迅速地转换高级速度，以使其迅速转换高级固定效果，从而迅速地转换了较高的速度，从而使高级固定型固定效果。 Furthermore, the \textit{UCS} incorporates a mask decoder that eliminates reliance on manual interaction through a dual-compression module: a Hierarchical Feature Compression module, which aggregates the outputs of the sampled encoder to enhance detail preservation, and a Guidance Feature Compression module, which extracts and compresses image-driven guidance features.在一个全面的多域数据集上进行了评估，其中包括一个涵盖八个天然曲线结构的内部数据集，\ textit {ucs}展示了跨医疗，工程，天然和植物图像的最新概括和开放设定的分段性能，并建立了通​​用CSS的新基准标记。

Title: Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Models

• Authors: Xuyang Guo, Zekai Huang, Jiayan Huo, Yingyu Liang, Zhenmei Shi, Zhao Song, Jiahao Zhang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.04051
• Pdf URL: https://arxiv.org/pdf/2504.04051
• Copy Paste: [[2504.04051]] Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Models(https://arxiv.org/abs/2504.04051)
• Keywords: generation, generative
• Abstract: Generative models have driven significant progress in a variety of AI tasks, including text-to-video generation, where models like Video LDM and Stable Video Diffusion can produce realistic, movie-level videos from textual instructions. Despite these advances, current text-to-video models still face fundamental challenges in reliably following human commands, particularly in adhering to simple numerical constraints. In this work, we present T2VCountBench, a specialized benchmark aiming at evaluating the counting capability of SOTA text-to-video models as of 2025. Our benchmark employs rigorous human evaluations to measure the number of generated objects and covers a diverse range of generators, covering both open-source and commercial models. Extensive experiments reveal that all existing models struggle with basic numerical tasks, almost always failing to generate videos with an object count of 9 or fewer. Furthermore, our comprehensive ablation studies explore how factors like video style, temporal dynamics, and multilingual inputs may influence counting performance. We also explore prompt refinement techniques and demonstrate that decomposing the task into smaller subtasks does not easily alleviate these limitations. Our findings highlight important challenges in current text-to-video generation and provide insights for future research aimed at improving adherence to basic numerical constraints.
• 摘要：生成模型在各种AI任务中都取得了重大进展，包括文本到视频生成，其中视频LDM和稳定的视频扩散可以从文本说明中产生现实的电影级视频。尽管有这些进步，但当前的文本对视频模型仍然在人类命令下可靠地面临着根本的挑战，尤其是在遵守简单的数值约束方面。在这项工作中，我们介绍了T2VCountBench，这是一种专门的基准测试，旨在评估截至2025年SOTA文本到视频模型的计数能力。我们的基准测试采用严格的人体评估来衡量生成的对象的数量并涵盖各种生成器，从而涵盖了开放式和商业模型。广泛的实验表明，所有现有模型都与基本的数值任务相努力，几乎总是无法生成对象数量为9或更少的视频。此外，我们的全面消融研究探讨了视频风格，时间动态和多语言输入等因素如何影响计数性能。我们还探讨了迅速的改进技术，并证明将任务分解为较小的子任务并不能轻易缓解这些局限性。我们的发现突出了当前文本到视频生成的重要挑战，并为未来的研究提供了见解，旨在改善对基本数值约束的依从性。

Title: UniRVQA: A Unified Framework for Retrieval-Augmented Vision Question Answering via Self-Reflective Joint Training

• Authors: Jiaqi Deng, Kaize Shi, Zonghan Wu, Huan Huo, Dingxian Wang, Guandong Xu
• Subjects: cs.CV, cs.IR, cs.MM
• Abstract URL: https://arxiv.org/abs/2504.04065
• Pdf URL: https://arxiv.org/pdf/2504.04065
• Copy Paste: [[2504.04065]] UniRVQA: A Unified Framework for Retrieval-Augmented Vision Question Answering via Self-Reflective Joint Training(https://arxiv.org/abs/2504.04065)
• Keywords: generation
• Abstract: Knowledge-based Vision Question Answering (KB-VQA) systems address complex visual-grounded questions requiring external knowledge, such as web-sourced encyclopedia articles. Existing methods often use sequential and separate frameworks for the retriever and the generator with limited parametric knowledge sharing. However, since both retrieval and generation tasks require accurate understanding of contextual and external information, such separation can potentially lead to suboptimal system performance. Another key challenge is the integration of multimodal information. General-purpose multimodal pre-trained models, while adept at multimodal representation learning, struggle with fine-grained retrieval required for knowledge-intensive visual questions. Recent specialized pre-trained models mitigate the issue, but are computationally expensive. To bridge the gap, we propose a Unified Retrieval-Augmented VQA framework (UniRVQA). UniRVQA adapts general multimodal pre-trained models for fine-grained knowledge-intensive tasks within a unified framework, enabling cross-task parametric knowledge sharing and the extension of existing multimodal representation learning capability. We further introduce a reflective-answering mechanism that allows the model to explicitly evaluate and refine its knowledge boundary. Additionally, we integrate late interaction into the retrieval-augmented generation joint training process to enhance fine-grained understanding of queries and documents. Our approach achieves competitive performance against state-of-the-art models, delivering a significant 4.7% improvement in answering accuracy, and brings an average 7.5% boost in base MLLMs' VQA performance.
• 摘要：基于知识的视觉问题答案（KB-VQA）系统解决了需要外部知识的复杂视觉接地问题，例如Web提供的百科全书文章。现有方法通常使用有限的参数知识共享的回收器和生成器使用顺序和单独的框架。但是，由于检索和生成任务都需要准确了解上下文和外部信息，因此这种分离可能会导致次优系统性能。另一个主要挑战是集成多模式信息。通用多模式预训练的模型虽然熟练于多模式表示学习，但在知识密集的视觉问题上与精细的检索作斗争。最近的专业预培训模型可以减轻该问题，但在计算上很昂贵。为了弥合差距，我们提出了一个统一的检索式VQA框架（UNIRVQA）。 UNIRVQA适应了一般的多模式预训练模型，以在统一的框架内完成精细的知识密集型任务，从而实现了交叉任务参数知识共享和现有多模式表示能力的扩展。我们进一步介绍了一种反射式交付机制，该机制允许该模型明确评估和完善其知识边界。此外，我们将较晚的互动整合到检索的一代联合培训过程中，以增强对查询和文档的细粒度理解。我们的方法可以针对最先进的模型实现竞争性能，在回答准确性方面取得了显着提高4.7％，并在基本MLLM的VQA性能中平均提高了7.5％。

Title: TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection

• Authors: Chunzhao Xie, Tongxuan Liu, Lei Jiang, Yuting Zeng, jinrong Guo, Yunheng Shen, Weizhe Huang, Jing Li, Xiaohua Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.04099
• Pdf URL: https://arxiv.org/pdf/2504.04099
• Copy Paste: [[2504.04099]] TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection(https://arxiv.org/abs/2504.04099)
• Keywords: generation
• Abstract: Large Vision-Language Models have demonstrated remarkable performance across various tasks; however, the challenge of hallucinations constrains their practical applications. The hallucination problem arises from multiple factors, including the inherent hallucinations in language models, the limitations of visual encoders in perception, and biases introduced by multimodal data. Extensive research has explored ways to mitigate hallucinations. For instance, OPERA prevents the model from overly focusing on "anchor tokens", thereby reducing hallucinations, whereas VCD mitigates hallucinations by employing a contrastive decoding approach. In this paper, we investigate the correlation between the decay of attention to image tokens and the occurrence of hallucinations. Based on this finding, we propose Temporal Attention Real-time Accumulative Connection (TARAC), a novel training-free method that dynamically accumulates and updates LVLMs' attention on image tokens during generation. By enhancing the model's attention to image tokens, TARAC mitigates hallucinations caused by the decay of attention on image tokens. We validate the effectiveness of TARAC across multiple models and datasets, demonstrating that our approach substantially mitigates hallucinations. In particular, TARAC reduces $C_S$ by 25.2 and $C_I$ by 8.7 compared to VCD on the CHAIR benchmark.
• 摘要：大型视觉模型在各种任务中表现出了出色的表现。但是，幻觉的挑战限制了其实际应用。幻觉问题来自多种因素，包括语言模型中的固有幻觉，感知中的视觉编码器的局限性以及多模式数据引入的偏见。广泛的研究探索了减轻幻觉的方法。例如，Opera阻止该模型过于关注“锚定令牌”，从而减少了幻觉，而VCD通过采用对比度解码方法来减轻幻觉。在本文中，我们研究了注意力令牌的衰减与幻觉的发生之间的相关性。基于这一发现，我们提出了时间关注实时累积连接（TARAC），这是一种新颖的无训练方法，可以动态积累并更新LVLMS在生成过程中对图像令牌的关注。通过增强模型对图像令牌的关注，Tarac减轻了由于图像令牌上的注意力而引起的幻觉。我们验证了TARAC在多个模型和数据集中的有效性，表明我们的方法大大减轻了幻觉。特别是，与椅子基准上的VCD相比，TARAC将$ C_S $减少25.2和$ C_I $。

Title: Multi-identity Human Image Animation with Structural Video Diffusion

• Authors: Zhenzhi Wang, Yixuan Li, Yanhong Zeng, Yuwei Guo, Dahua Lin, Tianfan Xue, Bo Dai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.04126
• Pdf URL: https://arxiv.org/pdf/2504.04126
• Copy Paste: [[2504.04126]] Multi-identity Human Image Animation with Structural Video Diffusion(https://arxiv.org/abs/2504.04126)
• Keywords: generation
• Abstract: Generating human videos from a single image while ensuring high visual quality and precise control is a challenging task, especially in complex scenarios involving multiple individuals and interactions with objects. Existing methods, while effective for single-human cases, often fail to handle the intricacies of multi-identity interactions because they struggle to associate the correct pairs of human appearance and pose condition and model the distribution of 3D-aware dynamics. To address these limitations, we present Structural Video Diffusion, a novel framework designed for generating realistic multi-human videos. Our approach introduces two core innovations: identity-specific embeddings to maintain consistent appearances across individuals and a structural learning mechanism that incorporates depth and surface-normal cues to model human-object interactions. Additionally, we expand existing human video dataset with 25K new videos featuring diverse multi-human and object interaction scenarios, providing a robust foundation for training. Experimental results demonstrate that Structural Video Diffusion achieves superior performance in generating lifelike, coherent videos for multiple subjects with dynamic and rich interactions, advancing the state of human-centric video generation.
• 摘要：从单个图像中生成人类视频，同时确保高视觉质量和精确控制是一项具有挑战性的任务，尤其是在涉及多个个人的复杂场景和与物体互动的情况下。现有方法虽然对单人类案例有效，但通常无法处理多个认同相互作用的复杂性，因为它们很难将人类外观和姿势状况的正确对并建模3D感知动力学的分布。为了解决这些局限性，我们提出了结构性视频扩散，这是一个新颖的框架，旨在生成现实的多人类视频。我们的方法介绍了两种核心创新：特定于身份的嵌入，以保持个人之间一致的外观和结构学习机制，该结构学习机制结合了深度和表面正常线索，以模拟人类对象相互作用。此外，我们通过25K新视频扩展了现有的人类视频数据集，其中包含多种多样的多人和对象互动方案，为培训提供了强大的基础。实验结果表明，结构视频扩散在产生栩栩如生的视频中，具有动态和丰富相互作用的多个受试者的连贯视频，可以提高以人为中心的视频的状态。

Title: Scaling Federated Learning Solutions with Kubernetes for Synthesizing Histopathology Images

• Authors: Andrei-Alexandru Preda, Iulian-Marius Tăiatu, Dumitru-Clementin Cercel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04130
• Pdf URL: https://arxiv.org/pdf/2504.04130
• Copy Paste: [[2504.04130]] Scaling Federated Learning Solutions with Kubernetes for Synthesizing Histopathology Images(https://arxiv.org/abs/2504.04130)
• Keywords: generative
• Abstract: In the field of deep learning, large architectures often obtain the best performance for many tasks, but also require massive datasets. In the histological domain, tissue images are expensive to obtain and constitute sensitive medical information, raising concerns about data scarcity and privacy. Vision Transformers are state-of-the-art computer vision models that have proven helpful in many tasks, including image classification. In this work, we combine vision Transformers with generative adversarial networks to generate histopathological images related to colorectal cancer and test their quality by augmenting a training dataset, leading to improved classification accuracy. Then, we replicate this performance using the federated learning technique and a realistic Kubernetes setup with multiple nodes, simulating a scenario where the training dataset is split among several hospitals unable to share their information directly due to privacy concerns.
• 摘要：在深度学习领域，大型体系结构通常可以为许多任务获得最佳性能，但也需要大量的数据集。在组织学领域，组织图像获得昂贵，构成敏感的医学信息，从而引起了人们对数据稀缺和隐私的担忧。视觉变压器是最先进的计算机视觉模型，已证明对许多任务（包括图像分类）有帮助。在这项工作中，我们将视觉变压器与生成的对抗网络相结合，以产生与结直肠癌相关的组织病理学图像，并通过增强培训数据集来测试其质量，从而提高了分类精度。然后，我们使用联合学习技术和具有多个节点的现实kubernetes设置来复制这种表现，模拟了一个场景，其中培训数据集被分配给几家由于隐私问题而无法直接共享其信息的医院。

Title: JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration

• Authors: Yunlong Lin, Zixu Lin, Haoyu Chen, Panwang Pan, Chenxin Li, Sixiang Chen, Yeying Jin, Wenbo Li, Xinghao Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04158
• Pdf URL: https://arxiv.org/pdf/2504.04158
• Copy Paste: [[2504.04158]] JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration(https://arxiv.org/abs/2504.04158)
• Keywords: restoration
• Abstract: Vision-centric perception systems struggle with unpredictable and coupled weather degradations in the wild. Current solutions are often limited, as they either depend on specific degradation priors or suffer from significant domain gaps. To enable robust and autonomous operation in real-world conditions, we propose JarvisIR, a VLM-powered agent that leverages the VLM as a controller to manage multiple expert restoration models. To further enhance system robustness, reduce hallucinations, and improve generalizability in real-world adverse weather, JarvisIR employs a novel two-stage framework consisting of supervised fine-tuning and human feedback alignment. Specifically, to address the lack of paired data in real-world scenarios, the human feedback alignment enables the VLM to be fine-tuned effectively on large-scale real-world data in an unsupervised manner. To support the training and evaluation of JarvisIR, we introduce CleanBench, a comprehensive dataset consisting of high-quality and large-scale instruction-responses pairs, including 150K synthetic entries and 80K real entries. Extensive experiments demonstrate that JarvisIR exhibits superior decision-making and restoration capabilities. Compared with existing methods, it achieves a 50% improvement in the average of all perception metrics on CleanBench-Real. Project page: this https URL.
• 摘要：以视觉为中心的感知系统与野外无法预测的耦合天气降解斗争。当前的解决方案通常受到限制，因为它们要么取决于特定的降解先验，要么遭受重要的域间隙。为了在现实世界中启用强大的自主操作，我们提出了一种由VLM驱动的代理Jarvisir，它利用VLM作为控制器来管理多个专家恢复模型。为了进一步增强系统的鲁棒性，减少幻觉并提高现实世界中不利天气的普遍性，Jarvisir采用了一种新颖的两阶段框架，包括受监督的微调和人类反馈对准。具体而言，为了解决现实情况下缺乏配对数据的缺乏，人类的反馈对齐使VLM能够以无聊的方式在大型现实世界中有效地进行微调。为了支持Jarvisir的培训和评估，我们介绍了Cleanbench，这是一个全面的数据集，该数据集由高质量和大规模的指令响应对组成，包括150K合成条目和80K真实条目。广泛的实验表明，Jarvisir具有出色的决策和恢复能力。与现有方法相比，它在清洁板现实的所有感知指标的平均值上取得了50％的提高。项目页面：此HTTPS URL。

Title: OrbitZoo: Multi-Agent Reinforcement Learning Environment for Orbital Dynamics

• Authors: Alexandre Oliveira, Katarina Dyreby, Francisco Caldas, Cláudia Soares
• Subjects: cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2504.04160
• Pdf URL: https://arxiv.org/pdf/2504.04160
• Copy Paste: [[2504.04160]] OrbitZoo: Multi-Agent Reinforcement Learning Environment for Orbital Dynamics(https://arxiv.org/abs/2504.04160)
• Keywords: generation
• Abstract: The increasing number of satellites and orbital debris has made space congestion a critical issue, threatening satellite safety and sustainability. Challenges such as collision avoidance, station-keeping, and orbital maneuvering require advanced techniques to handle dynamic uncertainties and multi-agent interactions. Reinforcement learning (RL) has shown promise in this domain, enabling adaptive, autonomous policies for space operations; however, many existing RL frameworks rely on custom-built environments developed from scratch, which often use simplified models and require significant time to implement and validate the orbital dynamics, limiting their ability to fully capture real-world complexities. To address this, we introduce OrbitZoo, a versatile multi-agent RL environment built on a high-fidelity industry standard library, that enables realistic data generation, supports scenarios like collision avoidance and cooperative maneuvers, and ensures robust and accurate orbital dynamics. The environment is validated against a real satellite constellation, Starlink, achieving a Mean Absolute Percentage Error (MAPE) of 0.16% compared to real-world data. This validation ensures reliability for generating high-fidelity simulations and enabling autonomous and independent satellite operations.
• 摘要：卫星和轨道碎屑的数量越来越多，使太空拥堵成为关键问题，威胁到卫星的安全性和可持续性。避免碰撞，维护站和轨道操纵等挑战需要高级技术来处理动态的不确定性和多代理相互作用。强化学习（RL）在该领域表现出了希望，从而实现了适应性的，自主政策的空间操作；但是，许多现有的RL框架依赖于从头开始开发的定制环境，这些环境通常使用简化的模型，并需要大量时间来实现和验证轨道动力学，从而限制了它们完全捕获现实世界中复杂性的能力。为了解决这个问题，我们介绍了Orbitzoo，这是一种基于高保真行业标准库的多功能多代理RL环境，可实现现实的数据生成，支持避免碰撞和合作操作等场景，并确保强大而准确的轨道动力学。与实际数据相比，与真实的卫星星座（Starlink）相比，该环境已被验证，其平均绝对百分比误差（MAPE）为0.16％。该验证可确保生成高保真模拟并实现自主和独立卫星操作的可靠性。

Title: SDEIT: Semantic-Driven Electrical Impedance Tomography

• Authors: Dong Liu, Yuanchao Wu, Bowen Tong, Jiansong Deng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04185
• Pdf URL: https://arxiv.org/pdf/2504.04185
• Copy Paste: [[2504.04185]] SDEIT: Semantic-Driven Electrical Impedance Tomography(https://arxiv.org/abs/2504.04185)
• Keywords: generation, generative
• Abstract: Regularization methods using prior knowledge are essential in solving ill-posed inverse problems such as Electrical Impedance Tomography (EIT). However, designing effective regularization and integrating prior information into EIT remains challenging due to the complexity and variability of anatomical structures. In this work, we introduce SDEIT, a novel semantic-driven framework that integrates Stable Diffusion 3.5 into EIT, marking the first use of large-scale text-to-image generation models in EIT. SDEIT employs natural language prompts as semantic priors to guide the reconstruction process. By coupling an implicit neural representation (INR) network with a plug-and-play optimization scheme that leverages SD-generated images as generative priors, SDEIT improves structural consistency and recovers fine details. Importantly, this method does not rely on paired training datasets, increasing its adaptability to varied EIT scenarios. Extensive experiments on both simulated and experimental data demonstrate that SDEIT outperforms state-of-the-art techniques, offering superior accuracy and robustness. This work opens a new pathway for integrating multimodal priors into ill-posed inverse problems like EIT.
• 摘要：使用先验知识的正则化方法对于解决不适合的逆问题（例如电阻抗断层扫描（EIT））至关重要。但是，由于解剖结构的复杂性和可变性，设计有效的正则化并将以前的信息整合到EIT中仍然具有挑战性。在这项工作中，我们介绍了SDEIT，这是一种新型的语义驱动框架，将稳定的扩散3.5集成到EIT中，标志着EIT中首次使用大型文本到图像生成模型。 Sdeit采用自然语言提示作为语义先验来指导重建过程。通过将隐式神经表示（INR）网络与利用SD生成的图像作为生成先验的插件优化方案耦合，SDEIT可以提高结构一致性并恢复细节。重要的是，此方法不依赖配对的培训数据集，从而增加了其对各种环境方案的适应性。对模拟和实验数据的广泛实验表明，SDEIT的表现优于最先进的技术，提供了卓越的准确性和鲁棒性。这项工作为将多模式先验整合到诸如EIT之类的逆问题中的新途径开辟了新的途径。

Title: Interpretable Single-View 3D Gaussian Splatting using Unsupervised Hierarchical Disentangled Representation Learning

• Authors: Yuyang Zhang, Baao Xie, Hu Zhu, Qi Wang, Huanting Guo, Xin Jin, Wenjun Zeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04190
• Pdf URL: https://arxiv.org/pdf/2504.04190
• Copy Paste: [[2504.04190]] Interpretable Single-View 3D Gaussian Splatting using Unsupervised Hierarchical Disentangled Representation Learning(https://arxiv.org/abs/2504.04190)
• Keywords: generation
• Abstract: Gaussian Splatting (GS) has recently marked a significant advancement in 3D reconstruction, delivering both rapid rendering and high-quality results. However, existing 3DGS methods pose challenges in understanding underlying 3D semantics, which hinders model controllability and interpretability. To address it, we propose an interpretable single-view 3DGS framework, termed 3DisGS, to discover both coarse- and fine-grained 3D semantics via hierarchical disentangled representation learning (DRL). Specifically, the model employs a dual-branch architecture, consisting of a point cloud initialization branch and a triplane-Gaussian generation branch, to achieve coarse-grained disentanglement by separating 3D geometry and visual appearance features. Subsequently, fine-grained semantic representations within each modality are further discovered through DRL-based encoder-adapters. To our knowledge, this is the first work to achieve unsupervised interpretable 3DGS. Evaluations indicate that our model achieves 3D disentanglement while preserving high-quality and rapid reconstruction.
• 摘要：高斯脱落（GS）最近在3D重建方面取得了显着进步，既可以呈现快速渲染和高质量的结果。但是，现有的3DGS方法在理解潜在的3D语义方面构成了挑战，这阻碍了模型的可控性和解释性。为了解决这个问题，我们提出了一个可解释的单视3DGS框架，称为3DISGS，以通过层次分离的表示形式学习（DRL）来发现粗粒和细粒3D语义。具体而言，该模型采用双分支架构，由点云初始化分支和三角形高斯生成分支组成，通过分开3D几何形状和视觉外观特征来实现粗粒度的分离。随后，通过基于DRL的编码适配器进一步发现了每种模式中细粒的语义表示。据我们所知，这是实现无监督的3DGS的第一项工作。评估表明，我们的模型在保留高质量和快速重建的同时，可以实现3D脱离。

Title: TrafficLLM: Enhancing Large Language Models for Network Traffic Analysis with Generic Traffic Representation

• Authors: Tianyu Cui, Xinjie Lin, Sijia Li, Miao Chen, Qilei Yin, Qi Li, Ke Xu
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2504.04222
• Pdf URL: https://arxiv.org/pdf/2504.04222
• Copy Paste: [[2504.04222]] TrafficLLM: Enhancing Large Language Models for Network Traffic Analysis with Generic Traffic Representation(https://arxiv.org/abs/2504.04222)
• Keywords: generation
• Abstract: Machine learning (ML) powered network traffic analysis has been widely used for the purpose of threat detection. Unfortunately, their generalization across different tasks and unseen data is very limited. Large language models (LLMs), known for their strong generalization capabilities, have shown promising performance in various domains. However, their application to the traffic analysis domain is limited due to significantly different characteristics of network traffic. To address the issue, in this paper, we propose TrafficLLM, which introduces a dual-stage fine-tuning framework to learn generic traffic representation from heterogeneous raw traffic data. The framework uses traffic-domain tokenization, dual-stage tuning pipeline, and extensible adaptation to help LLM release generalization ability on dynamic traffic analysis tasks, such that it enables traffic detection and traffic generation across a wide range of downstream tasks. We evaluate TrafficLLM across 10 distinct scenarios and 229 types of traffic. TrafficLLM achieves F1-scores of 0.9875 and 0.9483, with up to 80.12% and 33.92% better performance than existing detection and generation methods. It also shows strong generalization on unseen traffic with an 18.6% performance improvement. We further evaluate TrafficLLM in real-world scenarios. The results confirm that TrafficLLM is easy to scale and achieves accurate detection performance on enterprise traffic.
• 摘要：机器学习（ML）动力网络流量分析已被广泛用于威胁检测的目的。不幸的是，它们对不同任务和看不见的数据的概括非常有限。大型语言模型（LLMS）以其强大的概括能力而闻名，在各个领域都表现出了有希望的表现。但是，由于网络流量的特征明显不同，它们在流量分析域中的应用受到限制。为了解决这个问题，在本文中，我们提出了交通，该文章引入了双阶段微调框架，以从异构的原始流量数据中学习通用的流量表示。该框架使用交通域的令牌化，双阶段调整管道和可扩展的适应性，以帮助LLM释放概括能力对动态流量分析任务，从而在广泛的下游任务中启用交通检测和流量生成。我们在10种不同的方案和229种流量中评估流量。交通lllm的F1得分为0.9875和0.9483，比现有检测和发电方法高出80.12％和33.92％。它还显示出对看不见的流量的强烈概括，其性能提高了18.6％。我们在现实世界中进一步评估交通。结果证实，交通易于扩展，并在企业流量上实现准确的检测性能。

Title: Loss Functions in Deep Learning: A Comprehensive Review

• Authors: Omar Elharrouss, Yasir Mahmood, Yassine Bechqito, Mohamed Adel Serhani, Elarbi Badidi, Jamal Riffi, Hamid Tairi
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04242
• Pdf URL: https://arxiv.org/pdf/2504.04242
• Copy Paste: [[2504.04242]] Loss Functions in Deep Learning: A Comprehensive Review(https://arxiv.org/abs/2504.04242)
• Keywords: generative
• Abstract: Loss functions are at the heart of deep learning, shaping how models learn and perform across diverse tasks. They are used to quantify the difference between predicted outputs and ground truth labels, guiding the optimization process to minimize errors. Selecting the right loss function is critical, as it directly impacts model convergence, generalization, and overall performance across various applications, from computer vision to time series forecasting. This paper presents a comprehensive review of loss functions, covering fundamental metrics like Mean Squared Error and Cross-Entropy to advanced functions such as Adversarial and Diffusion losses. We explore their mathematical foundations, impact on model training, and strategic selection for various applications, including computer vision (Discriminative and generative), tabular data prediction, and time series forecasting. For each of these categories, we discuss the most used loss functions in the recent advancements of deep learning techniques. Also, this review explore the historical evolution, computational efficiency, and ongoing challenges in loss function design, underlining the need for more adaptive and robust solutions. Emphasis is placed on complex scenarios involving multi-modal data, class imbalances, and real-world constraints. Finally, we identify key future directions, advocating for loss functions that enhance interpretability, scalability, and generalization, leading to more effective and resilient deep learning models.
• 摘要：损失功能是深度学习的核心，塑造了模型如何跨不同任务学习和执行。它们用于量化预测的输出和地面真相标签之间的差异，从而指导优化过程以最大程度地减少错误。选择正确的损失函数至关重要，因为它直接影响了从计算机视觉到时间序列预测的各种应用程序的模型收敛，概括和整体性能。本文对损失函数进行了全面的综述，涵盖了基本指标，例如均方根误差和跨透镜，例如对抗和扩散损失等高级功能。我们探讨了他们的数学基础，对模型培训的影响以及针对各种应用程序的战略选择，包括计算机视觉（歧视性和生成性），表格数据预测以及时间序列预测。对于这些类别中的每个类别，我们在深度学习技术的最新进步中讨论了最常用的损失功能。此外，这篇评论探讨了损失功能设计中的历史演变，计算效率以及持续的挑战，强调了对更适应性和强大解决方案的需求。重点放在涉及多模式数据，类失衡和现实世界约束的复杂场景上。最后，我们确定未来的关键方向，提倡增强可解释性，可伸缩性和概括的损失功能，从而导致更有效和弹性的深度学习模型。

Title: Foundation Models for Environmental Science: A Survey of Emerging Frontiers

• Authors: Runlong Yu, Shengyu Chen, Yiqun Xie, Huaxiu Yao, Jared Willard, Xiaowei Jia
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2504.04280
• Pdf URL: https://arxiv.org/pdf/2504.04280
• Copy Paste: [[2504.04280]] Foundation Models for Environmental Science: A Survey of Emerging Frontiers(https://arxiv.org/abs/2504.04280)
• Keywords: generation
• Abstract: Modeling environmental ecosystems is essential for effective resource management, sustainable development, and understanding complex ecological processes. However, traditional data-driven methods face challenges in capturing inherently complex and interconnected processes and are further constrained by limited observational data in many environmental applications. Foundation models, which leverages large-scale pre-training and universal representations of complex and heterogeneous data, offer transformative opportunities for capturing spatiotemporal dynamics and dependencies in environmental processes, and facilitate adaptation to a broad range of applications. This survey presents a comprehensive overview of foundation model applications in environmental science, highlighting advancements in common environmental use cases including forward prediction, data generation, data assimilation, downscaling, inverse modeling, model ensembling, and decision-making across domains. We also detail the process of developing these models, covering data collection, architecture design, training, tuning, and evaluation. Through discussions on these emerging methods as well as their future opportunities, we aim to promote interdisciplinary collaboration that accelerates advancements in machine learning for driving scientific discovery in addressing critical environmental challenges.
• 摘要：建模环境生态系统对于有效的资源管理，可持续发展和了解复杂的生态过程至关重要。但是，传统数据驱动的方法在捕获固有的复杂和相互联系的过程时面临挑战，并且在许多环境应用中受到有限的观察数据进一步限制。基础模型利用复杂和异构数据的大规模预训练和普遍表示，为捕获环境过程中的时空动态和依赖性提供了变革的机会，并促进适应广泛的应用。这项调查介绍了环境科学中基础模型应用的全面概述，强调了常见环境用例的进步，包括远期预测，数据生成，数据同化，降低降低，逆向建模，模型结合，模型结合和跨域的决策。我们还详细介绍了开发这些模型的过程，涵盖数据收集，体系结构设计，培训，调整和评估。通过讨论这些新兴方法及其未来机会，我们旨在促进跨学科的合作，以加速机器学习的进步，以推动科学发现在应对关键的环境挑战方面。

Title: Variational Self-Supervised Learning

• Authors: Mehmet Can Yavuz, Berrin Yanikoglu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04318
• Pdf URL: https://arxiv.org/pdf/2504.04318
• Copy Paste: [[2504.04318]] Variational Self-Supervised Learning(https://arxiv.org/abs/2504.04318)
• Keywords: generative
• Abstract: We present Variational Self-Supervised Learning (VSSL), a novel framework that combines variational inference with self-supervised learning to enable efficient, decoder-free representation learning. Unlike traditional VAEs that rely on input reconstruction via a decoder, VSSL symmetrically couples two encoders with Gaussian outputs. A momentum-updated teacher network defines a dynamic, data-dependent prior, while the student encoder produces an approximate posterior from augmented views. The reconstruction term in the ELBO is replaced with a cross-view denoising objective, preserving the analytical tractability of Gaussian KL divergence. We further introduce cosine-based formulations of KL and log-likelihood terms to enhance semantic alignment in high-dimensional latent spaces. Experiments on CIFAR-10, CIFAR-100, and ImageNet-100 show that VSSL achieves competitive or superior performance to leading self-supervised methods, including BYOL and MoCo V3. VSSL offers a scalable, probabilistically grounded approach to learning transferable representations without generative reconstruction, bridging the gap between variational modeling and modern self-supervised techniques.
• 摘要：我们提出了各种自我监督学习（VSSL），这是一个新颖的框架，将各种推论与自我监督的学习相结合，以实现高效，无解码器的表示。与通过解码器依赖输入重建的传统VAE不同，VSSL对称地伴侣两个带有高斯输出的编码器。势头更新的教师网络定义了一个动态的，数据依赖的先验，而学生编码器则从增强视图中产生了近似的后部。 ELBO中的重建项被跨视图降解目标取代，从而保留了高斯KL差异的分析性障碍。我们进一步介绍了基于余弦的KL和对数样式术语，以增强高维潜在空间中的语义一致性。 CIFAR-10，CIFAR-100和IMAGENET-100的实验表明，VSSL具有竞争性或卓越的性能，即包括BYOL和MOCO V3在内的领先自我监督方法。 VSSL提供了一种可扩展的，概率为基础的方法，用于学习可转移的表示，而无需生成重建，从而弥合了变异建模和现代自我监督技术之间的差距。

Title: MedM-VL: What Makes a Good Medical LVLM?

• Authors: Yiming Shi, Shaoshuai Yang, Xun Zhu, Haoyu Wang, Miao Li, Ji Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04323
• Pdf URL: https://arxiv.org/pdf/2504.04323
• Copy Paste: [[2504.04323]] MedM-VL: What Makes a Good Medical LVLM?(https://arxiv.org/abs/2504.04323)
• Keywords: generation
• Abstract: Medical image analysis is a fundamental component. As deep learning progresses, the focus has shifted from single-task applications, such as classification and segmentation, to more complex multimodal tasks, including medical visual question answering and report generation. Traditional shallow and task-specific models are increasingly limited in addressing the complexity and scalability required in clinical practice. The emergence of large language models (LLMs) has driven the development of medical Large Vision-Language Models (LVLMs), offering a unified solution for diverse vision-language tasks. In this study, we investigate various architectural designs for medical LVLMs based on the widely adopted LLaVA framework, which follows an encoder-connector-LLM paradigm. We construct two distinct models targeting 2D and 3D modalities, respectively. These models are designed to support both general-purpose medical tasks and domain-specific fine-tuning, thereby serving as effective foundation models. To facilitate reproducibility and further research, we develop a modular and extensible codebase, MedM-VL, and release two LVLM variants: MedM-VL-2D for 2D medical image analysis and MedM-VL-CT-Chest for 3D CT-based applications. The code and models are available at: this https URL
• 摘要：医学图像分析是基本组成部分。随着深度学习的进展，重点已从单项任务应用程序（例如分类和细分）转变为更复杂的多模式任务，包括医学视觉问题答案和报告生成。传统的浅层和特定于任务模型在解决临床实践中所需的复杂性和可扩展性方面越来越有限。大型语言模型（LLM）的出现推动了医学大型视觉模型（LVLM）的发展，为各种视觉语言任务提供了统一的解决方案。在这项研究中，我们根据广泛采用的LLAVA框架研究了医疗LVLMS的各种建筑设计，该框架遵循编码器连接-llm范式。我们分别构建了针对2D和3D模式的两个不同的模型。这些模型旨在支持通用医疗任务和特定领域的微调，从而作为有效的基础模型。为了促进可重复性和进一步的研究，我们开发了一个模块化且可扩展的代码库MEDM-VL，并释放两个LVLM变体：用于2D医学图像分析的MEDM-VL-2D，用于3D CT的应用。代码和型号可用：此HTTPS URL

Title: AnomalyHybrid: A Domain-agnostic Generative Framework for General Anomaly Detection

• Authors: Ying Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04340
• Pdf URL: https://arxiv.org/pdf/2504.04340
• Copy Paste: [[2504.04340]] AnomalyHybrid: A Domain-agnostic Generative Framework for General Anomaly Detection(https://arxiv.org/abs/2504.04340)
• Keywords: generation, generative
• Abstract: Anomaly generation is an effective way to mitigate data scarcity for anomaly detection task. Most existing works shine at industrial anomaly generation with multiple specialists or large generative models, rarely generalizing to anomalies in other applications. In this paper, we present AnomalyHybrid, a domain-agnostic framework designed to generate authentic and diverse anomalies simply by combining the reference and target images. AnomalyHybrid is a Generative Adversarial Network(GAN)-based framework having two decoders that integrate the appearance of reference image into the depth and edge structures of target image respectively. With the help of depth decoders, AnomalyHybrid achieves authentic generation especially for the anomalies with depth values changing, such a s protrusion and dent. More, it relaxes the fine granularity structural control of the edge decoder and brings more diversity. Without using annotations, AnomalyHybrid is easily trained with sets of color, depth and edge of same images having different augmentations. Extensive experiments carried on HeliconiusButterfly, MVTecAD and MVTec3D datasets demonstrate that AnomalyHybrid surpasses the GAN-based state-of-the-art on anomaly generation and its downstream anomaly classification, detection and segmentation tasks. On MVTecAD dataset, AnomalyHybrid achieves 2.06/0.32 IS/LPIPS for anomaly generation, 52.6 Acc for anomaly classification with ResNet34, 97.3/72.9 AP for image/pixel-level anomaly detection with a simple UNet.
• 摘要：异常生成是减轻数据稀缺性检测任务的有效方法。大多数现有的作品在工业异常生成中发挥了多种专家或大型生成模型的发光，在其他应用程序中很少推广到异常。在本文中，我们提出了一种域 - 不合稳定框架，旨在仅通过组合参考图像和目标图像来产生真实和多样的异常。 Anomyhybrid是一种基于生成的对抗网络（GAN）的框架，其两个解码器分别将参考图像的外观整合到目标图像的深度和边缘结构中。在深度解码器的帮助下，异常的生成尤其是对于变化深度值的异常，例如s突出和凹痕。更重要的是，它放松了边缘解码器的精细颗粒状结构控制，并带来了更多的多样性。无需使用注释，很容易对具有不同增强的相同图像的颜色，深度和边缘进行训练。进行HeliconiusbutterFly，MVTECAD和MVTEC3D数据集进行的广泛实验表明，异常杂交超过基于GAN的关于异常生成及其下游异常分类，检测和分割任务的最先进。在MVTECAD数据集上，异常生成的Anomyhybrid可实现2.06/0.32 IS/LPIPS，使用RESNET34、97.3/72.9 AP进行异常分类为52.6 ACC，用于图像/像素级别的AP，可使用简单的UNET检测。

Title: Human-Level Competitive Pokémon via Scalable Offline Reinforcement Learning with Transformers

• Authors: Jake Grigsby, Yuqi Xie, Justin Sasek, Steven Zheng, Yuke Zhu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.04395
• Pdf URL: https://arxiv.org/pdf/2504.04395
• Copy Paste: [[2504.04395]] Human-Level Competitive Pokémon via Scalable Offline Reinforcement Learning with Transformers(https://arxiv.org/abs/2504.04395)
• Keywords: generation
• Abstract: Competitive Pokémon Singles (CPS) is a popular strategy game where players learn to exploit their opponent based on imperfect information in battles that can last more than one hundred stochastic turns. AI research in CPS has been led by heuristic tree search and online self-play, but the game may also create a platform to study adaptive policies trained offline on large datasets. We develop a pipeline to reconstruct the first-person perspective of an agent from logs saved from the third-person perspective of a spectator, thereby unlocking a dataset of real human battles spanning more than a decade that grows larger every day. This dataset enables a black-box approach where we train large sequence models to adapt to their opponent based solely on their input trajectory while selecting moves without explicit search of any kind. We study a progression from imitation learning to offline RL and offline fine-tuning on self-play data in the hardcore competitive setting of Pokémon's four oldest (and most partially observed) game generations. The resulting agents outperform a recent LLM Agent approach and a strong heuristic search engine. While playing anonymously in online battles against humans, our best agents climb to rankings inside the top 10% of active players.
• 摘要：竞争性的神奇宝贝单曲（CPS）是一款流行的战略游戏，玩家会根据可以持续超过一百个随机转弯的战斗中的不完美信息来利用对手。 CPS中的AI研究是由启发式树搜索和在线自我玩法领导的，但是该游戏也可能创建一个平台来研究在大型数据集上脱机训练的自适应政策。我们开发了一条管道，以从观众的第三人称角度保存的日志中重建代理商的第一人称视角，从而解开了跨越十多年来每天增长的真实人类战斗数据集。该数据集启用了一种黑框方法，我们在其中训练大型序列模型以适应其对手的输入轨迹，同时选择移动而无需明确搜索任何类型的动作。我们研究从模仿学习到离线RL和在神奇宝贝的四个最古老（也是部分观察到的）游戏的硬核竞争环境中的自我播放数据中进行微调的进步。最终的代理商的表现优于最近的LLM代理方法和强大的启发式搜索引擎。在与人类的在线战斗中匿名打球时，我们最好的代理商爬上了排名前10％的活跃球员的排名。

Title: UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding

• Authors: Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, Yu-Gang Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.04423
• Pdf URL: https://arxiv.org/pdf/2504.04423
• Copy Paste: [[2504.04423]] UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding(https://arxiv.org/abs/2504.04423)
• Keywords: generation
• Abstract: We introduce UniToken, an auto-regressive generation model that encodes visual inputs through a combination of discrete and continuous representations, enabling seamless integration of unified visual understanding and image generation tasks. Unlike previous approaches that rely on unilateral visual representations, our unified visual encoding framework captures both high-level semantics and low-level details, delivering multidimensional information that empowers heterogeneous tasks to selectively assimilate domain-specific knowledge based on their inherent characteristics. Through in-depth experiments, we uncover key principles for developing a unified model capable of both visual understanding and image generation. Extensive evaluations across a diverse range of prominent benchmarks demonstrate that UniToken achieves state-of-the-art performance, surpassing existing approaches. These results establish UniToken as a robust foundation for future research in this domain. The code and models are available at this https URL.
• 摘要：我们介绍了Unitoken，这是一种自动回归生成模型，该模型通过离散和连续表示的组合来编码视觉输入，从而使统一的视觉理解和图像生成任务无缝集成。与以前依赖单方面视觉表示的方法不同，我们统一的视觉编码框架既捕获了高级语义和低级细节，又提供了多维信息，从而使异质任务具有基于其内在特征的选择性同化域特有的知识。通过深入实验，我们发现了开发能够视觉理解和图像产生的统一模型的关键原理。对各种突出基准的广泛评估表明，Unitoken实现了最先进的性能，超过了现有方法。这些结果确立了unitoken作为该领域未来研究的强大基础。代码和模型可在此HTTPS URL上找到。

Title: FluentLip: A Phonemes-Based Two-stage Approach for Audio-Driven Lip Synthesis with Optical Flow Consistency

• Authors: Shiyan Liu, Rui Qu, Yan Jin
• Subjects: cs.CV, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2504.04427
• Pdf URL: https://arxiv.org/pdf/2504.04427
• Copy Paste: [[2504.04427]] FluentLip: A Phonemes-Based Two-stage Approach for Audio-Driven Lip Synthesis with Optical Flow Consistency(https://arxiv.org/abs/2504.04427)
• Keywords: generative
• Abstract: Generating consecutive images of lip movements that align with a given speech in audio-driven lip synthesis is a challenging task. While previous studies have made strides in synchronization and visual quality, lip intelligibility and video fluency remain persistent challenges. This work proposes FluentLip, a two-stage approach for audio-driven lip synthesis, incorporating three featured strategies. To improve lip synchronization and intelligibility, we integrate a phoneme extractor and encoder to generate a fusion of audio and phoneme information for multimodal learning. Additionally, we employ optical flow consistency loss to ensure natural transitions between image frames. Furthermore, we incorporate a diffusion chain during the training of Generative Adversarial Networks (GANs) to improve both stability and efficiency. We evaluate our proposed FluentLip through extensive experiments, comparing it with five state-of-the-art (SOTA) approaches across five metrics, including a proposed metric called Phoneme Error Rate (PER) that evaluates lip pose intelligibility and video fluency. The experimental results demonstrate that our FluentLip approach is highly competitive, achieving significant improvements in smoothness and naturalness. In particular, it outperforms these SOTA approaches by approximately $\textbf{16.3%}$ in Fréchet Inception Distance (FID) and $\textbf{35.2%}$ in PER.
• 摘要：在音频驱动的嘴唇合成中与给定语音保持一致的唇部运动的连续图像是一项艰巨的任务。尽管以前的研究在同步和视觉质量方面取得了进步，但唇彩和视频流利性仍然是持续的挑战。这项工作提出了Fluentlip，这是一种两阶段的方法，用于音频驱动的唇部合成，并结合了三种特色策略。为了提高唇部同步和清晰度，我们集成了音素提取器和编码器，以生成音频和音素信息的融合以进行多模式学习。此外，我们采用光流量损失，以确保图像框架之间的自然过渡。此外，我们在训练生成对抗网络（GAN）的过程中结合了一个扩散链，以提高稳定性和效率。我们通过广泛的实验评估了我们提出的Fluentlip，将其与五种五个指标的五种最先进的方法（SOTA）进行了比较，其中包括提出的称为音素错误率（PER）的指标，该指标评估了唇部的清晰度和视频流利性。实验结果表明，我们的Fluentlip方法具有很高的竞争力，可以显着改善平稳性和自然性。特别是，它在fréchetInception距离（FID）和$ \ textbf {35.2％} $中的fréchetinception inception inception inception inception Inception inception inception inception inception inception inception Inception Inception of textbf {16.3％} $均优于这些SOTA方法。

Title: PRISM: Probabilistic Representation for Integrated Shape Modeling and Generation

• Authors: Lei Cheng, Mahdi Saleh, Qing Cheng, Lu Sang, Hongli Xu, Daniel Cremers, Federico Tombari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04454
• Pdf URL: https://arxiv.org/pdf/2504.04454
• Copy Paste: [[2504.04454]] PRISM: Probabilistic Representation for Integrated Shape Modeling and Generation(https://arxiv.org/abs/2504.04454)
• Keywords: generation, generative
• Abstract: Despite the advancements in 3D full-shape generation, accurately modeling complex geometries and semantics of shape parts remains a significant challenge, particularly for shapes with varying numbers of parts. Current methods struggle to effectively integrate the contextual and structural information of 3D shapes into their generative processes. We address these limitations with PRISM, a novel compositional approach for 3D shape generation that integrates categorical diffusion models with Statistical Shape Models (SSM) and Gaussian Mixture Models (GMM). Our method employs compositional SSMs to capture part-level geometric variations and uses GMM to represent part semantics in a continuous space. This integration enables both high fidelity and diversity in generated shapes while preserving structural coherence. Through extensive experiments on shape generation and manipulation tasks, we demonstrate that our approach significantly outperforms previous methods in both quality and controllability of part-level operations. Our code will be made publicly available.
• 摘要：尽管在3D全形产生中取得了进步，但准确地对复杂的几何形状和形状零件的语义进行建模仍然是一个重大挑战，尤其是对于零件数量变化的形状而言。当前的方法很难有效地将3D形状的上下文和结构信息整合到其生成过程中。我们用Prism（一种新型的3D形状生成组成方法）来解决这些局限性，该方法将分类扩散模型与统计形状模型（SSM）和高斯混合物模型（GMM）集成在一起。我们的方法采用组成SSM来捕获零件级的几何变化，并使用GMM代表连续空间中的零件语义。这种整合使产生形状的高保真度和多样性在保持结构连贯性的同时。通过对形状产生和操纵任务的广泛实验，我们证明我们的方法在零件级操作的质量和可控性方面都显着优于以前的方法。我们的代码将公开可用。

Title: Attributed Synthetic Data Generation for Zero-shot Domain-specific Image Classification

• Authors: Shijian Wang, Linxin Song, Ryotaro Shimizu, Masayuki Goto, Hanqian Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04510
• Pdf URL: https://arxiv.org/pdf/2504.04510
• Copy Paste: [[2504.04510]] Attributed Synthetic Data Generation for Zero-shot Domain-specific Image Classification(https://arxiv.org/abs/2504.04510)
• Keywords: generation
• Abstract: Zero-shot domain-specific image classification is challenging in classifying real images without ground-truth in-domain training examples. Recent research involved knowledge from texts with a text-to-image model to generate in-domain training images in zero-shot scenarios. However, existing methods heavily rely on simple prompt strategies, limiting the diversity of synthetic training images, thus leading to inferior performance compared to real images. In this paper, we propose AttrSyn, which leverages large language models to generate attributed prompts. These prompts allow for the generation of more diverse attributed synthetic images. Experiments for zero-shot domain-specific image classification on two fine-grained datasets show that training with synthetic images generated by AttrSyn significantly outperforms CLIP's zero-shot classification under most situations and consistently surpasses simple prompt strategies.
• 摘要：零射击域特异性图像分类在没有地面真相内训练示例的情况下对真实图像进行分类方面具有挑战性。最近的研究涉及具有文本对图像模型的文本的知识，以在零照片方案中生成内域培训图像。但是，现有方法在很大程度上依赖于简单的及时策略，从而限制了合成训练图像的多样性，因此与真实图像相比导致性能较低。在本文中，我们提出了ATTRYN，该Attryn利用大型语言模型生成属性提示。这些提示允许产生更多样化的合成图像。在两个细粒数据集上进行零击域特异性图像分类的实验表明，在大多数情况下，Attryn生成的合成图像的训练显着超过了Clip的零摄影分类，并且一致地超过了简单的及时策略。

Title: Your Image Generator Is Your New Private Dataset

• Authors: Nicolo Resmini, Eugenio Lomurno, Cristian Sbrolli, Matteo Matteucci
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.04582
• Pdf URL: https://arxiv.org/pdf/2504.04582
• Copy Paste: [[2504.04582]] Your Image Generator Is Your New Private Dataset(https://arxiv.org/abs/2504.04582)
• Keywords: generation, generative
• Abstract: Generative diffusion models have emerged as powerful tools to synthetically produce training data, offering potential solutions to data scarcity and reducing labelling costs for downstream supervised deep learning applications. However, effectively leveraging text-conditioned image generation for building classifier training sets requires addressing key issues: constructing informative textual prompts, adapting generative models to specific domains, and ensuring robust performance. This paper proposes the Text-Conditioned Knowledge Recycling (TCKR) pipeline to tackle these challenges. TCKR combines dynamic image captioning, parameter-efficient diffusion model fine-tuning, and Generative Knowledge Distillation techniques to create synthetic datasets tailored for image classification. The pipeline is rigorously evaluated on ten diverse image classification benchmarks. The results demonstrate that models trained solely on TCKR-generated data achieve classification accuracies on par with (and in several cases exceeding) models trained on real images. Furthermore, the evaluation reveals that these synthetic-data-trained models exhibit substantially enhanced privacy characteristics: their vulnerability to Membership Inference Attacks is significantly reduced, with the membership inference AUC lowered by 5.49 points on average compared to using real training data, demonstrating a substantial improvement in the performance-privacy trade-off. These findings indicate that high-fidelity synthetic data can effectively replace real data for training classifiers, yielding strong performance whilst simultaneously providing improved privacy protection as a valuable emergent property. The code and trained models are available in the accompanying open-source repository.
• 摘要：生成扩散模型已成为合成生产培训数据的强大工具，为数据稀缺提供了潜在的解决方案，并降低了下游监督深度学习应用程序的标签成本。但是，有效利用文本条件的图像生成为建筑物分类器培训集需要解决关键问题：构建信息丰富的文本提示，将生成模型调整为特定域，并确保稳健的性能。本文提出了文本条件知识回收（TCKR）管道，以应对这些挑战。 TCKR结合了动态图像字幕，参数效率扩散模型微调和生成知识蒸馏技术，以创建用于图像分类的合成数据集。该管道对十个不同的图像分类基准进行了严格评估。结果表明，仅在TCKR生成的数据上训练的模型与在真实图像上训练的模型相当（并且在几种情况下）实现了分类精度。此外，评估表明，这些合成数据训练的模型具有显着增强的隐私特征：它们对会员推理攻击的脆弱性大大降低，与使用实际培训数据相比，成员推理平均降低了5.49点，这表明了绩效验证竞争优先折衷的实质性改善。这些发现表明，高保真综合数据可以有效地替换培训分类器的真实数据，从而产生强大的性能，同时提供改进的隐私保护作为宝贵的新兴属性。随附的开源存储库中可用代码和训练有素的模型。

Title: M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning in Large Vision-Language Models

• Authors: Yanshu Li, Hongyang He, Yi Cao, Qisen Cheng, Xiang Fu, Ruixiang Tang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.04633
• Pdf URL: https://arxiv.org/pdf/2504.04633
• Copy Paste: [[2504.04633]] M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning in Large Vision-Language Models(https://arxiv.org/abs/2504.04633)
• Keywords: generation
• Abstract: Multimodal in-context learning (ICL) is a vital capability for Large Vision-Language Models (LVLMs), allowing task adaptation via contextual prompts without parameter retraining. However, its application is hindered by the token-intensive nature of inputs and the high complexity of cross-modal few-shot learning, which limits the expressive power of representation methods. To tackle these challenges, we propose \textbf{M2IV}, a method that substitutes explicit demonstrations with learnable \textbf{I}n-context \textbf{V}ectors directly integrated into LVLMs. By exploiting the complementary strengths of multi-head attention (\textbf{M}HA) and multi-layer perceptrons (\textbf{M}LP), M2IV achieves robust cross-modal fidelity and fine-grained semantic distillation through training. This significantly enhances performance across diverse LVLMs and tasks and scales efficiently to many-shot scenarios, bypassing the context window limitations. We also introduce \textbf{VLibrary}, a repository for storing and retrieving M2IV, enabling flexible LVLM steering for tasks like cross-modal alignment, customized generation and safety improvement. Experiments across seven benchmarks and three LVLMs show that M2IV surpasses Vanilla ICL and prior representation engineering approaches, with an average accuracy gain of \textbf{3.74\%} over ICL with the same shot count, alongside substantial efficiency advantages.
• 摘要：多模式内下文学习（ICL）是大型视觉模型（LVLM）的重要能力，可以通过上下文提示进行任务适应而无需参数重新训练。然而，它的应用受到输入的令牌密集型性质和跨模式少数学习的高复杂性的阻碍，这限制了表示方法的表现力。为了应对这些挑战，我们提出\ textbf {m2iv}，一种用可学习的\ textbf {i} n-context \ textbf {v} eTors直接集成到lvlms中的明确演示的方法。通过利用多头注意的互补优势（\ textbf {m} ha）和多层感知器（\ textbf {m} lp），M2IV可以通过训练实现强大的交叉模式忠诚和精细的语义蒸馏。这大大提高了各种LVLM和任务的性能，并有效地缩放到了许多拍摄方案，从而绕开了上下文窗口限制。我们还介绍了\ textbf {vlibrary}，这是一个存储和检索M2IV的存储库，为跨模式对齐，自定义生成和安全性改进等任务启用灵活的LVLM转向。七个基准和三个LVLM的实验表明，M2IV超过了香草ICL和先前的代表工程方法，与ICL相同的平均准确性增长率为\ textbf {3.74 \％}，而相同的射击计数则具有相同的效率优势。

Title: ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedback

• Authors: Tasnia Rahman, Sathish A. P. Kumar, Sumit Jha, Arvind Ramanathan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.04657
• Pdf URL: https://arxiv.org/pdf/2504.04657
• Copy Paste: [[2504.04657]] ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedback(https://arxiv.org/abs/2504.04657)
• Keywords: generation
• Abstract: Automated Program Repair tools are developed for generating feedback and suggesting a repair method for erroneous code. State of the art (SOTA) code repair methods rely on data-driven approaches and often fail to deliver solution for complicated programming questions. To interpret the natural language of unprecedented programming problems, using Large Language Models (LLMs) for code-feedback generation is crucial. LLMs generate more comprehensible feedback than compiler-generated error messages, and Reinforcement Learning with Human Feedback (RLHF) further enhances quality by integrating human-in-the-loop which helps novice students to lean programming from scratch interactively. We are applying RLHF fine-tuning technique for an expected Socratic response such as a question with hint to solve the programming issue. We are proposing code feedback generation tool by fine-tuning LLM with RLHF, Automated Code Evaluation with RLHF (ACE-RLHF), combining two open-source LLM models with two different SOTA optimization techniques. The quality of feedback is evaluated on two benchmark datasets containing basic and competition-level programming questions where the later is proposed by us. We achieved 2-5% higher accuracy than RL-free SOTA techniques using Llama-3-7B-Proximal-policy optimization in automated evaluation and similar or slightly higher accuracy compared to reward model-free RL with AI Feedback (RLAIF). We achieved almost 40% higher accuracy with GPT-3.5 Best-of-n optimization while performing manual evaluation.
• 摘要：自动化程序维修工具是为了生成反馈的开发，并建议使用错误代码的维修方法。最新技术（SOTA）代码维修方法依赖于数据驱动的方法，并且通常无法为复杂的编程问题提供解决方案。为了解释前所未有的编程问题的自然语言，使用大型语言模型（LLMS）进行代码反馈生成至关重要。 LLM比编译器生成的错误消息产生更全面的反馈，而通过人类反馈（RLHF）进行加强学习，通过整合人类在循环中，进一步提高了质量，这可以帮助新手学生从划线上进行交互倾斜。我们正在将RLHF微调技术应用于预期的苏格拉底响应，例如提示解决编程问题的问题。我们正在通过使用RLHF进行微调LLM，使用RLHF（ACE-RLHF）进行自动代码评估来提出代码反馈生成工具，将两个开源的LLM模型与两种不同的SOTA优化技术相结合。在两个基准数据集上评估了反馈的质量，其中包含基本和竞争级的编程问题，我们提出了以后提出的问题。与使用AI反馈（RLAIF）相比，使用Llama-3-7B-Proximal-Policy优化优化的Llama-3-7b-Proximal-Policy优化，我们的精度比无RL的SOTA技术高2-5％。在执行手动评估时，我们通过GPT-3.5优化获得了近40％的精度。

Title: DeclutterNeRF: Generative-Free 3D Scene Recovery for Occlusion Removal

• Authors: Wanzhou Liu, Zhexiao Xiong, Xinyu Li, Nathan Jacobs
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04679
• Pdf URL: https://arxiv.org/pdf/2504.04679
• Copy Paste: [[2504.04679]] DeclutterNeRF: Generative-Free 3D Scene Recovery for Occlusion Removal(https://arxiv.org/abs/2504.04679)
• Keywords: generative
• Abstract: Recent novel view synthesis (NVS) techniques, including Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have greatly advanced 3D scene reconstruction with high-quality rendering and realistic detail recovery. Effectively removing occlusions while preserving scene details can further enhance the robustness and applicability of these techniques. However, existing approaches for object and occlusion removal predominantly rely on generative priors, which, despite filling the resulting holes, introduce new artifacts and blurriness. Moreover, existing benchmark datasets for evaluating occlusion removal methods lack realistic complexity and viewpoint variations. To address these issues, we introduce DeclutterSet, a novel dataset featuring diverse scenes with pronounced occlusions distributed across foreground, midground, and background, exhibiting substantial relative motion across viewpoints. We further introduce DeclutterNeRF, an occlusion removal method free from generative priors. DeclutterNeRF introduces joint multi-view optimization of learnable camera parameters, occlusion annealing regularization, and employs an explainable stochastic structural similarity loss, ensuring high-quality, artifact-free reconstructions from incomplete images. Experiments demonstrate that DeclutterNeRF significantly outperforms state-of-the-art methods on our proposed DeclutterSet, establishing a strong baseline for future research.
• 摘要：最近的新型视图合成（NVS）技术，包括神经辐射场（NERF）和3D高斯分裂（3DGS），具有高级3D场景重建，具有高质量的渲染和现实的细节恢复。有效地消除阻塞的同时保存场景细节可以进一步增强这些技术的鲁棒性和适用性。但是，现有的对象和遮挡去除的方法主要依赖生成的先验，尽管填充了所产生的孔，但仍引入了新的人工制品和模糊。此外，现有用于评估闭塞方法的基准数据集缺乏现实的复杂性和观点变化。为了解决这些问题，我们介绍了Declutterset，这是一个新颖的数据集，其中包含各种场景，这些场景具有明显的遮挡，分布在前景，中间和背景上，跨越了跨视点的相对运动。我们进一步介绍了Declutternerf，这是一种无生成先验的遮挡方法。 Dectutternerf引入了可学习的摄像机参数的联合多视图优化，遮挡放电正规化，并采用了可解释的随机结构相似性损失，从而确保了来自不完整图像的高质量，无伪影的无伪影。实验表明，Declutternerf在我们拟议的Declutterset上明显胜过最先进的方法，为将来的研究建立了强大的基线。

Title: Bridging Knowledge Gap Between Image Inpainting and Large-Area Visible Watermark Removal

• Authors: Yicheng Leng, Chaowei Fang, Junye Chen, Yixiang Fang, Sheng Li, Guanbin Li
• Subjects: cs.CV, cs.AI, cs.MM, eess.IV
• Abstract URL: https://arxiv.org/abs/2504.04687
• Pdf URL: https://arxiv.org/pdf/2504.04687
• Copy Paste: [[2504.04687]] Bridging Knowledge Gap Between Image Inpainting and Large-Area Visible Watermark Removal(https://arxiv.org/abs/2504.04687)
• Keywords: restoration
• Abstract: Visible watermark removal which involves watermark cleaning and background content restoration is pivotal to evaluate the resilience of watermarks. Existing deep neural network (DNN)-based models still struggle with large-area watermarks and are overly dependent on the quality of watermark mask prediction. To overcome these challenges, we introduce a novel feature adapting framework that leverages the representation modeling capacity of a pre-trained image inpainting model. Our approach bridges the knowledge gap between image inpainting and watermark removal by fusing information of the residual background content beneath watermarks into the inpainting backbone model. We establish a dual-branch system to capture and embed features from the residual background content, which are merged into intermediate features of the inpainting backbone model via gated feature fusion modules. Moreover, for relieving the dependence on high-quality watermark masks, we introduce a new training paradigm by utilizing coarse watermark masks to guide the inference process. This contributes to a visible image removal model which is insensitive to the quality of watermark mask during testing. Extensive experiments on both a large-scale synthesized dataset and a real-world dataset demonstrate that our approach significantly outperforms existing state-of-the-art methods. The source code is available in the supplementary materials.
• 摘要：可见的水印去除涉及水印清洁和背景内容修复是评估水印的弹性的关键。现有的深神经网络（DNN）的模型仍在大区域水印上挣扎，并且过于依赖水印掩码预测的质量。为了克服这些挑战，我们介绍了一个新型的功能调整框架，该框架利用了预训练的图像授予模型的表示形式建模能力。我们的方法通过将水印下的残留背景内容融合到嵌入式骨干模型中，从而弥合了图像覆盖和去除水印之间的知识差距。我们建立了一个双分支系统，以捕获和嵌入剩余背景内容的特征，这些功能通过封闭的特征融合模块合并为嵌入式骨干模型的中间特征。此外，为了减轻对高质量水印面膜的依赖，我们通过利用粗糙的水印面膜来指导推理过程，引入了新的训练范式。这有助于可见的图像去除模型，该模型对测试过程中水印面膜的质量不敏感。大规模合成数据集和现实世界数据集的广泛实验表明，我们的方法显着胜过现有的最新方法。源代码在补充材料中可用。

Title: SapiensID: Foundation for Human Recognition

• Authors: Minchul Kim, Dingqiang Ye, Yiyang Su, Feng Liu, Xiaoming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04708
• Pdf URL: https://arxiv.org/pdf/2504.04708
• Copy Paste: [[2504.04708]] SapiensID: Foundation for Human Recognition(https://arxiv.org/abs/2504.04708)
• Keywords: generation
• Abstract: Existing human recognition systems often rely on separate, specialized models for face and body analysis, limiting their effectiveness in real-world scenarios where pose, visibility, and context vary widely. This paper introduces SapiensID, a unified model that bridges this gap, achieving robust performance across diverse settings. SapiensID introduces (i) Retina Patch (RP), a dynamic patch generation scheme that adapts to subject scale and ensures consistent tokenization of regions of interest, (ii) a masked recognition model (MRM) that learns from variable token length, and (iii) Semantic Attention Head (SAH), an module that learns pose-invariant representations by pooling features around key body parts. To facilitate training, we introduce WebBody4M, a large-scale dataset capturing diverse poses and scale variations. Extensive experiments demonstrate that SapiensID achieves state-of-the-art results on various body ReID benchmarks, outperforming specialized models in both short-term and long-term scenarios while remaining competitive with dedicated face recognition systems. Furthermore, SapiensID establishes a strong baseline for the newly introduced challenge of Cross Pose-Scale ReID, demonstrating its ability to generalize to complex, real-world conditions.
• 摘要：现有的人类识别系统通常依靠单独的专业模型进行面部和身体分析，从而限制了它们在现实情况下的有效性，在现实情况下，姿势，可见性和环境差异很大。本文介绍了Sapiensid，这是一个统一的模型，它弥合了这一差距，从而在各种环境中实现了强劲的性能。 sapiensid引入（i）视网膜斑点（RP），这是一种动态的补丁生成方案，可适应主题尺度并确保感兴趣区域的持续象征化，（ii）掩盖的识别模型（MRM）从可变的令牌长度中学习，从可变的令牌长度中学习，并且（iii）语义注意力（iii）的语义注意力（SAH），一个模块化，以构成体积的体现，并具有pose-Invariant的特征。为了促进培训，我们介绍了WebBody4M，这是一个大规模的数据集，可捕获多样化的姿势和规模变化。广泛的实验表明，Sapiensid在各种身体的基准测试基准上取得了最新的结果，在短期和长期情景下都优于专业模型，同时与专用的面部识别系统保持竞争力。此外，Sapiensid为新引入的跨姿势尺度REID挑战建立了强大的基准，这表明了其将其推广到复杂的现实世界条件的能力。

Title: AnyArtisticGlyph: Multilingual Controllable Artistic Glyph Generation

• Authors: Xiongbo Lu, Yaxiong Chen, Shengwu Xiong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04743
• Pdf URL: https://arxiv.org/pdf/2504.04743
• Copy Paste: [[2504.04743]] AnyArtisticGlyph: Multilingual Controllable Artistic Glyph Generation(https://arxiv.org/abs/2504.04743)
• Keywords: generation
• Abstract: Artistic Glyph Image Generation (AGIG) differs from current creativity-focused generation models by offering finely controllable deterministic generation. It transfers the style of a reference image to a source while preserving its content. Although advanced and promising, current methods may reveal flaws when scrutinizing synthesized image details, often producing blurred or incorrect textures, posing a significant challenge. Hence, we introduce AnyArtisticGlyph, a diffusion-based, multilingual controllable artistic glyph generation model. It includes a font fusion and embedding module, which generates latent features for detailed structure creation, and a vision-text fusion and embedding module that uses the CLIP model to encode references and blends them with transformation caption embeddings for seamless global image generation. Moreover, we incorporate a coarse-grained feature-level loss to enhance generation accuracy. Experiments show that it produces natural, detailed artistic glyph images with state-of-the-art performance. Our project will be open-sourced on this https URL to advance text generation technology.
• 摘要：艺术字形图像产生（AGIG）通过提供可控制的确定性生成而与当前以创造力为中心的生成模型不同。它在保留其内容的同时将参考图像的样式转移到源。尽管高级且有希望，但是当仔细检查合成图像细节时，当前的方法通常会产生模糊或不正确的纹理，从而带来重大挑战。因此，我们介绍了Anyartisticglyph，这是一种基于扩散的多语言可控艺术雕文生成模型。它包括一个字体融合和嵌入模块，该模块生成了用于详细结构创建的潜在特征，以及使用剪辑模型编码引用并将其与转换字幕嵌入无缝全局图像生成的转换字幕嵌入的视觉文本融合和嵌入模块。此外，我们结合了粗粒的特征级损失，以提高发电精度。实验表明，它会产生具有最先进的性能的天然，详细的艺术镜像图像。我们的项目将在此HTTPS URL上开源，以推进文本生成技术。

Title: CADCrafter: Generating Computer-Aided Design Models from Unconstrained Images

• Authors: Cheng Chen, Jiacheng Wei, Tianrun Chen, Chi Zhang, Xiaofeng Yang, Shangzhan Zhang, Bingchen Yang, Chuan-Sheng Foo, Guosheng Lin, Qixing Huang, Fayao Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04753
• Pdf URL: https://arxiv.org/pdf/2504.04753
• Copy Paste: [[2504.04753]] CADCrafter: Generating Computer-Aided Design Models from Unconstrained Images(https://arxiv.org/abs/2504.04753)
• Keywords: generation
• Abstract: Creating CAD digital twins from the physical world is crucial for manufacturing, design, and simulation. However, current methods typically rely on costly 3D scanning with labor-intensive post-processing. To provide a user-friendly design process, we explore the problem of reverse engineering from unconstrained real-world CAD images that can be easily captured by users of all experiences. However, the scarcity of real-world CAD data poses challenges in directly training such models. To tackle these challenges, we propose CADCrafter, an image-to-parametric CAD model generation framework that trains solely on synthetic textureless CAD data while testing on real-world images. To bridge the significant representation disparity between images and parametric CAD models, we introduce a geometry encoder to accurately capture diverse geometric features. Moreover, the texture-invariant properties of the geometric features can also facilitate the generalization to real-world scenarios. Since compiling CAD parameter sequences into explicit CAD models is a non-differentiable process, the network training inherently lacks explicit geometric supervision. To impose geometric validity constraints, we employ direct preference optimization (DPO) to fine-tune our model with the automatic code checker feedback on CAD sequence quality. Furthermore, we collected a real-world dataset, comprised of multi-view images and corresponding CAD command sequence pairs, to evaluate our method. Experimental results demonstrate that our approach can robustly handle real unconstrained CAD images, and even generalize to unseen general objects.
• 摘要：从物理世界中创建CAD数字双胞胎对于制造，设计和模拟至关重要。但是，当前的方法通常依赖于昂贵的3D扫描，并通过劳动密集型后处理。为了提供一个用户友好的设计过程，我们探讨了从无约束的现实世界CAD图像进行反向工程的问题，这些图像很容易被所有体验的用户捕获。但是，现实世界中CAD数据的稀缺性在直接培训此类模型时构成了挑战。为了应对这些挑战，我们提出了Cadcrafter，这是一种图像到参数CAD模型生成框架，该框架仅根据合成的无纹理CAD数据进行训练，同时测试现实世界图像。为了弥合图像和参数CAD模型之间的显着表示差异，我们引入了一个几何编码器，以准确捕获多种几何特征。此外，几何特征的纹理不变特性也可以促进对现实情况的概括。由于将CAD参数序列汇编为显式CAD模型是一个非不同的过程，因此网络训练本质上缺乏明确的几何监督。为了施加几何有效性限制，我们采用直接偏好优化（DPO）来通过自动代码检查器的CAD序列质量反馈来微调我们的模型。此外，我们收集了一个由多视图图像和相应的CAD命令序列对组成的真实数据集，以评估我们的方法。实验结果表明，我们的方法可以鲁棒地处理实际不受限制的CAD图像，甚至可以推广到看不见的一般对象。

Title: Continuous Locomotive Crowd Behavior Generation

• Authors: Inhwan Bae, Junoh Lee, Hae-Gon Jeon
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2504.04756
• Pdf URL: https://arxiv.org/pdf/2504.04756
• Copy Paste: [[2504.04756]] Continuous Locomotive Crowd Behavior Generation(https://arxiv.org/abs/2504.04756)
• Keywords: generation
• Abstract: Modeling and reproducing crowd behaviors are important in various domains including psychology, robotics, transport engineering and virtual environments. Conventional methods have focused on synthesizing momentary scenes, which have difficulty in replicating the continuous nature of real-world crowds. In this paper, we introduce a novel method for automatically generating continuous, realistic crowd trajectories with heterogeneous behaviors and interactions among individuals. We first design a crowd emitter model. To do this, we obtain spatial layouts from single input images, including a segmentation map, appearance map, population density map and population probability, prior to crowd generation. The emitter then continually places individuals on the timeline by assigning independent behavior characteristics such as agents' type, pace, and start/end positions using diffusion models. Next, our crowd simulator produces their long-term locomotions. To simulate diverse actions, it can augment their behaviors based on a Markov chain. As a result, our overall framework populates the scenes with heterogeneous crowd behaviors by alternating between the proposed emitter and simulator. Note that all the components in the proposed framework are user-controllable. Lastly, we propose a benchmark protocol to evaluate the realism and quality of the generated crowds in terms of the scene-level population dynamics and the individual-level trajectory accuracy. We demonstrate that our approach effectively models diverse crowd behavior patterns and generalizes well across different geographical environments. Code is publicly available at this https URL .
• 摘要：在包括心理学，机器人技术，运输工程和虚拟环境在内的各个领域中，建模和再现人群行为很重要。传统方法集中在综合瞬时场景上，这些场景难以复制现实世界人群的连续性。在本文中，我们介绍了一种新颖的方法，用于自动产生连续的，逼真的人群轨迹，具有异质行为和个人之间的相互作用。我们首先设计一个人群发射极模型。为此，我们从单个输入图像中获得空间布局，包括分割图，外观图，人口密度图和人口概率，在人群生成之前。然后，发射极不断地将个人置于时间表上，通过使用扩散模型分配独立的行为特征，例如代理的类型，速度和开始/结束位置。接下来，我们的人群模拟器会产生他们的长期运动。为了模拟各种行动，它可以根据马尔可夫链来增强其行为。结果，我们的整体框架通过在提出的发射极和模拟器之间交替，以异质的人群行为填充场景。请注意，建议的框架中的所有组件都是用户控制的。最后，我们提出了一种基准协议，以根据场景级别的人口动态和个体级别的轨迹准确性来评估产生的人群的现实主义和质量。我们证明，我们的方法有效地对各种人群行为模式进行了建模，并在不同的地理环境中很好地概括了。代码在此HTTPS URL上公开可用。

Title: Bidirectional Hierarchical Protein Multi-Modal Representation Learning

• Authors: Xuefeng Liu, Songhao Jiang, Chih-chan Tien, Jinbo Xu, Rick Stevens
• Subjects: cs.LG, cs.AI, q-bio.MN
• Abstract URL: https://arxiv.org/abs/2504.04770
• Pdf URL: https://arxiv.org/pdf/2504.04770
• Copy Paste: [[2504.04770]] Bidirectional Hierarchical Protein Multi-Modal Representation Learning(https://arxiv.org/abs/2504.04770)
• Keywords: quality assessment
• Abstract: Protein representation learning is critical for numerous biological tasks. Recently, large transformer-based protein language models (pLMs) pretrained on large scale protein sequences have demonstrated significant success in sequence-based tasks. However, pLMs lack structural information. Conversely, graph neural networks (GNNs) designed to leverage 3D structural information have shown promising generalization in protein-related prediction tasks, but their effectiveness is often constrained by the scarcity of labeled structural data. Recognizing that sequence and structural representations are complementary perspectives of the same protein entity, we propose a multimodal bidirectional hierarchical fusion framework to effectively merge these modalities. Our framework employs attention and gating mechanisms to enable effective interaction between pLMs-generated sequential representations and GNN-extracted structural features, improving information exchange and enhancement across layers of the neural network. Based on the framework, we further introduce local Bi-Hierarchical Fusion with gating and global Bi-Hierarchical Fusion with multihead self-attention approaches. Through extensive experiments on a diverse set of protein-related tasks, our method demonstrates consistent improvements over strong baselines and existing fusion techniques in a variety of protein representation learning benchmarks, including react (enzyme/EC classification), model quality assessment (MQA), protein-ligand binding affinity prediction (LBA), protein-protein binding site prediction (PPBS), and B cell epitopes prediction (BCEs). Our method establishes a new state-of-the-art for multimodal protein representation learning, emphasizing the efficacy of BIHIERARCHICAL FUSION in bridging sequence and structural modalities.
• 摘要：蛋白质表示学习对于众多生物学任务至关重要。最近，在大规模蛋白质序列上预测的基于变压器的大型蛋白质语言模型（PLM）在基于序列的任务上表现出了显着成功。但是，PLM缺乏结构信息。相反，旨在利用3D结构信息的图形神经网络（GNN）显示出有希望的蛋白质相关预测任务的概括，但是它们的有效性通常受到标记的结构数据的稀缺性的限制。认识到序列和结构表示是同一蛋白质实体的互补观点，我们提出了多模式双向分层融合框架，以有效合并这些方式。我们的框架采用了注意力和门控机制，以实现PLMS生成的顺序表示与GNN提取的结构特征之间的有效相互作用，从而改善了神经网络层的信息交换和增强。基于该框架，我们进一步引入了局部双层融合，并采用门控和全球双层结构融合，并采用多头自我注意方法。 Through extensive experiments on a diverse set of protein-related tasks, our method demonstrates consistent improvements over strong baselines and existing fusion techniques in a variety of protein representation learning benchmarks, including react (enzyme/EC classification), model quality assessment (MQA), protein-ligand binding affinity prediction (LBA), protein-protein binding site prediction (PPBS), and B cell epitopes prediction （BCE）。我们的方法建立了一种用于多模式蛋白表示学习的新最新技术，强调了双层融合在桥接序列和结构方式中的功效。

Title: Feedback-Enhanced Hallucination-Resistant Vision-Language Model for Real-Time Scene Understanding

• Authors: Zahir Alsulaimawi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.04772
• Pdf URL: https://arxiv.org/pdf/2504.04772
• Copy Paste: [[2504.04772]] Feedback-Enhanced Hallucination-Resistant Vision-Language Model for Real-Time Scene Understanding(https://arxiv.org/abs/2504.04772)
• Keywords: generation
• Abstract: Real-time scene comprehension is a key advance in artificial intelligence, enhancing robotics, surveillance, and assistive tools. However, hallucination remains a challenge. AI systems often misinterpret visual inputs, detecting nonexistent objects or describing events that never happened. These errors, far from minor, threaten reliability in critical areas like security and autonomous navigation where accuracy is essential. Our approach tackles this by embedding self-awareness into the AI. Instead of trusting initial outputs, our framework continuously assesses them in real time, adjusting confidence thresholds dynamically. When certainty falls below a solid benchmark, it suppresses unreliable claims. Combining YOLOv5's object detection strength with VILA1.5-3B's controlled language generation, we tie descriptions to confirmed visual data. Strengths include dynamic threshold tuning for better accuracy, evidence-based text to reduce hallucination, and real-time performance at 18 frames per second. This feedback-driven design cuts hallucination by 37 percent over traditional methods. Fast, flexible, and reliable, it excels in applications from robotic navigation to security monitoring, aligning AI perception with reality.
• 摘要：实时场景理解是人工智能，增强机器人技术，监视和辅助工具的关键进步。但是，幻觉仍然是一个挑战。 AI系统通常会误解视觉输入，检测不存在的对象或描述从未发生的事件。这些错误远非较小的，威胁到准确性至关重要的关键领域的可靠性。我们的方法通过将自我意识纳入AI来解决这一问题。我们的框架没有信任初始输出，而是连续地实时评估它们，从而动态调整置信度阈值。当确定性低于稳定的基准时，它会抑制不可靠的主张。将Yolov5的对象检测强度与VILA1.5-3B的受控语言生成相结合，我们将描述与确认的视觉数据相结合。优势包括动态阈值调整，以提高准确性，基于证据的文本以减少幻觉和每秒18帧的实时性能。这种反馈驱动的设计比传统方法减少了37％的幻觉。快速，灵活且可靠，它在从机器人导航到安全监控的应用中擅长，使AI感知与现实保持一致。

Title: Playing Non-Embedded Card-Based Games with Reinforcement Learning

• Authors: Tianyang Wu, Lipeng Wan, Yuhang Wang, Qiang Wan, Xuguang Lan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.04783
• Pdf URL: https://arxiv.org/pdf/2504.04783
• Copy Paste: [[2504.04783]] Playing Non-Embedded Card-Based Games with Reinforcement Learning(https://arxiv.org/abs/2504.04783)
• Keywords: generative
• Abstract: Significant progress has been made in AI for games, including board games, MOBA, and RTS games. However, complex agents are typically developed in an embedded manner, directly accessing game state information, unlike human players who rely on noisy visual data, leading to unfair competition. Developing complex non-embedded agents remains challenging, especially in card-based RTS games with complex features and large state spaces. We propose a non-embedded offline reinforcement learning training strategy using visual inputs to achieve real-time autonomous gameplay in the RTS game Clash Royale. Due to the lack of a object detection dataset for this game, we designed an efficient generative object detection dataset for training. We extract features using state-of-the-art object detection and optical character recognition models. Our method enables real-time image acquisition, perception feature fusion, decision-making, and control on mobile devices, successfully defeating built-in AI opponents. All code is open-sourced at this https URL.
• 摘要：AI在包括棋盘游戏，MOBA和RTS游戏在内的AI中取得了重大进展。但是，与依靠嘈杂的视觉数据的人类玩家不同，复杂的代理通常以嵌入式方式开发，直接访问游戏状态信息，从而导致不公平的竞争。开发复杂的非安装代理仍然具有挑战性，尤其是在具有复杂功能和较大状态空间的基于卡的RTS游戏中。我们建议使用视觉输入来实现RTS Game Clash Royale中实时自主游戏玩法的非安装离线增强学习训练策略。由于缺乏该游戏的对象检测数据集，我们设计了一个有效的生成对象检测数据集用于培训。我们使用最新的对象检测和光学特征识别模型提取功能。我们的方法使实时图像获取，感知功能融合，决策和移动设备的控制成功，成功击败了内置的AI对手。所有代码均在此HTTPS URL上开源。

Title: TabRep: Training Tabular Diffusion Models with a Simple and Effective Continuous Representation

• Authors: Jacob Si, Zijing Ou, Mike Qu, Zhengrui Xiang, Yingzhen Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.04798
• Pdf URL: https://arxiv.org/pdf/2504.04798
• Copy Paste: [[2504.04798]] TabRep: Training Tabular Diffusion Models with a Simple and Effective Continuous Representation(https://arxiv.org/abs/2504.04798)
• Keywords: generation, generative
• Abstract: Diffusion models have been the predominant generative model for tabular data generation. However, they face the conundrum of modeling under a separate versus a unified data representation. The former encounters the challenge of jointly modeling all multi-modal distributions of tabular data in one model. While the latter alleviates this by learning a single representation for all features, it currently leverages sparse suboptimal encoding heuristics and necessitates additional computation costs. In this work, we address the latter by presenting TabRep, a tabular diffusion architecture trained with a unified continuous representation. To motivate the design of our representation, we provide geometric insights into how the data manifold affects diffusion models. The key attributes of our representation are composed of its density, flexibility to provide ample separability for nominal features, and ability to preserve intrinsic relationships. Ultimately, TabRep provides a simple yet effective approach for training tabular diffusion models under a continuous data manifold. Our results showcase that TabRep achieves superior performance across a broad suite of evaluations. It is the first to synthesize tabular data that exceeds the downstream quality of the original datasets while preserving privacy and remaining computationally efficient.
• 摘要：扩散模型一直是表格数据生成的主要生成模型。但是，他们面临着在单独的数据表示和统一数据表示下的建模的难题。前者遇到了一个模型中共同建模表格数据的所有多模式分布的挑战。尽管后者通过学习所有功能的单一表示来减轻这一点，但目前它利用了稀疏的次优编码启发式方法，并需要额外的计算成本。在这项工作中，我们通过介绍TabRep来解决后者，TabRep是一种表达式扩散体系结构，该体系结构训练有统一的连续表示。为了激发我们表示的设计，我们提供了有关数据歧管如何影响扩散模型的几何见解。我们表示的关键属性由其密度组成，灵活性以提供象征特征的足够可分离性以及保持内在关系的能力。最终，TABREP提供了一种简单而有效的方法，用于在连续数据歧管下训练表格扩散模型。我们的结果表明，TABREP在广泛的评估中取得了卓越的性能。它是第一个合成超过原始数据集质量下游质量的表格数据的同时保留隐私并保持计算效率效率的数据。

Title: Topological Schrödinger Bridge Matching

• Authors: Maosheng Yang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2504.04799
• Pdf URL: https://arxiv.org/pdf/2504.04799
• Copy Paste: [[2504.04799]] Topological Schrödinger Bridge Matching(https://arxiv.org/abs/2504.04799)
• Keywords: generative
• Abstract: Given two boundary distributions, the Schrödinger Bridge (SB) problem seeks the ``most likely`` random evolution between them with respect to a reference process. It has revealed rich connections to recent machine learning methods for generative modeling and distribution matching. While these methods perform well in Euclidean domains, they are not directly applicable to topological domains such as graphs and simplicial complexes, which are crucial for data defined over network entities, such as node signals and edge flows. In this work, we propose the Topological Schrödinger Bridge problem (TSBP) for matching signal distributions on a topological domain. We set the reference process to follow some linear tractable topology-aware stochastic dynamics such as topological heat diffusion. For the case of Gaussian boundary distributions, we derive a closed-form topological SB (TSB) in terms of its time-marginal and stochastic differential. In the general case, leveraging the well-known result, we show that the optimal process follows the forward-backward topological dynamics governed by some unknowns. Building on these results, we develop TSB-based models for matching topological signals by parameterizing the unknowns in the optimal process as (topological) neural networks and learning them through likelihood training. We validate the theoretical results and demonstrate the practical applications of TSB-based models on both synthetic and real-world networks, emphasizing the role of topology. Additionally, we discuss the connections of TSB-based models to other emerging models, and outline future directions for topological signal matching.
• 摘要：给定两个边界分布，Schrödinger桥（SB）问题寻求``最有可能的''与参考过程之间的随机演变。它揭示了与最近的机器学习方法的丰富联系，用于生成建模和分配匹配。尽管这些方法在欧几里得域中表现良好，但它们并不直接适用于拓扑结构域，例如图形和简单络合物，这些域对于在网络实体（例如节点信号和边缘流）上定义的数据至关重要。在这项工作中，我们提出了拓扑结构桥问题（TSBP），以匹配拓扑结构域上的信号分布。我们设置了参考过程，以遵循一些线性拓扑感知的随机动力学，例如拓扑热扩散。对于高斯边界分布的情况，我们根据其时间 - 边界和随机差异来得出封闭形式的拓扑SB（TSB）。在一般情况下，利用众所周知的结果，我们表明最佳过程遵循一些未知数控制的前向拓扑动态。在这些结果的基础上，我们开发了基于TSB的模型，以通过将最佳过程中的未知数参数为（拓扑）神经网络并通过可能性训练来学习它们，从而匹配拓扑信号。我们验证了理论结果，并证明了基于TSB的模型在合成和现实世界网络上的实际应用，从而强调了拓扑的作用。此外，我们讨论了基于TSB的模型与其他新兴模型的连接，并概述了拓扑信号匹配的未来方向。

Title: From Specificity to Generality: Revisiting Generalizable Artifacts in Detecting Face Deepfakes

• Authors: Long Ma, Zhiyuan Yan, Yize Chen, Jin Xu, Qinglang Guo, Hu Huang, Yong Liao, Hui Lin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.04827
• Pdf URL: https://arxiv.org/pdf/2504.04827
• Copy Paste: [[2504.04827]] From Specificity to Generality: Revisiting Generalizable Artifacts in Detecting Face Deepfakes(https://arxiv.org/abs/2504.04827)
• Keywords: super-resolution, generation
• Abstract: Detecting deepfakes has been an increasingly important topic, especially given the rapid development of AI generation techniques. In this paper, we ask: How can we build a universal detection framework that is effective for most facial deepfakes? One significant challenge is the wide variety of deepfake generators available, resulting in varying forgery artifacts (e.g., lighting inconsistency, color mismatch, etc). But should we ``teach" the detector to learn all these artifacts separately? It is impossible and impractical to elaborate on them all. So the core idea is to pinpoint the more common and general artifacts across different deepfakes. Accordingly, we categorize deepfake artifacts into two distinct yet complementary types: Face Inconsistency Artifacts (FIA) and Up-Sampling Artifacts (USA). FIA arise from the challenge of generating all intricate details, inevitably causing inconsistencies between the complex facial features and relatively uniform surrounding areas. USA, on the other hand, are the inevitable traces left by the generator's decoder during the up-sampling process. This categorization stems from the observation that all existing deepfakes typically exhibit one or both of these artifacts. To achieve this, we propose a new data-level pseudo-fake creation framework that constructs fake samples with only the FIA and USA, without introducing extra less-general artifacts. Specifically, we employ a super-resolution to simulate the USA, while design a Blender module that uses image-level self-blending on diverse facial regions to create the FIA. We surprisingly found that, with this intuitive design, a standard image classifier trained only with our pseudo-fake data can non-trivially generalize well to unseen deepfakes.
• 摘要：检测深击是一个越来越重要的话题，尤其是考虑到AI生成技术的快速发展。在本文中，我们问：我们如何构建一个对大多数面部深层效果有效的通用检测框架？一个重大的挑战是可用的多种深泡产生器，导致伪造的伪像（例如，照明不一致，颜色不匹配等）。但是，我们应该``教导探测器分别学习所有这些文物吗？详细说明所有这些文物是不可能且不切实际的。因此，核心的想法是，核心的想法是在不同的深层捕获范围内查明更为普遍和一般的伪像。因此，我们将深层伪像属于两种截然不同的类型：面对不一致的企业（Facia confia）（FACIA）（fiia）（fiia foria foria foria foria foria foria frofia）（美国）的作品（又是ARISAIS）。为了产生所有复杂的细节，不可避免地会在周围的面积和相对均匀的区域之间造成不一致。框架仅使用FIA和美国构建假样品，而没有引入额外的较少的人工制品，我们采用了超级分辨率来模拟美国，同时设计了使用图像级别的自我融合的搅拌机模块来创建FIA。我们出乎意料地发现，通过这种直观的设计，仅使用我们的伪捕获数据训练的标准图像分类器才能在非平凡的概括中概括为看不见的深击。

Title: FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis

• Authors: Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, Mu Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04842
• Pdf URL: https://arxiv.org/pdf/2504.04842
• Copy Paste: [[2504.04842]] FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis(https://arxiv.org/abs/2504.04842)
• Keywords: generation
• Abstract: Creating a realistic animatable avatar from a single static portrait remains challenging. Existing approaches often struggle to capture subtle facial expressions, the associated global body movements, and the dynamic background. To address these limitations, we propose a novel framework that leverages a pretrained video diffusion transformer model to generate high-fidelity, coherent talking portraits with controllable motion dynamics. At the core of our work is a dual-stage audio-visual alignment strategy. In the first stage, we employ a clip-level training scheme to establish coherent global motion by aligning audio-driven dynamics across the entire scene, including the reference portrait, contextual objects, and background. In the second stage, we refine lip movements at the frame level using a lip-tracing mask, ensuring precise synchronization with audio signals. To preserve identity without compromising motion flexibility, we replace the commonly used reference network with a facial-focused cross-attention module that effectively maintains facial consistency throughout the video. Furthermore, we integrate a motion intensity modulation module that explicitly controls expression and body motion intensity, enabling controllable manipulation of portrait movements beyond mere lip motion. Extensive experimental results show that our proposed approach achieves higher quality with better realism, coherence, motion intensity, and identity preservation. Ours project page: this https URL.
• 摘要：从单个静态肖像中创建一个可逼真的动画头像仍然具有挑战性。现有的方法通常难以捕获微妙的面部表情，相关的全球身体运动和动态背景。为了解决这些局限性，我们提出了一个新型框架，该框架利用了验证的视频扩散变压器模型来生成具有可控运动动力学的高保真性，连贯的说话肖像。我们工作的核心是双阶段的视听策略。在第一阶段，我们采用剪辑级训练方案来建立连贯的全局运动，通过整个场景中的音频驱动动态，包括参考肖像，上下文对象和背景。在第二阶段，我们使用唇部跟踪掩码在框架级别提炼唇部运动，从而确保与音频信号的精确同步。为了保持身份而不损害运动灵活性，我们用面部注重的跨意识模块代替了常用的参考网络，该模块在整个视频中有效地保持面部一致性。此外，我们集成了一个运动强度调制模块，该模块明确控制了表达和身体运动强度，从​​而可以控制对超出唇部运动的肖像运动的控制。广泛的实验结果表明，我们提出的方法可以通过更好的现实主义，连贯性，运动强度和身份保存来达到更高的质量。我们的项目页面：此HTTPS URL。

Title: Content-Aware Transformer for All-in-one Image Restoration

• Authors: Gang Wu, Junjun Jiang, Kui Jiang, Xianming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.04869
• Pdf URL: https://arxiv.org/pdf/2504.04869
• Copy Paste: [[2504.04869]] Content-Aware Transformer for All-in-one Image Restoration(https://arxiv.org/abs/2504.04869)
• Keywords: restoration
• Abstract: Image restoration has witnessed significant advancements with the development of deep learning models. Although Transformer architectures have progressed considerably in recent years, challenges remain, particularly the limited receptive field in window-based self-attention. In this work, we propose DSwinIR, a Deformable Sliding window Transformer for Image Restoration. DSwinIR introduces a novel deformable sliding window self-attention that adaptively adjusts receptive fields based on image content, enabling the attention mechanism to focus on important regions and enhance feature extraction aligned with salient features. Additionally, we introduce a central ensemble pattern to reduce the inclusion of irrelevant content within attention windows. In this way, the proposed DSwinIR model integrates the deformable sliding window Transformer and central ensemble pattern to amplify the strengths of both CNNs and Transformers while mitigating their limitations. Extensive experiments on various image restoration tasks demonstrate that DSwinIR achieves state-of-the-art performance. For example, in image deraining, compared to DRSformer on the SPA dataset, DSwinIR achieves a 0.66 dB PSNR improvement. In all-in-one image restoration, compared to PromptIR, DSwinIR achieves over a 0.66 dB and 1.04 dB improvement on three-task and five-task settings, respectively. Pretrained models and code are available at our project this https URL.
• 摘要：图像恢复已经看到了深度学习模型的发展取得了重大进步。尽管近年来变压器架构取得了长足的发展，但仍存在挑战，尤其是基于窗户的自我注意力中的受欢迎领域有限。在这项工作中，我们提出了Dswinir，这是一种可变形的滑动窗变压器，用于图像恢复。 Dswinir引入了一种新型的可变形滑动窗口自我发作，该自适应根据图像内容适应接受场，从而使注意机制能够专注于重要区域，并增强与显着特征对齐的特征提取。此外，我们引入了一种中央合奏模式，以减少注意力窗口中无关的内容。这样，提出的DSWINIR模型集成了可变形的滑动窗口变压器和中央集合模式，以放大CNN和变压器的强度，同时减轻其局限性。关于各种图像恢复任务的广泛实验表明，Dswinir实现了最先进的性能。例如，与SPA数据集上的DRSFormer相比，DSWINIR可实现0.66 dB PSNR的改进。与Pictectir相比，在多合一的图像恢复中，Dswinir在三任任务和五任务设置上分别实现了0.66 dB和1.04 dB的改进。验证的模型和代码可在我们的项目此HTTPS URL上找到。

Title: Lumina-OmniLV: A Unified Multimodal Framework for General Low-Level Vision

• Authors: Yuandong Pu, Le Zhuo, Kaiwen Zhu, Liangbin Xie, Wenlong Zhang, Xiangyu Chen, Pneg Gao, Yu Qiao, Chao Dong, Yihao Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.04903
• Pdf URL: https://arxiv.org/pdf/2504.04903
• Copy Paste: [[2504.04903]] Lumina-OmniLV: A Unified Multimodal Framework for General Low-Level Vision(https://arxiv.org/abs/2504.04903)
• Keywords: restoration, generative
• Abstract: We present Lunima-OmniLV (abbreviated as OmniLV), a universal multimodal multi-task framework for low-level vision that addresses over 100 sub-tasks across four major categories: image restoration, image enhancement, weak-semantic dense prediction, and stylization. OmniLV leverages both textual and visual prompts to offer flexible and user-friendly interactions. Built on Diffusion Transformer (DiT)-based generative priors, our framework supports arbitrary resolutions -- achieving optimal performance at 1K resolution -- while preserving fine-grained details and high fidelity. Through extensive experiments, we demonstrate that separately encoding text and visual instructions, combined with co-training using shallow feature control, is essential to mitigate task ambiguity and enhance multi-task generalization. Our findings also reveal that integrating high-level generative tasks into low-level vision models can compromise detail-sensitive restoration. These insights pave the way for more robust and generalizable low-level vision systems.
• 摘要：我们提出了Lunima-Omnilv（缩写为Omnilv），这是一种通用的多模式多模式的多模式多任务框架，用于低级视觉，以四个主要类别介绍100个子任务：图像恢复，图像增强，图像增强，虚弱的电子设施密集预测和风格化。 Omnilv利用文本和视觉提示提供灵活和用户友好的交互。我们的框架建立在基于扩散的变压器（DIT）基于生成的先验的基础上，支持任意决议 - 以1K分辨率实现最佳性能 - 同时保留细粒度的细节和高忠诚度。通过广泛的实验，我们证明，分别编码文本和视觉说明，并结合使用浅色特征控制的共同训练，这对于减轻任务歧义并增强多任务概括至关重要。我们的发现还表明，将高级生成任务整合到低级视觉模型中会损害细节敏感的恢复。这些见解为更健壮和可推广的低级视力系统铺平了道路。

Title: Video-Bench: Human-Aligned Video Generation Benchmark

• Authors: Hui Han, Siyuan Li, Jiaqi Chen, Yiwen Yuan, Yuling Wu, Chak Tou Leong, Hanwen Du, Junchen Fu, Youhua Li, Jie Zhang, Chi Zhang, Li-jia Li, Yongxin Ni
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.04907
• Pdf URL: https://arxiv.org/pdf/2504.04907
• Copy Paste: [[2504.04907]] Video-Bench: Human-Aligned Video Generation Benchmark(https://arxiv.org/abs/2504.04907)
• Keywords: generation, generative
• Abstract: Video generation assessment is essential for ensuring that generative models produce visually realistic, high-quality videos while aligning with human expectations. Current video generation benchmarks fall into two main categories: traditional benchmarks, which use metrics and embeddings to evaluate generated video quality across multiple dimensions but often lack alignment with human judgments; and large language model (LLM)-based benchmarks, though capable of human-like reasoning, are constrained by a limited understanding of video quality metrics and cross-modal consistency. To address these challenges and establish a benchmark that better aligns with human preferences, this paper introduces Video-Bench, a comprehensive benchmark featuring a rich prompt suite and extensive evaluation dimensions. This benchmark represents the first attempt to systematically leverage MLLMs across all dimensions relevant to video generation assessment in generative models. By incorporating few-shot scoring and chain-of-query techniques, Video-Bench provides a structured, scalable approach to generated video evaluation. Experiments on advanced models including Sora demonstrate that Video-Bench achieves superior alignment with human preferences across all dimensions. Moreover, in instances where our framework's assessments diverge from human evaluations, it consistently offers more objective and accurate insights, suggesting an even greater potential advantage over traditional human judgment.
• 摘要：视频生成评估对于确保生成模型产生视觉现实，高质量的视频而与人类期望保持一致至关重要。当前的视频生成基准分为两个主要类别：传统基准测试，它们使用指标和嵌入来评估跨多个维度的产生的视频质量，但通常与人类判断不符；大型语言模型（LLM）基于基于类似人类的推理的基准受到对视频质量指标和跨模式一致性的有限理解的限制。为了应对这些挑战并建立一个更好地与人类偏好保持一致的基准，本文介绍了视频基础，这是一个全面的基准测试，具有丰富的及时套件和广泛的评估维度。该基准是在与生成模型中与视频生成评估相关的所有维度中系统地利用MLLM的首次尝试。通过结合几次射击的评分和杂货技术，视频基础提供了一种结构化的，可扩展的方法来生成视频评估。包括SORA在内的高级模型的实验表明，视频基础座位与所有维度的人类偏好达到了卓越的对齐。此外，在我们的框架评估与人类评估有所不同的情况下，它始终提供了更客观和准确的见解，这表明比传统的人类判断具有更大的潜在优势。

Title: A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization

• Authors: Wenyuan Xu, Xiaochen Zuo, Chao Xin, Yu Yue, Lin Yan, Yonghui Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.04950
• Pdf URL: https://arxiv.org/pdf/2504.04950
• Copy Paste: [[2504.04950]] A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization(https://arxiv.org/abs/2504.04950)
• Keywords: generative
• Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a important paradigm for aligning large language models (LLMs) with human preferences during post-training. This framework typically involves two stages: first, training a reward model on human preference data, followed by optimizing the language model using reinforcement learning algorithms. However, current RLHF approaches may constrained by two limitations. First, existing RLHF frameworks often rely on Bradley-Terry models to assign scalar rewards based on pairwise comparisons of individual responses. However, this approach imposes significant challenges on reward model (RM), as the inherent variability in prompt-response pairs across different contexts demands robust calibration capabilities from the RM. Second, reward models are typically initialized from generative foundation models, such as pre-trained or supervised fine-tuned models, despite the fact that reward models perform discriminative tasks, creating a mismatch. This paper introduces Pairwise-RL, a RLHF framework that addresses these challenges through a combination of generative reward modeling and a pairwise proximal policy optimization (PPO) algorithm. Pairwise-RL unifies reward model training and its application during reinforcement learning within a consistent pairwise paradigm, leveraging generative modeling techniques to enhance reward model performance and score calibration. Experimental evaluations demonstrate that Pairwise-RL outperforms traditional RLHF frameworks across both internal evaluation datasets and standard public benchmarks, underscoring its effectiveness in improving alignment and model behavior.
• 摘要：从人类反馈（RLHF）中学习的强化已成为将大语言模型（LLMS）与培训期间人类偏好保持一致的重要范式。该框架通常涉及两个阶段：首先，在人类偏好数据上培训奖励模型，然后使用增强学习算法优化语言模型。但是，当前的RLHF方法可能受到两个限制的约束。首先，现有的RLHF框架通常依靠Bradley-Terry模型来基于单个响应的成对比较来分配标量奖励。但是，这种方法对奖励模型（RM）施加了重大挑战，因为跨不同环境的及时响应对的固有可变性需要RM的强大校准功能。其次，尽管奖励模型执行歧视性任务，从而造成了不匹配，但奖励模型通常是从​​生成基础模型（例如预训练或监督的微调模型）初始化的。本文介绍了Pairwise-RL，这是一个RLHF框架，该框架通过生成奖励建模和成对近端策略优化（PPO）算法的结合来解决这些挑战。成对RL在一致的成对范式内统一奖励模型训练及其在加固学习过程中的应用，利用生成建模技术来增强奖励模型性能和得分校准。实验评估表明，成对RL的表现优于内部评估数据集和标准公共基准的传统RLHF框架，从而强调了其在改善对齐和模型行为方面的有效性。

Title: RS-RAG: Bridging Remote Sensing Imagery and Comprehensive Knowledge with a Multi-Modal Dataset and Retrieval-Augmented Generation Model

• Authors: Congcong Wen, Yiting Lin, Xiaokang Qu, Nan Li, Yong Liao, Hui Lin, Xiang Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.04988
• Pdf URL: https://arxiv.org/pdf/2504.04988
• Copy Paste: [[2504.04988]] RS-RAG: Bridging Remote Sensing Imagery and Comprehensive Knowledge with a Multi-Modal Dataset and Retrieval-Augmented Generation Model(https://arxiv.org/abs/2504.04988)
• Keywords: generation
• Abstract: Recent progress in VLMs has demonstrated impressive capabilities across a variety of tasks in the natural image domain. Motivated by these advancements, the remote sensing community has begun to adopt VLMs for remote sensing vision-language tasks, including scene understanding, image captioning, and visual question answering. However, existing remote sensing VLMs typically rely on closed-set scene understanding and focus on generic scene descriptions, yet lack the ability to incorporate external knowledge. This limitation hinders their capacity for semantic reasoning over complex or context-dependent queries that involve domain-specific or world knowledge. To address these challenges, we first introduced a multimodal Remote Sensing World Knowledge (RSWK) dataset, which comprises high-resolution satellite imagery and detailed textual descriptions for 14,141 well-known landmarks from 175 countries, integrating both remote sensing domain knowledge and broader world knowledge. Building upon this dataset, we proposed a novel Remote Sensing Retrieval-Augmented Generation (RS-RAG) framework, which consists of two key components. The Multi-Modal Knowledge Vector Database Construction module encodes remote sensing imagery and associated textual knowledge into a unified vector space. The Knowledge Retrieval and Response Generation module retrieves and re-ranks relevant knowledge based on image and/or text queries, and incorporates the retrieved content into a knowledge-augmented prompt to guide the VLM in producing contextually grounded responses. We validated the effectiveness of our approach on three representative vision-language tasks, including image captioning, image classification, and visual question answering, where RS-RAG significantly outperformed state-of-the-art baselines.
• 摘要：VLMS的最新进展表现出了自然图像域中各种任务的令人印象深刻的能力。受这些进步的促进，遥感社区已开始采用VLM来进行遥感视觉语言任务，包括场景理解，图像字幕和视觉问题的回答。但是，现有的遥感VLM通常依赖于封闭场景的理解并专注于通用场景描述，但缺乏合并外部知识的能力。这种限制阻碍了他们在涉及特定领域或世界知识的复杂或上下文依赖性查询上进行语义推理的能力。为了应对这些挑战，我们首先引入了多模式遥感世界知识（RSWK）数据集，该数据集包括高分辨率卫星图像和1411个来自175个国家的1411个著名地标的详细文本描述，既整合远程感应域知识又整合了更广泛的世界知识。在此数据集的基础上，我们提出了一种新颖的遥感检索型生成（RS-rag）框架，该框架由两个关键组成部分组成。多模式知识矢量数据库构造模块将遥感图像和相关的文本知识编码为统一的矢量空间。知识检索和响应生成模块根据图像和/或文本查询检索并重新排列相关的知识，并将检索到的内容纳入知识增强的提示中，以指导VLM产生上下文基础的响应。我们验证了我们的方法对三个代表性视觉语言任务的有效性，包括图像字幕，图像分类和视觉问题答案，其中RS-rag在其中明显优于最先进的基线。

Title: Mixture-of-Personas Language Models for Population Simulation

• Authors: Ngoc Bui, Hieu Trung Nguyen, Shantanu Kumar, Julian Theodore, Weikang Qiu, Viet Anh Nguyen, Rex Ying
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05019
• Pdf URL: https://arxiv.org/pdf/2504.05019
• Copy Paste: [[2504.05019]] Mixture-of-Personas Language Models for Population Simulation(https://arxiv.org/abs/2504.05019)
• Keywords: generation
• Abstract: Advances in Large Language Models (LLMs) paved the way for their emerging applications in various domains, such as human behavior simulations, where LLMs could augment human-generated data in social science research and machine learning model training. However, pretrained LLMs often fail to capture the behavioral diversity of target populations due to the inherent variability across individuals and groups. To address this, we propose \textit{Mixture of Personas} (MoP), a \textit{probabilistic} prompting method that aligns the LLM responses with the target population. MoP is a contextual mixture model, where each component is an LM agent characterized by a persona and an exemplar representing subpopulation behaviors. The persona and exemplar are randomly chosen according to the learned mixing weights to elicit diverse LLM responses during simulation. MoP is flexible, requires no model finetuning, and is transferable across base models. Experiments for synthetic data generation show that MoP outperforms competing methods in alignment and diversity metrics.
• 摘要：大语言模型（LLMS）的进步为在各种领域（例如人类行为模拟）中的新兴应用铺平了道路，LLM可以在社会科学研究和机器学习模型培训中增加人类生成的数据。但是，由于个人和群体之间固有的可变性，验证的LLM通常无法捕获目标人群的行为多样性。为了解决这个问题，我们提出了\ textit {personas}（mop）的混合物，\ textit {probabilistic}提示方法将LLM响应与目标总体保持一致。 MOP是一种上下文混合模型，其中每个组件都是LM代理，其特征是角色和代表亚种群行为的示例。根据所学的混合权重以在模拟过程中引起不同的LLM响应，随机选择角色和示例。 MOP是灵活的，不需要模型登录，并且可以在基本模型之间转移。合成数据生成的实验表明，拖把在对齐和多样性指标中的表现优于竞争方法。

Title: InstructionBench: An Instructional Video Understanding Benchmark

• Authors: Haiwan Wei, Yitian Yuan, Xiaohan Lan, Wei Ke, Lin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.05040
• Pdf URL: https://arxiv.org/pdf/2504.05040
• Copy Paste: [[2504.05040]] InstructionBench: An Instructional Video Understanding Benchmark(https://arxiv.org/abs/2504.05040)
• Keywords: generation
• Abstract: Despite progress in video large language models (Video-LLMs), research on instructional video understanding, crucial for enhancing access to instructional content, remains insufficient. To address this, we introduce InstructionBench, an Instructional video understanding Benchmark, which challenges models' advanced temporal reasoning within instructional videos characterized by their strict step-by-step flow. Employing GPT-4, we formulate Q\&A pairs in open-ended and multiple-choice formats to assess both Coarse-Grained event-level and Fine-Grained object-level reasoning. Our filtering strategies exclude questions answerable purely by common-sense knowledge, focusing on visual perception and analysis when evaluating Video-LLM models. The benchmark finally contains 5k questions across over 700 videos. We evaluate the latest Video-LLMs on our InstructionBench, finding that closed-source models outperform open-source ones. However, even the best model, GPT-4o, achieves only 53.42\% accuracy, indicating significant gaps in temporal reasoning. To advance the field, we also develop a comprehensive instructional video dataset with over 19k Q\&A pairs from nearly 2.5k videos, using an automated data generation framework, thereby enriching the community's research resources.
• 摘要：尽管视频大型语言模型（视频-LLM）取得了进展，但有关教学视频理解的研究，对于增强获得教学内容的访问至关重要，仍然不足。为了解决这个问题，我们介绍了教学视频理解基准的TenchentionBench，该基准挑战了模型在其严格逐步流动为特征的教学视频中的高级时间推理。使用GPT-4，我们采用开放式和多项选择格式制定Q \＆A对，以评估粗粒度的事件级别和细粒度的对象级别的推理。我们的过滤策略排除了完全通过常识知识回答的问题，在评估视频-LLM模型时着重于视觉感知和分析。基准最终在700多个视频中包含5K问题。我们在UndertionBench上评估了最新的视频插件，发现封闭源模型的表现要优于开源量。但是，即使是最佳模型GPT-4O，也仅达到53.42 \％的精度，表明时间推理的差距很大。为了推进该领域，我们还使用自动数据生成框架开发了一个超过2.5k视频的19k Q \＆A对，从而获得了超过19k Q \＆A对，从而丰富了社区的研究资源。

Title: Content-Distortion High-Order Interaction for Blind Image Quality Assessment

• Authors: Shuai Liu, Qingyu Mao, Chao Li, Jiacong Chen, Fanyang Meng, Yonghong Tian, Yongsheng Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.05076
• Pdf URL: https://arxiv.org/pdf/2504.05076
• Copy Paste: [[2504.05076]] Content-Distortion High-Order Interaction for Blind Image Quality Assessment(https://arxiv.org/abs/2504.05076)
• Keywords: quality assessment
• Abstract: The content and distortion are widely recognized as the two primary factors affecting the visual quality of an image. While existing No-Reference Image Quality Assessment (NR-IQA) methods have modeled these factors, they fail to capture the complex interactions between content and distortions. This shortfall impairs their ability to accurately perceive quality. To confront this, we analyze the key properties required for interaction modeling and propose a robust NR-IQA approach termed CoDI-IQA (Content-Distortion high-order Interaction for NR-IQA), which aggregates local distortion and global content features within a hierarchical interaction framework. Specifically, a Progressive Perception Interaction Module (PPIM) is proposed to explicitly simulate how content and distortions independently and jointly influence image quality. By integrating internal interaction, coarse interaction, and fine interaction, it achieves high-order interaction modeling that allows the model to properly represent the underlying interaction patterns. To ensure sufficient interaction, multiple PPIMs are employed to hierarchically fuse multi-level content and distortion features at different granularities. We also tailor a training strategy suited for CoDI-IQA to maintain interaction stability. Extensive experiments demonstrate that the proposed method notably outperforms the state-of-the-art methods in terms of prediction accuracy, data efficiency, and generalization ability.
• 摘要：内容和失真被广泛认为是影响图像视觉质量的两个主要因素。尽管现有的无参考图像质量评估（NR-IQA）方法已经建模了这些因素，但它们无法捕获内容和扭曲之间的复杂相互作用。这种短缺会损害他们准确感知质量的能力。为了面对这一点，我们分析了相互作用建模所需的关键属性，并提出了称为CODI-IQA的强大NR-IQA方法（NR-IQA的内容延伸高阶相互作用），该方法在层次交互框架中汇总了局部失真和全局内容特征。具体而言，提出了渐进感知相互作用模块（PPIM），以明确模拟内容和扭曲如何独立和共同影响图像质量。通过整合内部相互作用，粗糙的相互作用和精细的相互作用，它可以实现高阶相互作用建模，从而允许模型正确地表示潜在的相互作用模式。为了确保足够的相互作用，在不同的粒度上使用多个PPIM用于层次融合的多级含量和失真特征。我们还量身定制适合Codi-IQA的培训策略，以保持交互稳定性。广泛的实验表明，所提出的方法在预测准确性，数据效率和概括能力方面尤其优于最先进的方法。

Title: DA2Diff: Exploring Degradation-aware Adaptive Diffusion Priors for All-in-One Weather Restoration

• Authors: Jiamei Xiong, Xuefeng Yan, Yongzhen Wang, Wei Zhao, Xiao-Ping Zhang, Mingqiang Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.05135
• Pdf URL: https://arxiv.org/pdf/2504.05135
• Copy Paste: [[2504.05135]] DA2Diff: Exploring Degradation-aware Adaptive Diffusion Priors for All-in-One Weather Restoration(https://arxiv.org/abs/2504.05135)
• Keywords: restoration
• Abstract: Image restoration under adverse weather conditions is a critical task for many vision-based applications. Recent all-in-one frameworks that handle multiple weather degradations within a unified model have shown potential. However, the diversity of degradation patterns across different weather conditions, as well as the complex and varied nature of real-world degradations, pose significant challenges for multiple weather removal. To address these challenges, we propose an innovative diffusion paradigm with degradation-aware adaptive priors for all-in-one weather restoration, termed DA2Diff. It is a new exploration that applies CLIP to perceive degradation-aware properties for better multi-weather restoration. Specifically, we deploy a set of learnable prompts to capture degradation-aware representations by the prompt-image similarity constraints in the CLIP space. By aligning the snowy/hazy/rainy images with snow/haze/rain prompts, each prompt contributes to different weather degradation characteristics. The learned prompts are then integrated into the diffusion model via the designed weather specific prompt guidance module, making it possible to restore multiple weather types. To further improve the adaptiveness to complex weather degradations, we propose a dynamic expert selection modulator that employs a dynamic weather-aware router to flexibly dispatch varying numbers of restoration experts for each weather-distorted image, allowing the diffusion model to restore diverse degradations adaptively. Experimental results substantiate the favorable performance of DA2Diff over state-of-the-arts in quantitative and qualitative evaluation. Source code will be available after acceptance.
• 摘要：对于许多基于视觉的应用程序，在不利天气条件下的图像恢复是一项关键任务。在统一模型中处理多种天气降解的最新一合化框架已显示出潜力。然而，在不同天气条件下的退化模式的多样性以及现实世界下降的复杂性和多样性的性质，对多次拆除构成了重大挑战。为了应对这些挑战，我们提出了一种创新的扩散范式，以降解感知的适应性先验，用于多合一天气恢复，称为DA2DIFF。这是一种新的探索，将剪辑应用于感知降解感知的特性，以更好地恢复多天气。具体来说，我们通过剪辑空间中的及时图像相似性约束来捕获一组可学习的提示，以捕获降解感知表示。通过将雪/朦胧/多雨的图像与雪/霾/雨水提示对齐，每个提示都会有助于不同的天气退化特征。然后，通过设计的天气及时指导模块将学习的提示集成到扩散模型中，从而可以恢复多种天气类型。为了进一步提高对复杂天气降低的适应性，我们提出了一个动态的专家选择调节器，该调制器采用动态的天气感知路由器来灵活地针对每个天气不足的图像派遣不同数量的恢复专家，从而使扩散模型适应了多样化的降解。实验结果证实了DA2DIFF在定量和定性评估方面的优惠表现。源代码将在接受后可用。

Title: PanoDreamer: Consistent Text to 360-Degree Scene Generation

• Authors: Zhexiao Xiong, Zhang Chen, Zhong Li, Yi Xu, Nathan Jacobs
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.05152
• Pdf URL: https://arxiv.org/pdf/2504.05152
• Copy Paste: [[2504.05152]] PanoDreamer: Consistent Text to 360-Degree Scene Generation(https://arxiv.org/abs/2504.05152)
• Keywords: generation
• Abstract: Automatically generating a complete 3D scene from a text description, a reference image, or both has significant applications in fields like virtual reality and gaming. However, current methods often generate low-quality textures and inconsistent 3D structures. This is especially true when extrapolating significantly beyond the field of view of the reference image. To address these challenges, we propose PanoDreamer, a novel framework for consistent, 3D scene generation with flexible text and image control. Our approach employs a large language model and a warp-refine pipeline, first generating an initial set of images and then compositing them into a 360-degree panorama. This panorama is then lifted into 3D to form an initial point cloud. We then use several approaches to generate additional images, from different viewpoints, that are consistent with the initial point cloud and expand/refine the initial point cloud. Given the resulting set of images, we utilize 3D Gaussian Splatting to create the final 3D scene, which can then be rendered from different viewpoints. Experiments demonstrate the effectiveness of PanoDreamer in generating high-quality, geometrically consistent 3D scenes.
• 摘要：从文本描述，参考图像或两者都在虚拟现实和游戏等字段中自动生成完整的3D场景。但是，当前方法通常会产生低质量的纹理和不一致的3D结构。当显着超出参考图像视野之外，尤其如此。为了应对这些挑战，我们提出了Panodreamer，这是一个具有灵活文本和图像控制的一致，3D场景生成的新颖框架。我们的方法采用大型语言模型和扭曲的管道，首先生成一组初始图像，然后将其合成为360度全景。然后将此全景提升为3D，形成初始点云。然后，我们使用几种方法从不同的角度生成其他图像，这些图像与初始点云相一致，并扩展/完善初始点云。鉴于所得的图像集，我们利用3D高斯脱落来创建最终的3D场景，然后可以从不同的角度渲染。实验证明了Panodreamer在产生高质量，几何一致的3D场景方面的有效性。

Title: Learning symmetries in datasets

• Authors: Veronica Sanz
• Subjects: cs.LG, hep-ph
• Abstract URL: https://arxiv.org/abs/2504.05174
• Pdf URL: https://arxiv.org/pdf/2504.05174
• Copy Paste: [[2504.05174]] Learning symmetries in datasets(https://arxiv.org/abs/2504.05174)
• Keywords: generative
• Abstract: We investigate how symmetries present in datasets affect the structure of the latent space learned by Variational Autoencoders (VAEs). By training VAEs on data originating from simple mechanical systems and particle collisions, we analyze the organization of the latent space through a relevance measure that identifies the most meaningful latent directions. We show that when symmetries or approximate symmetries are present, the VAE self-organizes its latent space, effectively compressing the data along a reduced number of latent variables. This behavior captures the intrinsic dimensionality determined by the symmetry constraints and reveals hidden relations among the features. Furthermore, we provide a theoretical analysis of a simple toy model, demonstrating how, under idealized conditions, the latent space aligns with the symmetry directions of the data manifold. We illustrate these findings with examples ranging from two-dimensional datasets with $O(2)$ symmetry to realistic datasets from electron-positron and proton-proton collisions. Our results highlight the potential of unsupervised generative models to expose underlying structures in data and offer a novel approach to symmetry discovery without explicit supervision.
• 摘要：我们研究了数据集中存在的对称性如何影响变异自动编码器（VAE）学到的潜在空间的结构。通过训练来自简单机械系统和粒子碰撞的数据的VAE，我们通过标识最有意义的潜在方向的相关度量来分析潜在空间的组织。我们表明，当存在对称性或近似对称性时，VAE会自组织其潜在空间，从而有效地沿减少的潜在变量压缩数据。这种行为捕获了由对称约束确定的内在维度，并揭示了特征之间的隐藏关系。此外，我们提供了一个简单的玩具模型的理论分析，该模型表明，在理想化的条件下，潜在空间如何与数据歧管的对称方向保持一致。我们用$ O（2）$对称性的二维数据集到来自Electron-Positron和Proton-Proton碰撞的现实数据集的示例来说明这些发现。我们的结果强调了无监督生成模型在数据中暴露基础结构的潜力，并提供了一种新颖的对称方法，而无需明确的监督。

Title: BRIDGES: Bridging Graph Modality and Large Language Models within EDA Tasks

• Authors: Wei Li, Yang Zou, Christopher Ellis, Ruben Purdy, Shawn Blanton, José M. F. Moura
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.05180
• Pdf URL: https://arxiv.org/pdf/2504.05180
• Copy Paste: [[2504.05180]] BRIDGES: Bridging Graph Modality and Large Language Models within EDA Tasks(https://arxiv.org/abs/2504.05180)
• Keywords: generation
• Abstract: While many EDA tasks already involve graph-based data, existing LLMs in EDA primarily either represent graphs as sequential text, or simply ignore graph-structured data that might be beneficial like dataflow graphs of RTL code. Recent studies have found that LLM performance suffers when graphs are represented as sequential text, and using additional graph information significantly boosts performance. To address these challenges, we introduce BRIDGES, a framework designed to incorporate graph modality into LLMs for EDA tasks. BRIDGES integrates an automated data generation workflow, a solution that combines graph modality with LLM, and a comprehensive evaluation suite. First, we establish an LLM-driven workflow to generate RTL and netlist-level data, converting them into dataflow and netlist graphs with function descriptions. This workflow yields a large-scale dataset comprising over 500,000 graph instances and more than 1.5 billion tokens. Second, we propose a lightweight cross-modal projector that encodes graph representations into text-compatible prompts, enabling LLMs to effectively utilize graph data without architectural modifications. Experimental results demonstrate 2x to 10x improvements across multiple tasks compared to text-only baselines, including accuracy in design retrieval, type prediction and perplexity in function description, with negligible computational overhead (<1% model weights increase and <30% additional runtime overhead). Even without additional LLM finetuning, our results outperform text-only by a large margin. We plan to release BRIDGES, including the dataset, models, and training flow.
• 摘要：尽管许多EDA任务已经涉及基于图的数据，但是EDA中的现有LLMS主要表示图形为顺序文本，或者只是忽略了图形结构的数据，这些数据可能像RTL代码的数据流图一样有益。最近的研究发现，当图形表示为顺序文本时，LLM的性能会受到影响，并且使用其他图形信息会显着提高性能。为了应对这些挑战，我们介绍了桥梁，该框架旨在将图形模式纳入LLMS以进行EDA任务。 Bridges集成了自动数据生成工作流程，将图形模式与LLM结合在一起的解决方案以及全面的评估套件。首先，我们建立了一个LLM驱动的工作流程，以生成RTL和NetList级数据，将它们转换为具有功能描述的数据流和NetList图。该工作流程产生一个大规模的数据集，其中包括超过500,000个图形实例和超过15亿个令牌。其次，我们提出了一个轻巧的跨模式投影仪，该投影仪将图形表示编码为文本兼容提示，从而使LLMS能够有效地利用图形数据而无需架构修改。实验结果表明，与仅文本基线相比，多个任务之间的两次改进为2倍至10倍，包括设计检索，类型预测和功能描述中的困惑的准确性，计算上可以忽略不计（<1％的模型权重增加和<30％的额外运行时开销）。即使没有额外的LLM登录，我们的结果也以很大的幅度优于文本。我们计划释放桥梁，包括数据集，模型和训练流。

Title: Federated Learning for Medical Image Classification: A Comprehensive Benchmark

• Authors: Zhekai Zhou, Guibo Luo, Mingzhi Chen, Zhenyu Weng, Yuesheng Zhu
• Subjects: cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2504.05238
• Pdf URL: https://arxiv.org/pdf/2504.05238
• Copy Paste: [[2504.05238]] Federated Learning for Medical Image Classification: A Comprehensive Benchmark(https://arxiv.org/abs/2504.05238)
• Keywords: generative
• Abstract: The federated learning paradigm is wellsuited for the field of medical image analysis, as it can effectively cope with machine learning on isolated multicenter data while protecting the privacy of participating parties. However, current research on optimization algorithms in federated learning often focuses on limited datasets and scenarios, primarily centered around natural images, with insufficient comparative experiments in medical contexts. In this work, we conduct a comprehensive evaluation of several state-of-the-art federated learning algorithms in the context of medical imaging. We conduct a fair comparison of classification models trained using various federated learning algorithms across multiple medical imaging datasets. Additionally, we evaluate system performance metrics, such as communication cost and computational efficiency, while considering different federated learning architectures. Our findings show that medical imaging datasets pose substantial challenges for current federated learning optimization algorithms. No single algorithm consistently delivers optimal performance across all medical federated learning scenarios, and many optimization algorithms may underperform when applied to these datasets. Our experiments provide a benchmark and guidance for future research and application of federated learning in medical imaging contexts. Furthermore, we propose an efficient and robust method that combines generative techniques using denoising diffusion probabilistic models with label smoothing to augment datasets, widely enhancing the performance of federated learning on classification tasks across various medical imaging datasets. Our code will be released on GitHub, offering a reliable and comprehensive benchmark for future federated learning studies in medical imaging.
• 摘要：联合学习范式可用于医学图像分析领域，因为它可以有效地应对孤立的多中心数据的机器学习，同时保护参与各方的隐私。但是，联合学习中对优化算法的当前研究通常集中在有限的数据集和方案上，主要围绕自然图像，而在医学环境中的比较实验不足。在这项工作中，我们对医学成像背景下的几种最新联邦学习算法进行了全面评估。我们对使用多个医学成像数据集的各种联合学习算法进行培训的分类模型进行了公平的比较。此外，我们评估了系统性能指标，例如通信成本和计算效率，同时考虑了不同的联合学习体系结构。我们的发现表明，医学成像数据集对当前联邦学习优化算法构成了重大挑战。没有任何单一算法始终在所有医疗联合学习方案中提供最佳性能，并且在应用于这些数据集时，许多优化算法的表现可能不佳。我们的实验为在医学成像环境中联合学习的未来研究和应用提供了基准和指导。此外，我们提出了一种高效且可靠的方法，该方法使用deno的扩散概率模型与标签平滑度结合生成技术来增强数据集，从而广泛增强了在各种医学成像数据集中的分类任务上联合学习的性能。我们的代码将在Github上发布，为将来的医学成像学习研究提供可靠且全面的基准。

Title: Explaining Low Perception Model Competency with High-Competency Counterfactuals

• Authors: Sara Pohland, Claire Tomlin
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.05254
• Pdf URL: https://arxiv.org/pdf/2504.05254
• Copy Paste: [[2504.05254]] Explaining Low Perception Model Competency with High-Competency Counterfactuals(https://arxiv.org/abs/2504.05254)
• Keywords: generation
• Abstract: There exist many methods to explain how an image classification model generates its decision, but very little work has explored methods to explain why a classifier might lack confidence in its prediction. As there are various reasons the classifier might lose confidence, it would be valuable for this model to not only indicate its level of uncertainty but also explain why it is uncertain. Counterfactual images have been used to visualize changes that could be made to an image to generate a different classification decision. In this work, we explore the use of counterfactuals to offer an explanation for low model competency--a generalized form of predictive uncertainty that measures confidence. Toward this end, we develop five novel methods to generate high-competency counterfactual images, namely Image Gradient Descent (IGD), Feature Gradient Descent (FGD), Autoencoder Reconstruction (Reco), Latent Gradient Descent (LGD), and Latent Nearest Neighbors (LNN). We evaluate these methods across two unique datasets containing images with six known causes for low model competency and find Reco, LGD, and LNN to be the most promising methods for counterfactual generation. We further evaluate how these three methods can be utilized by pre-trained Multimodal Large Language Models (MLLMs) to generate language explanations for low model competency. We find that the inclusion of a counterfactual image in the language model query greatly increases the ability of the model to generate an accurate explanation for the cause of low model competency, thus demonstrating the utility of counterfactual images in explaining low perception model competency.
• 摘要：存在许多方法来解释图像分类模型如何产生其决策，但是很少的工作探讨了分类器为什么对其预测缺乏信心的方法。由于分类器可能会失去信心的原因有多种原因，因此该模型不仅表明其不确定性水平，而且还解释了为什么它不确定是有价值的。反事实图像已用于可视化可以对图像进行的更改，以生成不同的分类决策。在这项工作中，我们探讨了反事实的使用来为低模型能力提供解释 - 一种衡量置信度的预测不确定性形式。为此，我们开发了五种新型方法来生成高竞争性反事实图像，即图像梯度下降（IGD），特征梯度下降（FGD），自动编码器重建（RECO），潜在梯度下降（LGD）和潜在的邻居（LNN）。我们在两个独特的数据集中评估了这些方法，这些数据集包含具有六个已知原因的图像低模型能力，并发现Reco，LGD和LNN是反事实生成的最有希望的方法。我们进一步评估了如何通过预训练的多模式模型（MLLM）来利用这三种方法来生成低模型能力的语言解释。我们发现，在语言模型中包含反事实图像可以大大提高模型为低模型能力的原因产生准确的解释的能力，从而证明了反事实图像在解释低感知模型能力中的实用性。

Title: From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Models

• Authors: German Barquero, Nadine Bertsch, Manojkumar Marramreddy, Carlos Chacón, Filippo Arcadu, Ferran Rigual, Nicky Sijia He, Cristina Palmero, Sergio Escalera, Yuting Ye, Robin Kips
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.05265
• Pdf URL: https://arxiv.org/pdf/2504.05265
• Copy Paste: [[2504.05265]] From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Models(https://arxiv.org/abs/2504.05265)
• Keywords: generation
• Abstract: In extended reality (XR), generating full-body motion of the users is important to understand their actions, drive their virtual avatars for social interaction, and convey a realistic sense of presence. While prior works focused on spatially sparse and always-on input signals from motion controllers, many XR applications opt for vision-based hand tracking for reduced user friction and better immersion. Compared to controllers, hand tracking signals are less accurate and can even be missing for an extended period of time. To handle such unreliable inputs, we present Rolling Prediction Model (RPM), an online and real-time approach that generates smooth full-body motion from temporally and spatially sparse input signals. Our model generates 1) accurate motion that matches the inputs (i.e., tracking mode) and 2) plausible motion when inputs are missing (i.e., synthesis mode). More importantly, RPM generates seamless transitions from tracking to synthesis, and vice versa. To demonstrate the practical importance of handling noisy and missing inputs, we present GORP, the first dataset of realistic sparse inputs from a commercial virtual reality (VR) headset with paired high quality body motion ground truth. GORP provides >14 hours of VR gameplay data from 28 people using motion controllers (spatially sparse) and hand tracking (spatially and temporally sparse). We benchmark RPM against the state of the art on both synthetic data and GORP to highlight how we can bridge the gap for real-world applications with a realistic dataset and by handling unreliable input signals. Our code, pretrained models, and GORP dataset are available in the project webpage.
• 摘要：在扩展现实（XR）中，产生用户的全身运动对于理解自己的行为，推动其虚拟化身进行社交互动并传达现实的存在感很重要。虽然先前的作品着重于运动控制器的空间稀疏和始终连接输入信号，但许多XR应用程序选择基于视觉的手动跟踪，以减少用户摩擦和更好的沉浸感。与控制器相比，手动跟踪信号的准确性较低，甚至可能会在很长的时间内丢失。为了处理此类不可靠的输入，我们提出了滚动预测模型（RPM），这是一种在线和实时方法，可从时间和空间上稀疏的输入信号产生平滑的全身运动。我们的模型生成1）与输入（即跟踪模式）匹配的准确运动和2）当缺失输入时（即合成模式）时合理的运动。更重要的是，RPM生成从跟踪到合成的无缝过渡，反之亦然。为了证明处理嘈杂和缺少输入的实际重要性，我们提出了大戈普（Gorp），这是来自商业虚拟现实（VR）耳机的现实稀疏输入的第一个数据集，并配对了高质量的身体运动地面真相。 GORP使用运动控制器（空间稀疏）和手动跟踪（在空间和时间上稀疏）提供了来自28个人的VR游戏数据> 14小时。我们在综合数据和gorp上基于rpm对抗最新技术，以突出显示如何使用现实数据集和处理不可靠的输入信号来弥合实际应用程序的差距。我们的代码，预处理的模型和GORP数据集可在项目网页中找到。

Title: One-Minute Video Generation with Test-Time Training

• Authors: Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, Tatsunori Hashimoto, Sanmi Koyejo, Yejin Choi, Yu Sun, Xiaolong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.05298
• Pdf URL: https://arxiv.org/pdf/2504.05298
• Copy Paste: [[2504.05298]] One-Minute Video Generation with Test-Time Training(https://arxiv.org/abs/2504.05298)
• Keywords: generation
• Abstract: Transformers today still struggle to generate one-minute videos because self-attention layers are inefficient for long context. Alternatives such as Mamba layers struggle with complex multi-scene stories because their hidden states are less expressive. We experiment with Test-Time Training (TTT) layers, whose hidden states themselves can be neural networks, therefore more expressive. Adding TTT layers into a pre-trained Transformer enables it to generate one-minute videos from text storyboards. For proof of concept, we curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers generate much more coherent videos that tell complex stories, leading by 34 Elo points in a human evaluation of 100 videos per method. Although promising, results still contain artifacts, likely due to the limited capability of the pre-trained 5B model. The efficiency of our implementation can also be improved. We have only experimented with one-minute videos due to resource constraints, but the approach can be extended to longer videos and more complex stories. Sample videos, code and annotations are available at: this https URL
• 摘要：如今，变形金刚仍然难以生成一分钟的视频，因为自我发挥的层面对于长篇小说效率低下。诸如Mamba层之类的替代方案与复杂的多场景故事斗争，因为它们的隐藏状态不那么表现力。我们尝试测试时间培训（TTT）层，其隐藏状态本身可以是神经网络，因此更具表现力。将TTT图层添加到预训练的变压器中，使其能够从文本故事板中生成一分钟的视频。为了获得概念证明，我们策划了一个基于Tom和Jerry卡通的数据集。与Mamba〜2，门控的Deltanet和滑动窗口注意的层相比，TTT层产生了更连贯的视频，这些视频讲述了复杂的故事，在人类对每种方法100个视频的评估中，以34个ELO点为导致。尽管很有希望，但结果仍然包含伪影，这可能是由于预先训练的5B模型的能力有限。我们实施的效率也可以提高。由于资源限制，我们仅尝试了一分钟的视频，但是该方法可以扩展到更长的视频和更复杂的故事。示例视频，代码和注释可在以下网址提供：此HTTPS URL

Title: Dimension-Free Convergence of Diffusion Models for Approximate Gaussian Mixtures

• Authors: Gen Li, Changxiao Cai, Yuting Wei
• Subjects: cs.LG, math.NA, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2504.05300
• Pdf URL: https://arxiv.org/pdf/2504.05300
• Copy Paste: [[2504.05300]] Dimension-Free Convergence of Diffusion Models for Approximate Gaussian Mixtures(https://arxiv.org/abs/2504.05300)
• Keywords: generation, generative
• Abstract: Diffusion models are distinguished by their exceptional generative performance, particularly in producing high-quality samples through iterative denoising. While current theory suggests that the number of denoising steps required for accurate sample generation should scale linearly with data dimension, this does not reflect the practical efficiency of widely used algorithms like Denoising Diffusion Probabilistic Models (DDPMs). This paper investigates the effectiveness of diffusion models in sampling from complex high-dimensional distributions that can be well-approximated by Gaussian Mixture Models (GMMs). For these distributions, our main result shows that DDPM takes at most $\widetilde{O}(1/\varepsilon)$ iterations to attain an $\varepsilon$-accurate distribution in total variation (TV) distance, independent of both the ambient dimension $d$ and the number of components $K$, up to logarithmic factors. Furthermore, this result remains robust to score estimation errors. These findings highlight the remarkable effectiveness of diffusion models in high-dimensional settings given the universal approximation capability of GMMs, and provide theoretical insights into their practical success.
• 摘要：扩散模型以其特殊的生成性能为特征，尤其是在通过迭代降解生成高质量的样品时。虽然当前的理论表明，准确样品产生所需的剥离步骤的数量应随数据维度线性扩展，但这并不能反映广泛使用算法（例如Deoing denoing扩散概率模型（DDPMS））的实践效率。本文研究了扩散模型在从复杂的高维分布中取样中的有效性，这些分布可以通过高斯混合模型（GMM）充分x氧化。对于这些分布，我们的主要结果表明，ddpm最多取决于$ \ widetilde {o}（1/\ varepsilon）$迭代，以达到$ \ varepsilon $ - accurate分布（电视）距离（电视）距离（电视）距离，与环境尺寸$ d $ d $ d $ d $ d $ d $ d $ d $ k $ k $ k $ k $ k $，logry to logrith castic factor to logrith fives faction。此外，该结果仍然可靠地得分估计误差。这些发现突出了GMM的普遍近似能力，扩散模型在高维设置中的出色效率，并为其实践成功提供了理论上的见解。

Title: Gaussian Mixture Flow Matching Models

• Authors: Hansheng Chen, Kai Zhang, Hao Tan, Zexiang Xu, Fujun Luan, Leonidas Guibas, Gordon Wetzstein, Sai Bi
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2504.05304
• Pdf URL: https://arxiv.org/pdf/2504.05304
• Copy Paste: [[2504.05304]] Gaussian Mixture Flow Matching Models(https://arxiv.org/abs/2504.05304)
• Keywords: generation
• Abstract: Diffusion models approximate the denoising distribution as a Gaussian and predict its mean, whereas flow matching models reparameterize the Gaussian mean as flow velocity. However, they underperform in few-step sampling due to discretization error and tend to produce over-saturated colors under classifier-free guidance (CFG). To address these limitations, we propose a novel Gaussian mixture flow matching (GMFlow) model: instead of predicting the mean, GMFlow predicts dynamic Gaussian mixture (GM) parameters to capture a multi-modal flow velocity distribution, which can be learned with a KL divergence loss. We demonstrate that GMFlow generalizes previous diffusion and flow matching models where a single Gaussian is learned with an $L_2$ denoising loss. For inference, we derive GM-SDE/ODE solvers that leverage analytic denoising distributions and velocity fields for precise few-step sampling. Furthermore, we introduce a novel probabilistic guidance scheme that mitigates the over-saturation issues of CFG and improves image generation quality. Extensive experiments demonstrate that GMFlow consistently outperforms flow matching baselines in generation quality, achieving a Precision of 0.942 with only 6 sampling steps on ImageNet 256$\times$256.
• 摘要：扩散模型将其作为高斯的脱氧分布近似，并预测其平均值，而流量匹配模型将高斯平均值重新分配为流速。但是，由于离散误差，它们的表现不佳，在几步中采样，并且在无分类器指导（CFG）下倾向于产生过饱和的颜色。为了解决这些局限性，我们提出了一种新型的高斯混合物流量匹配（GMFLOW）模型：GMFlow预测动态高斯混合物（GM）参数，而不是预测平均值，以捕获多模式流量速度分布，可以通过KL差异损失来学习。我们证明了GMFlow概括了以前的扩散和流匹配模型，在这些模型中，以$ l_2 $ denoising损失学习了单个高斯。为了推断，我们得出了GM-SDE/ODE求解器，该求解器利用分析性降解分布和速度字段进行精确的少量抽样。此外，我们介绍了一种新颖的概率指导方案，该方案减轻了CFG的过度饱和问题并提高了图像生成质量。广泛的实验表明，GMFlow在发电质量方面始终优于流量匹配基线，在Imagenet 256 $ \ times $ 256上只有6个采样步骤，达到0.942的精度。

Title: URECA: Unique Region Caption Anything

• Authors: Sangbeom Lim, Junwan Kim, Heeji Yoon, Jaewoo Jung, Seungryong Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.05305
• Pdf URL: https://arxiv.org/pdf/2504.05305
• Copy Paste: [[2504.05305]] URECA: Unique Region Caption Anything(https://arxiv.org/abs/2504.05305)
• Keywords: generation
• Abstract: Region-level captioning aims to generate natural language descriptions for specific image regions while highlighting their distinguishing features. However, existing methods struggle to produce unique captions across multi-granularity, limiting their real-world applicability. To address the need for detailed region-level understanding, we introduce URECA dataset, a large-scale dataset tailored for multi-granularity region captioning. Unlike prior datasets that focus primarily on salient objects, URECA dataset ensures a unique and consistent mapping between regions and captions by incorporating a diverse set of objects, parts, and background elements. Central to this is a stage-wise data curation pipeline, where each stage incrementally refines region selection and caption generation. By leveraging Multimodal Large Language Models (MLLMs) at each stage, our pipeline produces distinctive and contextually grounded captions with improved accuracy and semantic diversity. Building upon this dataset, we present URECA, a novel captioning model designed to effectively encode multi-granularity regions. URECA maintains essential spatial properties such as position and shape through simple yet impactful modifications to existing MLLMs, enabling fine-grained and semantically rich region descriptions. Our approach introduces dynamic mask modeling and a high-resolution mask encoder to enhance caption uniqueness. Experiments show that URECA achieves state-of-the-art performance on URECA dataset and generalizes well to existing region-level captioning benchmarks.
• 摘要：区域级字幕旨在为特定图像区域生成自然语言描述，同时突出其区别特征。但是，现有的方法难以在多范围内产生独特的标题，从而限制了其现实世界的适用性。为了满足详细的区域级别理解的需求，我们介绍了UECA数据集，这是一个针对多晶体区域字幕的大规模数据集。与主要关注显着对象的先前数据集不同，Ureca数据集通过合并各种对象，零件和背景元素来确保区域和字幕之间的独特且一致的映射。核心是阶段的数据策划管道，每个阶段都会逐步完善区域选择和字幕生成。通过在每个阶段利用多模式的大语言模型（MLLM），我们的管道以提高的准确性和语义多样性产生独特的和上下文的字幕。在此数据集的基础上，我们提出了乌雷卡（Ureca），这是一种新颖的字幕模型，旨在有效编码多晶状体区域。尿素通过对现有MLLM的简单而有影响力的修改来维持基本的空间特性，例如位置和形状，从而实现了精细元素和语义丰富的区域描述。我们的方法引入了动态面具建模和高分辨率掩模编码器，以增强字幕唯一性。实验表明，乌雷卡（Ureca）在乌雷卡（Ureca）数据集上实现了最先进的性能，并将其概括为现有的区域级字幕基准。

Title: CREA: A Collaborative Multi-Agent Framework for Creative Content Generation with Diffusion Models

• Authors: Kavana Venkatesh, Connor Dunlop, Pinar Yanardag
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.05306
• Pdf URL: https://arxiv.org/pdf/2504.05306
• Copy Paste: [[2504.05306]] CREA: A Collaborative Multi-Agent Framework for Creative Content Generation with Diffusion Models(https://arxiv.org/abs/2504.05306)
• Keywords: generation, generative
• Abstract: Creativity in AI imagery remains a fundamental challenge, requiring not only the generation of visually compelling content but also the capacity to add novel, expressive, and artistically rich transformations to images. Unlike conventional editing tasks that rely on direct prompt-based modifications, creative image editing demands an autonomous, iterative approach that balances originality, coherence, and artistic intent. To address this, we introduce CREA, a novel multi-agent collaborative framework that mimics the human creative process. Our framework leverages a team of specialized AI agents who dynamically collaborate to conceptualize, generate, critique, and enhance images. Through extensive qualitative and quantitative evaluations, we demonstrate that CREA significantly outperforms state-of-the-art methods in diversity, semantic alignment, and creative transformation. By structuring creativity as a dynamic, agentic process, CREA redefines the intersection of AI and art, paving the way for autonomous AI-driven artistic exploration, generative design, and human-AI co-creation. To the best of our knowledge, this is the first work to introduce the task of creative editing.
• 摘要：AI图像中的创造力仍然是一个根本的挑战，不仅需要产生视觉上引人入胜的内容，而且还需要在图像中增加新颖，表现力和艺术丰富的转换的能力。与依赖基于直接及时的直接修改的常规编辑任务不同，创意图像编辑需要一种自主，迭代的方法，以平衡独创性，连贯性和艺术意图。为了解决这个问题，我们介绍了CREA，这是一个模仿人类创作过程的新型多代理协作框架。我们的框架利用了一个专业的AI代理团队，他们动态合作以概念化，生成，批评和增强图像。通过广泛的定性和定量评估，我们证明了CREA在多样性，语义一致性和创造性转型方面显着优于最先进的方法。通过将创造力构造为动态的，代理过程，CREA重新定义了人工智能和艺术的交集，为自主的AI驱动的艺术探索，生成设计和人类ai共同创造铺平了道路。据我们所知，这是介绍创意编辑任务的第一部作品。