2025-02-04

Title: Leveraging Large Language Models to Enhance Machine Learning Interpretability and Predictive Performance: A Case Study on Emergency Department Returns for Mental Health Patients

• Authors: Abdulaziz Ahmed, Mohammad Saleem, Mohammed Alzeen, Badari Birur, Rachel E Fargason, Bradley G Burk, Hannah Rose Harkins, Ahmed Alhassan, Mohammed Ali Al-Garadi
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.00025
• Pdf URL: https://arxiv.org/pdf/2502.00025
• Copy Paste: [[2502.00025]] Leveraging Large Language Models to Enhance Machine Learning Interpretability and Predictive Performance: A Case Study on Emergency Department Returns for Mental Health Patients(https://arxiv.org/abs/2502.00025)
• Keywords: generation
• Abstract: Objective: To evaluate whether integrating large language models (LLMs) with traditional machine learning approaches improves both the predictive accuracy and clinical interpretability of ED mental health returns risk models. Methods: This retrospective cohort study analyzed 42,464 ED visits for 27,904 unique mental health patients at an Academic Medical Center in the deep South of the United States between January 2018 and December 2022. Main Outcomes and Measures: Two primary outcomes were evaluated: (1) 30 days ED return prediction accuracy and (2) model interpretability through a novel retrieval-augmented generation (RAG) framework integrating SHAP (SHapley Additive exPlanations) values with contextual clinical knowledge. Results: The proposed machine learning interpretability framework, leveraging LLM, achieved 99% accuracy in translating complex model predictions into clinically relevant explanations. Integration of LLM-extracted features enhanced predictive performance, improving the XGBoost model area under the curve (AUC) from 0.73 to 0.76. The LLM-based feature extraction using 10-shot learning significantly outperformed traditional approaches, achieving an accuracy of 0.882 and an F1 score of 0.86 for chief complaint classification (compared to conventional methods with an accuracy range of 0.59 to 0.63) and demonstrating accuracy values ranging from 0.65 to 0.93 across multiple SDoH categories, underscoring its robust performance in extracting features from clinical notes. Conclusions and Relevance: Integrating LLMs with traditional machine learning models yielded modest but consistent improvements in ED return prediction accuracy while substantially enhancing model interpretability through automated, clinically relevant explanations. This approach offers a framework for translating complex predictive analytics into actionable clinical insights.
• 摘要：目的：评估将大型语言模型 (LLM) 与传统机器学习方法相结合是否能提高 ED 心理健康回报风险模型的预测准确性和临床可解释性。方法：这项回顾性队列研究分析了 2018 年 1 月至 2022 年 12 月期间美国南部某学术医学中心 27,904 名独特心理健康患者的 42,464 次 ED 就诊情况。主要结果和措施：评估了两个主要结果：(1) 30 天 ED 返回预测准确性和 (2) 通过将 SHAP (SHapley Additive exPlanations) 值与上下文临床知识相结合的新型检索增强生成 (RAG) 框架实现模型可解释性。结果：提出的机器学习可解释性框架利用 LLM，在将复杂模型预测转化为临床相关解释方面实现了 99% 的准确率。整合 LLM 提取的特征可增强预测性能，将 XGBoost 模型曲线下面积 (AUC) 从 0.73 提高到 0.76。使用 10 次学习的基于 LLM 的特征提取显著优于传统方法，对主诉分类的准确率达到 0.882，F1 得分为 0.86（传统方法的准确率范围为 0.59 到 0.63），在多个 SDoH 类别中的准确率值范围为 0.65 到 0.93，凸显了其在从临床记录中提取特征方面的强大性能。结论和相关性：将 LLM 与传统机器学习模型相结合，可适度但持续地提高 ED 返回预测准确率，同时通过自动化、临床相关的解释显着提高模型的可解释性。这种方法提供了一个将复杂的预测分析转化为可操作的临床见解的框架。

Title: AIN: The Arabic INclusive Large Multimodal Model

• Authors: Ahmed Heakl, Sara Ghaboura, Omkar Thawkar, Fahad Shahbaz Khan, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan
• Subjects: cs.CV, cs.AI, cs.CL, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00094
• Pdf URL: https://arxiv.org/pdf/2502.00094
• Copy Paste: [[2502.00094]] AIN: The Arabic INclusive Large Multimodal Model(https://arxiv.org/abs/2502.00094)
• Keywords: generative
• Abstract: Amid the swift progress of large language models (LLMs) and their evolution into large multimodal models (LMMs), significant strides have been made in high-resource languages such as English and Chinese. While Arabic LLMs have seen notable progress, Arabic LMMs remain largely unexplored, often narrowly focusing on a few specific aspects of the language and visual understanding. To bridge this gap, we introduce AIN-the Arabic Inclusive Multimodal Model-designed to excel across diverse domains. AIN is an English-Arabic bilingual LMM designed to excel in English and Arabic, leveraging carefully constructed 3.6 million high-quality Arabic-English multimodal data samples. AIN demonstrates state-of-the-art Arabic performance, while also possessing strong English-language visual capabilities. On the recent CAMEL-Bench benchmark comprising 38 sub-domains including, multi-image understanding, complex visual perception, handwritten document understanding, video understanding, medical imaging, plant diseases, and remote sensing-based land use understanding, our AIN demonstrates strong performance with the 7B model outperforming GPT-4o by an absolute gain of 3.4% averaged over eight domains and 38 sub-domains. AIN's superior capabilities position it as a significant step toward empowering Arabic speakers with advanced multimodal generative AI tools across diverse applications.
• 摘要：随着大型语言模型 (LLM) 的快速发展及其向大型多模态模型 (LMM) 的演变，英语和中文等资源丰富的语言取得了重大进展。虽然阿拉伯语 LLM 取得了显著进展，但阿拉伯语 LMM 仍未得到充分开发，通常只关注语言和视觉理解的几个特定方面。为了弥补这一差距，我们推出了 AIN（阿拉伯语包容性多模态模型），旨在在不同领域取得卓越成就。AIN 是一种英语-阿拉伯语双语 LMM，旨在在英语和阿拉伯语方面表现出色，利用精心构建的 360 万个高质量阿拉伯语-英语多模态数据样本。AIN 展示了最先进的阿拉伯语性能，同时还拥有强大的英语视觉能力。在最近的 CAMEL-Bench 基准测试中，该基准测试包含 38 个子域，包括多图像理解、复杂视觉感知、手写文档理解、视频理解、医学成像、植物疾病和基于遥感的土地使用理解，我们的 AIN 表现出色，7B 模型的表现优于 GPT-4o，在 8 个域和 38 个子域中平均绝对增益为 3.4%。AIN 的卓越功能使其成为向阿拉伯语使用者提供适用于各种应用的高级多模式生成 AI 工具的重要一步。

Title: ProtoSnap: Prototype Alignment for Cuneiform Signs

• Authors: Rachel Mikulinsky, Morris Alper, Shai Gordin, Enrique Jiménez, Yoram Cohen, Hadar Averbuch-Elor
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00129
• Pdf URL: https://arxiv.org/pdf/2502.00129
• Copy Paste: [[2502.00129]] ProtoSnap: Prototype Alignment for Cuneiform Signs(https://arxiv.org/abs/2502.00129)
• Keywords: generative
• Abstract: The cuneiform writing system served as the medium for transmitting knowledge in the ancient Near East for a period of over three thousand years. Cuneiform signs have a complex internal structure which is the subject of expert paleographic analysis, as variations in sign shapes bear witness to historical developments and transmission of writing and culture over time. However, prior automated techniques mostly treat sign types as categorical and do not explicitly model their highly varied internal configurations. In this work, we present an unsupervised approach for recovering the fine-grained internal configuration of cuneiform signs by leveraging powerful generative models and the appearance and structure of prototype font images as priors. Our approach, ProtoSnap, enforces structural consistency on matches found with deep image features to estimate the diverse configurations of cuneiform characters, snapping a skeleton-based template to photographed cuneiform signs. We provide a new benchmark of expert annotations and evaluate our method on this task. Our evaluation shows that our approach succeeds in aligning prototype skeletons to a wide variety of cuneiform signs. Moreover, we show that conditioning on structures produced by our method allows for generating synthetic data with correct structural configurations, significantly boosting the performance of cuneiform sign recognition beyond existing techniques, in particular over rare signs. Our code, data, and trained models are available at the project page: this https URL
• 摘要：楔形文字系统是古代近东地区传播知识的媒介，已有三千多年的历史。楔形文字符号具有复杂的内部结构，这是专家古文字学分析的主题，因为符号形状的变化见证了历史发展以及文字和文化随着时间的推移而传播的过程。然而，以前的自动化技术大多将符号类型视为分类，并没有明确地模拟它们高度多样化的内部配置。在这项工作中，我们提出了一种无监督方法，通过利用强大的生成模型和原型字体图像的外观和结构作为先验，恢复楔形文字符号的细粒度内部配置。我们的方法 ProtoSnap 强制使用深度图像特征匹配的结构一致性来估计楔形文字字符的不同配置，将基于骨架的模板捕捉到拍摄的楔形文字符号上。我们提供了专家注释的新基准，并评估了我们在此任务上的方法。我们的评估表明，我们的方法成功地将原型骨架与各种各样的楔形文字符号对齐。此外，我们表明，通过我们的方法生成的结构进行条件化可以生成具有正确结构配置的合成数据，从而大大提高楔形符号识别的性能，超越现有技术，尤其是对罕见符号的识别。我们的代码、数据和经过训练的模型可在项目页面上找到：此 https URL

Title: EcoWeedNet: A Lightweight and Automated Weed Detection Method for Sustainable Next-Generation Agricultural Consumer Electronics

• Authors: Omar H. Khater, Abdul Jabbar Siddiqui, M. Shamim Hossain
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00205
• Pdf URL: https://arxiv.org/pdf/2502.00205
• Copy Paste: [[2502.00205]] EcoWeedNet: A Lightweight and Automated Weed Detection Method for Sustainable Next-Generation Agricultural Consumer Electronics(https://arxiv.org/abs/2502.00205)
• Keywords: generation
• Abstract: Sustainable agriculture plays a crucial role in ensuring world food security for consumers. A critical challenge faced by sustainable precision agriculture is weed growth, as weeds share essential resources with the crops, such as water, soil nutrients, and sunlight, which notably affect crop yields. The traditional methods employed to combat weeds include the usage of chemical herbicides and manual weed removal methods. However, these could damage the environment and pose health hazards. The adoption of automated computer vision technologies and ground agricultural consumer electronic vehicles in precision agriculture offers sustainable, low-carbon solutions. However, prior works suffer from issues such as low accuracy and precision and high computational expense. This work proposes EcoWeedNet, a novel model with enhanced weed detection performance without adding significant computational complexity, aligning with the goals of low-carbon agricultural practices. Additionally, our model is lightweight and optimal for deployment on ground-based consumer electronic agricultural vehicles and robots. The effectiveness of the proposed model is demonstrated through comprehensive experiments on the CottonWeedDet12 benchmark dataset reflecting real-world scenarios. EcoWeedNet achieves performance close to that of large models yet with much fewer parameters. (approximately 4.21% of the parameters and 6.59% of the GFLOPs of YOLOv4). This work contributes effectively to the development of automated weed detection methods for next-generation agricultural consumer electronics featuring lower energy consumption and lower carbon footprint. This work paves the way forward for sustainable agricultural consumer technologies.
• 摘要：可持续农业在确保世界消费者粮食安全方面发挥着至关重要的作用。可持续精准农业面临的一个关键挑战是杂草生长，因为杂草与农作物共享水、土壤养分和阳光等基本资源，这些资源对农作物产量有显著影响。传统的除草方法包括使用化学除草剂和人工除草方法。然而，这些方法可能会破坏环境并造成健康危害。在精准农业中采用自动化计算机视觉技术和地面农业消费电子车辆提供了可持续的低碳解决方案。然而，先前的研究存在准确度和精度低、计算成本高等问题。这项研究提出了 EcoWeedNet，这是一种新型模型，它增强了杂草检测性能，而不会增加显著的计算复杂度，符合低碳农业实践的目标。此外，我们的模型重量轻，最适合部署在地面消费电子农业车辆和机器人上。通过对反映真实场景的 CottonWeedDet12 基准数据集进行全面实验，证明了所提模型的有效性。EcoWeedNet 的性能接近大型模型，但参数要少得多。 （约占 YOLOv4 的 4.21% 参数和 6.59% GFLOP）。这项工作为开发具有更低能耗和更低碳足迹的下一代农业消费电子产品的自动杂草检测方法做出了有效贡献。这项工作为可持续农业消费技术铺平了道路。

Title: Beyond Limited Data: Self-play LLM Theorem Provers with Iterative Conjecturing and Proving

• Authors: Kefan Dong, Tengyu Ma
• Subjects: cs.LG, cs.AI, cs.LO
• Abstract URL: https://arxiv.org/abs/2502.00212
• Pdf URL: https://arxiv.org/pdf/2502.00212
• Copy Paste: [[2502.00212]] Beyond Limited Data: Self-play LLM Theorem Provers with Iterative Conjecturing and Proving(https://arxiv.org/abs/2502.00212)
• Keywords: generation
• Abstract: A fundamental challenge in formal theorem proving by LLMs is the lack of high-quality training data. Although reinforcement learning or expert iteration partially mitigates this issue by alternating between LLM generating proofs and finetuning them on correctly generated ones, performance quickly plateaus due to the scarcity of correct proofs (sparse rewards). To keep improving the models with limited data, we draw inspiration from mathematicians, who continuously develop new results, partly by proposing novel conjectures or exercises (which are often variants of known results) and attempting to solve them. We design the Self-play Theorem Prover (STP) that simultaneously takes on two roles, conjecturer and prover, each providing training signals to the other. The conjecturer is trained iteratively on previously generated conjectures that are barely provable by the current prover, which incentivizes it to generate increasingly challenging conjectures over time. The prover attempts to prove the conjectures with standard expert iteration. We evaluate STP with both Lean and Isabelle formal versifiers. With 19.8 billion tokens generated during the training in Lean, STP proves 26.3% of the statements in the LeanWorkbook dataset, doubling the previous best result of 13.2% achieved through expert iteration. The final model achieves state-of-the-art performance among whole-proof generation methods on miniF2F-test (61.1%, pass@3200), Proofnet-test (23.1%, pass@3200) and PutnamBench (8/644, pass@64).
• 摘要：LLM 形式化定理证明的一个基本挑战是缺乏高质量的训练数据。虽然强化学习或专家迭代通过交替使用 LLM 生成证明并在正确生成的证明上进行微调部分缓解了这个问题，但由于正确证明的稀缺（稀疏奖励），性能很快就会停滞不前。为了在有限的数据下不断改进模型，我们从数学家那里汲取灵感，他们不断开发新的结果，部分是通过提出新的猜想或练习（通常是已知结果的变体）并尝试解决它们。我们设计了自玩定理证明器 (STP)，它同时承担两个角色，猜想者和证明者，每个角色都为对方提供训练信号。猜想者在当前证明者几乎无法证明的先前生成的猜想上进行迭代训练，这激励它随着时间的推移生成越来越具有挑战性的猜想。证明者尝试使用标准专家迭代来证明猜想。我们使用 Lean 和 Isabelle 形式化验证器对 STP 进行评估。在 Lean 的训练过程中，STP 生成了 198 亿个 token，验证了 LeanWorkbook 数据集中 26.3% 的语句，是之前通过专家迭代获得的最佳结果 13.2% 的两倍。最终模型在 miniF2F-test（61.1%，pass@3200）、Proofnet-test（23.1%，pass@3200）和 PutnamBench（8/644，pass@64）上实现了整体证明生成方法中的最佳性能。

Title: Should You Use Your Large Language Model to Explore or Exploit?

• Authors: Keegan Harris, Aleksandrs Slivkins
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00225
• Pdf URL: https://arxiv.org/pdf/2502.00225
• Copy Paste: [[2502.00225]] Should You Use Your Large Language Model to Explore or Exploit?(https://arxiv.org/abs/2502.00225)
• Keywords: generation
• Abstract: We evaluate the ability of the current generation of large language models (LLMs) to help a decision-making agent facing an exploration-exploitation tradeoff. We use LLMs to explore and exploit in silos in various (contextual) bandit tasks. We find that while the current LLMs often struggle to exploit, in-context mitigations may be used to substantially improve performance for small-scale tasks. However even then, LLMs perform worse than a simple linear regression. On the other hand, we find that LLMs do help at exploring large action spaces with inherent semantics, by suggesting suitable candidates to explore.
• 摘要：我们评估了当前一代大型语言模型 (LLM) 帮助面临探索-利用权衡的决策代理的能力。我们使用 LLM 在各种（上下文）老虎机任务中以孤岛形式进行探索和利用。我们发现，虽然当前的 LLM 通常难以利用，但上下文缓解措施可用于大幅提高小规模任务的性能。然而即便如此，LLM 的表现也比简单的线性回归差。另一方面，我们发现 LLM 确实有助于探索具有固有语义的大型动作空间，因为它会建议合适的候选对象进行探索。

Title: Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms

• Authors: Yinuo Ren, Haoxuan Chen, Yuchen Zhu, Wei Guo, Yongxin Chen, Grant M. Rotskoff, Molei Tao, Lexing Ying
• Subjects: cs.LG, cs.CV, math.NA, physics.comp-ph, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.00234
• Pdf URL: https://arxiv.org/pdf/2502.00234
• Copy Paste: [[2502.00234]] Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms(https://arxiv.org/abs/2502.00234)
• Keywords: generation, generative
• Abstract: Discrete diffusion models have emerged as a powerful generative modeling framework for discrete data with successful applications spanning from text generation to image synthesis. However, their deployment faces challenges due to the high dimensionality of the state space, necessitating the development of efficient inference algorithms. Current inference approaches mainly fall into two categories: exact simulation and approximate methods such as $\tau$-leaping. While exact methods suffer from unpredictable inference time and redundant function evaluations, $\tau$-leaping is limited by its first-order accuracy. In this work, we advance the latter category by tailoring the first extension of high-order numerical inference schemes to discrete diffusion models, enabling larger step sizes while reducing error. We rigorously analyze the proposed schemes and establish the second-order accuracy of the $\theta$-trapezoidal method in KL divergence. Empirical evaluations on GPT-2 level text and ImageNet-level image generation tasks demonstrate that our method achieves superior sample quality compared to existing approaches under equivalent computational constraints.
• 摘要：离散扩散模型已成为离散数据的强大生成建模框架，其成功应用范围从文本生成到图像合成。然而，由于状态空间的高维性，它们的部署面临挑战，因此需要开发高效的推理算法。当前的推理方法主要分为两类：精确模拟和近似方法，例如$\tau$-leaping。虽然精确方法存在不可预测的推理时间和冗余函数评估的问题，但$\tau$-leaping 受到其一阶精度的限制。在这项工作中，我们通过将高阶数值推理方案的第一个扩展定制为离散扩散模型来推进后一类，从而实现更大的步长同时减少误差。我们严格分析了所提出的方案，并建立了 KL 散度中$\theta$-梯形方法的二阶精度。在 GPT-2 级文本和 ImageNet 级图像生成任务上的经验评估表明，与现有方法相比，在等效计算约束下，我们的方法实现了卓越的样本质量。

Title: Contrastive Private Data Synthesis via Weighted Multi-PLM Fusion

• Authors: Tianyuan Zou, Yang Liu, Peng Li, Yufei Xiong, Jianqing Zhang, Jingjing Liu, Xiaozhou Ye, Ye Ouyang, Ya-Qin Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00245
• Pdf URL: https://arxiv.org/pdf/2502.00245
• Copy Paste: [[2502.00245]] Contrastive Private Data Synthesis via Weighted Multi-PLM Fusion(https://arxiv.org/abs/2502.00245)
• Keywords: generation, generative
• Abstract: Substantial quantity and high quality are the golden rules of making a good training dataset with sample privacy protection equally important. Generating synthetic samples that resemble high-quality private data while ensuring Differential Privacy (DP), a formal privacy guarantee, promises scalability and practicality. However, existing methods relying on pre-trained models for data synthesis %that avoid fine-tuning large pre-trained generative models often struggle in data-deficient scenarios, suffering from limited sample size, inevitable generation noise and existing pre-trained model bias. To address these challenges, we propose a novel contrAstive private data Synthesis via Weighted multiple Pre-trained language models (PLM) framework, named as WASP. WASP utilizes limited private samples for more accurate private data distribution estimation via a Top-Q voting mechanism, and leverages low-quality synthetic samples for contrastive generation via collaboration among dynamically weighted multiple pre-trained this http URL experiments on 6 well-developed datasets with 6 open-source and 3 closed-source PLMs demonstrate the superiority of WASP in improving model performance over diverse downstream tasks. Code is available at this https URL.
• 摘要：数量和质量是制作优质训练数据集的黄金法则，样本隐私保护也同样重要。生成与高质量隐私数据相似的合成样本，同时确保差分隐私（DP），这是一种正式的隐私保证，具有可扩展性和实用性。然而，现有的数据合成方法依赖于预训练模型，避免对大型预训练生成模型进行微调，通常在数据不足的情况下会遇到困难，因为样本量有限、生成噪声不可避免，并且存在预训练模型偏差。为了应对这些挑战，我们提出了一种新的基于加权多个预训练语言模型（PLM）框架的对比隐私数据合成，称为 WASP。 WASP 利用有限的隐私样本，通过 Top-Q 投票机制对隐私数据分布进行更准确的估计，并通过动态加权的多个预训练模型之间的协作，利用低质量的合成样本进行对比生成。在 6 个成熟的数据集上进行的实验（使用 6 个开源和 3 个闭源 PLM）证明了 WASP 在提高模型性能方面优于各种下游任务。代码可从此 https URL 获取。

Title: MCM: Multi-layer Concept Map for Efficient Concept Learning from Masked Images

• Authors: Yuwei Sun, Lu Mi, Ippei Fujisawa, Ryota Kanai
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00266
• Pdf URL: https://arxiv.org/pdf/2502.00266
• Copy Paste: [[2502.00266]] MCM: Multi-layer Concept Map for Efficient Concept Learning from Masked Images(https://arxiv.org/abs/2502.00266)
• Keywords: generation
• Abstract: Masking strategies commonly employed in natural language processing are still underexplored in vision tasks such as concept learning, where conventional methods typically rely on full images. However, using masked images diversifies perceptual inputs, potentially offering significant advantages in concept learning with large-scale Transformer models. To this end, we propose Multi-layer Concept Map (MCM), the first work to devise an efficient concept learning method based on masked images. In particular, we introduce an asymmetric concept learning architecture by establishing correlations between different encoder and decoder layers, updating concept tokens using backward gradients from reconstruction tasks. The learned concept tokens at various levels of granularity help either reconstruct the masked image patches by filling in gaps or guide the reconstruction results in a direction that reflects specific concepts. Moreover, we present both quantitative and qualitative results across a wide range of metrics, demonstrating that MCM significantly reduces computational costs by training on fewer than 75% of the total image patches while enhancing concept prediction performance. Additionally, editing specific concept tokens in the latent space enables targeted image generation from masked images, aligning both the visible contextual patches and the provided concepts. By further adjusting the testing time mask ratio, we could produce a range of reconstructions that blend the visible patches with the provided concepts, proportional to the chosen ratios.
• 摘要：自然语言处理中常用的掩蔽策略在概念学习等视觉任务中仍未得到充分探索，因为传统方法通常依赖于完整图像。然而，使用掩蔽图像可以使感知输入多样化，从而可能为使用大规​​模 Transformer 模型的概念学习提供显着优势。为此，我们提出了多层概念图 (MCM)，这是第一项基于掩蔽图像设计高效概念学习方法的工作。具体而言，我们通过在不同的编码器和解码器层之间建立相关性来引入非对称概念学习架构，使用来自重建任务的后向梯度来更新概念标记。在不同粒度级别上学习到的概念标记有助于通过填补空白来重建掩蔽图像块，或者将重建结果引导到反映特定概念的方向上。此外，我们在广泛的指标中展示了定量和定性结果，表明 MCM 通过对不到 75% 的总图像块进行训练显着降低了计算成本，同时提高了概念预测性能。此外，在潜在空间中编辑特定概念标记可以从掩码图像生成有针对性的图像，从而对齐可见的上下文块和提供的概念。通过进一步调整测试时间掩码比率，我们可以生成一系列将可见块与提供的概念融合在一起的重建，与所选比率成比例。

Title: Regularized Langevin Dynamics for Combinatorial Optimization

• Authors: Shengyu Feng, Yiming Yang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.00277
• Pdf URL: https://arxiv.org/pdf/2502.00277
• Copy Paste: [[2502.00277]] Regularized Langevin Dynamics for Combinatorial Optimization(https://arxiv.org/abs/2502.00277)
• Keywords: generative
• Abstract: This work proposes a simple yet effective sampling framework for combinatorial optimization (CO). Our method builds on discrete Langevin dynamics (LD), an efficient gradient-guided generative algorithm. However, we observed that directly applying LD often leads to limited exploration. To overcome this limitation, we propose the Regularized Langevin Dynamics (RLD), which enforces an expected distance between the sampled and current solutions, effectively avoiding local minima. We develop two CO solvers on top of RLD, one based on simulated annealing (SA) and the other one based on neural network (NN). Empirical results on three classical CO problems demonstrate that both of our methods can achieve comparable or better performance against the previous state-of-the-art (SOTA) SA and NN-based solvers. In particular, our SA algorithm reduces the running time of the previous SOTA SA method by up to 80\%, while achieving equal or superior performance. In summary, RLD offers a promising framework for enhancing both traditional heuristics and NN models to solve CO problems.
• 摘要：这项工作提出了一个简单而有效的组合优化 (CO) 采样框架。我们的方法建立在离散朗之万动力学 (LD) 的基础上，这是一种高效的梯度引导生成算法。然而，我们观察到直接应用 LD 通常会导致有限的探索。为了克服这一限制，我们提出了正则化朗之万动力学 (RLD)，它强制采样和当前解决方案之间的预期距离，有效地避免了局部最小值。我们在 RLD 之上开发了两个 CO 求解器，一个基于模拟退火 (SA)，另一个基于神经网络 (NN)。对三个经典 CO 问题的经验结果表明，我们的两种方法都可以实现与以前最先进的 (SOTA) SA 和基于 NN 的求解器相当或更好的性能。特别是，我们的 SA 算法将以前的 SOTA SA 方法的运行时间减少了高达 80\%，同时实现了相同或更好的性能。总之，RLD 提供了一个有前途的框架，可以增强传统启发式和 NN 模型来解决 CO 问题。

Title: From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation

• Authors: Xingchen Wan, Han Zhou, Ruoxi Sun, Hootan Nakhost, Ke Jiang, Sercan Ö. Arık
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.00330
• Pdf URL: https://arxiv.org/pdf/2502.00330
• Copy Paste: [[2502.00330]] From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation(https://arxiv.org/abs/2502.00330)
• Keywords: generation
• Abstract: Recent advances in long-context large language models (LLMs) have led to the emerging paradigm of many-shot in-context learning (ICL), where it is observed that scaling many more demonstrating examples beyond the conventional few-shot setup in the context can lead to performance benefits. However, despite its promise, it is unclear what aspects dominate the benefits and whether simply scaling to more examples is the most effective way of improving many-shot ICL. In this work, we first provide an analysis of the factors driving many-shot ICL, and we find that 1) many-shot performance can still be attributed to often a few disproportionately influential examples and 2) identifying such influential examples ("optimize") and using them as demonstrations to regenerate new examples ("generate") can lead to further improvements. Inspired by the findings, we propose BRIDGE, an algorithm that alternates between the optimize step with Bayesian optimization to discover the influential sets of examples and the generate step to reuse this set to expand the reasoning paths of the examples back to the many-shot regime automatically. On Gemini, Claude, and Mistral LLMs of different sizes, we show that BRIDGE to significant improvements across a diverse set of tasks, including symbolic reasoning, numerical reasoning, and code generation.
• 摘要：长上下文大型语言模型 (LLM) 的最新进展导致了多样本上下文学习 (ICL) 的新兴范式，其中观察到，在上下文中扩展更多演示示例，超越传统的少样本设置，可以带来性能优势。然而，尽管前景光明，但尚不清楚哪些方面主导了这些好处，以及仅仅扩展到更多示例是否是改进多样本 ICL 的最有效方法。在这项工作中，我们首先对推动多样本 ICL 的因素进行了分析，我们发现 1) 多样本性能仍然可以归因于通常少数不成比例的有影响力的示例，2) 识别这些有影响力的示例（“优化”）并使用它们作为演示来重新生成新示例（“生成”）可以带来进一步的改进。受这些发现的启发，我们提出了 BRIDGE，这是一种在优化步骤与贝叶斯优化之间交替的算法，以发现有影响力的示例集，并在生成步骤中重用该集以自动将示例的推理路径扩展回多样本机制。在不同规模的 Gemini、Claude 和 Mistral LLM 上，我们表明 BRIDGE 在一系列多样化任务（包括符号推理、数值推理和代码生成）中取得了显著的进步。

Title: BiMaCoSR: Binary One-Step Diffusion Model Leveraging Flexible Matrix Compression for Real Super-Resolution

• Authors: Kai Liu, Kaicheng Yang, Zheng Chen, Zhiteng Li, Yong Guo, Wenbo Li, Linghe Kong, Yulun Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.00333
• Pdf URL: https://arxiv.org/pdf/2502.00333
• Copy Paste: [[2502.00333]] BiMaCoSR: Binary One-Step Diffusion Model Leveraging Flexible Matrix Compression for Real Super-Resolution(https://arxiv.org/abs/2502.00333)
• Keywords: super-resolution
• Abstract: While super-resolution (SR) methods based on diffusion models (DM) have demonstrated inspiring performance, their deployment is impeded due to the heavy request of memory and computation. Recent researchers apply two kinds of methods to compress or fasten the DM. One is to compress the DM into 1-bit, aka binarization, alleviating the storage and computation pressure. The other distills the multi-step DM into only one step, significantly speeding up inference process. Nonetheless, it remains impossible to deploy DM to resource-limited edge devices. To address this problem, we propose BiMaCoSR, which combines binarization and one-step distillation to obtain extreme compression and acceleration. To prevent the catastrophic collapse of the model caused by binarization, we proposed sparse matrix branch (SMB) and low rank matrixbranch (LRM). Both auxiliary branches pass the full-precision (FP) information but in different ways. SMB absorbs the extreme values and its output is high rank, carrying abundant FP information. Whereas, the design of LRMB is inspired by LoRA and is initialized with the top r SVD components, outputting low rank representation. The computation and storage overhead of our proposed branches can be safely ignored. Comprehensive comparison experiments are conducted to exhibit BiMaCoSR outperforms current state-of-the-art binarization methods and gains competitive performance compared with FP one-step model. BiMaCoSR achieves a 23.8x compression ratio and a 27.4x speedup ratio compared to FP counterpart. Our code and model are available at this https URL.
• 摘要：基于扩散模型 (DM) 的超分辨率 (SR) 方法虽然表现出鼓舞人心的性能，但由于对内存和计算的要求很高，其部署受到阻碍。最近的研究人员采用两种方法来压缩或加速 DM。一种是将 DM 压缩为 1 位，即二值化，以减轻存储和计算压力。另一种方法将多步骤 DM 提炼为仅一步，大大加快推理过程。尽管如此，仍然不可能将 DM 部署到资源有限的边缘设备。为了解决这个问题，我们提出了 BiMaCoSR，它结合了二值化和一步提炼以获得极端的压缩和加速。为了防止二值化导致模型灾难性崩溃，我们提出了稀疏矩阵分支 (SMB) 和低秩矩阵分支 (LRM)。两个辅助分支都传递全精度 (FP) 信息，但方式不同。SMB 吸收极值，其输出是高秩的，携带丰富的 FP 信息。而 LRMB 的设计灵感来自 LoRA，并使用前 r 个 SVD 组件进行初始化，输出低秩表示。我们提出的分支的计算和存储开销可以安全地忽略。进行了全面的比较实验，以表明 BiMaCoSR 优于当前最先进的二值化方法，并且与 FP 单步模型相比具有竞争力。与 FP 相比，BiMaCoSR 实现了 23.8 倍的压缩比和 27.4 倍的加速比。我们的代码和模型可在此 https URL 上找到。

Title: Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curves

• Authors: Anand Jerry George, Rodrigo Veiga, Nicolas Macris
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.00336
• Pdf URL: https://arxiv.org/pdf/2502.00336
• Copy Paste: [[2502.00336]] Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curves(https://arxiv.org/abs/2502.00336)
• Keywords: generative
• Abstract: We derive asymptotically precise expressions for test and train errors of denoising score matching (DSM) in generative diffusion models. The score function is parameterized by random features neural networks, with the target distribution being $d$-dimensional standard Gaussian. We operate in a regime where the dimension $d$, number of data samples $n$, and number of features $p$ tend to infinity while keeping the ratios $\psi_n=\frac{n}{d}$ and $\psi_p=\frac{p}{d}$ fixed. By characterizing the test and train errors, we identify regimes of generalization and memorization in diffusion models. Furthermore, our work sheds light on the conditions enhancing either generalization or memorization. Consistent with prior empirical observations, our findings indicate that the model complexity ($p$) and the number of noise samples per data sample ($m$) used during DSM significantly influence generalization and memorization behaviors.
• 摘要：我们推导出生成扩散模型中去噪分数匹配 (DSM) 的测试和训练误差的渐近精确表达式。分数函数由随机特征神经网络参数化，目标分布为 $d$ 维标准高斯分布。我们在维度 $d$、数据样本数量 $n$ 和特征数量 $p$ 趋于无穷大的状态下操作，同时保持比率 $\psi_n=\frac{n}{d}$ 和 $\psi_p=\frac{p}{d}$ 固定。通过表征测试和训练误差，我们确定了扩散模型中的泛化和记忆机制。此外，我们的工作揭示了增强泛化或记忆的条件。与之前的经验观察一致，我们的研究结果表明，DSM 期间使用的模型复杂度 ($p$) 和每个数据样本的噪声样本数量 ($m$) 显著影响泛化和记忆行为。

Title: Exploring Representation-Aligned Latent Space for Better Generation

• Authors: Wanghan Xu, Xiaoyu Yue, Zidong Wang, Yao Teng, Wenlong Zhang, Xihui Liu, Luping Zhou, Wanli Ouyang, Lei Bai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00359
• Pdf URL: https://arxiv.org/pdf/2502.00359
• Copy Paste: [[2502.00359]] Exploring Representation-Aligned Latent Space for Better Generation(https://arxiv.org/abs/2502.00359)
• Keywords: generation, generative
• Abstract: Generative models serve as powerful tools for modeling the real world, with mainstream diffusion models, particularly those based on the latent diffusion model paradigm, achieving remarkable progress across various tasks, such as image and video synthesis. Latent diffusion models are typically trained using Variational Autoencoders (VAEs), interacting with VAE latents rather than the real samples. While this generative paradigm speeds up training and inference, the quality of the generated outputs is limited by the latents' quality. Traditional VAE latents are often seen as spatial compression in pixel space and lack explicit semantic representations, which are essential for modeling the real world. In this paper, we introduce ReaLS (Representation-Aligned Latent Space), which integrates semantic priors to improve generation performance. Extensive experiments show that fundamental DiT and SiT trained on ReaLS can achieve a 15% improvement in FID metric. Furthermore, the enhanced semantic latent space enables more perceptual downstream tasks, such as segmentation and depth estimation.
• 摘要：生成模型是建模现实世界的强大工具，主流扩散模型，尤其是基于潜在扩散模型范式的模型，在图像和视频合成等各种任务中取得了显著进展。潜在扩散模型通常使用变分自编码器 (VAE) 进行训练，与 VAE 潜在样本而不是真实样本进行交互。虽然这种生成范式加快了训练和推理速度，但生成的输出质量受到潜在样本质量的限制。传统的 VAE 潜在样本通常被视为像素空间中的空间压缩，缺乏明确的语义表示，而这对于建模现实世界至关重要。在本文中，我们引入了 ReaLS（表示对齐潜在空间），它集成了语义先验以提高生成性能。大量实验表明，在 ReaLS 上训练的基本 DiT 和 SiT 可以将 FID 指标提高 15%。此外，增强的语义潜在空间可以实现更多感知下游任务，例如分割和深度估计。

Title: Shape from Semantics: 3D Shape Generation from Multi-View Semantics

• Authors: Liangchen Li, Caoliwen Wang, Yuqi Zhou, Bailin Deng, Juyong Zhang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2502.00360
• Pdf URL: https://arxiv.org/pdf/2502.00360
• Copy Paste: [[2502.00360]] Shape from Semantics: 3D Shape Generation from Multi-View Semantics(https://arxiv.org/abs/2502.00360)
• Keywords: restoration, generation, generative
• Abstract: We propose ``Shape from Semantics'', which is able to create 3D models whose geometry and appearance match given semantics when observed from different views. Traditional ``Shape from X'' tasks usually use visual input (e.g., RGB images or depth maps) to reconstruct geometry, imposing strict constraints that limit creative explorations. As applications, works like Shadow Art and Wire Art often struggle to grasp the embedded semantics of their design through direct observation and rely heavily on specific setups for proper display. To address these limitations, our framework uses semantics as input, greatly expanding the design space to create objects that integrate multiple semantic elements and are easily discernible by observers. Considering that this task requires a rich imagination, we adopt various generative models and structure-to-detail pipelines. Specifically, we adopt multi-semantics Score Distillation Sampling (SDS) to distill 3D geometry and appearance from 2D diffusion models, ensuring that the initial shape is consistent with the semantic input. We then use image restoration and video generation models to add more details as supervision. Finally, we introduce neural signed distance field (SDF) representation to achieve detailed shape reconstruction. Our framework generates meshes with complex details, well-structured geometry, coherent textures, and smooth transitions, resulting in visually appealing and eye-catching designs. Project page: this https URL
• 摘要：我们提出了“从语义中获取形状”的概念，它能够创建 3D 模型，从不同角度观察时，其几何形状和外观与给定的语义相匹配。传统的“从 X 中获取形状”任务通常使用视觉输入（例如 RGB 图像或深度图）来重建几何形状，施加了严格的约束，限制了创造性探索。作为应用程序，像 Shadow Art 和 Wire Art 这样的作品通常很难通过直接观察来掌握其设计的嵌入语义，并且严重依赖特定设置才能正确显示。为了解决这些限制，我们的框架使用语义作为输入，大大扩展了设计空间，以创建集成多个语义元素且易于观察者辨别的对象。考虑到这项任务需要丰富的想象力，我们采用了各种生成模型和从结构到细节的流程。具体来说，我们采用多语义分数蒸馏采样 (SDS) 从 2D 扩散模型中提取 3D 几何形状和外观，确保初始形状与语义输入一致。然后，我们使用图像恢复和视频生成模型添加更多细节作为监督。最后，我们引入神经符号距离场 (SDF) 表示来实现详细的形状重建。我们的框架生成具有复杂细节、结构良好的几何形状、连贯的纹理和平滑过渡的网格，从而产生具有视觉吸引力和引人注​​目的设计。项目页面：此 https URL

Title: Scalable Framework for Classifying AI-Generated Content Across Modalities

• Authors: Anh-Kiet Duong, Petra Gomez-Krämer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.00375
• Pdf URL: https://arxiv.org/pdf/2502.00375
• Copy Paste: [[2502.00375]] Scalable Framework for Classifying AI-Generated Content Across Modalities(https://arxiv.org/abs/2502.00375)
• Keywords: generative
• Abstract: The rapid growth of generative AI technologies has heightened the importance of effectively distinguishing between human and AI-generated content, as well as classifying outputs from diverse generative models. This paper presents a scalable framework that integrates perceptual hashing, similarity measurement, and pseudo-labeling to address these challenges. Our method enables the incorporation of new generative models without retraining, ensuring adaptability and robustness in dynamic scenarios. Comprehensive evaluations on the Defactify4 dataset demonstrate competitive performance in text and image classification tasks, achieving high accuracy across both distinguishing human and AI-generated content and classifying among generative methods. These results highlight the framework's potential for real-world applications as generative AI continues to evolve. Source codes are publicly available at this https URL.
• 摘要：生成式人工智能技术的快速发展提高了有效区分人类和人工智能生成的内容以及对来自不同生成模型的输出进行分类的重要性。本文提出了一个可扩展的框架，该框架集成了感知哈希、相似性测量和伪标签以应对这些挑战。我们的方法无需重新训练即可整合新的生成模型，确保了动态场景中的适应性和稳健性。对 Defactify4 数据集的全面评估表明，它在文本和图像分类任务中具有竞争力，在区分人类和人工智能生成的内容以及对生成方法进行分类方面都具有很高的准确性。这些结果凸显了该框架在生成式人工智能不断发展的过程中在现实世界应用中的潜力。源代码可在此 https URL 上公开获取。

Title: Masked Generative Nested Transformers with Decode Time Scaling

• Authors: Sahil Goyal, Debapriya Tula, Gagan Jain, Pradeep Shenoy, Prateek Jain, Sujoy Paul
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00382
• Pdf URL: https://arxiv.org/pdf/2502.00382
• Copy Paste: [[2502.00382]] Masked Generative Nested Transformers with Decode Time Scaling(https://arxiv.org/abs/2502.00382)
• Keywords: generation, generative
• Abstract: Recent advances in visual generation have made significant strides in producing content of exceptional quality. However, most methods suffer from a fundamental problem - a bottleneck of inference computational efficiency. Most of these algorithms involve multiple passes over a transformer model to generate tokens or denoise inputs. However, the model size is kept consistent throughout all iterations, which makes it computationally expensive. In this work, we aim to address this issue primarily through two key ideas - (a) not all parts of the generation process need equal compute, and we design a decode time model scaling schedule to utilize compute effectively, and (b) we can cache and reuse some of the computation. Combining these two ideas leads to using smaller models to process more tokens while large models process fewer tokens. These different-sized models do not increase the parameter size, as they share parameters. We rigorously experiment with ImageNet256$\times$256 , UCF101, and Kinetics600 to showcase the efficacy of the proposed method for image/video generation and frame prediction. Our experiments show that with almost $3\times$ less compute than baseline, our model obtains competitive performance.
• 摘要：视觉生成领域的最新进展在生成高质量内容方面取得了重大进展。然而，大多数方法都存在一个根本问题——推理计算效率的瓶颈。这些算法中的大多数都涉及对变压器模型进行多次传递以生成标记或去噪输入。然而，模型大小在所有迭代中保持一致，这使得计算成本高昂。在这项工作中，我们旨在主要通过两个关键思想来解决这个问题——(a) 生成过程的所有部分都需要相同的计算，我们设计了一个解码时间模型扩展计划来有效利用计算，(b) 我们可以缓存和重用部分计算。结合这两个想法会导致使用较小的模型来处理更多的标记，而大型模型处理较少的标记。这些不同大小的模型不会增加参数大小，因为它们共享参数。我们对 ImageNet256$\times$256、UCF101 和 Kinetics600 进行了严格的实验，以展示所提出的方法在图像/视频生成和帧预测方面的有效性。我们的实验表明，与基线相比，我们的模型在计算量减少了近 3 倍的情况下仍获得了具有竞争力的性能。

Title: Exploring Linear Attention Alternative for Single Image Super-Resolution

• Authors: Rongchang Lu, Changyu Li, Donghang Li, Guojing Zhang, Jianqiang Huang, Xilai Li
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2502.00404
• Pdf URL: https://arxiv.org/pdf/2502.00404
• Copy Paste: [[2502.00404]] Exploring Linear Attention Alternative for Single Image Super-Resolution(https://arxiv.org/abs/2502.00404)
• Keywords: super-resolution
• Abstract: Deep learning-based single-image super-resolution (SISR) technology focuses on enhancing low-resolution (LR) images into high-resolution (HR) ones. Although significant progress has been made, challenges remain in computational complexity and quality, particularly in remote sensing image processing. To address these issues, we propose our Omni-Scale RWKV Super-Resolution (OmniRWKVSR) model which presents a novel approach that combines the Receptance Weighted Key Value (RWKV) architecture with feature extraction techniques such as Visual RWKV Spatial Mixing (VRSM) and Visual RWKV Channel Mixing (VRCM), aiming to overcome the limitations of existing methods and achieve superior SISR performance. This work has proved able to provide effective solutions for high-quality image reconstruction. Under the 4x Super-Resolution tasks, compared to the MambaIR model, we achieved an average improvement of 0.26% in PSNR and 0.16% in SSIM.
• 摘要：基于深度学习的单图像超分辨率 (SISR) 技术专注于将低分辨率 (LR) 图像增强为高分辨率 (HR) 图像。尽管取得了重大进展，但在计算复杂性和质量方面仍然存在挑战，特别是在遥感图像处理方面。为了解决这些问题，我们提出了全尺度 RWKV 超分辨率 (OmniRWKVSR) 模型，该模型提出了一种新颖的方法，将接受加权键值 (RWKV) 架构与特征提取技术（例如视觉 RWKV 空间混合 (VRSM) 和视觉 RWKV 通道混合 (VRCM)）相结合，旨在克服现有方法的局限性并实现卓越的 SISR 性能。事实证明，这项工作能够为高质量图像重建提供有效的解决方案。在 4 倍超分辨率任务下，与 MambaIR 模型相比，我们在 PSNR 方面平均提高了 0.26%，在 SSIM 方面平均提高了 0.16%。

Title: CAT Pruning: Cluster-Aware Token Pruning For Text-to-Image Diffusion Models

• Authors: Xinle Cheng, Zhuoming Chen, Zhihao Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.00433
• Pdf URL: https://arxiv.org/pdf/2502.00433
• Copy Paste: [[2502.00433]] CAT Pruning: Cluster-Aware Token Pruning For Text-to-Image Diffusion Models(https://arxiv.org/abs/2502.00433)
• Keywords: generative
• Abstract: Diffusion models have revolutionized generative tasks, especially in the domain of text-to-image synthesis; however, their iterative denoising process demands substantial computational resources. In this paper, we present a novel acceleration strategy that integrates token-level pruning with caching techniques to tackle this computational challenge. By employing noise relative magnitude, we identify significant token changes across denoising iterations. Additionally, we enhance token selection by incorporating spatial clustering and ensuring distributional balance. Our experiments demonstrate reveal a 50%-60% reduction in computational costs while preserving the performance of the model, thereby markedly increasing the efficiency of diffusion models. The code is available at this https URL
• 摘要：扩散模型彻底改变了生成任务，尤其是在文本到图像合成领域；然而，它们的迭代去噪过程需要大量的计算资源。在本文中，我们提出了一种新颖的加速策略，将 token 级修剪与缓存技术相结合，以应对这一计算挑战。通过使用噪声相对幅度，我们可以识别出去噪迭代过程中显著的 token 变化。此外，我们通过结合空间聚类并确保分布平衡来增强 token 选择。我们的实验表明，在保持模型性能的同时，计算成本降低了 50%-60%，从而显著提高了扩散模型的效率。代码可从此 https URL 获得

Title: Weak-to-Strong Diffusion with Reflection

• Authors: Lichen Bai, Masashi Sugiyama, Zeke Xie
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.00473
• Pdf URL: https://arxiv.org/pdf/2502.00473
• Copy Paste: [[2502.00473]] Weak-to-Strong Diffusion with Reflection(https://arxiv.org/abs/2502.00473)
• Keywords: generative
• Abstract: The goal of diffusion generative models is to align the learned distribution with the real data distribution through gradient score matching. However, inherent limitations in training data quality, modeling strategies, and architectural design lead to inevitable gap between generated outputs and real data. To reduce this gap, we propose Weak-to-Strong Diffusion (W2SD), a novel framework that utilizes the estimated difference between existing weak and strong models (i.e., weak-to-strong difference) to approximate the gap between an ideal model and a strong model. By employing a reflective operation that alternates between denoising and inversion with weak-to-strong difference, we theoretically understand that W2SD steers latent variables along sampling trajectories toward regions of the real data distribution. W2SD is highly flexible and broadly applicable, enabling diverse improvements through the strategic selection of weak-to-strong model pairs (e.g., DreamShaper vs. SD1.5, good experts vs. bad experts in MoE). Extensive experiments demonstrate that W2SD significantly improves human preference, aesthetic quality, and prompt adherence, achieving SOTA performance across various modalities (e.g., image, video), architectures (e.g., UNet-based, DiT-based, MoE), and benchmarks. For example, Juggernaut-XL with W2SD can improve with the HPSv2 winning rate up to 90% over the original results. Moreover, the performance gains achieved by W2SD markedly outweigh its additional computational overhead, while the cumulative improvements from different weak-to-strong difference further solidify its practical utility and deployability.
• 摘要：扩散生成模型的目标是通过梯度得分匹配将学习到的分布与真实数据分布对齐。然而，训练数据质量、建模策略和架构设计中的固有限制导致生成的输出与真实数据之间不可避免地存在差距。为了缩小这一差距，我们提出了弱到强扩散 (W2SD)，这是一个新颖的框架，它利用现有弱模型和强模型之间的估计差异（即弱到强差异）来近似理想模型和强模型之间的差距。通过采用在弱到强差异之间交替进行去噪和反演的反射操作，我们从理论上了解到 W2SD 会沿着采样轨迹将潜在变量引导至真实数据分布的区域。W2SD 高度灵活且适用范围广泛，通过对弱到强模型对的战略性选择（例如，DreamShaper 与 SD1.5，MoE 中的好专家与坏专家）实现多种改进。大量实验表明，W2SD 显著改善了人类偏好、审美质量和及时遵守，在各种模态（例如图像、视频）、架构（例如基于 UNet、基于 DiT、MoE）和基准上实现了 SOTA 性能。例如，使用 W2SD 的 Juggernaut-XL 可以将 HPSv2 获胜率提高到原始结果的 90% 以上。此外，W2SD 实现的性能提升明显超过了其额外的计算开销，而不同弱到强差异带来的累积改进进一步巩固了其实用性和可部署性。

Title: A framework for river connectivity classification using temporal image processing and attention based neural networks

• Authors: Timothy James Becker, Derin Gezgin, Jun Yi He Wu, Mary Becker
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2502.00474
• Pdf URL: https://arxiv.org/pdf/2502.00474
• Copy Paste: [[2502.00474]] A framework for river connectivity classification using temporal image processing and attention based neural networks(https://arxiv.org/abs/2502.00474)
• Keywords: generative
• Abstract: Measuring the connectivity of water in rivers and streams is essential for effective water resource management. Increased extreme weather events associated with climate change can result in alterations to river and stream connectivity. While traditional stream flow gauges are costly to deploy and limited to large river bodies, trail camera methods are a low-cost and easily deployed alternative to collect hourly data. Image capturing, however requires stream ecologists to manually curate (select and label) tens of thousands of images per year. To improve this workflow, we developed an automated instream trail camera image classification system consisting of three parts: (1) image processing, (2) image augmentation and (3) machine learning. The image preprocessing consists of seven image quality filters, foliage-based luma variance reduction, resizing and bottom-center cropping. Images are balanced using variable amount of generative augmentation using diffusion models and then passed to a machine learning classification model in labeled form. By using the vision transformer architecture and temporal image enhancement in our framework, we are able to increase the 75% base accuracy to 90% for a new unseen site image. We make use of a dataset captured and labeled by staff from the Connecticut Department of Energy and Environmental Protection between 2018-2020. Our results indicate that a combination of temporal image processing and attention-based models are effective at classifying unseen river connectivity images.
• 摘要：测量河流和溪流中水的连通性对于有效的水资源管理至关重要。与气候变化相关的极端天气事件增多可能导致河流和溪流连通性的改变。虽然传统的河流流量计部署成本高昂且仅限于大型河流，但追踪摄像机方法是一种低成本且易于部署的替代方法，可以收集每小时数据。然而，图像捕获需要河流生态学家每年手动整理（选择和标记）数万张图像。为了改进这个工作流程，我们开发了一个自动化的河流追踪摄像机图像分类系统，该系统由三部分组成：（1）图像处理，（2）图像增强和（3）机器学习。图像预处理包括七个图像质量过滤器、基于树叶的亮度方差减少、调整大小和底部中心裁剪。使用扩散模型使用不同数量的生成增强来平衡图像，然后以标记形式传递给机器学习分类模型。通过在我们的框架中使用视觉转换器架构和时间图像增强，我们能够将新的未见过的站点图像的基本准确率从 75% 提高到 90%。我们利用了康涅狄格州能源和环境保护部工作人员在 2018-2020 年期间捕获和标记的数据集。我们的结果表明，时间图像处理和基于注意力的模型相结合可以有效地对未见过的河流连通性图像进行分类。

Title: Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolation

• Authors: Yang Cao, Zhao Song, Chiwun Yang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00500
• Pdf URL: https://arxiv.org/pdf/2502.00500
• Copy Paste: [[2502.00500]] Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolation(https://arxiv.org/abs/2502.00500)
• Keywords: generation
• Abstract: This paper considers an efficient video modeling process called Video Latent Flow Matching (VLFM). Unlike prior works, which randomly sampled latent patches for video generation, our method relies on current strong pre-trained image generation models, modeling a certain caption-guided flow of latent patches that can be decoded to time-dependent video frames. We first speculate multiple images of a video are differentiable with respect to time in some latent space. Based on this conjecture, we introduce the HiPPO framework to approximate the optimal projection for polynomials to generate the probability path. Our approach gains the theoretical benefits of the bounded universal approximation error and timescale robustness. Moreover, VLFM processes the interpolation and extrapolation abilities for video generation with arbitrary frame rates. We conduct experiments on several text-to-video datasets to showcase the effectiveness of our method.
• 摘要：本文探讨了一种称为视频潜在流匹配 (VLFM) 的高效视频建模过程。与之前随机采样潜在块以生成视频的工作不同，我们的方法依赖于当前强大的预训练图像生成模型，对可以解码为时间相关视频帧的特定字幕引导潜在块流进行建模。我们首先推测视频的多个图像在某些潜在空间中关于时间是可微的。基于这一猜想，我们引入了 HiPPO 框架来近似多项式的最佳投影以生成概率路径。我们的方法获得了有界通用近似误差和时间尺度鲁棒性的理论优势。此外，VLFM 处理具有任意帧速率的视频生成插值和外推能力。我们在几个文本到视频数据集上进行了实验，以展示我们方法的有效性。

Title: Bridging Internal Probability and Self-Consistency for Effective and Efficient LLM Reasoning

• Authors: Zhi Zhou, Tan Yuhao, Zenan Li, Yuan Yao, Lan-Zhe Guo, Xiaoxing Ma, Yu-Feng Li
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00511
• Pdf URL: https://arxiv.org/pdf/2502.00511
• Copy Paste: [[2502.00511]] Bridging Internal Probability and Self-Consistency for Effective and Efficient LLM Reasoning(https://arxiv.org/abs/2502.00511)
• Keywords: generation
• Abstract: Recent advancements in large language models (LLMs) have demonstrated remarkable reasoning capabilities. However, single-shot inference often yields unreliable results for complex reasoning tasks, leading researchers to explore multiple reasoning paths through methods such as perplexity and self-consistency. In this paper, we present the first theoretical error decomposition analysis of these techniques, breaking down their error into estimation error and model error. Our analysis reveals a fundamental trade-off: perplexity methods suffer from substantial model error due to the absence of a proper consistency function, while self-consistency exhibits high estimation error due to a slow error convergence rate. To overcome these limitations, we propose Reasoning-Pruning Perplexity Consistency (RPC). This approach combines Perplexity Consistency, which seamlessly integrates LLM perplexity with self-consistency, and Reasoning Pruning, which eliminates low-probability reasoning paths to effectively prevent the degeneration of estimation error reduction. Theoretical analysis demonstrates that RPC not only accelerates the convergence rate of estimation error to an exponential level but also holds strong potential for further reducing model error. Extensive empirical evaluations on seven benchmark datasets confirm that RPC can significantly improve reasoning performance, sample efficiency, and confidence reliability.
• 摘要：大型语言模型 (LLM) 的最新进展已展现出卓越的推理能力。然而，单次推理通常会对复杂的推理任务产生不可靠的结果，这促使研究人员通过困惑度和自洽度等方法探索多种推理路径。在本文中，我们首次对这些技术进行了理论误差分解分析，将其误差分解为估计误差和模型误差。我们的分析揭示了一个根本的权衡：由于缺乏适当的一致性函数，困惑度方法会遭受大量模型误差，而由于误差收敛速度慢，自洽度会表现出较高的估计误差。为了克服这些限制，我们提出了推理修剪困惑度一致性 (RPC)。这种方法结合了困惑度一致性（将 LLM 困惑度与自洽度无缝集成）和推理修剪（消除低概率推理路径，有效防止估计误差减少的退化）。理论分析表明，RPC不仅将估计误差的收敛速度加速到指数级，而且具有进一步降低模型误差的强大潜力。在七个基准数据集上的大量实证评估证实，RPC可以显著提高推理性能、样本效率和置信度。

Title: Generating crossmodal gene expression from cancer histopathology improves multimodal AI predictions

• Authors: Samiran Dey, Christopher R.S. Banerji, Partha Basuchowdhuri, Sanjoy K. Saha, Deepak Parashar, Tapabrata Chakraborti
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00568
• Pdf URL: https://arxiv.org/pdf/2502.00568
• Copy Paste: [[2502.00568]] Generating crossmodal gene expression from cancer histopathology improves multimodal AI predictions(https://arxiv.org/abs/2502.00568)
• Keywords: generative
• Abstract: Emerging research has highlighted that artificial intelligence based multimodal fusion of digital pathology and transcriptomic features can improve cancer diagnosis (grading/subtyping) and prognosis (survival risk) prediction. However, such direct fusion for joint decision is impractical in real clinical settings, where histopathology is still the gold standard for diagnosis and transcriptomic tests are rarely requested, at least in the public healthcare system. With our novel diffusion based crossmodal generative AI model PathoGen, we show that genomic expressions synthesized from digital histopathology jointly predicts cancer grading and patient survival risk with high accuracy (state-of-the-art performance), certainty (through conformal coverage guarantee) and interpretability (through distributed attention maps). PathoGen code is available for open use by the research community through GitHub at this https URL.
• 摘要：新兴研究强调，基于人工智能的数字病理学和转录组特征的多模态融合可以改善癌症诊断（分级/亚型）和预后（生存风险）预测。然而，这种直接融合进行联合决策在实际临床环境中是不切实际的，因为组织病理学仍然是诊断的黄金标准，而且很少要求进行转录组测试，至少在公共医疗系统中是这样。借助我们基于扩散的新型跨模态生成 AI 模型 PathoGen，我们表明从数字组织病理学合成的基因组表达可以联合预测癌症分级和患者生存风险，具有高准确度（最先进的性能）、确定性（通过保形覆盖保证）和可解释性（通过分布式注意力图）。PathoGen 代码可通过 GitHub 的 https URL 供研究社区开放使用。

Title: Zeroth-order Informed Fine-Tuning for Diffusion Model: A Recursive Likelihood Ratio Optimizer

• Authors: Tao Ren, Zishi Zhang, Zehao Li, Jingyang Jiang, Shentao Qin, Guanghao Li, Yan Li, Yi Zheng, Xinping Li, Min Zhan, Yijie Peng
• Subjects: cs.CV, cs.AI, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.00639
• Pdf URL: https://arxiv.org/pdf/2502.00639
• Copy Paste: [[2502.00639]] Zeroth-order Informed Fine-Tuning for Diffusion Model: A Recursive Likelihood Ratio Optimizer(https://arxiv.org/abs/2502.00639)
• Keywords: generation
• Abstract: The probabilistic diffusion model (DM), generating content by inferencing through a recursive chain structure, has emerged as a powerful framework for visual generation. After pre-training on enormous unlabeled data, the model needs to be properly aligned to meet requirements for downstream applications. How to efficiently align the foundation DM is a crucial task. Contemporary methods are either based on Reinforcement Learning (RL) or truncated Backpropagation (BP). However, RL and truncated BP suffer from low sample efficiency and biased gradient estimation respectively, resulting in limited improvement or, even worse, complete training failure. To overcome the challenges, we propose the Recursive Likelihood Ratio (RLR) optimizer, a zeroth-order informed fine-tuning paradigm for DM. The zeroth-order gradient estimator enables the computation graph rearrangement within the recursive diffusive chain, making the RLR's gradient estimator an unbiased one with the lower variance than other methods. We provide theoretical guarantees for the performance of the RLR. Extensive experiments are conducted on image and video generation tasks to validate the superiority of the RLR. Furthermore, we propose a novel prompt technique that is natural for the RLR to achieve a synergistic effect.
• 摘要：概率扩散模型 (DM) 通过递归链结构进行推理来生成内容，已成为视觉生成的强大框架。在对大量未标记数据进行预训练后，需要对模型进行适当的调整以满足下游应用的要求。如何有效地调整基础 DM 是一项至关重要的任务。当代方法要么基于强化学习 (RL)，要么基于截断反向传播 (BP)。然而，RL 和截断 BP 分别存在样本效率低和梯度估计有偏的问题，导致改进有限，甚至更糟，训练完全失败。为了克服这些挑战，我们提出了递归似然比 (RLR) 优化器，这是一种零阶知情 DM 微调范式。零阶梯度估计器可以在递归扩散链内重新排列计算图，使 RLR 的梯度估计器成为无偏的，并且方差低于其他方法。我们为 RLR 的性能提供了理论保证。我们在图像和视频生成任务上进行了大量的实验，以验证 RLR 的优越性。此外，我们提出了一种新颖的提示技术，使 RLR 自然地发挥协同效应。

Title: Mitigating the Modality Gap: Few-Shot Out-of-Distribution Detection with Multi-modal Prototypes and Image Bias Estimation

• Authors: Yimu Wang, Evelien Riddell, Adrian Chow, Sean Sedwards, Krzysztof Czarnecki
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00662
• Pdf URL: https://arxiv.org/pdf/2502.00662
• Copy Paste: [[2502.00662]] Mitigating the Modality Gap: Few-Shot Out-of-Distribution Detection with Multi-modal Prototypes and Image Bias Estimation(https://arxiv.org/abs/2502.00662)
• Keywords: generation
• Abstract: Existing vision-language model (VLM)-based methods for out-of-distribution (OOD) detection typically rely on similarity scores between input images and in-distribution (ID) text prototypes. However, the modality gap between image and text often results in high false positive rates, as OOD samples can exhibit high similarity to ID text prototypes. To mitigate the impact of this modality gap, we propose incorporating ID image prototypes along with ID text prototypes. We present theoretical analysis and empirical evidence indicating that this approach enhances VLM-based OOD detection performance without any additional training. To further reduce the gap between image and text, we introduce a novel few-shot tuning framework, SUPREME, comprising biased prompts generation (BPG) and image-text consistency (ITC) modules. BPG enhances image-text fusion and improves generalization by conditioning ID text prototypes on the Gaussian-based estimated image domain bias; ITC reduces the modality gap by minimizing intra- and inter-modal distances. Moreover, inspired by our theoretical and empirical findings, we introduce a novel OOD score $S_{\textit{GMP}}$, leveraging uni- and cross-modal similarities. Finally, we present extensive experiments to demonstrate that SUPREME consistently outperforms existing VLM-based OOD detection methods.
• 摘要：现有的基于视觉语言模型 (VLM) 的分布外 (OOD) 检测方法通常依赖于输入图像与分布内 (ID) 文本原型之间的相似度得分。然而，图像和文本之间的模态差距往往会导致高假阳性率，因为 OOD 样本可能与 ID 文本原型表现出很高的相似度。为了减轻这种模态差距的影响，我们建议将 ID 图像原型与 ID 文本原型结合起来。我们提出的理论分析和实证证据表明，这种方法无需任何额外训练即可提高基于 VLM 的 OOD 检测性能。为了进一步缩小图像和文本之间的差距，我们引入了一个新颖的少样本调整框架 SUPREME，包括有偏提示生成 (BPG) 和图像文本一致性 (ITC) 模块。BPG 通过在基于高斯的估计图像域偏差上调节 ID 文本原型来增强图像文本融合并提高泛化能力；ITC 通过最小化模态内和模态间距离来缩小模态差距。此外，受我们的理论和实证研究结果的启发，我们引入了一种新颖的 OOD 分数 $S_{\textit{GMP}}$，利用单模态和跨模态相似性。最后，我们进行了广泛的实验，以证明 SUPREME 始终优于现有的基于 VLM 的 OOD 检测方法。

Title: High-Order Matching for One-Step Shortcut Diffusion Models

• Authors: Bo Chen, Chengyue Gong, Xiaoyu Li, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song, Mingda Wan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00688
• Pdf URL: https://arxiv.org/pdf/2502.00688
• Copy Paste: [[2502.00688]] High-Order Matching for One-Step Shortcut Diffusion Models(https://arxiv.org/abs/2502.00688)
• Keywords: generation, generative
• Abstract: One-step shortcut diffusion models [Frans, Hafner, Levine and Abbeel, ICLR 2025] have shown potential in vision generation, but their reliance on first-order trajectory supervision is fundamentally limited. The Shortcut model's simplistic velocity-only approach fails to capture intrinsic manifold geometry, leading to erratic trajectories, poor geometric alignment, and instability-especially in high-curvature regions. These shortcomings stem from its inability to model mid-horizon dependencies or complex distributional features, leaving it ill-equipped for robust generative modeling. In this work, we introduce HOMO (High-Order Matching for One-Step Shortcut Diffusion), a game-changing framework that leverages high-order supervision to revolutionize distribution transportation. By incorporating acceleration, jerk, and beyond, HOMO not only fixes the flaws of the Shortcut model but also achieves unprecedented smoothness, stability, and geometric precision. Theoretically, we prove that HOMO's high-order supervision ensures superior approximation accuracy, outperforming first-order methods. Empirically, HOMO dominates in complex settings, particularly in high-curvature regions where the Shortcut model struggles. Our experiments show that HOMO delivers smoother trajectories and better distributional alignment, setting a new standard for one-step generative models.
• 摘要：一步捷径扩散模型 [Frans、Hafner、Levine 和 Abbeel，ICLR 2025] 在视觉生成方面表现出潜力，但它们对一阶轨迹监督的依赖从根本上受到限制。捷径模型的简单速度方法无法捕捉内在的流形几何，导致轨迹不稳定、几何对齐不良和不稳定 - 尤其是在高曲率区域。这些缺点源于它无法对中期依赖关系或复杂的分布特征进行建模，因此无法进行稳健的生成建模。在这项工作中，我们引入了 HOMO（一步捷径扩散的高阶匹配），这是一个改变游戏规则的框架，它利用高阶监督来彻底改变配送运输。通过结合加速度、急动度等，HOMO 不仅修复了捷径模型的缺陷，而且还实现了前所未有的平滑度、稳定性和几何精度。从理论上讲，我们证明了 HOMO 的高阶监督可确保出色的近似精度，优于一阶方法。从经验上讲，HOMO 在复杂环境中占主导地位，特别是在 Shortcut 模型难以应对的高曲率区域。我们的实验表明，HOMO 可提供更平滑的轨迹和更好的分布对齐，为一步生成模型树立了新标准。

Title: PhiP-G: Physics-Guided Text-to-3D Compositional Scene Generation

• Authors: Qixuan Li, Chao Wang, Zongjin He, Yan Peng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00708
• Pdf URL: https://arxiv.org/pdf/2502.00708
• Copy Paste: [[2502.00708]] PhiP-G: Physics-Guided Text-to-3D Compositional Scene Generation(https://arxiv.org/abs/2502.00708)
• Keywords: generation
• Abstract: Text-to-3D asset generation has achieved significant optimization under the supervision of 2D diffusion priors. However, when dealing with compositional scenes, existing methods encounter several challenges: 1). failure to ensure that composite scene layouts comply with physical laws; 2). difficulty in accurately capturing the assets and relationships described in complex scene descriptions; 3). limited autonomous asset generation capabilities among layout approaches leveraging large language models (LLMs). To avoid these compromises, we propose a novel framework for compositional scene generation, PhiP-G, which seamlessly integrates generation techniques with layout guidance based on a world model. Leveraging LLM-based agents, PhiP-G analyzes the complex scene description to generate a scene graph, and integrating a multimodal 2D generation agent and a 3D Gaussian generation method for targeted assets creation. For the stage of layout, PhiP-G employs a physical pool with adhesion capabilities and a visual supervision agent, forming a world model for layout prediction and planning. Extensive experiments demonstrate that PhiP-G significantly enhances the generation quality and physical rationality of the compositional scenes. Notably, PhiP-G attains state-of-the-art (SOTA) performance in CLIP scores, achieves parity with the leading methods in generation quality as measured by the T$^3$Bench, and improves efficiency by 24x.
• 摘要：在二维扩散先验的监督下，文本到三维资产生成已经实现了显著的优化。然而，在处理组合场景时，现有方法遇到了几个挑战：1）无法确保复合场景布局符合物理定律；2）难以准确捕捉复杂场景描述中描述的资产和关系；3）利用大型语言模型（LLM）的布局方法中自主资产生成能力有限。为了避免这些妥协，我们提出了一个新颖的组合场景生成框架 PhiP-G，它将生成技术与基于世界模型的布局指导无缝集成。利用基于 LLM 的代理，PhiP-G 分析复杂场景描述以生成场景图，并集成多模态二维生成代理和三维高斯生成方法以创建有针对性的资产。对于布局阶段，PhiP-G 采用具有粘附能力的物理池和视觉监督代理，形成布局预测和规划的世界模型。大量实验表明，PhiP-G 显著提高了组合场景的生成质量和物理合理性。值得注意的是，PhiP-G 在 CLIP 分数中达到了最先进 (SOTA) 的性能，在 T$^3$Bench 测量的生成质量上与领先方法达到了同等水平，并且效率提高了 24 倍。

Title: MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction

• Authors: Chao Wang, Jianming Yang, Yang Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.00717
• Pdf URL: https://arxiv.org/pdf/2502.00717
• Copy Paste: [[2502.00717]] MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction(https://arxiv.org/abs/2502.00717)
• Keywords: generation
• Abstract: Hallucination has been a long-standing and inevitable problem that hinders the application of Large Vision-Language Models (LVLMs) in domains that require high reliability. Various methods focus on improvement depending on data annotations or training strategies, yet place less emphasis on LLM's inherent problems. To fill this gap, we delve into the attention mechanism of the decoding process in the LVLM. Intriguingly, our investigation uncovers the prevalent attention redundancy within the hierarchical architecture of the LVLM, manifesting as overextended image processing in deep layers and an overabundance of non-essential image tokens. Stemming from the observation, we thus propose MINT, a novel training-free decoding strategy, MItigating hallucinations via tokeN reducTion. Specifically, we dynamically intensify the LVLM's local perception capability by masking its attention to irrelevant image tokens. In addition, we use contrastive decoding that pushes the model to focus more on those key image regions. Our full method aims to guide the model in concentrating more on key visual elements during generation. Extensive experimental results on several popular public benchmarks show that our approach achieves a 4% improvement in mitigating hallucinations caused by distracted perception compared to original models. Meanwhile, our approach is demonstrated to make the model perceive 5% more visual points even though we reduce a suite of image tokens.
• 摘要：幻觉是一个长期存在且不可避免的问题，阻碍了大型视觉语言模型 (LVLM) 在需要高可靠性的领域中的应用。各种方法都侧重于根据数据注释或训练策略进行改进，而较少关注 LLM 的固有问题。为了填补这一空白，我们深入研究了 LVLM 中解码过程的注意力机制。有趣的是，我们的研究发现了 LVLM 的分层架构中普遍存在的注意力冗余，表现为深层图像处理过度和非必要图像标记过多。基于这一观察，我们提出了 MINT，一种新颖的无训练解码策略，通过标记减少来缓解幻觉。具体来说，我们通过掩盖其对不相关图像标记的注意力来动态增强 LVLM 的局部感知能力。此外，我们使用对比解码，促使模型更多地关注那些关键图像区域。我们的完整方法旨在引导模型在生成过程中更加专注于关键的视觉元素。在几个流行的公共基准上进行的大量实验结果表明，与原始模型相比，我们的方法在缓解因感知分散而引起的幻觉方面取得了 4% 的改善。同时，即使我们减少了一组图像标记，我们的方法也能使模型感知到 5% 以上的视觉点。

Title: Understanding and Mitigating the High Computational Cost in Path Data Diffusion

• Authors: Dingyuan Shi, Lulu Zhang, Yongxin Tong, Ke Xu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00725
• Pdf URL: https://arxiv.org/pdf/2502.00725
• Copy Paste: [[2502.00725]] Understanding and Mitigating the High Computational Cost in Path Data Diffusion(https://arxiv.org/abs/2502.00725)
• Keywords: generation
• Abstract: Advancements in mobility services, navigation systems, and smart transportation technologies have made it possible to collect large amounts of path data. Modeling the distribution of this path data, known as the Path Generation (PG) problem, is crucial for understanding urban mobility patterns and developing intelligent transportation systems. Recent studies have explored using diffusion models to address the PG problem due to their ability to capture multimodal distributions and support conditional generation. A recent work devises a diffusion process explicitly in graph space and achieves state-of-the-art performance. However, this method suffers a high computation cost in terms of both time and memory, which prohibits its application. In this paper, we analyze this method both theoretically and experimentally and find that the main culprit of its high computation cost is its explicit design of the diffusion process in graph space. To improve efficiency, we devise a Latent-space Path Diffusion (LPD) model, which operates in latent space instead of graph space. Our LPD significantly reduces both time and memory costs by up to 82.8% and 83.1%, respectively. Despite these reductions, our approach does not suffer from performance degradation. It outperforms the state-of-the-art method in most scenarios by 24.5%~34.0%.
• 摘要：移动服务、导航系统和智能交通技术的进步使得收集大量路径数据成为可能。对这些路径数据的分布进行建模，即所谓的路径生成 (PG) 问题，对于理解城市移动模式和开发智能交通系统至关重要。最近的研究探索了使用扩散模型来解决 PG 问题，因为它们能够捕捉多模态分布并支持条件生成。最近的一项研究在图形空间中明确设计了一个扩散过程，并实现了最先进的性能。然而，这种方法在时间和内存方面的计算成本很高，这阻碍了它的应用。在本文中，我们从理论和实验两个方面分析了这种方法，发现其高计算成本的主要原因是它在图形空间中明确设计了扩散过程。为了提高效率，我们设计了一个潜在空间路径扩散 (LPD) 模型，它在潜在空间而不是图形空间中运行。我们的 LPD 分别显著降低了高达 82.8% 和 83.1% 的时间和内存成本。尽管有这些减少，我们的方法并没有遭受性能下降。在大多数情况下，它比最先进的方法高出 24.5%~34.0%。

Title: A method for estimating forest carbon storage distribution density via artificial intelligence generated content model

• Authors: Zhenyu Yu, Jinnian Wang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2502.00783
• Pdf URL: https://arxiv.org/pdf/2502.00783
• Copy Paste: [[2502.00783]] A method for estimating forest carbon storage distribution density via artificial intelligence generated content model(https://arxiv.org/abs/2502.00783)
• Keywords: restoration, generative
• Abstract: Forest is the most significant land-based carbon storage mechanism. The forest carbon sink can effectively decrease the atmospheric CO2 concentration and mitigate climate change. Remote sensing estimation not only ensures high accuracy of data, but also enables large-scale area observation. Optical images provide the possibility for long-term monitoring, which is a potential issue in the future carbon storage estimation research. We chose Huize County, Qujing City, Yunnan Province, China as the study area, took GF-1 WFV satellite image as the data, introduced the KD-VGG module to extract the initial features, and proposed the improved implicit diffusion model (IIDM). The results showed that: (1) The VGG-19 module after knowledge distillation can realize the initial feature extraction, reduce the inference time and improve the accuracy in the case of reducing the number of model parameters. (2) The Attention + MLP module was added for feature fusion to obtain the relationship between global and local features and realized the restoration of high-fidelity images in the continuous scale range. (3) The IIDM model proposed in this paper had the highest estimation accuracy, with RMSE of 28.68, which was 13.16 higher than that of the regression model, about 31.45%. In the estimation of carbon storage, the generative model can extract deeper features, and its performance was significantly better than other models. It demonstrated the feasibility of artificial intelligence-generated content (AIGC) in the field of quantitative remote sensing and provided valuable insights for the study of carbon neutralization effect. By combining the actual characteristics of the forest, the regional carbon storage estimation with a resolution of 16-meter was utilized to provide a significant theoretical basis for the formulation of forest carbon sink regulation.
• 摘要：森林是陆地上最显著的碳储存机制，森林碳汇能有效降低大气CO2浓度，减缓气候变化。遥感估测不仅保证了数据的高精度，而且可以进行大范围的区域观测，而光学影像为长期监测提供了可能，是未来碳储量估测研究的潜在课题。以云南省曲靖市会泽县为研究区域，以GF-1 WFV卫星影像为数据，引入KD-VGG模块提取初始特征，提出改进的隐式扩散模型（IIDM）。结果表明：（1）经过知识蒸馏后的VGG-19模块可实现初始特征提取，在减少模型参数数量的情况下减少推理时间并提高精度；（2）加入Attention+MLP模块进行特征融合，获得全局特征与局部特征之间的关系，实现连续尺度范围内高保真图像的恢复。 （3）本文提出的IIDM模型估算精度最高，RMSE为28.68，比回归模型高13.16，约31.45%。在碳储量估算中，生成模型能够提取更深层次的特征，其性能明显优于其他模型。证明了人工智能生成内容（AIGC）在定量遥感领域的可行性，为碳中和效果研究提供了有价值的启示。结合森林实际特点，以16米分辨率进行区域碳储量估算，为森林碳汇调控的制定提供重要的理论依据。

Title: Adversarial Semantic Augmentation for Training Generative Adversarial Networks under Limited Data

• Authors: Mengping Yang, Zhe Wang, Ziqiu Chi, Dongdong Li, Wenli Du
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2502.00800
• Pdf URL: https://arxiv.org/pdf/2502.00800
• Copy Paste: [[2502.00800]] Adversarial Semantic Augmentation for Training Generative Adversarial Networks under Limited Data(https://arxiv.org/abs/2502.00800)
• Keywords: generative
• Abstract: Generative adversarial networks (GANs) have made remarkable achievements in synthesizing images in recent years. Typically, training GANs requires massive data, and the performance of GANs deteriorates significantly when training data is limited. To improve the synthesis performance of GANs in low-data regimes, existing approaches use various data augmentation techniques to enlarge the training sets. However, it is identified that these augmentation techniques may leak or even alter the data distribution. To remedy this, we propose an adversarial semantic augmentation (ASA) technique to enlarge the training data at the semantic level instead of the image level. Concretely, considering semantic features usually encode informative information of images, we estimate the covariance matrices of semantic features for both real and generated images to find meaningful transformation directions. Such directions translate original features to another semantic representation, e.g., changing the backgrounds or expressions of the human face dataset. Moreover, we derive an upper bound of the expected adversarial loss. By optimizing the upper bound, our semantic augmentation is implicitly achieved. Such design avoids redundant sampling of the augmented features and introduces negligible computation overhead, making our approach computation efficient. Extensive experiments on both few-shot and large-scale datasets demonstrate that our method consistently improve the synthesis quality under various data regimes, and further visualized and analytic results suggesting satisfactory versatility of our proposed method.
• 摘要：近年来，生成对抗网络（GAN）在图像合成方面取得了令人瞩目的成就。通常，训练 GAN 需要大量数据，而当训练数据有限时，GAN 的性能会显著下降。为了提高 GAN 在低数据环境下的合成性能，现有方法使用各种数据增强技术来扩大训练集。然而，我们发现这些增强技术可能会泄漏甚至改变数据分布。为了解决这个问题，我们提出了一种对抗语义增强（ASA）技术来在语义层面而不是图像层面扩大训练数据。具体而言，考虑到语义特征通常编码图像的信息，我们估计真实图像和生成图像的语义特征的协方差矩阵以找到有意义的转换方向。这些方向将原始特征转换为另一种语义表示，例如，改变人脸数据集的背景或表情。此外，我们推导出预期对抗损失的上限。通过优化上限，我们的语义增强得以隐式实现。这种设计避免了对增强特征的冗余采样，并且引入了可忽略不计的计算开销，使我们的方法计算效率高。在小样本和大规模数据集上进行的大量实验表明，我们的方法在各种数据环境下都能持续提高合成质量，进一步的可视化和分析结果表明我们提出的方法具有令人满意的多功能性。

Title: UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphs

• Authors: Yufei He, Yuan Sui, Xiaoxin He, Yue Liu, Yifei Sun, Bryan Hooi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00806
• Pdf URL: https://arxiv.org/pdf/2502.00806
• Copy Paste: [[2502.00806]] UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphs(https://arxiv.org/abs/2502.00806)
• Keywords: generative
• Abstract: Existing foundation models, such as CLIP, aim to learn a unified embedding space for multimodal data, enabling a wide range of downstream web-based applications like search, recommendation, and content classification. However, these models often overlook the inherent graph structures in multimodal datasets, where entities and their relationships are crucial. Multimodal graphs (MMGs) represent such graphs where each node is associated with features from different modalities, while the edges capture the relationships between these entities. On the other hand, existing graph foundation models primarily focus on text-attributed graphs (TAGs) and are not designed to handle the complexities of MMGs. To address these limitations, we propose UniGraph2, a novel cross-domain graph foundation model that enables general representation learning on MMGs, providing a unified embedding space. UniGraph2 employs modality-specific encoders alongside a graph neural network (GNN) to learn a unified low-dimensional embedding space that captures both the multimodal information and the underlying graph structure. We propose a new cross-domain multi-graph pre-training algorithm at scale to ensure effective transfer learning across diverse graph domains and modalities. Additionally, we adopt a Mixture of Experts (MoE) component to align features from different domains and modalities, ensuring coherent and robust embeddings that unify the information across modalities. Extensive experiments on a variety of multimodal graph tasks demonstrate that UniGraph2 significantly outperforms state-of-the-art models in tasks such as representation learning, transfer learning, and multimodal generative tasks, offering a scalable and flexible solution for learning on MMGs.
• 摘要：现有的基础模型（例如 CLIP）旨在学习多模态数据的统一嵌入空间，从而支持搜索、推荐和内容分类等广泛的下游 Web 应用。然而，这些模型往往忽略了多模态数据集中固有的图结构，而实体及其关系在其中至关重要。多模态图 (MMG) 表示这样的图，其中每个节点都与来自不同模态的特征相关联，而边则捕获这些实体之间的关系。另一方面，现有的图基础模型主要关注文本属性图 (TAG)，并非设计用于处理 MMG 的复杂性。为了解决这些限制，我们提出了 UniGraph2，这是一种新颖的跨域图基础模型，它支持在 MMG 上进行通用表示学习，从而提供统一的嵌入空间。UniGraph2 使用模态特定的编码器和图神经网络 (GNN) 来学习统一的低维嵌入空间，该空间可同时捕获多模态信息和底层图结构。我们提出了一种新的跨域多图预训练算法，以确保在不同的图域和模态中进行有效的迁移学习。此外，我们采用了混合专家 (MoE) 组件来对齐来自不同域和模态的特征，确保统一跨模态信息的连贯和稳健的嵌入。对各种多模态图任务的大量实验表明，UniGraph2 在表示学习、迁移学习和多模态生成任务等任务中的表现明显优于最先进的模型，为 MMG 上的学习提供了可扩展且灵活的解决方案。

Title: Synthetic Artifact Auditing: Tracing LLM-Generated Synthetic Data Usage in Downstream Applications

• Authors: Yixin Wu, Ziqing Yang, Yun Shen, Michael Backes, Yang Zhang
• Subjects: cs.LG, cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.00808
• Pdf URL: https://arxiv.org/pdf/2502.00808
• Copy Paste: [[2502.00808]] Synthetic Artifact Auditing: Tracing LLM-Generated Synthetic Data Usage in Downstream Applications(https://arxiv.org/abs/2502.00808)
• Keywords: generation
• Abstract: Large language models (LLMs) have facilitated the generation of high-quality, cost-effective synthetic data for developing downstream models and conducting statistical analyses in various domains. However, the increased reliance on synthetic data may pose potential negative impacts. Numerous studies have demonstrated that LLM-generated synthetic data can perpetuate and even amplify societal biases and stereotypes, and produce erroneous outputs known as ``hallucinations'' that deviate from factual knowledge. In this paper, we aim to audit artifacts, such as classifiers, generators, or statistical plots, to identify those trained on or derived from synthetic data and raise user awareness, thereby reducing unexpected consequences and risks in downstream applications. To this end, we take the first step to introduce synthetic artifact auditing to assess whether a given artifact is derived from LLM-generated synthetic data. We then propose an auditing framework with three methods including metric-based auditing, tuning-based auditing, and classification-based auditing. These methods operate without requiring the artifact owner to disclose proprietary training details. We evaluate our auditing framework on three text classification tasks, two text summarization tasks, and two data visualization tasks across three training scenarios. Our evaluation demonstrates the effectiveness of all proposed auditing methods across all these tasks. For instance, black-box metric-based auditing can achieve an average accuracy of $0.868 \pm 0.071$ for auditing classifiers and $0.880 \pm 0.052$ for auditing generators using only 200 random queries across three scenarios. We hope our research will enhance model transparency and regulatory compliance, ensuring the ethical and responsible use of synthetic data.
• 摘要：大型语言模型 (LLM) 促进了高质量、高性价比的合成数据的生成，可用于开发下游模型并在各个领域进行统计分析。然而，对合成数据的依赖性增加可能会带来潜在的负面影响。大量研究表明，LLM 生成的合成数据可以延续甚至放大社会偏见和刻板印象，并产生偏离事实知识的错误输出，即所谓的“幻觉”。在本文中，我们旨在审核分类器、生成器或统计图等工件，以识别那些在合成数据上训练或从合成数据中衍生的工件，并提高用户意识，从而减少下游应用中的意外后果和风险。为此，我们迈出了第一步，引入合成工件审计，以评估给定工件是否来自 LLM 生成的合成数据。然后，我们提出了一个审计框架，其中包含三种方法，包括基于度量的审计、基于调整的审计和基于分类的审计。这些方法的运行不需要工件所有者披露专有的培训细节。我们在三个训练场景中对三个文本分类任务、两个文本摘要任务和两个数据可视化任务评估了我们的审计框架。我们的评估证明了所有提出的审计方法在所有这些任务中的有效性。例如，基于黑盒度量的审计在三个场景中仅使用 200 个随机查询即可实现审计分类器的平均准确率 $0.868 \pm 0.071$，审计生成器的平均准确率 $0.880 \pm 0.052$。我们希望我们的研究能够提高模型透明度和法规遵从性，确保以合乎道德和负责任的方式使用合成数据。

Title: Sundial: A Family of Highly Capable Time Series Foundation Models

• Authors: Yong Liu, Guo Qin, Zhiyuan Shi, Zhi Chen, Caiyin Yang, Xiangdong Huang, Jianmin Wang, Mingsheng Long
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00816
• Pdf URL: https://arxiv.org/pdf/2502.00816
• Copy Paste: [[2502.00816]] Sundial: A Family of Highly Capable Time Series Foundation Models(https://arxiv.org/abs/2502.00816)
• Keywords: generative
• Abstract: We introduce Sundial, a family of native, flexible, and scalable time series foundation models. To predict the next-patch's distribution, we propose a TimeFlow Loss based on flow-matching, which facilitates native pre-training of Transformers on time series without discrete tokenization. Conditioned on arbitrary-length time series, our model is pre-trained without specifying any prior distribution and can generate multiple probable predictions, achieving flexibility in representation learning beyond using parametric densities. Towards time series foundation models, we leverage minimal but crucial adaptations of Transformers and curate TimeBench with 1 trillion time points, comprising mostly real-world datasets and synthetic data. By mitigating mode collapse through TimeFlow Loss, we pre-train a family of Sundial models on TimeBench, which exhibit unprecedented model capacity and generalization performance on zero-shot forecasting. In addition to presenting good scaling behavior, Sundial achieves new state-of-the-art on both point forecasting and probabilistic forecasting benchmarks. We believe that Sundial's pioneering generative paradigm will facilitate a wide variety of forecasting scenarios.
• 摘要：我们推出了 Sundial，这是一系列原生、灵活且可扩展的时间序列基础模型。为了预测下一个补丁的分布，我们提出了基于流匹配的 TimeFlow Loss，这有助于在时间序列上对 Transformers 进行原生预训练，而无需离散标记。以任意长度的时间序列为条件，我们的模型无需指定任何先验分布即可进行预训练，并且可以生成多个可能的预测，从而实现除使用参数密度之外的表示学习灵活性。对于时间序列基础模型，我们利用 Transformers 的最小但关键的改编，并使用 1 万亿个时间点来整理 TimeBench，其中主要包括现实世界的数据集和合成数据。通过 TimeFlow Loss 缓解模式崩溃，我们在 TimeBench 上预训练了一系列 Sundial 模型，这些模型在零样本预测中表现出前所未有的模型容量和泛化性能。除了表现出良好的扩展行为外，Sundial 还在点预测和概率预测基准上实现了新的最先进水平。我们相信，Sundial 的开创性生成范式将促进各种各样的预测场景。

Title: OOD Detection with immature Models

• Authors: Behrooz Montazeran, Ullrich Köthe
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.00820
• Pdf URL: https://arxiv.org/pdf/2502.00820
• Copy Paste: [[2502.00820]] OOD Detection with immature Models(https://arxiv.org/abs/2502.00820)
• Keywords: generative
• Abstract: Likelihood-based deep generative models (DGMs) have gained significant attention for their ability to approximate the distributions of high-dimensional data. However, these models lack a performance guarantee in assigning higher likelihood values to in-distribution (ID) inputs, data the models are trained on, compared to out-of-distribution (OOD) inputs. This counter-intuitive behaviour is particularly pronounced when ID inputs are more complex than OOD data points. One potential approach to address this challenge involves leveraging the gradient of a data point with respect to the parameters of the DGMs. A recent OOD detection framework proposed estimating the joint density of layer-wise gradient norms for a given data point as a model-agnostic method, demonstrating superior performance compared to the Typicality Test across likelihood-based DGMs and image dataset pairs. In particular, most existing methods presuppose access to fully converged models, the training of which is both time-intensive and computationally demanding. In this work, we demonstrate that using immature models,stopped at early stages of training, can mostly achieve equivalent or even superior results on this downstream task compared to mature models capable of generating high-quality samples that closely resemble ID data. This novel finding enhances our understanding of how DGMs learn the distribution of ID data and highlights the potential of leveraging partially trained models for downstream tasks. Furthermore, we offer a possible explanation for this unexpected behaviour through the concept of support overlap.
• 摘要：基于似然的深度生成模型 (DGM) 因其能够近似高维数据分布的能力而备受关注。然而，与分布外 (OOD) 输入相比，这些模型在为分布内 (ID) 输入（模型训练所用的数据）分配更高的似然值方面缺乏性能保证。当 ID 输入比 OOD 数据点更复杂时，这种违反直觉的行为尤其明显。解决这一挑战的一种潜在方法是利用数据点相对于 DGM 参数的梯度。最近的 OOD 检测框架提出了一种与模型无关的方法，即估计给定数据点的分层梯度范数的联合密度，与基于似然的 DGM 和图像数据集对的典型性测试相比，它表现出了卓越的性能。特别是，大多数现有方法都假设可以访问完全收敛的模型，而这种模型的训练既耗时又耗计算。在这项研究中，我们证明了使用在训练早期阶段停止的不成熟模型，在这项下游任务上取得的结果，大多与能够生成与 ID 数据非常相似的高质量样本的成熟模型相当，甚至更好。这一新发现增强了我们对 DGM 如何学习 ID 数据分布的理解，并凸显了利用部分训练模型完成下游任务的潜力。此外，我们通过支持重叠的概念为这种意外行为提供了一种可能的解释。

Title: RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning

• Authors: Yuanhuiyi Lyu, Xu Zheng, Lutao Jiang, Yibo Yan, Xin Zou, Huiyu Zhou, Linfeng Zhang, Xuming Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.00848
• Pdf URL: https://arxiv.org/pdf/2502.00848
• Copy Paste: [[2502.00848]] RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning(https://arxiv.org/abs/2502.00848)
• Keywords: generation, generative
• Abstract: Recent text-to-image generative models, e.g., Stable Diffusion V3 and Flux, have achieved notable progress. However, these models are strongly restricted to their limited knowledge, a.k.a., their own fixed parameters, that are trained with closed datasets. This leads to significant hallucinations or distortions when facing fine-grained and unseen novel real-world objects, e.g., the appearance of the Tesla Cybertruck. To this end, we present the first real-object-based retrieval-augmented generation framework (RealRAG), which augments fine-grained and unseen novel object generation by learning and retrieving real-world images to overcome the knowledge gaps of generative models. Specifically, to integrate missing memory for unseen novel object generation, we train a reflective retriever by self-reflective contrastive learning, which injects the generator's knowledge into the sef-reflective negatives, ensuring that the retrieved augmented images compensate for the model's missing knowledge. Furthermore, the real-object-based framework integrates fine-grained visual knowledge for the generative models, tackling the distortion problem and improving the realism for fine-grained object generation. Our Real-RAG is superior in its modular application to all types of state-of-the-art text-to-image generative models and also delivers remarkable performance boosts with all of them, such as a gain of 16.18% FID score with the auto-regressive model on the Stanford Car benchmark.
• 摘要：最近的文本到图像生成模型，例如 Stable Diffusion V3 和 Flux，取得了显著进展。然而，这些模型被严格限制在有限的知识中，也就是它们自己的固定参数，这些参数是用封闭数据集训练的。当面对细粒度和看不见的新颖现实世界物体时，例如特斯拉 Cyber​​truck 的外观，这会导致严重的幻觉或扭曲。为此，我们提出了第一个基于真实对象的检索增强生成框架 (RealRAG)，它通过学习和检索真实世界图像来增强细粒度和看不见的新颖物体的生成，以克服生成模型的知识空白。具体来说，为了整合看不见的新颖物体生成的缺失记忆，我们通过自反射对比学习训练了一个反射检索器，它将生成器的知识注入到自反射负片中，确保检索到的增强图像弥补了模型缺失的知识。此外，基于真实对象的框架为生成模型集成了细粒度的视觉知识，解决了失真问题并提高了细粒度对象生成的真实度。我们的 Real-RAG 在模块化应用方面表现出色，可应用于所有类型的最先进的文本到图像生成模型，并且可为所有这些模型带来显著的性能提升，例如在斯坦福汽车基准测试中使用自回归模型可将 FID 得分提高 16.18%。

Title: Towards Automation of Cognitive Modeling using Large Language Models

• Authors: Milena Rmus, Akshay K. Jagadish, Marvin Mathony, Tobias Ludwig, Eric Schulz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00879
• Pdf URL: https://arxiv.org/pdf/2502.00879
• Copy Paste: [[2502.00879]] Towards Automation of Cognitive Modeling using Large Language Models(https://arxiv.org/abs/2502.00879)
• Keywords: generation
• Abstract: Computational cognitive models, which formalize theories of cognition, enable researchers to quantify cognitive processes and arbitrate between competing theories by fitting models to behavioral data. Traditionally, these models are handcrafted, which requires significant domain knowledge, coding expertise, and time investment. Previous work has demonstrated that Large Language Models (LLMs) are adept at pattern recognition in-context, solving complex problems, and generating executable code. In this work, we leverage these abilities to explore the potential of LLMs in automating the generation of cognitive models based on behavioral data. We evaluated the LLM in two different tasks: model identification (relating data to a source model), and model generation (generating the underlying cognitive model). We performed these tasks across two cognitive domains - decision making and learning. In the case of data simulated from canonical cognitive models, we found that the LLM successfully identified and generated the ground truth model. In the case of human data, where behavioral noise and lack of knowledge of the true underlying process pose significant challenges, the LLM generated models that are identical or close to the winning model from cognitive science literature. Our findings suggest that LLMs can have a transformative impact on cognitive modeling. With this project, we aim to contribute to an ongoing effort of automating scientific discovery in cognitive science.
• 摘要：计算认知模型将认知理论形式化，使研究人员能够量化认知过程，并通过将模型与行为数据拟合来在相互竞争的理论之间进行仲裁。传统上，这些模型是手工制作的，需要大量的领域知识、编码专业知识和时间投入。之前的研究表明，大型语言模型 (LLM) 擅长上下文模式识别、解决复杂问题和生成可执行代码。在这项工作中，我们利用这些能力来探索 LLM 在基于行为数据自动生成认知模型方面的潜力。我们在两个不同的任务中评估了 LLM：模型识别（将数据与源模型相关联）和模型生成（生成底层认知模型）。我们在两个认知领域（决策和学习）执行了这些任务。对于从典型认知模型模拟的数据，我们发现 LLM 成功识别并生成了基本事实模型。对于人类数据，行为噪声和对真实潜在过程的缺乏了解构成了重大挑战，法学硕士生成的模型与认知科学文献中的获胜模型相同或接近。我们的研究结果表明，法学硕士可以对认知建模产生变革性影响。通过这个项目，我们旨在为认知科学领域持续的自动化科学发现做出贡献。

Title: Blink of an eye: a simple theory for feature localization in generative models

• Authors: Marvin Li, Aayush Karan, Sitan Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00921
• Pdf URL: https://arxiv.org/pdf/2502.00921
• Copy Paste: [[2502.00921]] Blink of an eye: a simple theory for feature localization in generative models(https://arxiv.org/abs/2502.00921)
• Keywords: generation, generative
• Abstract: Large language models (LLMs) can exhibit undesirable and unexpected behavior in the blink of an eye. In a recent Anthropic demo, Claude switched from coding to Googling pictures of Yellowstone, and these sudden shifts in behavior have also been observed in reasoning patterns and jailbreaks. This phenomenon is not unique to autoregressive models: in diffusion models, key features of the final output are decided in narrow ``critical windows'' of the generation process. In this work we develop a simple, unifying theory to explain this phenomenon. We show that it emerges generically as the generation process localizes to a sub-population of the distribution it models. While critical windows have been studied at length in diffusion models, existing theory heavily relies on strong distributional assumptions and the particulars of Gaussian diffusion. In contrast to existing work our theory (1) applies to autoregressive and diffusion models; (2) makes no distributional assumptions; (3) quantitatively improves previous bounds even when specialized to diffusions; and (4) requires basic tools and no stochastic calculus or statistical physics-based machinery. We also identify an intriguing connection to the all-or-nothing phenomenon from statistical inference. Finally, we validate our predictions empirically for LLMs and find that critical windows often coincide with failures in problem solving for various math and reasoning benchmarks.
• 摘要：大型语言模型 (LLM) 可以在眨眼之间表现出不良和意外的行为。在最近的 Anthropic 演示中，Claude 从编码切换到谷歌搜索黄石公园的图片，这些突然的行为转变也在推理模式和越狱中观察到。这种现象并非自回归模型所独有：在扩散模型中，最终输出的关键特征由生成过程的狭窄“关键窗口”决定。在这项工作中，我们开发了一个简单、统一的理论来解释这种现象。我们表明，当生成过程定位到它所建模的分布的子群体时，它就会普遍出现。虽然关键窗口在扩散模型中已经得到了深入研究，但现有理论严重依赖于强分布假设和高斯扩散的细节。与现有工作相比，我们的理论 (1) 适用于自回归和扩散模型；(2) 不做任何分布假设；(3) 即使专门用于扩散，也能定量改进以前的界限； （4）需要基本工具，不需要随机微积分或基于统计物理的机器。我们还从统计推断中发现了与全有或全无现象的有趣联系。最后，我们通过经验验证了我们对法学硕士的预测，发现关键窗口通常与各种数学和推理基准的问题解决失败相吻合。

Title: PDE-Controller: LLMs for Autoformalization and Reasoning of PDEs

• Authors: Mauricio Soroco, Jialin Song, Mengzhou Xia, Kye Emond, Weiran Sun, Wuyang Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00963
• Pdf URL: https://arxiv.org/pdf/2502.00963
• Copy Paste: [[2502.00963]] PDE-Controller: LLMs for Autoformalization and Reasoning of PDEs(https://arxiv.org/abs/2502.00963)
• Keywords: generation
• Abstract: While recent AI-for-math has made strides in pure mathematics, areas of applied mathematics, particularly PDEs, remain underexplored despite their significant real-world applications. We present PDE-Controller, a framework that enables large language models (LLMs) to control systems governed by partial differential equations (PDEs). Our approach enables LLMs to transform informal natural language instructions into formal specifications, and then execute reasoning and planning steps to improve the utility of PDE control. We build a holistic solution comprising datasets (both human-written cases and 2 million synthetic samples), math-reasoning models, and novel evaluation metrics, all of which require significant effort. Our PDE-Controller significantly outperforms prompting the latest open-source and GPT models in reasoning, autoformalization, and program synthesis, achieving up to a 62% improvement in utility gain for PDE control. By bridging the gap between language generation and PDE systems, we demonstrate the potential of LLMs in addressing complex scientific and engineering challenges. We will release all data, model checkpoints, and code at this https URL.
• 摘要：虽然最近的人工智能数学在纯数学方面取得了长足进步，但应用数学领域，尤其是偏微分方程，尽管在现实世界中有着重要的应用，但仍未得到充分探索。我们提出了 PDE-Controller，这是一个框架，它使大型语言模型 (LLM) 能够控制由偏微分方程 (PDE) 控制的系统。我们的方法使 LLM 能够将非正式的自然语言指令转换为正式的规范，然后执行推理和规划步骤以提高 PDE 控制的效用。我们构建了一个整体解决方案，包括数据集（包括人工编写的案例和 200 万个合成样本）、数学推理模型和新颖的评估指标，所有这些都需要付出巨大的努力。我们的 PDE-Controller 在推理、自动形式化和程序综合方面的表现明显优于最新的开源和 GPT 模型，使 PDE 控制的效用增益提高了 62%。通过弥合语言生成和 PDE 系统之间的差距，我们展示了 LLM 在解决复杂科学和工程挑战方面的潜力。我们将在此 https URL 发布所有数据、模型检查点和代码。

Title: Pushing the Boundaries of State Space Models for Image and Video Generation

• Authors: Yicong Hong, Long Mai, Yuan Yao, Feng Liu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00972
• Pdf URL: https://arxiv.org/pdf/2502.00972
• Copy Paste: [[2502.00972]] Pushing the Boundaries of State Space Models for Image and Video Generation(https://arxiv.org/abs/2502.00972)
• Keywords: generation
• Abstract: While Transformers have become the dominant architecture for visual generation, linear attention models, such as the state-space models (SSM), are increasingly recognized for their efficiency in processing long visual sequences. However, the essential efficiency of these models comes from formulating a limited recurrent state, enforcing causality among tokens that are prone to inconsistent modeling of N-dimensional visual data, leaving questions on their capacity to generate long non-causal sequences. In this paper, we explore the boundary of SSM on image and video generation by building the largest-scale diffusion SSM-Transformer hybrid model to date (5B parameters) based on the sub-quadratic bi-directional Hydra and self-attention, and generate up to 2K images and 360p 8 seconds (16 FPS) videos. Our results demonstrate that the model can produce faithful results aligned with complex text prompts and temporal consistent videos with high dynamics, suggesting the great potential of using SSMs for visual generation tasks.
• 摘要：虽然 Transformer 已成为视觉生成的主导架构，但线性注意模型（例如状态空间模型 (SSM)）因其在处理长视觉序列方面的效率而受到越来越多的认可。然而，这些模型的本质效率来自于制定有限的循环状态，在容易导致 N 维视觉数据建模不一致的 token 之间强制因果关系，这使得它们生成长非因果序列的能力受到质疑。在本文中，我们通过基于亚二次双向 Hydra 和自注意力构建迄今为止最大规模的扩散 SSM-Transformer 混合模型（5B 参数）探索 SSM 在图像和视频生成方面的边界，并生成高达 2K 的图像和 360p 8 秒（16 FPS）的视频。我们的结果表明，该模型可以产生与复杂文本提示一致的忠实结果和具有高动态的时间一致视频，这表明 SSM 在视觉生成任务中具有巨大潜力。

Title: FCBoost-Net: A Generative Network for Synthesizing Multiple Collocated Outfits via Fashion Compatibility Boosting

• Authors: Dongliang Zhou, Haijun Zhang, Jianghong Ma, Jicong Fan, Zhao Zhang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2502.00992
• Pdf URL: https://arxiv.org/pdf/2502.00992
• Copy Paste: [[2502.00992]] FCBoost-Net: A Generative Network for Synthesizing Multiple Collocated Outfits via Fashion Compatibility Boosting(https://arxiv.org/abs/2502.00992)
• Keywords: generation, generative
• Abstract: Outfit generation is a challenging task in the field of fashion technology, in which the aim is to create a collocated set of fashion items that complement a given set of items. Previous studies in this area have been limited to generating a unique set of fashion items based on a given set of items, without providing additional options to users. This lack of a diverse range of choices necessitates the development of a more versatile framework. However, when the task of generating collocated and diversified outfits is approached with multimodal image-to-image translation methods, it poses a challenging problem in terms of non-aligned image translation, which is hard to address with existing methods. In this research, we present FCBoost-Net, a new framework for outfit generation that leverages the power of pre-trained generative models to produce multiple collocated and diversified outfits. Initially, FCBoost-Net randomly synthesizes multiple sets of fashion items, and the compatibility of the synthesized sets is then improved in several rounds using a novel fashion compatibility booster. This approach was inspired by boosting algorithms and allows the performance to be gradually improved in multiple steps. Empirical evidence indicates that the proposed strategy can improve the fashion compatibility of randomly synthesized fashion items as well as maintain their diversity. Extensive experiments confirm the effectiveness of our proposed framework with respect to visual authenticity, diversity, and fashion compatibility.
• 摘要：服装生成是时尚技术领域的一项具有挑战性的任务，其目的是创建一组搭配的时尚单品来补充给定的单品。该领域的先前研究仅限于根据给定的单品集生成一组独特的时尚单品，而不向用户提供额外的选项。这种缺乏多样化选择的范围需要开发一个更通用的框架。然而，当使用多模态图像到图像转换方法来完成生成搭配和多样化服装的任务时，它在非对齐图像转换方面提出了一个具有挑战性的问题，这很难用现有方法解决。在这项研究中，我们提出了 FCBoost-Net，这是一个用于服装生成的新框架，它利用预先训练的生成模型的强大功能来生成多个搭配和多样化的服装。最初，FCBoost-Net 随机合成多组时尚单品，然后使用新颖的时尚兼容性增强器在几轮中提高合成集的兼容性。该方法的灵感来自增强算法，允许通过多个步骤逐步提高性能。经验证据表明，所提出的策略可以提高随机合成时尚物品的时尚兼容性，并保持其多样性。大量实验证实了我们提出的框架在视觉真实性、多样性和时尚兼容性方面的有效性。

Title: Vessel segmentation for X-separation

• Authors: Taechang Kim, Sooyeon Ji, Kyeongseon Min, Minjun Kim, Jonghyo Youn, Chungseok Oh, Jiye Kim, Jongho Lee
• Subjects: cs.CV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2502.01023
• Pdf URL: https://arxiv.org/pdf/2502.01023
• Copy Paste: [[2502.01023]] Vessel segmentation for X-separation(https://arxiv.org/abs/2502.01023)
• Keywords: generation
• Abstract: $\chi$-separation is an advanced quantitative susceptibility mapping (QSM) method that is designed to generate paramagnetic ($\chi_{para}$) and diamagnetic ($|\chi_{dia}|$) susceptibility maps, reflecting the distribution of iron and myelin in the brain. However, vessels have shown artifacts, interfering with the accurate quantification of iron and myelin in applications. To address this challenge, a new vessel segmentation method for $\chi$-separation is developed. The method comprises three steps: 1) Seed generation from $\textit{R}_2^*$ and the product of $\chi_{para}$ and $|\chi_{dia}|$ maps; 2) Region growing, guided by vessel geometry, creating a vessel mask; 3) Refinement of the vessel mask by excluding non-vessel structures. The performance of the method was compared to conventional vessel segmentation methods both qualitatively and quantitatively. To demonstrate the utility of the method, it was tested in two applications: quantitative evaluation of a neural network-based $\chi$-separation reconstruction method ($\chi$-sepnet-$\textit{R}_2^*$) and population-averaged region of interest (ROI) analysis. The proposed method demonstrates superior performance to the conventional vessel segmentation methods, effectively excluding the non-vessel structures, achieving the highest Dice score coefficient. For the applications, applying vessel masks report notable improvements for the quantitative evaluation of $\chi$-sepnet-$\textit{R}_2^*$ and statistically significant differences in population-averaged ROI analysis. These applications suggest excluding vessels when analyzing the $\chi$-separation maps provide more accurate evaluations. The proposed method has the potential to facilitate various applications, offering reliable analysis through the generation of a high-quality vessel mask.
• 摘要：$\chi$-separation 是一种先进的定量磁化率映射 (QSM) 方法，旨在生成顺磁性 ($\chi_{para}$) 和抗磁性 ($|\chi_{dia}|$) 磁化率图，反映脑内铁和髓鞘的分布。然而，血管显示出伪影，干扰了应用中铁和髓鞘的准确量化。为了应对这一挑战，开发了一种用于 $\chi$-separation 的新型血管分割方法。该方法包括三个步骤：1) 从 $\textit{R}_2^*$ 和 $\chi_{para}$ 与 $|\chi_{dia}|$ 图的乘积生成种子；2) 以血管几何为指导的区域增长，创建血管掩模；3) 通过排除非血管结构来细化血管掩模。将该方法的性能与传统血管分割方法进行了定性和定量比较。为了证明该方法的实用性，在两个应用中对其进行了测试：基于神经网络的 $\chi$ 分离重建方法 ($\chi$-sepnet-$\textit{R}_2^*$) 的定量评估和群体平均感兴趣区域 (ROI) 分析。所提出的方法表现出优于传统血管分割方法的性能，有效地排除了非血管结构，获得了最高的 Dice 得分系数。对于应用，应用血管掩模报告了 $\chi$-sepnet-$\textit{R}_2^*$ 定量评估的显着改进和群体平均 ROI 分析的统计显着差异。这些应用建议在分析 $\chi$ 分离图时排除血管可提供更准确的评估。所提出的方法有可能促进各种应用，通过生成高质量的血管掩模提供可靠的分析。

Title: WonderHuman: Hallucinating Unseen Parts in Dynamic 3D Human Reconstruction

• Authors: Zilong Wang, Zhiyang Dou, Yuan Liu, Cheng Lin, Xiao Dong, Yunhui Guo, Chenxu Zhang, Xin Li, Wenping Wang, Xiaohu Guo
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2502.01045
• Pdf URL: https://arxiv.org/pdf/2502.01045
• Copy Paste: [[2502.01045]] WonderHuman: Hallucinating Unseen Parts in Dynamic 3D Human Reconstruction(https://arxiv.org/abs/2502.01045)
• Keywords: generative
• Abstract: In this paper, we present WonderHuman to reconstruct dynamic human avatars from a monocular video for high-fidelity novel view synthesis. Previous dynamic human avatar reconstruction methods typically require the input video to have full coverage of the observed human body. However, in daily practice, one typically has access to limited viewpoints, such as monocular front-view videos, making it a cumbersome task for previous methods to reconstruct the unseen parts of the human avatar. To tackle the issue, we present WonderHuman, which leverages 2D generative diffusion model priors to achieve high-quality, photorealistic reconstructions of dynamic human avatars from monocular videos, including accurate rendering of unseen body parts. Our approach introduces a Dual-Space Optimization technique, applying Score Distillation Sampling (SDS) in both canonical and observation spaces to ensure visual consistency and enhance realism in dynamic human reconstruction. Additionally, we present a View Selection strategy and Pose Feature Injection to enforce the consistency between SDS predictions and observed data, ensuring pose-dependent effects and higher fidelity in the reconstructed avatar. In the experiments, our method achieves SOTA performance in producing photorealistic renderings from the given monocular video, particularly for those challenging unseen parts. The project page and source code can be found at this https URL.
• 摘要：在本文中，我们提出了 WonderHuman，用于从单目视频重建动态人体头像，以实现高保真新视图合成。以前的动态人体头像重建方法通常要求输入视频完全覆盖观察到的人体。然而，在日常实践中，人们通常只能获得有限的视点，例如单目正面视频，这使得以前的方法很难重建人体头像中看不见的部分。为了解决这个问题，我们提出了 WonderHuman，它利用 2D 生成扩散模型先验从单目视频中实现高质量、逼真的动态人体头像重建，包括准确渲染看不见的身体部位。我们的方法引入了一种双空间优化技术，在规范空间和观察空间中应用分数蒸馏采样 (SDS)，以确保视觉一致性并增强动态人体重建的真实感。此外，我们提出了视图选择策略和姿势特征注入，以加强 SDS 预测与观察到的数据之间的一致性，确保重建头像的姿势相关效果和更高的保真度。在实验中，我们的方法在从给定的单目视频生成照片级逼真的渲染方面实现了 SOTA 性能，特别是对于那些具有挑战性的看不见的部分。项目页面和源代码可以在这个 https URL 中找到。

Title: OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models

• Authors: Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, Chao Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01061
• Pdf URL: https://arxiv.org/pdf/2502.01061
• Copy Paste: [[2502.01061]] OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models(https://arxiv.org/abs/2502.01061)
• Keywords: generation
• Abstract: End-to-end human animation, such as audio-driven talking human generation, has undergone notable advancements in the recent few years. However, existing methods still struggle to scale up as large general video generation models, limiting their potential in real applications. In this paper, we propose OmniHuman, a Diffusion Transformer-based framework that scales up data by mixing motion-related conditions into the training phase. To this end, we introduce two training principles for these mixed conditions, along with the corresponding model architecture and inference strategy. These designs enable OmniHuman to fully leverage data-driven motion generation, ultimately achieving highly realistic human video generation. More importantly, OmniHuman supports various portrait contents (face close-up, portrait, half-body, full-body), supports both talking and singing, handles human-object interactions and challenging body poses, and accommodates different image styles. Compared to existing end-to-end audio-driven methods, OmniHuman not only produces more realistic videos, but also offers greater flexibility in inputs. It also supports multiple driving modalities (audio-driven, video-driven and combined driving signals). Video samples are provided on the ttfamily project page (this https URL)
• 摘要：端到端人体动画，例如音频驱动的说话人生成，在最近几年取得了显着的进步。然而，现有的方法仍然难以扩展为大型通用视频生成模型，限制了它们在实际应用中的潜力。在本文中，我们提出了 OmniHuman，这是一个基于 Diffusion Transformer 的框架，通过将与运动相关的条件混合到训练阶段来扩展数据。为此，我们介绍了针对这些混合条件的两种训练原则，以及相应的模型架构和推理策略。这些设计使 OmniHuman 能够充分利用数据驱动的运动生成，最终实现高度逼真的人体视频生成。更重要的是，OmniHuman 支持各种肖像内容（面部特写、肖像、半身、全身），支持说话和唱歌，处理人与物体的交互和具有挑战性的身体姿势，并适应不同的图像风格。与现有的端到端音频驱动方法相比，OmniHuman 不仅可以制作更逼真的视频，而且在输入方面也提供了更大的灵活性。它还支持多种驾驶模式（音频驱动、视频驱动和组合驱动信号）。ttfamily 项目页面（此 https URL）上提供了视频示例

Title: BC-GAN: A Generative Adversarial Network for Synthesizing a Batch of Collocated Clothing

• Authors: Dongliang Zhou, Haijun Zhang, Jianghong Ma, Jianyang Shi
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2502.01080
• Pdf URL: https://arxiv.org/pdf/2502.01080
• Copy Paste: [[2502.01080]] BC-GAN: A Generative Adversarial Network for Synthesizing a Batch of Collocated Clothing(https://arxiv.org/abs/2502.01080)
• Keywords: generation, generative
• Abstract: Collocated clothing synthesis using generative networks has become an emerging topic in the field of fashion intelligence, as it has significant potential economic value to increase revenue in the fashion industry. In previous studies, several works have attempted to synthesize visually-collocated clothing based on a given clothing item using generative adversarial networks (GANs) with promising results. These works, however, can only accomplish the synthesis of one collocated clothing item each time. Nevertheless, users may require different clothing items to meet their multiple choices due to their personal tastes and different dressing scenarios. To address this limitation, we introduce a novel batch clothing generation framework, named BC-GAN, which is able to synthesize multiple visually-collocated clothing images simultaneously. In particular, to further improve the fashion compatibility of synthetic results, BC-GAN proposes a new fashion compatibility discriminator in a contrastive learning perspective by fully exploiting the collocation relationship among all clothing items. Our model was examined in a large-scale dataset with compatible outfits constructed by ourselves. Extensive experiment results confirmed the effectiveness of our proposed BC-GAN in comparison to state-of-the-art methods in terms of diversity, visual authenticity, and fashion compatibility.
• 摘要：使用生成网络进行搭配服装合成已成为时尚智能领域的一个新兴课题，因为它具有巨大的潜在经济价值，可以增加时尚行业的收入。在之前的研究中，一些研究尝试使用生成对抗网络 (GAN) 根据给定的服装项目合成视觉搭配的服装，并取得了令人鼓舞的结果。然而，这些研究每次只能完成一件搭配服装的合成。然而，由于用户的个人品味和不同的穿衣场景，他们可能需要不同的服装来满足他们的多种选择。为了解决这一限制，我们引入了一种新颖的批量服装生成框架，称为 BC-GAN，它能够同时合成多个视觉搭配的服装图像。具体而言，为了进一步提高合成结果的时尚兼容性，BC-GAN 通过充分利用所有服装之间的搭配关系，从对比学习的角度提出了一种新的时尚兼容性判别器。我们的模型在一个由我们自己构建的兼容服装的大规模数据集中进行了测试。大量实验结果证实了我们提出的 BC-GAN 在多样性、视觉真实性和时尚兼容性方面与最先进的方法相比的有效性。

Title: SatFlow: Generative model based framework for producing High Resolution Gap Free Remote Sensing Imagery

• Authors: Bharath Irigireddy, Varaprasad Bandaru
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01098
• Pdf URL: https://arxiv.org/pdf/2502.01098
• Copy Paste: [[2502.01098]] SatFlow: Generative model based framework for producing High Resolution Gap Free Remote Sensing Imagery(https://arxiv.org/abs/2502.01098)
• Keywords: generative
• Abstract: Frequent, high-resolution remote sensing imagery is crucial for agricultural and environmental monitoring. Satellites from the Landsat collection offer detailed imagery at 30m resolution but with lower temporal frequency, whereas missions like MODIS and VIIRS provide daily coverage at coarser resolutions. Clouds and cloud shadows contaminate about 55\% of the optical remote sensing observations, posing additional challenges. To address these challenges, we present SatFlow, a generative model-based framework that fuses low-resolution MODIS imagery and Landsat observations to produce frequent, high-resolution, gap-free surface reflectance imagery. Our model, trained via Conditional Flow Matching, demonstrates better performance in generating imagery with preserved structural and spectral integrity. Cloud imputation is treated as an image inpainting task, where the model reconstructs cloud-contaminated pixels and fills gaps caused by scan lines during inference by leveraging the learned generative processes. Experimental results demonstrate the capability of our approach in reliably imputing cloud-covered regions. This capability is crucial for downstream applications such as crop phenology tracking, environmental change detection etc.,
• 摘要：频繁的高分辨率遥感图像对于农业和环境监测至关重要。 Landsat 系列卫星提供 30 米分辨率的详细图像，但时间频率较低，而 MODIS 和 VIIRS 等任务则以较粗的分辨率提供每日覆盖。云和云阴影污染了约 55% 的光学遥感观测，带来了额外的挑战。为了应对这些挑战，我们提出了 SatFlow，这是一个基于生成模型的框架，它融合了低分辨率 MODIS 图像和 Landsat 观测，以生成频繁、高分辨率、无间隙的表面反射图像。我们的模型通过条件流匹配进行训练，在生成具有保留结构和光谱完整性的图像方面表现出更好的性能。云插补被视为图像修复任务，其中模型利用学习到的生成过程重建云污染像素并填补推理过程中扫描线造成的间隙。实验结果证明了我们的方法能够可靠地插补云覆盖区域。这种能力对于作物物候跟踪、环境变化检测等下游应用至关重要。

Title: VidSketch: Hand-drawn Sketch-Driven Video Generation with Diffusion Control

• Authors: Lifan Jiang, Shuang Chen, Boxi Wu, Xiaotong Guan, Jiahui Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01101
• Pdf URL: https://arxiv.org/pdf/2502.01101
• Copy Paste: [[2502.01101]] VidSketch: Hand-drawn Sketch-Driven Video Generation with Diffusion Control(https://arxiv.org/abs/2502.01101)
• Keywords: generation, generative
• Abstract: With the advancement of generative artificial intelligence, previous studies have achieved the task of generating aesthetic images from hand-drawn sketches, fulfilling the public's needs for drawing. However, these methods are limited to static images and lack the ability to control video animation generation using hand-drawn sketches. To address this gap, we propose VidSketch, the first method capable of generating high-quality video animations directly from any number of hand-drawn sketches and simple text prompts, bridging the divide between ordinary users and professional artists. Specifically, our method introduces a Level-Based Sketch Control Strategy to automatically adjust the guidance strength of sketches during the generation process, accommodating users with varying drawing skills. Furthermore, a TempSpatial Attention mechanism is designed to enhance the spatiotemporal consistency of generated video animations, significantly improving the coherence across frames. You can find more detailed cases on our official website.
• 摘要：随着生成式人工智能的进步，先前的研究已经实现了从手绘草图生成具有美感的图像，满足了大众对于绘画的需求。然而这些方法局限于静态图像，缺乏对手绘草图生成视频动画的控制能力。针对这一问题，我们提出了 VidSketch，这是第一个能够直接从任意数量的手绘草图和简单的文本提示生成高质量视频动画的方法，弥合了普通用户和专业艺术家之间的鸿沟。具体来说，我们的方法引入了基于级别的草图控制策略，在生成过程中自动调整草图的引导强度，以适应不同绘画水平的用户。此外，我们设计了 TempSpatial Attention 机制来增强生成的视频动画的时空一致性，显著提高跨帧连贯性。您可以在我们的官方网站上找到更多详细案例。

Title: LayerTracer: Cognitive-Aligned Layered SVG Synthesis via Diffusion Transformer

• Authors: Yiren Song, Danze Chen, Mike Zheng Shou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01105
• Pdf URL: https://arxiv.org/pdf/2502.01105
• Copy Paste: [[2502.01105]] LayerTracer: Cognitive-Aligned Layered SVG Synthesis via Diffusion Transformer(https://arxiv.org/abs/2502.01105)
• Keywords: generation
• Abstract: Generating cognitive-aligned layered SVGs remains challenging due to existing methods' tendencies toward either oversimplified single-layer outputs or optimization-induced shape redundancies. We propose LayerTracer, a diffusion transformer based framework that bridges this gap by learning designers' layered SVG creation processes from a novel dataset of sequential design operations. Our approach operates in two phases: First, a text-conditioned DiT generates multi-phase rasterized construction blueprints that simulate human design workflows. Second, layer-wise vectorization with path deduplication produces clean, editable SVGs. For image vectorization, we introduce a conditional diffusion mechanism that encodes reference images into latent tokens, guiding hierarchical reconstruction while preserving structural integrity. Extensive experiments demonstrate LayerTracer's superior performance against optimization-based and neural baselines in both generation quality and editability, effectively aligning AI-generated vectors with professional design cognition.
• 摘要：由于现有方法倾向于过于简单的单层输出或优化引起的形状冗余，因此生成与认知一致的分层 SVG 仍然具有挑战性。我们提出了一个基于扩散变压器的框架 LayerTracer，它通过从一个新的顺序设计操作数据集中学习设计师的分层 SVG 创建过程来弥补这一差距。我们的方法分为两个阶段：首先，文本条件 DiT 生成模拟人类设计工作流程的多阶段栅格化构造蓝图。其次，具有路径重复数据删除的逐层矢量化可生成干净、可编辑的 SVG。对于图像矢量化，我们引入了一种条件扩散机制，将参考图像编码为潜在标记，在保持结构完整性的同时指导分层重建。大量实验表明，LayerTracer 在生成质量和可编辑性方面均优于基于优化和神经基线，有效地将 AI 生成的矢量与专业设计认知相结合。

Title: GTG: Generalizable Trajectory Generation Model for Urban Mobility

• Authors: Jingyuan Wang, Yujing Lin, Yudong Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01107
• Pdf URL: https://arxiv.org/pdf/2502.01107
• Copy Paste: [[2502.01107]] GTG: Generalizable Trajectory Generation Model for Urban Mobility(https://arxiv.org/abs/2502.01107)
• Keywords: generation
• Abstract: Trajectory data mining is crucial for smart city management. However, collecting large-scale trajectory datasets is challenging due to factors such as commercial conflicts and privacy regulations. Therefore, we urgently need trajectory generation techniques to address this issue. Existing trajectory generation methods rely on the global road network structure of cities. When the road network structure changes, these methods are often not transferable to other cities. In fact, there exist invariant mobility patterns between different cities: 1) People prefer paths with the minimal travel cost; 2) The travel cost of roads has an invariant relationship with the topological features of the road network. Based on the above insight, this paper proposes a Generalizable Trajectory Generation model (GTG). The model consists of three parts: 1) Extracting city-invariant road representation based on Space Syntax method; 2) Cross-city travel cost prediction through disentangled adversarial training; 3) Travel preference learning by shortest path search and preference update. By learning invariant movement patterns, the model is capable of generating trajectories in new cities. Experiments on three datasets demonstrates that our model significantly outperforms existing models in terms of generalization ability.
• 摘要：轨迹数据挖掘对于智慧城市管理至关重要。然而，由于商业冲突和隐私法规等因素，收集大规模轨迹数据集具有挑战性。因此，我们迫切需要轨迹生成技术来解决这一问题。现有的轨迹生成方法依赖于城市的全局道路网络结构。当道路网络结构发生变化时，这些方法往往不能转移到其他城市。事实上，不同城市之间存在不变的流动模式：1）人们喜欢出行成本最小的路径；2）道路的出行成本与道路网络的拓扑特征具有不变的关系。基于以上见解，本文提出了一种可泛化轨迹生成模型（GTG）。该模型由三部分组成：1）基于空间句法方法提取城市不变的道路表示；2）通过解缠对抗训练进行跨城市旅行成本预测；3）通过最短路径搜索和偏好更新进行出行偏好学习。通过学习不变的运动模式，该模型能够生成新城市的轨迹。在三个数据集上的实验表明，我们的模型在泛化能力方面明显优于现有模型。

Title: Learning to Learn Weight Generation via Trajectory Diffusion

• Authors: Yunchuan Guan, Yu Liu, Ke Zhou, Zhiqi Shen, Serge Belongie, Jenq-Neng Hwang, Lei Li
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01117
• Pdf URL: https://arxiv.org/pdf/2502.01117
• Copy Paste: [[2502.01117]] Learning to Learn Weight Generation via Trajectory Diffusion(https://arxiv.org/abs/2502.01117)
• Keywords: generation
• Abstract: Diffusion-based algorithms have emerged as promising techniques for weight generation, particularly in scenarios like multi-task learning that require frequent weight updates. However, existing solutions suffer from limited cross-task transferability. In addition, they only utilize optimal weights as training samples, ignoring the value of other weights in the optimization process. To address these issues, we propose Lt-Di, which integrates the diffusion algorithm with meta-learning to generate weights for unseen tasks. Furthermore, we extend the vanilla diffusion algorithm into a trajectory diffusion algorithm to utilize other weights along the optimization trajectory. Trajectory diffusion decomposes the entire diffusion chain into multiple shorter ones, improving training and inference efficiency. We analyze the convergence properties of the weight generation paradigm and improve convergence efficiency without additional time overhead. Our experiments demonstrate Lt-Di's higher accuracy while reducing computational overhead across various tasks, including zero-shot and few-shot learning, multi-domain generalization, and large-scale language model this http URL code is released at this https URL.
• 摘要：基于扩散的算法已成为权重生成的有前途的技术，特别是在需要频繁更新权重的多任务学习场景中。然而，现有的解决方案在跨任务可转移性方面有限。此外，它们仅使用最佳权重作为训练样本，而忽略了优化过程中其他权重的值。为了解决这些问题，我们提出了 Lt-Di，它将扩散算法与元学习相结合，为未见过的任务生成权重。此外，我们将原始扩散算法扩展为轨迹扩散算法，以利用优化轨迹上的其他权重。轨迹扩散将整个扩散链分解为多个较短的扩散链，从而提高训练和推理效率。我们分析了权重生成范式的收敛特性，并在不增加时间开销的情况下提高了收敛效率。我们的实验证明了 Lt-Di 的准确性更高，同时降低了各种任务的计算开销，包括零样本和小样本学习、多领域泛化和大规模语言模型。此 http URL 代码在此 https URL 发布。

Title: AtmosSci-Bench: Evaluating the Recent Advance of Large Language Model for Atmospheric Science

• Authors: Chenyue Li, Wen Deng, Mengqian Lu, Binhang Yuan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01159
• Pdf URL: https://arxiv.org/pdf/2502.01159
• Copy Paste: [[2502.01159]] AtmosSci-Bench: Evaluating the Recent Advance of Large Language Model for Atmospheric Science(https://arxiv.org/abs/2502.01159)
• Keywords: generation
• Abstract: The rapid advancements in large language models (LLMs), particularly in their reasoning capabilities, hold transformative potential for addressing complex challenges in atmospheric science. However, leveraging LLMs effectively in this domain requires a robust and comprehensive evaluation benchmark. To address this need, we present AtmosSci-Bench, a novel benchmark designed to systematically assess LLM performance across five core categories of atmospheric science problems: hydrology, atmospheric dynamics, atmospheric physics, geophysics, and physical oceanography. We employ a template-based question generation framework, enabling scalable and diverse multiple-choice questions curated from graduate-level atmospheric science problems. We conduct a comprehensive evaluation of representative LLMs, categorized into four groups: instruction-tuned models, advanced reasoning models, math-augmented models, and domain-specific climate models. Our analysis provides some interesting insights into the reasoning and problem-solving capabilities of LLMs in atmospheric science. We believe AtmosSci-Bench can serve as a critical step toward advancing LLM applications in climate service by offering a standard and rigorous evaluation framework. Our source codes are currently available at this https URL.
• 摘要：大型语言模型 (LLM) 的快速发展，尤其是其推理能力的快速发展，为解决大气科学中的复杂挑战提供了变革性的潜力。然而，要有效地利用 LLM 来应对这一领域，需要一个强大而全面的评估基准。为了满足这一需求，我们提出了 AtmosSci-Bench，这是一种新颖的基准，旨在系统地评估 LLM 在五大核心大气科学问题类别中的表现：水文学、大气动力学、大气物理学、地球物理学和物理海洋学。我们采用基于模板的问题生成框架，从研究生级大气科学问题中精选出可扩展且多样化的多项选择题。我们对代表性的 LLM 进行了全面评估，分为四类：指令调整模型、高级推理模型、数学增强模型和领域特定气候模型。我们的分析为 LLM 在大气科学中的推理和解决问题的能力提供了一些有趣的见解。我们相信，通过提供标准而严格的评估框架，AtmosSci-Bench 可以成为推进 LLM 在气候服务中的应用的关键一步。我们的源代码目前可通过此 https URL 获得。

Title: Dance recalibration for dance coherency with recurrent convolution block

• Authors: Seungho Eum, Ihjoon Cho, Junghyeon Kim
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01190
• Pdf URL: https://arxiv.org/pdf/2502.01190
• Copy Paste: [[2502.01190]] Dance recalibration for dance coherency with recurrent convolution block(https://arxiv.org/abs/2502.01190)
• Keywords: generation, generative
• Abstract: With the recent advancements in generative AI such as GAN, Diffusion, and VAE, the use of generative AI for dance generation has seen significant progress and received considerable interest. In this study, We propose R-Lodge, an enhanced version of Lodge. R-Lodge incorporates Recurrent Sequential Representation Learning named Dance Recalibration to original coarse-to-fine long dance generation model. R-Lodge utilizes Dance Recalibration method using $N$ Dance Recalibration Block to address the lack of consistency in the coarse dance representation of the Lodge model. By utilizing this method, each generated dance motion incorporates a bit of information from the previous dance motions. We evaluate R-Lodge on FineDance dataset and the results show that R-Lodge enhances the consistency of the whole generated dance motions.
• 摘要：随着 GAN、Diffusion 和 VAE 等生成式 AI 的最新进展，生成式 AI 在舞蹈生成中的应用取得了重大进展并引起了广泛关注。在本研究中，我们提出了 Lodge 的增强版 R-Lodge。R-Lodge 将名为“舞蹈重新校准”的循环顺序表征学习融入到原始的由粗到细的长舞蹈生成模型中。R-Lodge 利用舞蹈重新校准方法（使用 $N$ 舞蹈重新校准块）来解决 Lodge 模型粗舞蹈表征缺乏一致性的问题。通过利用这种方法，每个生成的舞蹈动作都包含了来自之前舞蹈动作的一些信息。我们在 FineDance 数据集上评估了 R-Lodge，结果表明 R-Lodge 增强了整个生成的舞蹈动作的一致性。

Title: One-to-Normal: Anomaly Personalization for Few-shot Anomaly Detection

• Authors: Yiyue Li, Shaoting Zhang, Kang Li, Qicheng Lao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01201
• Pdf URL: https://arxiv.org/pdf/2502.01201
• Copy Paste: [[2502.01201]] One-to-Normal: Anomaly Personalization for Few-shot Anomaly Detection(https://arxiv.org/abs/2502.01201)
• Keywords: generation
• Abstract: Traditional Anomaly Detection (AD) methods have predominantly relied on unsupervised learning from extensive normal data. Recent AD methods have evolved with the advent of large pre-trained vision-language models, enhancing few-shot anomaly detection capabilities. However, these latest AD methods still exhibit limitations in accuracy improvement. One contributing factor is their direct comparison of a query image's features with those of few-shot normal images. This direct comparison often leads to a loss of precision and complicates the extension of these techniques to more complex domains--an area that remains underexplored in a more refined and comprehensive manner. To address these limitations, we introduce the anomaly personalization method, which performs a personalized one-to-normal transformation of query images using an anomaly-free customized generation model, ensuring close alignment with the normal manifold. Moreover, to further enhance the stability and robustness of prediction results, we propose a triplet contrastive anomaly inference strategy, which incorporates a comprehensive comparison between the query and generated anomaly-free data pool and prompt information. Extensive evaluations across eleven datasets in three domains demonstrate our model's effectiveness compared to the latest AD methods. Additionally, our method has been proven to transfer flexibly to other AD methods, with the generated image data effectively improving the performance of other AD methods.
• 摘要：传统的异常检测 (AD) 方法主要依赖于从大量正常数据中进行无监督学习。随着大型预训练视觉语言模型的出现，最近的 AD 方法得到了发展，增强了少量样本异常检测能力。然而，这些最新的 AD 方法在准确度提升方面仍然存在局限性。其中一个因素是它们直接将查询图像的特征与少量样本正常图像的特征进行比较。这种直接比较通常会导致精度损失，并使这些技术难以扩展到更复杂的领域——这一领域仍未得到更精细和全面的探索。为了解决这些限制，我们引入了异常个性化方法，该方法使用无异常定制生成模型对查询图像执行个性化的一对正常转换，确保与正常流形紧密对齐。此外，为了进一步增强预测结果的稳定性和稳健性，我们提出了一种三重对比异常推理策略，该策略结合了查询和生成的无异常数据池与提示信息之间的全面比较。在三个领域的 11 个数据集上进行的广泛评估证明了我们的模型与最新的 AD 方法相比的有效性。此外，我们的方法已被证明可以灵活地转移到其他 AD 方法，生成的图像数据可以有效提高其他 AD 方法的性能。

Title: Land Surface Temperature Super-Resolution with a Scale-Invariance-Free Neural Approach: Application to MODIS

• Authors: Romuald Ait-Bachir (ODYSSEY, IMT Atlantique - MEE, Lab-STICC\_OSE), Carlos Granero-Belinchon (ODYSSEY, IMT Atlantique - MEE, Lab-STICC\_OSE), Aurélie Michel, Julien Michel (CESBIO, CNES), Xavier Briottet, Lucas Drumetz (Lab-STICC\_OSE, IMT Atlantique - MEE, ODYSSEY)
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01204
• Pdf URL: https://arxiv.org/pdf/2502.01204
• Copy Paste: [[2502.01204]] Land Surface Temperature Super-Resolution with a Scale-Invariance-Free Neural Approach: Application to MODIS(https://arxiv.org/abs/2502.01204)
• Keywords: super-resolution
• Abstract: Due to the trade-off between the temporal and spatial resolution of thermal spaceborne sensors, super-resolution methods have been developed to provide fine-scale Land SurfaceTemperature (LST) maps. Most of them are trained at low resolution but applied at fine resolution, and so they require a scale-invariance hypothesis that is not always adapted. Themain contribution of this work is the introduction of a Scale-Invariance-Free approach for training Neural Network (NN) models, and the implementation of two NN models, calledScale-Invariance-Free Convolutional Neural Network for Super-Resolution (SIF-CNN-SR) for the super-resolution of MODIS LST products. The Scale-Invariance-Free approach consists ontraining the models in order to provide LST maps at high spatial resolution that recover the initial LST when they are degraded at low resolution and that contain fine-scale texturesinformed by the high resolution NDVI. The second contribution of this work is the release of a test database with ASTER LST images concomitant with MODIS ones that can be usedfor evaluation of super-resolution algorithms. We compare the two proposed models, SIF-CNN-SR1 and SIF-CNN-SR2, with four state-of-the-art methods, Bicubic, DMS, ATPRK, Tsharp,and a CNN sharing the same architecture as SIF-CNN-SR but trained under the scale-invariance hypothesis. We show that SIF-CNN-SR1 outperforms the state-of-the-art methods and the other two CNN models as evaluated with LPIPS and Fourier space metrics focusing on the analysis of textures. These results and the available ASTER-MODIS database for evaluation are promising for future studies on super-resolution of LST.
• 摘要：由于热空间传感器的时间分辨率和空间分辨率之间的权衡，已经开发了超分辨率方法来提供精细尺度的陆地表面温度 (LST) 地图。它们中的大多数都是在低分辨率下训练的，但应用于高分辨率，因此它们需要一个并不总是适应的尺度不变性假设。这项工作的主要贡献是引入了一种无尺度不变性的方法来训练神经网络 (NN) 模型，并实现了两个 NN 模型，称为用于超分辨率的无尺度不变性卷积神经网络 (SIF-CNN-SR)，用于 MODIS LST 产品的超分辨率。无尺度不变性方法包括训练模型以提供高空间分辨率的 LST 地图，当 LST 在低分辨率下退化时，这些地图可以恢复初始 LST，并且包含由高分辨率 NDVI 提供的精细纹理。这项工作的第二个贡献是发布了一个测试数据库，其中包含 ASTER LST 图像和 MODIS 图像，可用于评估超分辨率算法。我们将两个提出的模型 SIF-CNN-SR1 和 SIF-CNN-SR2 与四种最先进的方法 Bicubic、DMS、ATPRK、Tsharp 以及与 SIF-CNN-SR 共享相同架构但在尺度不变性假设下训练的 CNN 进行了比较。我们表明，在使用 LPIPS 和专注于纹理分析的傅里叶空间指标进行评估时，SIF-CNN-SR1 优于最先进的方法和其他两种 CNN 模型。这些结果和可用于评估的 ASTER-MODIS 数据库对未来 LST 超分辨率研究大有裨益。

Title: Almost Surely Safe Alignment of Large Language Models at Inference-Time

• Authors: Xiaotong Ji, Shyam Sundhar Ramesh, Matthieu Zimmer, Ilija Bogunovic, Jun Wang, Haitham Bou Ammar
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01208
• Pdf URL: https://arxiv.org/pdf/2502.01208
• Copy Paste: [[2502.01208]] Almost Surely Safe Alignment of Large Language Models at Inference-Time(https://arxiv.org/abs/2502.01208)
• Keywords: generation
• Abstract: Even highly capable large language models (LLMs) can produce biased or unsafe responses, and alignment techniques, such as RLHF, aimed at mitigating this issue, are expensive and prone to overfitting as they retrain the LLM. This paper introduces a novel inference-time alignment approach that ensures LLMs generate safe responses almost surely, i.e., with a probability approaching one. We achieve this by framing the safe generation of inference-time responses as a constrained Markov decision process within the LLM's latent space. Crucially, we augment a safety state that tracks the evolution of safety constraints and enables us to demonstrate formal safety guarantees upon solving the MDP in the latent space. Building on this foundation, we propose InferenceGuard, a practical implementation that safely aligns LLMs without modifying the model weights. Empirically, we demonstrate InferenceGuard effectively balances safety and task performance, outperforming existing inference-time alignment methods in generating safe and aligned responses.
• 摘要：即使是功能强大的大型语言模型 (LLM) 也会产生有偏差或不安全的响应，而旨在缓解此问题的对齐技术（例如 RLHF）成本高昂，并且在重新训练 LLM 时容易过度拟合。本文介绍了一种新颖的推理时间对齐方法，可确保 LLM 几乎肯定地（即概率接近 1）生成安全响应。我们通过将推理时间响应的安全生成构建为 LLM 潜在空间内的受约束马尔可夫决策过程来实现这一点。至关重要的是，我们增强了一种安全状态，该状态可跟踪安全约束的演变，并使我们能够在解决潜在空间中的 MDP 时展示正式的安全保证。在此基础上，我们提出了 InferenceGuard，这是一种在不修改模型权重的情况下安全地对齐 LLM 的实用实现。从经验上讲，我们证明 InferenceGuard 有效地平衡了安全性和任务性能，在生成安全和对齐的响应方面优于现有的推理时间对齐方法。

Title: Heterogeneous Image GNN: Graph-Conditioned Diffusion for Image Synthesis

• Authors: Rupert Menneer, Christos Margadji, Sebastian W. Pattinson
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01309
• Pdf URL: https://arxiv.org/pdf/2502.01309
• Copy Paste: [[2502.01309]] Heterogeneous Image GNN: Graph-Conditioned Diffusion for Image Synthesis(https://arxiv.org/abs/2502.01309)
• Keywords: generation
• Abstract: We introduce a novel method for conditioning diffusion-based image synthesis models with heterogeneous graph data. Existing approaches typically incorporate conditioning variables directly into model architectures, either through cross-attention layers that attend to text latents or image concatenation that spatially restrict generation. However, these methods struggle to handle complex scenarios involving diverse, relational conditioning variables, which are more naturally represented as unstructured graphs. This paper presents Heterogeneous Image Graphs (HIG), a novel representation that models conditioning variables and target images as two interconnected graphs, enabling efficient handling of variable-length conditioning inputs and their relationships. We also propose a magnitude-preserving GNN that integrates the HIG into the existing EDM2 diffusion model using a ControlNet approach. Our approach improves upon the SOTA on a variety of conditioning inputs for the COCO-stuff and Visual Genome datasets, and showcases the ability to condition on graph attributes and relationships represented by edges in the HIG.
• 摘要：我们介绍了一种使用异构图数据调节基于扩散的图像合成模型的新方法。现有方法通常将条件变量直接合并到模型架构中，要么通过关注文本潜能的交叉注意层，要么通过空间限制生成的图像连接。然而，这些方法难以处理涉及各种关系条件变量的复杂场景，这些变量更自然地表示为非结构化图。本文介绍了异构图像图 (HIG)，这是一种新颖的表示形式，它将条件变量和目标图像建模为两个相互连接的图，从而能够有效处理可变长度的条件输入及其关系。我们还提出了一种保持幅度的 GNN，使用 ControlNet 方法将 HIG 集成到现有的 EDM2 扩散模型中。我们的方法改进了 COCO-stuff 和 Visual Genome 数据集的各种条件输入的 SOTA，并展示了对 HIG 中边表示的图属性和关系进行条件处理的能力。

Title: A Statistical Learning Perspective on Semi-dual Adversarial Neural Optimal Transport Solvers

• Authors: Roman Tarasov, Petr Mokrov, Milena Gazdieva, Evgeny Burnaev, Alexander Korotin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01310
• Pdf URL: https://arxiv.org/pdf/2502.01310
• Copy Paste: [[2502.01310]] A Statistical Learning Perspective on Semi-dual Adversarial Neural Optimal Transport Solvers(https://arxiv.org/abs/2502.01310)
• Keywords: super-resolution, generative
• Abstract: Neural network based Optimal Transport (OT) is a recent and fruitful direction in the generative modeling community. It finds its applications in various fields such as domain translation, image super-resolution, computational biology and others. Among the existing approaches to OT, of considerable interest are adversarial minimax solvers based on semi-dual formulations of OT problems. While promising, these methods lack theoretical investigation from a statistical learning perspective. Our work fills this gap by establishing upper bounds on the generalization error of an approximate OT map recovered by the minimax quadratic OT solver. Importantly, the bounds we derive depend solely on some standard statistical and mathematical properties of the considered functional classes (neural networks). While our analysis focuses on the quadratic OT, we believe that similar bounds could be derived for more general OT formulations, paving the promising direction for future research.
• 摘要：基于神经网络的最佳传输 (OT) 是生成建模社区中一个近期且富有成果的方向。它可应用于领域翻译、图像超分辨率、计算生物学等各个领域。在现有的 OT 方法中，基于 OT 问题半对偶公式的对抗性极小极大求解器引起了广泛关注。这些方法虽然很有前景，但缺乏从统计学习角度进行的理论研究。我们的工作通过建立由极小极大二次 OT 求解器恢复的近似 OT 映射的泛化误差的上限来填补这一空白。重要的是，我们得出的界限仅取决于所考虑的功能类（神经网络）的一些标准统计和数学属性。虽然我们的分析侧重于二次 OT，但我们相信可以为更一般的 OT 公式得出类似的界限，为未来的研究铺平了有希望的方向。

Title: ConceptVAE: Self-Supervised Fine-Grained Concept Disentanglement from 2D Echocardiographies

• Authors: Costin F. Ciusdel, Alex Serban, Tiziano Passerini
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01335
• Pdf URL: https://arxiv.org/pdf/2502.01335
• Copy Paste: [[2502.01335]] ConceptVAE: Self-Supervised Fine-Grained Concept Disentanglement from 2D Echocardiographies(https://arxiv.org/abs/2502.01335)
• Keywords: generation
• Abstract: While traditional self-supervised learning methods improve performance and robustness across various medical tasks, they rely on single-vector embeddings that may not capture fine-grained concepts such as anatomical structures or organs. The ability to identify such concepts and their characteristics without supervision has the potential to improve pre-training methods, and enable novel applications such as fine-grained image retrieval and concept-based outlier detection. In this paper, we introduce ConceptVAE, a novel pre-training framework that detects and disentangles fine-grained concepts from their style characteristics in a self-supervised manner. We present a suite of loss terms and model architecture primitives designed to discretise input data into a preset number of concepts along with their local style. We validate ConceptVAE both qualitatively and quantitatively, demonstrating its ability to detect fine-grained anatomical structures such as blood pools and septum walls from 2D cardiac echocardiographies. Quantitatively, ConceptVAE outperforms traditional self-supervised methods in tasks such as region-based instance retrieval, semantic segmentation, out-of-distribution detection, and object detection. Additionally, we explore the generation of in-distribution synthetic data that maintains the same concepts as the training data but with distinct styles, highlighting its potential for more calibrated data generation. Overall, our study introduces and validates a promising new pre-training technique based on concept-style disentanglement, opening multiple avenues for developing models for medical image analysis that are more interpretable and explainable than black-box approaches.
• 摘要：虽然传统的自监督学习方法可以提高各种医疗任务的性能和稳健性，但它们依赖于单向量嵌入，而单向量嵌入可能无法捕获诸如解剖结构或器官之类的细粒度概念。无需监督即可识别此类概念及其特征的能力有可能改进预训练方法，并实现诸如细粒度图像检索和基于概念的异常值检测等新应用。在本文中，我们介绍了 ConceptVAE，这是一种新颖的预训练框架，它以自监督的方式检测细粒度概念并将其从其风格特征中分离出来。我们提出了一套损失项和模型架构原语，旨在将输入数据离散为预设数量的概念及其局部风格。我们对 ConceptVAE 进行了定性和定量验证，证明了它能够从 2D 心脏超声心动图中检测血池和隔膜壁等细粒度解剖结构。从定量上看，ConceptVAE 在基于区域的实例检索、语义分割、分布外检测和对象检测等任务中的表现优于传统的自监督方法。此外，我们还探索了分布内合成数据的生成，这些数据与训练数据保持相同的概念，但风格不同，突出了其生成更精确的数据的潜力。总的来说，我们的研究引入并验证了一种基于概念风格解缠的有前途的新型预训练技术，为开发比黑盒方法更易于解释和说明的医学图像分析模型开辟了多种途径。

Title: Inverse Bridge Matching Distillation

• Authors: Nikita Gushchin, David Li, Daniil Selikhanovych, Evgeny Burnaev, Dmitry Baranchuk, Alexander Korotin
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01362
• Pdf URL: https://arxiv.org/pdf/2502.01362
• Copy Paste: [[2502.01362]] Inverse Bridge Matching Distillation(https://arxiv.org/abs/2502.01362)
• Keywords: restoration, super-resolution, generation
• Abstract: Learning diffusion bridge models is easy; making them fast and practical is an art. Diffusion bridge models (DBMs) are a promising extension of diffusion models for applications in image-to-image translation. However, like many modern diffusion and flow models, DBMs suffer from the problem of slow inference. To address it, we propose a novel distillation technique based on the inverse bridge matching formulation and derive the tractable objective to solve it in practice. Unlike previously developed DBM distillation techniques, the proposed method can distill both conditional and unconditional types of DBMs, distill models in a one-step generator, and use only the corrupted images for training. We evaluate our approach for both conditional and unconditional types of bridge matching on a wide set of setups, including super-resolution, JPEG restoration, sketch-to-image, and other tasks, and show that our distillation technique allows us to accelerate the inference of DBMs from 4x to 100x and even provide better generation quality than used teacher model depending on particular setup.
• 摘要：学习扩散桥模型很容易；让它们快速实用是一门艺术。扩散桥模型 (DBM) 是扩散模型的一个有前途的扩展，可用于图像到图像的转换。然而，与许多现代扩散和流动模型一样，DBM 也存在推理速度慢的问题。为了解决这个问题，我们提出了一种基于逆桥匹配公式的新型蒸馏技术，并推导出在实践中解决该问题的可处理目标。与以前开发的 DBM 蒸馏技术不同，所提出的方法可以蒸馏条件和无条件类型的 DBM，在一步生成器中蒸馏模型，并且仅使用损坏的图像进行训练。我们在一系列设置上评估了我们的方法对条件和无条件类型的桥匹配，包括超分辨率、JPEG 恢复、草图到图像和其他任务，并表明我们的蒸馏技术使我们能够将 DBM 的推理速度从 4 倍加速到 100 倍，甚至根据特定设置提供比使用的教师模型更好的生成质量。

Title: InfoBridge: Mutual Information estimation via Bridge Matching

• Authors: Sergei Kholkin, Ivan Butakov, Evgeny Burnaev, Nikita Gushchin, Alexander Korotin
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.01383
• Pdf URL: https://arxiv.org/pdf/2502.01383
• Copy Paste: [[2502.01383]] InfoBridge: Mutual Information estimation via Bridge Matching(https://arxiv.org/abs/2502.01383)
• Keywords: generative
• Abstract: Diffusion bridge models have recently become a powerful tool in the field of generative modeling. In this work, we leverage their power to address another important problem in machine learning and information theory - the estimation of the mutual information (MI) between two random variables. We show that by using the theory of diffusion bridges, one can construct an unbiased estimator for data posing difficulties for conventional MI estimators. We showcase the performance of our estimator on a series of standard MI estimation benchmarks.
• 摘要：扩散桥模型最近已成为生成建模领域的一个强大工具。在这项工作中，我们利用它们的力量来解决机器学习和信息论中的另一个重要问题——两个随机变量之间的互信息 (MI) 的估计。我们表明，通过使用扩散桥理论，可以为传统 MI 估计器难以处理的数据构建无偏估计器。我们在一系列标准 MI 估计基准上展示了我们的估计器的性能。

Title: Learning Traffic Anomalies from Generative Models on Real-Time Observations

• Authors: Fotis I. Giasemis, Alexandros Sopasakis
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01391
• Pdf URL: https://arxiv.org/pdf/2502.01391
• Copy Paste: [[2502.01391]] Learning Traffic Anomalies from Generative Models on Real-Time Observations(https://arxiv.org/abs/2502.01391)
• Keywords: generative
• Abstract: Accurate detection of traffic anomalies is crucial for effective urban traffic management and congestion mitigation. We use the Spatiotemporal Generative Adversarial Network (STGAN) framework combining Graph Neural Networks and Long Short-Term Memory networks to capture complex spatial and temporal dependencies in traffic data. We apply STGAN to real-time, minute-by-minute observations from 42 traffic cameras across Gothenburg, Sweden, collected over several months in 2020. The images are processed to compute a flow metric representing vehicle density, which serves as input for the model. Training is conducted on data from April to November 2020, and validation is performed on a separate dataset from November 14 to 23, 2020. Our results demonstrate that the model effectively detects traffic anomalies with high precision and low false positive rates. The detected anomalies include camera signal interruptions, visual artifacts, and extreme weather conditions affecting traffic flow.
• 摘要：准确检测交通异常对于有效的城市交通管理和缓解交通拥堵至关重要。我们使用时空生成对抗网络 (STGAN) 框架，结合图神经网络和长短期记忆网络来捕获交通数据中复杂的空间和时间依赖关系。我们将 STGAN 应用于 2020 年几个月内从瑞典哥德堡的 42 个交通摄像头收集的实时、逐分钟观测数据。处理图像以计算代表车辆密度的流量指标，该指标作为模型的输入。训练是在 2020 年 4 月至 11 月的数据上进行的，验证是在 2020 年 11 月 14 日至 23 日对单独的数据集进行的。我们的结果表明，该模型可以有效地检测交通异常，精度高，误报率低。检测到的异常包括摄像头信号中断、视觉伪影和影响交通流量的极端天气条件。

Title: Human Body Restoration with One-Step Diffusion Model and A New Benchmark

• Authors: Jue Gong, Jingkai Wang, Zheng Chen, Xing Liu, Hong Gu, Yulun Zhang, Xiaokang Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01411
• Pdf URL: https://arxiv.org/pdf/2502.01411
• Copy Paste: [[2502.01411]] Human Body Restoration with One-Step Diffusion Model and A New Benchmark(https://arxiv.org/abs/2502.01411)
• Keywords: restoration
• Abstract: Human body restoration, as a specific application of image restoration, is widely applied in practice and plays a vital role across diverse fields. However, thorough research remains difficult, particularly due to the lack of benchmark datasets. In this study, we propose a high-quality dataset automated cropping and filtering (HQ-ACF) pipeline. This pipeline leverages existing object detection datasets and other unlabeled images to automatically crop and filter high-quality human images. Using this pipeline, we constructed a person-based restoration with sophisticated objects and natural activities (\emph{PERSONA}) dataset, which includes training, validation, and test sets. The dataset significantly surpasses other human-related datasets in both quality and content richness. Finally, we propose \emph{OSDHuman}, a novel one-step diffusion model for human body restoration. Specifically, we propose a high-fidelity image embedder (HFIE) as the prompt generator to better guide the model with low-quality human image information, effectively avoiding misleading prompts. Experimental results show that OSDHuman outperforms existing methods in both visual quality and quantitative metrics. The dataset and code will at this https URL.
• 摘要：人体复原作为图像复原的一个具体应用，在实践中得到了广泛的应用，并在各个领域发挥着至关重要的作用。然而，深入的研究仍然很困难，特别是由于缺乏基准数据集。在本研究中，我们提出了一种高质量数据集自动裁剪和过滤（HQ-ACF）流程。该流程利用现有的物体检测数据集和其他未标记的图像来自动裁剪和过滤高质量的人体图像。利用这个流程，我们构建了一个基于复杂物体和自然活动的人体复原（\emph{PERSONA}）数据集，其中包括训练、验证和测试集。该数据集在质量和内容丰富度方面都远远超过其他与人类相关的数据集。最后，我们提出了一种用于人体复原的新型一步扩散模型\emph{OSDHuman}。具体来说，我们提出了一个高保真图像嵌入器（HFIE）作为提示生成器，以便更好地用低质量的人体图像信息引导模型，有效避免误导性提示。实验结果表明，OSDHuman 在视觉质量和定量指标方面均优于现有方法。数据集和代码将位于此 https URL。

Title: Categorical Schr\"odinger Bridge Matching

• Authors: Grigoriy Ksenofontov, Alexander Korotin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01416
• Pdf URL: https://arxiv.org/pdf/2502.01416
• Copy Paste: [[2502.01416]] Categorical Schr\"odinger Bridge Matching(https://arxiv.org/abs/2502.01416)
• Keywords: generative
• Abstract: The Schrödinger Bridge (SB) is a powerful framework for solving generative modeling tasks such as unpaired domain translation. Most SB-related research focuses on continuous data space $\mathbb{R}^{D}$ and leaves open theoretical and algorithmic questions about applying SB methods to discrete data, e.g, on finite spaces $\mathbb{S}^{D}$. Notable examples of such sets $\mathbb{S}$ are codebooks of vector-quantized (VQ) representations of modern autoencoders, tokens in texts, categories of atoms in molecules, etc. In this paper, we provide a theoretical and algorithmic foundation for solving SB in discrete spaces using the recently introduced Iterative Markovian Fitting (IMF) procedure. Specifically, we theoretically justify the convergence of discrete-time IMF (D-IMF) to SB in discrete spaces. This enables us to develop a practical computational algorithm for SB which we call Categorical Schrödinger Bridge Matching (CSBM). We show the performance of CSBM via a series of experiments with synthetic data and VQ representations of images.
• 摘要：薛定谔桥 (SB) 是一个强大的框架，可用于解决生成建模任务，例如非成对域转换。大多数与 SB 相关的研究都集中在连续数据空间 $\mathbb{R}^{D}$ 上，而将 SB 方法应用于离散数据（例如有限空间 $\mathbb{S}^{D}$）的理论和算法问题尚未解决。此类集合 $\mathbb{S}$ 的显著示例是现代自动编码器的矢量量化 (VQ) 表示的代码本、文本中的标记、分子中的原子类别等。在本文中，我们使用最近引入的迭代马尔可夫拟合 (IMF) 程序为在离散空间中解决 SB 提供了理论和算法基础。具体而言，我们从理论上证明了离散时间 IMF (D-IMF) 在离散空间中向 SB 的收敛性。这使我们能够开发一种实用的 SB 计算算法，我们称之为分类薛定谔桥匹配 (CSBM)。我们通过使用合成数据和图像的 VQ 表示进行的一系列实验展示了 CSBM 的性能。

Title: Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models

• Authors: Mingi Jung, Saehuyng Lee, Eunji Kim, Sungroh Yoon
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01419
• Pdf URL: https://arxiv.org/pdf/2502.01419
• Copy Paste: [[2502.01419]] Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models(https://arxiv.org/abs/2502.01419)
• Keywords: generation
• Abstract: Detailed image captioning is essential for tasks like data generation and aiding visually impaired individuals. High-quality captions require a balance between precision and recall, which remains challenging for current multimodal large language models (MLLMs). In this work, we hypothesize that this limitation stems from weakening and increasingly noisy visual attention as responses lengthen. To address this issue, we propose SPARC (Selective Progressive Attention ReCalibration), a training-free method that enhances the contribution of visual tokens during decoding. SPARC is founded on three key observations: (1) increasing the influence of all visual tokens reduces recall; thus, SPARC selectively amplifies visual tokens; (2) as captions lengthen, visual attention becomes noisier, so SPARC identifies critical visual tokens by leveraging attention differences across time steps; (3) as visual attention gradually weakens, SPARC reinforces it to preserve its influence. Our experiments, incorporating both automated and human evaluations, demonstrate that existing methods improve the precision of MLLMs at the cost of recall. In contrast, our proposed method enhances both precision and recall with minimal computational overhead.
• 摘要：详细的图像字幕对于数据生成和帮助视障人士等任务至关重要。高质量的字幕需要在准确率和召回率之间取得平衡，这对当前的多模态大型语言模型 (MLLM) 来说仍然具有挑战性。在这项工作中，我们假设这种限制源于随着响应时间延长，视觉注意力会减弱且噪声越来越大。为了解决这个问题，我们提出了 SPARC（选择性渐进注意力重新校准），这是一种无需训练的方法，可在解码过程中增强视觉标记的贡献。SPARC 基于三个关键观察结果：（1）增加所有视觉标记的影响力会降低召回率；因此，SPARC 会选择性地放大视觉标记；（2）随着字幕时间延长，视觉注意力会变得更加嘈杂，因此 SPARC 会利用时间步骤之间的注意力差异来识别关键的视觉标记；（3）随着视觉注意力逐渐减弱，SPARC 会加强它以保持其影响力。我们的实验结合了自动和人工评估，表明现有方法以召回率为代价提高了 MLLM 的准确率。相比之下，我们提出的方法以最小的计算开销提高了准确率和召回率。

Title: Improved Training Technique for Latent Consistency Models

• Authors: Quan Dao, Khanh Doan, Di Liu, Trung Le, Dimitris Metaxas
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01441
• Pdf URL: https://arxiv.org/pdf/2502.01441
• Copy Paste: [[2502.01441]] Improved Training Technique for Latent Consistency Models(https://arxiv.org/abs/2502.01441)
• Keywords: generation, generative
• Abstract: Consistency models are a new family of generative models capable of producing high-quality samples in either a single step or multiple steps. Recently, consistency models have demonstrated impressive performance, achieving results on par with diffusion models in the pixel space. However, the success of scaling consistency training to large-scale datasets, particularly for text-to-image and video generation tasks, is determined by performance in the latent space. In this work, we analyze the statistical differences between pixel and latent spaces, discovering that latent data often contains highly impulsive outliers, which significantly degrade the performance of iCT in the latent space. To address this, we replace Pseudo-Huber losses with Cauchy losses, effectively mitigating the impact of outliers. Additionally, we introduce a diffusion loss at early timesteps and employ optimal transport (OT) coupling to further enhance performance. Lastly, we introduce the adaptive scaling-$c$ scheduler to manage the robust training process and adopt Non-scaling LayerNorm in the architecture to better capture the statistics of the features and reduce outlier impact. With these strategies, we successfully train latent consistency models capable of high-quality sampling with one or two steps, significantly narrowing the performance gap between latent consistency and diffusion models. The implementation is released here: this https URL
• 摘要：一致性模型是一类新的生成模型，能够通过一步或多步生成高质量样本。最近，一致性模型表现出了令人印象深刻的性能，在像素空间中取得了与扩散模型相当的结果。然而，将一致性训练扩展到大规模数据集的成功，特别是对于文本到图像和视频生成任务，取决于潜在空间的性能。在这项工作中，我们分析了像素和潜在空间之间的统计差异，发现潜在数据通常包含高度脉冲的异常值，这显著降低了 iCT 在潜在空间中的性能。为了解决这个问题，我们用柯西损失代替了伪 Huber 损失，有效地减轻了异常值的影响。此外，我们在早期时间步引入了扩散损失，并采用最佳传输 (OT) 耦合来进一步提高性能。最后，我们引入了自适应缩放-$c$ 调度程序来管理稳健的训练过程，并在架构中采用非缩放 LayerNorm 来更好地捕获特征的统计数据并减少异常值的影响。通过这些策略，我们成功地训练出能够通过一到两步进行高质量采样的潜在一致性模型，显著缩小了潜在一致性和扩散模型之间的性能差距。实现在此处发布：此 https URL

Title: MoireDB: Formula-generated Interference-fringe Image Dataset

• Authors: Yuto Matsuo, Ryo Hayamizu, Hirokatsu Kataoka, Akio Nakamura
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01490
• Pdf URL: https://arxiv.org/pdf/2502.01490
• Copy Paste: [[2502.01490]] MoireDB: Formula-generated Interference-fringe Image Dataset(https://arxiv.org/abs/2502.01490)
• Keywords: generative
• Abstract: Image recognition models have struggled to treat recognition robustness to real-world degradations. In this context, data augmentation methods like PixMix improve robustness but rely on generative arts and feature visualizations (FVis), which have copyright, drawing cost, and scalability issues. We propose MoireDB, a formula-generated interference-fringe image dataset for image augmentation enhancing robustness. MoireDB eliminates copyright concerns, reduces dataset assembly costs, and enhances robustness by leveraging illusory patterns. Experiments show that MoireDB augmented images outperforms traditional Fractal arts and FVis-based augmentations, making it a scalable and effective solution for improving model robustness against real-world degradations.
• 摘要：图像识别模型一直难以处理识别鲁棒性以应对现实世界的退化。在这种情况下，像 PixMix 这样的数据增强方法可以提高鲁棒性，但依赖于生成艺术和特征可视化 (FVis)，而这些艺术和特征可视化存在版权、绘制成本和可扩展性问题。我们提出了 MoireDB，这是一个公式生成的干涉条纹图像数据集，用于图像增强以增强鲁棒性。MoireDB 消除了版权问题，降低了数据集组装成本，并通过利用虚幻模式增强了鲁棒性。实验表明，MoireDB 增强图像优于传统的分形艺术和基于 FVis 的增强，使其成为一种可扩展且有效的解决方案，可提高模型对现实世界退化的鲁棒性。

Title: End-to-end Training for Text-to-Image Synthesis using Dual-Text Embeddings

• Authors: Yeruru Asrar Ahmed, Anurag Mittal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01507
• Pdf URL: https://arxiv.org/pdf/2502.01507
• Copy Paste: [[2502.01507]] End-to-end Training for Text-to-Image Synthesis using Dual-Text Embeddings(https://arxiv.org/abs/2502.01507)
• Keywords: generative
• Abstract: Text-to-Image (T2I) synthesis is a challenging task that requires modeling complex interactions between two modalities ( i.e., text and image). A common framework adopted in recent state-of-the-art approaches to achieving such multimodal interactions is to bootstrap the learning process with pre-trained image-aligned text embeddings trained using contrastive loss. Furthermore, these embeddings are typically trained generically and reused across various synthesis models. In contrast, we explore an approach to learning text embeddings specifically tailored to the T2I synthesis network, trained in an end-to-end fashion. Further, we combine generative and contrastive training and use two embeddings, one optimized to enhance the photo-realism of the generated images, and the other seeking to capture text-to-image alignment. A comprehensive set of experiments on three text-to-image benchmark datasets (Oxford-102, Caltech-UCSD, and MS-COCO) reveal that having two separate embeddings gives better results than using a shared one and that such an approach performs favourably in comparison with methods that use text representations from a pre-trained text encoder trained using a discriminative approach. Finally, we demonstrate that such learned embeddings can be used in other contexts as well, such as text-to-image manipulation.
• 摘要：文本到图像 (T2I) 合成是一项具有挑战性的任务，需要对两种模态（即文本和图像）之间的复杂交互进行建模。近期最先进的方法为实现此类多模态交互而采用的常见框架是使用对比损失训练的预训练图像对齐文本嵌入来引导学习过程。此外，这些嵌入通常经过通用训练并在各种合成模型中重复使用。相比之下，我们探索了一种专门针对 T2I 合成网络的文本嵌入学习方法，以端到端的方式进行训练。此外，我们结合生成训练和对比训练并使用两个嵌入，一个经过优化以增强生成图像的照片真实感，另一个用于捕捉文本到图像对齐。在三个文本转图像基准数据集（Oxford-102、Caltech-UCSD 和 MS-COCO）上进行的一系列全面实验表明，使用两个单独的嵌入比使用共享的嵌入效果更好，并且与使用使用判别方法训练的预训练文本编码器的文本表示的方法相比，这种方法表现更佳。最后，我们证明了这种学习到的嵌入也可以用于其他环境，例如文本转图像处理。

Title: BD-Diff: Generative Diffusion Model for Image Deblurring on Unknown Domains with Blur-Decoupled Learning

• Authors: Junhao Cheng, Wei-Ting Chen, Xi Lu, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01522
• Pdf URL: https://arxiv.org/pdf/2502.01522
• Copy Paste: [[2502.01522]] BD-Diff: Generative Diffusion Model for Image Deblurring on Unknown Domains with Blur-Decoupled Learning(https://arxiv.org/abs/2502.01522)
• Keywords: restoration, generative
• Abstract: Generative diffusion models trained on large-scale datasets have achieved remarkable progress in image synthesis. In favor of their ability to supplement missing details and generate aesthetically pleasing contents, recent works have applied them to image deblurring tasks via training an adapter on blurry-sharp image pairs to provide structural conditions for restoration. However, acquiring substantial amounts of realistic paired data is challenging and costly in real-world scenarios. On the other hand, relying solely on synthetic data often results in overfitting, leading to unsatisfactory performance when confronted with unseen blur patterns. To tackle this issue, we propose BD-Diff, a generative-diffusion-based model designed to enhance deblurring performance on unknown domains by decoupling structural features and blur patterns through joint training on three specially designed tasks. We employ two Q-Formers as structural representations and blur patterns extractors separately. The features extracted by them will be used for the supervised deblurring task on synthetic data and the unsupervised blur-transfer task by leveraging unpaired blurred images from the target domain simultaneously. Furthermore, we introduce a reconstruction task to make the structural features and blur patterns complementary. This blur-decoupled learning process enhances the generalization capabilities of BD-Diff when encountering unknown domain blur patterns. Experiments on real-world datasets demonstrate that BD-Diff outperforms existing state-of-the-art methods in blur removal and structural preservation in various challenging scenarios. The codes will be released in this https URL
• 摘要：在大型数据集上训练的生成式扩散模型在图像合成方面取得了显著进展。由于它们能够补充缺失的细节并生成美观的内容，最近的研究将它们应用于图像去模糊任务，通过在模糊-清晰图像对上训练适配器来提供恢复的结构条件。然而，在现实场景中，获取大量真实的配对数据具有挑战性且成本高昂。另一方面，仅仅依靠合成数据通常会导致过度拟合，导致在面对看不见的模糊模式时性能不尽如人意。为了解决这个问题，我们提出了 BD-Diff，这是一个基于生成式扩散的模型，旨在通过在三个专门设计的任务上进行联合训练，将结构特征和模糊模式解耦，从而增强对未知领域的去模糊性能。我们分别采用两个 Q-Former 作为结构表示和模糊模式提取器。它们提取的特征将用于合成数据的监督去模糊任务和无监督模糊传输任务，通过同时利用目标域中未配对的模糊图像。此外，我们引入了重建任务，使结构特征和模糊模式互补。这种模糊解耦学习过程增强了 BD-Diff 在遇到未知域模糊模式时的泛化能力。在真实数据集上的实验表明，BD-Diff 在各种具有挑战性的场景中，在模糊去除和结构保存方面均优于现有的最先进方法。代码将在此 https URL 中发布

Title: Federated Learning with Discriminative Naive Bayes Classifier

• Authors: Pablo Torrijos, Juan C. Alfaro, José A. Gámez, José M. Puerta
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01532
• Pdf URL: https://arxiv.org/pdf/2502.01532
• Copy Paste: [[2502.01532]] Federated Learning with Discriminative Naive Bayes Classifier(https://arxiv.org/abs/2502.01532)
• Keywords: generative
• Abstract: Federated Learning has emerged as a promising approach to train machine learning models on decentralized data sources while preserving data privacy. This paper proposes a new federated approach for Naive Bayes (NB) classification, assuming discrete variables. Our approach federates a discriminative variant of NB, sharing meaningless parameters instead of conditional probability tables. Therefore, this process is more reliable against possible attacks. We conduct extensive experiments on 12 datasets to validate the efficacy of our approach, comparing federated and non-federated settings. Additionally, we benchmark our method against the generative variant of NB, which serves as a baseline for comparison. Our experimental results demonstrate the effectiveness of our method in achieving accurate classification.
• 摘要：联邦学习已成为一种有前途的方法，可以在分散数据源上训练机器学习模型，同时保护数据隐私。本文提出了一种新的联合方法，用于朴素贝叶斯 (NB) 分类，假设离散变量。我们的方法联合了 NB 的判别变体，共享无意义的参数而不是条件概率表。因此，此过程对可能的攻击更可靠。我们在 12 个数据集上进行了广泛的实验，以验证我们方法的有效性，比较了联合和非联合设置。此外，我们将我们的方法与 NB 的生成变体进行基准测试，作为比较的基线。我们的实验结果证明了我们的方法在实现准确分类方面的有效性。

Title: MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generation

• Authors: Yiren Song, Cheng Liu, Mike Zheng Shou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01572
• Pdf URL: https://arxiv.org/pdf/2502.01572
• Copy Paste: [[2502.01572]] MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generation(https://arxiv.org/abs/2502.01572)
• Keywords: generation
• Abstract: A hallmark of human intelligence is the ability to create complex artifacts through structured multi-step processes. Generating procedural tutorials with AI is a longstanding but challenging goal, facing three key obstacles: (1) scarcity of multi-task procedural datasets, (2) maintaining logical continuity and visual consistency between steps, and (3) generalizing across multiple domains. To address these challenges, we propose a multi-domain dataset covering 21 tasks with over 24,000 procedural sequences. Building upon this foundation, we introduce MakeAnything, a framework based on the diffusion transformer (DIT), which leverages fine-tuning to activate the in-context capabilities of DIT for generating consistent procedural sequences. We introduce asymmetric low-rank adaptation (LoRA) for image generation, which balances generalization capabilities and task-specific performance by freezing encoder parameters while adaptively tuning decoder layers. Additionally, our ReCraft model enables image-to-process generation through spatiotemporal consistency constraints, allowing static images to be decomposed into plausible creation sequences. Extensive experiments demonstrate that MakeAnything surpasses existing methods, setting new performance benchmarks for procedural generation tasks.
• 摘要：人类智能的一个标志是能够通过结构化的多步骤流程创建复杂的工件。使用 AI 生成程序教程是一个长期但具有挑战性的目标，面临三个主要障碍：（1）多任务程序数据集的稀缺性，（2）保持步骤之间的逻辑连续性和视觉一致性，以及（3）跨多个领域进行泛化。为了应对这些挑战，我们提出了一个多领域数据集，涵盖 21 个任务，包含超过 24,000 个程序序列。在此基础上，我们引入了 MakeAnything，这是一个基于扩散变换器 (DIT) 的框架，它利用微调来激活 DIT 的上下文功能以生成一致的程序序列。我们为图像生成引入了非对称低秩自适应 (LoRA)，它通过冻结编码器参数同时自适应地调整解码器层来平衡泛化能力和特定于任务的性能。此外，我们的 ReCraft 模型通过时空一致性约束实现图像到过程的生成，从而允许将静态图像分解为合理的创建序列。大量实验表明，MakeAnything 超越了现有的方法，为程序生成任务设定了新的性能基准。

Title: MFP-VTON: Enhancing Mask-Free Person-to-Person Virtual Try-On via Diffusion Transformer

• Authors: Le Shen, Yanting Kang, Rong Huang, Zhijie Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01626
• Pdf URL: https://arxiv.org/pdf/2502.01626
• Copy Paste: [[2502.01626]] MFP-VTON: Enhancing Mask-Free Person-to-Person Virtual Try-On via Diffusion Transformer(https://arxiv.org/abs/2502.01626)
• Keywords: generative
• Abstract: The garment-to-person virtual try-on (VTON) task, which aims to generate fitting images of a person wearing a reference garment, has made significant strides. However, obtaining a standard garment is often more challenging than using the garment already worn by the person. To improve ease of use, we propose MFP-VTON, a Mask-Free framework for Person-to-Person VTON. Recognizing the scarcity of person-to-person data, we adapt a garment-to-person model and dataset to construct a specialized dataset for this task. Our approach builds upon a pretrained diffusion transformer, leveraging its strong generative capabilities. During mask-free model fine-tuning, we introduce a Focus Attention loss to emphasize the garment of the reference person and the details outside the garment of the target person. Experimental results demonstrate that our model excels in both person-to-person and garment-to-person VTON tasks, generating high-fidelity fitting images.
• 摘要：服装对人虚拟试穿 (VTON) 任务旨在生成穿着参考服装的人的试穿图像，该任务取得了重大进展。然而，获得标准服装通常比使用该人已经穿过的服装更具挑战性。为了提高易用性，我们提出了 MFP-VTON，一种用于人对人 VTON 的无口罩框架。认识到人对人数据的稀缺性，我们调整了服装对人模型和数据集来构建专门用于此任务的数据集。我们的方法建立在预训练的扩散变换器之上，利用其强大的生成能力。在无口罩模型微调期间，我们引入了 Focus Attention 损失来强调参考人的服装和目标人服装外部的细节。实验结果表明，我们的模型在人对人和服装对人 VTON 任务中都表现出色，可以生成高保真试穿图像。