2025-02-26

Title: RAG-Enhanced Collaborative LLM Agents for Drug Discovery

• Authors: Namkyeong Lee, Edward De Brouwer, Ehsan Hajiramezanali, Chanyoung Park, Gabriele Scalia
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.17506
• Pdf URL: https://arxiv.org/pdf/2502.17506
• Copy Paste: [[2502.17506]] RAG-Enhanced Collaborative LLM Agents for Drug Discovery(https://arxiv.org/abs/2502.17506)
• Keywords: generation
• Abstract: Recent advances in large language models (LLMs) have shown great potential to accelerate drug discovery. However, the specialized nature of biochemical data often necessitates costly domain-specific fine-tuning, posing critical challenges. First, it hinders the application of more flexible general-purpose LLMs in cutting-edge drug discovery tasks. More importantly, it impedes the rapid integration of the vast amounts of scientific data continuously generated through experiments and research. To investigate these challenges, we propose CLADD, a retrieval-augmented generation (RAG)-empowered agentic system tailored to drug discovery tasks. Through the collaboration of multiple LLM agents, CLADD dynamically retrieves information from biomedical knowledge bases, contextualizes query molecules, and integrates relevant evidence to generate responses -- all without the need for domain-specific fine-tuning. Crucially, we tackle key obstacles in applying RAG workflows to biochemical data, including data heterogeneity, ambiguity, and multi-source integration. We demonstrate the flexibility and effectiveness of this framework across a variety of drug discovery tasks, showing that it outperforms general-purpose and domain-specific LLMs as well as traditional deep learning approaches.
• 摘要：大型语言模型（LLM）的最新进展显示出巨大的加速药物发现潜力。但是，生化数据的专业性质通常需要昂贵的领域特定微调，从而带来关键的挑战。首先，它阻碍了更灵活的通用LLM在尖端的药物发现任务中的应用。更重要的是，它阻碍了通过实验和研究不断生成的大量科学数据的快速整合。为了调查这些挑战，我们提出了Cladd，这是一种根据药物发现任务量身定制的检索增强发电（RAG）授权的代理系统。通过多个LLM代理的协作，Cladd可以从生物医学知识库中动态检索信息，将查询分子进行环境化，并整合相关证据以产生响应 - 所有这些都不需要特定领域的微调。至关重要的是，我们解决了将抹布工作流应用于生化数据的关键障碍，包括数据异质性，歧义和多源集成。我们证明了该框架在各种药物发现任务中的灵活性和有效性，这表明它的表现优于通用和域特异性LLM以及传统的深度学习方法。

Title: Towards User-level Private Reinforcement Learning with Human Feedback

• Authors: Jiaming Zhang, Mingxi Lei, Meng Ding, Mengdi Li, Zihang Xiang, Difei Xu, Jinhui Xu, Di Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.17515
• Pdf URL: https://arxiv.org/pdf/2502.17515
• Copy Paste: [[2502.17515]] Towards User-level Private Reinforcement Learning with Human Feedback(https://arxiv.org/abs/2502.17515)
• Keywords: generation
• Abstract: Reinforcement Learning with Human Feedback (RLHF) has emerged as an influential technique, enabling the alignment of large language models (LLMs) with human preferences. Despite the promising potential of RLHF, how to protect user preference privacy has become a crucial issue. Most previous work has focused on using differential privacy (DP) to protect the privacy of individual data. However, they have concentrated primarily on item-level privacy protection and have unsatisfactory performance for user-level privacy, which is more common in RLHF. This study proposes a novel framework, AUP-RLHF, which integrates user-level label DP into RLHF. We first show that the classical random response algorithm, which achieves an acceptable performance in item-level privacy, leads to suboptimal utility when in the user-level settings. We then establish a lower bound for the user-level label DP-RLHF and develop the AUP-RLHF algorithm, which guarantees $(\varepsilon, \delta)$ user-level privacy and achieves an improved estimation error. Experimental results show that AUP-RLHF outperforms existing baseline methods in sentiment generation and summarization tasks, achieving a better privacy-utility trade-off.
• 摘要：通过人类反馈（RLHF）的增强学习已成为一种有影响力的技术，从而使大语模型（LLMS）与人类偏好保持一致。尽管RLHF具有有希望的潜力，但如何保护用户偏好隐私已成为一个至关重要的问题。以前的大多数工作都集中在使用差异隐私（DP）来保护单个数据的隐私。但是，它们主要集中于项目级隐私保护，并且在用户级隐私方面的性能不令人满意，这在RLHF中更为常见。这项研究提出了一个新颖的框架AUP-RLHF，该框架将用户级标签DP集成到RLHF中。我们首先表明，在用户级设置中，可以在项目级隐私方面实现可接受性能的经典随机响应算法。然后，我们为用户级标签DP-RLHF建立了一个下限，并开发AUP-RLHF算法，该算法保证$（\ varepsilon，\ delta）$用户级隐私并实现改进的估计错误。实验结果表明，AUP-RLHF在情感生成和摘要任务中的现有基线方法的表现，实现了更好的隐私性 - 实用性权衡。

Title: A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models

• Authors: Zihao Lin, Samyadeep Basu, Mohammad Beigi, Varun Manjunatha, Ryan A. Rossi, Zichao Wang, Yufan Zhou, Sriram Balasubramanian, Arman Zarei, Keivan Rezaei, Ying Shen, Barry Menglong Yao, Zhiyang Xu, Qin Liu, Yuxiang Zhang, Yan Sun, Shilong Liu, Li Shen, Hongxuan Li, Soheil Feizi, Lifu Huang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.17516
• Pdf URL: https://arxiv.org/pdf/2502.17516
• Copy Paste: [[2502.17516]] A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models(https://arxiv.org/abs/2502.17516)
• Keywords: generative
• Abstract: The rise of foundation models has transformed machine learning research, prompting efforts to uncover their inner workings and develop more efficient and reliable applications for better control. While significant progress has been made in interpreting Large Language Models (LLMs), multimodal foundation models (MMFMs) - such as contrastive vision-language models, generative vision-language models, and text-to-image models - pose unique interpretability challenges beyond unimodal frameworks. Despite initial studies, a substantial gap remains between the interpretability of LLMs and MMFMs. This survey explores two key aspects: (1) the adaptation of LLM interpretability methods to multimodal models and (2) understanding the mechanistic differences between unimodal language models and crossmodal systems. By systematically reviewing current MMFM analysis techniques, we propose a structured taxonomy of interpretability methods, compare insights across unimodal and multimodal architectures, and highlight critical research gaps.
• 摘要：基础模型的兴起已经改变了机器学习研究，促使努力揭示其内部运作，并开发更高效，更可靠的应用程序以更好地控制。虽然在解释大语模型（LLM），多模式基础模型（MMFM）（例如对比度视觉模型，生成视觉语言模型和文本到图像模型）方面取得了重大进展，构成了独特的解释性挑战。框架。尽管进行了初步研究，但LLM和MMFM的可解释性之间仍然存在很大的差距。该调查探讨了两个关键方面：（1）LLM可解释性方法对多模型模型的适应，以及（2）了解单峰语言模型和跨模式系统之间的机械差异。通过系统地审查当前的MMFM分析技术，我们提出了一种结构化分类法的解释性分类法，比较单峰和多模式架构之间的见解，并突出关键的研究差距。

Title: The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?

• Authors: Zhenheng Tang, Xiang Liu, Qian Wang, Peijie Dong, Bingsheng He, Xiaowen Chu, Bo Li
• Subjects: cs.LG, cs.AI, cs.CL, cs.FL
• Abstract URL: https://arxiv.org/abs/2502.17535
• Pdf URL: https://arxiv.org/pdf/2502.17535
• Copy Paste: [[2502.17535]] The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?(https://arxiv.org/abs/2502.17535)
• Keywords: generation
• Abstract: Motivated by reducing the computational and storage costs of LLMs, model compression and KV cache compression have attracted much attention from researchers. However, current methods predominantly emphasize maintaining the performance of compressed LLMs, as measured by perplexity or simple accuracy on tasks of common sense knowledge QA and basic arithmetic reasoning. In this blog, we present a brief review of recent advancements in LLMs related to retrieval-augmented generation, multi-step reasoning, external tools, and computational expressivity, all of which substantially enhance LLM performance. Then, we propose a lottery LLM hypothesis suggesting that for a given LLM and task, there exists a smaller lottery LLM capable of producing the same performance as the original LLM with the assistance of multi-step reasoning and external tools. Based on the review of current progress in LLMs, we discuss and summarize the essential capabilities that the lottery LLM and KV cache compression must possess, which are currently overlooked in existing methods.
• 摘要：通过降低LLM的计算和存储成本，模型压缩和KV缓存压缩引起了研究人员的广泛关注。但是，当前方法主要强调保持压缩LLM的性能，这是通过在常识知识质量质量质量质量质量警察和基本算术推理的任务上进行的困惑或简单的精度来衡量的。在此博客中，我们对LLMS的最新进展进行了简要回顾，与检索型发电，多步推理，外部工具和计算表达性有关，所有这些都显着提高了LLM的性能。然后，我们提出了一个彩票LLM假设，表明对于给定的LLM和任务，有一个较小的彩票LLM能够在多步推理和外部工具的帮助下与原始LLM产生相同的性能。根据对LLM中当前进展的综述，我们讨论并总结了彩票LLM和KV缓存压缩必须具有的基本功能，这些功能目前在现有方法中被忽略了。

Title: On the Vulnerability of Concept Erasure in Diffusion Models

• Authors: Lucas Beerens, Alex D. Richardson, Kaicheng Zhang, Dongdong Chen
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2502.17537
• Pdf URL: https://arxiv.org/pdf/2502.17537
• Copy Paste: [[2502.17537]] On the Vulnerability of Concept Erasure in Diffusion Models(https://arxiv.org/abs/2502.17537)
• Keywords: generation
• Abstract: The proliferation of text-to-image diffusion models has raised significant privacy and security concerns, particularly regarding the generation of copyrighted or harmful images. To address these issues, research on machine unlearning has developed various concept erasure methods, which aim to remove the effect of unwanted data through post-hoc training. However, we show these erasure techniques are vulnerable, where images of supposedly erased concepts can still be generated using adversarially crafted prompts. We introduce RECORD, a coordinate-descent-based algorithm that discovers prompts capable of eliciting the generation of erased content. We demonstrate that RECORD significantly beats the attack success rate of current state-of-the-art attack methods. Furthermore, our findings reveal that models subjected to concept erasure are more susceptible to adversarial attacks than previously anticipated, highlighting the urgency for more robust unlearning approaches. We open source all our code at this https URL
• 摘要：文本到图像扩散模型的扩散引起了严重的隐私和安全问题，尤其是在受版权或有害图像的产生方面。为了解决这些问题，对机器学习的研究开发了各种概念擦除方法，旨在通过事后培训来消除不需要的数据的效果。但是，我们表明这些擦除技术很容易受到伤害，其中仍然可以使用对抗制作的提示来生成所谓的擦除概念的图像。我们介绍了一种基于坐标的算法记录，该算法会发现能够引起擦除内容的发起的提示。我们证明，记录可以显着击败当前最新攻击方法的攻击成功率。此外，我们的发现表明，经过概念擦除的模型比以前的预期更容易受到对抗攻击的影响，这突出了更强大的未学习方法的紧迫性。我们在此HTTPS URL上为所有代码开放了所有代码

Title: Synthetic Text Generation for Training Large Language Models via Gradient Matching

• Authors: Dang Nguyen, Zeman Li, Mohammadhossein Bateni, Vahab Mirrokni, Meisam Razaviyayn, Baharan Mirzasoleiman
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.17607
• Pdf URL: https://arxiv.org/pdf/2502.17607
• Copy Paste: [[2502.17607]] Synthetic Text Generation for Training Large Language Models via Gradient Matching(https://arxiv.org/abs/2502.17607)
• Keywords: generation
• Abstract: Synthetic data has the potential to improve the performance, training efficiency, and privacy of real training examples. Nevertheless, existing approaches for synthetic text generation are mostly heuristics and cannot generate human-readable text without compromising the privacy of real data or provide performance guarantees for training Large Language Models (LLMs). In this work, we propose the first theoretically rigorous approach for generating synthetic human-readable text that guarantees the convergence and performance of LLMs during fine-tuning on a target task. To do so, we leverage Alternating Direction Method of Multipliers (ADMM) that iteratively optimizes the embeddings of synthetic examples to match the gradient of the target training or validation data, and maps them to a sequence of text tokens with low perplexity. In doing so, the generated synthetic text can guarantee convergence of the model to a close neighborhood of the solution obtained by fine-tuning on real data. Experiments on various classification tasks confirm the effectiveness of our proposed approach.
• 摘要：合成数据有可能提高实际培训实例的性能，训练效率和隐私。然而，现有的合成文本生成方法主要是启发式方法，并且不能在不损害真实数据的隐私或为培训大语言模型（LLMS）提供绩效保证的情况下产生可读的文本。在这项工作中，我们提出了一种理论上严格的方法，用于生成合成的人类可读文本，以确保在目标任务进行微调期间LLM的收敛性和性能。为此，我们利用乘数的交替方向方法（ADMM）迭代优化合成示例的嵌入，以匹配目标训练或验证数据的梯度，并将它们映射到一系列具有低相处的文本令牌。这样一来，生成的合成文本可以保证模型通过微调实际数据获得的解决方案的近距离收敛。各种分类任务的实验证实了我们提出的方法的有效性。

Title: Flexible Counterfactual Explanations with Generative Models

• Authors: Stig Hellemans, Andres Algaba, Sam Verboven, Vincent Ginis
• Subjects: cs.LG, cs.AI, stat.ME
• Abstract URL: https://arxiv.org/abs/2502.17613
• Pdf URL: https://arxiv.org/pdf/2502.17613
• Copy Paste: [[2502.17613]] Flexible Counterfactual Explanations with Generative Models(https://arxiv.org/abs/2502.17613)
• Keywords: generative
• Abstract: Counterfactual explanations provide actionable insights to achieve desired outcomes by suggesting minimal changes to input features. However, existing methods rely on fixed sets of mutable features, which makes counterfactual explanations inflexible for users with heterogeneous real-world constraints. Here, we introduce Flexible Counterfactual Explanations, a framework incorporating counterfactual templates, which allows users to dynamically specify mutable features at inference time. In our implementation, we use Generative Adversarial Networks (FCEGAN), which align explanations with user-defined constraints without requiring model retraining or additional optimization. Furthermore, FCEGAN is designed for black-box scenarios, leveraging historical prediction datasets to generate explanations without direct access to model internals. Experiments across economic and healthcare datasets demonstrate that FCEGAN significantly improves counterfactual explanations' validity compared to traditional benchmark methods. By integrating user-driven flexibility and black-box compatibility, counterfactual templates support personalized explanations tailored to user constraints.
• 摘要：反事实解释提供了可行的见解，可以通过提出最小的输入特征变化来实现预期的结果。但是，现有方法依赖于固定的可变特征集，这使得对具有异构现实世界约束的用户的反事实解释不灵活。在这里，我们介绍了灵活的反事实解释，这是一个包含反事实模板的框架，该框架允许用户在推理时间动态指定可突变的功能。在我们的实施中，我们使用生成对抗网络（FCEGAN），该网络与用户定义的约束不需要模型再培训或其他优化。此外，FCEGAN是为黑框方案而设计的，利用历史预测数据集生成解释，而无需直接访问模型内​​部设备。经济和医疗保健数据集的实验表明，与传统基准方法相比，FCEGAN显着提高了反事实解释的有效性。通过集成用户驱动的灵活性和黑框兼容性，反事实模板支持针对用户约束的个性化解释。

Title: METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scaling

• Authors: Bingxuan Li, Yiwei Wang, Jiuxiang Gu, Kai-Wei Chang, Nanyun Peng
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.17651
• Pdf URL: https://arxiv.org/pdf/2502.17651
• Copy Paste: [[2502.17651]] METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scaling(https://arxiv.org/abs/2502.17651)
• Keywords: generation
• Abstract: Chart generation aims to generate code to produce charts satisfying the desired visual properties, e.g., texts, layout, color, and type. It has great potential to empower the automatic professional report generation in financial analysis, research presentation, education, and healthcare. In this work, we build a vision-language model (VLM) based multi-agent framework for effective automatic chart generation. Generating high-quality charts requires both strong visual design skills and precise coding capabilities that embed the desired visual properties into code. Such a complex multi-modal reasoning process is difficult for direct prompting of VLMs. To resolve these challenges, we propose METAL, a multi-agent framework that decomposes the task of chart generation into the iterative collaboration among specialized agents. METAL achieves 5.2% improvement in accuracy over the current best result in the chart generation task. The METAL framework exhibits the phenomenon of test-time scaling: its performance increases monotonically as the logarithmic computational budget grows from 512 to 8192 tokens. In addition, we find that separating different modalities during the critique process of METAL boosts the self-correction capability of VLMs in the multimodal context.
• 摘要：图表生成旨在生成代码以产生满足所需的视觉属性的图表，例如文本，布局，颜色和类型。它具有巨大的潜力，可以在财务分析，研究表现，教育和医疗保健中赋予自动专业报告的能力。在这项工作中，我们建立了一个基于视觉语言模型（VLM）的多代理框架，以生成有效的自动图表。生成高质量的图表需要强大的视觉设计技能和精确的编码功能，将所需的视觉属性嵌入代码中。对于直接提示VLM，很难进行如此复杂的多模式推理过程。为了解决这些挑战，我们提出了金属，这是一个多代理框架，将图表生成的任务分解为专业代理之间的迭代协作。金属在图表生成任务中的当前最佳结果的准确性提高了5.2％。金属框架表现出测试时间缩放的现象：随着对数计算预算从512个令牌增长到8192代币，其性能会单调增加。此外，我们发现在金属的批评过程中分离不同的方式可以增强VLM在多模式背景下的自我纠正能力。

Title: Contrastive Visual Data Augmentation

• Authors: Yu Zhou, Bingxuan Li, Mohan Tang, Xiaomeng Jin, Te-Lin Wu, Kuan-Hao Huang, Heng Ji, Kai-Wei Chang, Nanyun Peng
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2502.17709
• Pdf URL: https://arxiv.org/pdf/2502.17709
• Copy Paste: [[2502.17709]] Contrastive Visual Data Augmentation(https://arxiv.org/abs/2502.17709)
• Keywords: generative
• Abstract: Large multimodal models (LMMs) often struggle to recognize novel concepts, as they rely on pre-trained knowledge and have limited ability to capture subtle visual details. Domain-specific knowledge gaps in training also make them prone to confusing visually similar, commonly misrepresented, or low-resource concepts. To help LMMs better align nuanced visual features with language, improving their ability to recognize and reason about novel or rare concepts, we propose a Contrastive visual Data Augmentation (CoDA) strategy. CoDA extracts key contrastive textual and visual features of target concepts against the known concepts they are misrecognized as, and then uses multimodal generative models to produce targeted synthetic data. Automatic filtering of extracted features and augmented images is implemented to guarantee their quality, as verified by human annotators. We show the effectiveness and efficiency of CoDA on low-resource concept and diverse scene recognition datasets including INaturalist and SUN. We additionally collect NovelSpecies, a benchmark dataset consisting of newly discovered animal species that are guaranteed to be unseen by LMMs. LLaVA-1.6 1-shot updating results on these three datasets show CoDA significantly improves SOTA visual data augmentation strategies by 12.3% (NovelSpecies), 5.1% (SUN), and 6.0% (iNat) absolute gains in accuracy.
• 摘要：大型多模型模型（LMM）通常很难识别新颖的概念，因为它们依赖于预训练的知识，并且能够捕获微妙的视觉细节的能力。培训中特定于领域的知识差距也使它们容易混淆视觉相似，通常歪曲或低资源概念。为了帮助LMM更好地使细微差别的视觉特征与语言保持一致，提高了他们识别和理由的新颖或稀有概念的能力，我们提出了一种对比性视觉数据增强（CODA）策略。 CODA提取目标概念的关键对比文本和视觉特征与已知的概念被误认为为已知概念，然后使用多模式生成模型生成目标的合成数据。如人类注释者所验证的那样，实施了提取的功能和增强图像的自动过滤，以确保其质量。我们显示了尾声对低资源概念以及包括Inaturalist和Sun在内的各种场景识别数据集的有效性和效率。我们还收集了小说《小说》，这是一个由新发现的动物物种组成的基准数据集，这些动物物种保证不会被LMMS看到。 LLAVA-1.6这三个数据集上的1-shot更新结果表明，CODA显着将SOTA视觉数据增强策略提高了12.3％（小说类），5.1％（Sun）和6.0％（inatat）的准确性（INAT）绝对获得。

Title: Aligning Compound AI Systems via System-level DPO

• Authors: Xiangwen Wang, Yibo Jacky Zhang, Zhoujie Ding, Katherine Tsai, Sanmi Koyejo
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2502.17721
• Pdf URL: https://arxiv.org/pdf/2502.17721
• Copy Paste: [[2502.17721]] Aligning Compound AI Systems via System-level DPO(https://arxiv.org/abs/2502.17721)
• Keywords: generation
• Abstract: Compound AI systems, comprising multiple interacting components such as LLM agents and external tools, demonstrate state-of-the-art results across diverse tasks. It is hence crucial to align components within the system to produce consistent results that match human expectations. However, conventional alignment methods, such as Direct Preference Optimization (DPO), are not directly applicable to compound AI systems. These challenges include the non-differentiable interactions between components, making end-to-end gradient optimization infeasible. Additionally, system-level preferences cannot be directly translated into component-level preferences, further complicating alignment. We address the issues by formulating compound AI systems as Directed Acyclic Graphs (DAGs), capturing the connections between agents and the data generation processes. We propose a system-level DPO (SysDPO) to jointly align compound systems by adapting the DPO to operate on these DAGs. We study the joint alignment of an LLM and a diffusion model to demonstrate the effectiveness of our approach. Our exploration provides insights into the alignment of compound AI systems and lays a foundation for future advancements.
• 摘要：复合AI系统包括多个相互作用的组件，例如LLM代理和外部工具，可以在各种任务中展示最先进的结果。因此，对于在系统中的组件对齐成分至关重要，以产生与人类期望相匹配的一致结果。但是，常规的对齐方法（例如直接偏好优化（DPO））并不直接适用于化合物AI系统。这些挑战包括组件之间的非差异相互作用，使端到端梯度优化不可行。另外，系统级偏好不能直接转化为组件级偏好，从而使对齐变得更加复杂。我们通过指示无环形图（DAG）来捕获代理与数据生成过程之间的连接来解决问题。我们建议通过调整DPO在这些DAG上操作，提出一个系统级DPO（SYSDPO）以共同对齐化合物系统。我们研究了LLM和扩散模型的联合对准，以证明我们方法的有效性。我们的探索提供了对化合物AI系统对齐的见解，并为未来的进步奠定了基础。

Title: Can Score-Based Generative Modeling Effectively Handle Medical Image Classification?

• Authors: Sushmita Sarker, Prithul Sarker, George Bebis, Alireza Tavakkoli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.17727
• Pdf URL: https://arxiv.org/pdf/2502.17727
• Copy Paste: [[2502.17727]] Can Score-Based Generative Modeling Effectively Handle Medical Image Classification?(https://arxiv.org/abs/2502.17727)
• Keywords: generative
• Abstract: The remarkable success of deep learning in recent years has prompted applications in medical image classification and diagnosis tasks. While classification models have demonstrated robustness in classifying simpler datasets like MNIST or natural images such as ImageNet, this resilience is not consistently observed in complex medical image datasets where data is more scarce and lacks diversity. Moreover, previous findings on natural image datasets have indicated a potential trade-off between data likelihood and classification accuracy. In this study, we explore the use of score-based generative models as classifiers for medical images, specifically mammographic images. Our findings suggest that our proposed generative classifier model not only achieves superior classification results on CBIS-DDSM, INbreast and Vin-Dr Mammo datasets, but also introduces a novel approach to image classification in a broader context. Our code is publicly available at this https URL
• 摘要：近年来，深度学习的显着成功促使了医学图像分类和诊断任务的应用。虽然分类模型在对更简单的数据集进行分类（例如MNIST或自然图像（例如Imagenet））时表现出了鲁棒性，但在复杂的医学图像数据集中并未始终如一地观察到这种弹性，因为数据更稀缺并且缺乏多样性。此外，自然图像数据集的先前发现表明，数据可能性和分类准确性之间存在潜在的权衡。在这项研究中，我们探讨了将基于得分的生成模型作为医学图像的分类器，特别是乳房X线照片图像的使用。我们的发现表明，我们提出的生成分类器模型不仅在CBIS-DDSM，INBREAST和VIN-DR MAMMO数据集上实现了卓越的分类结果，而且还引入了一种在更广泛的环境中进行图像分类的新方法。我们的代码在此HTTPS URL上公开可用

Title: A General Framework to Enhance Fine-tuning-based LLM Unlearning

• Authors: Jie Ren, Zhenwei Dai, Xianfeng Tang, Hui Liu, Jingying Zeng, Zhen Li, Rahul Goutam, Suhang Wang, Yue Xing, Qi He, Hui Liu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.17823
• Pdf URL: https://arxiv.org/pdf/2502.17823
• Copy Paste: [[2502.17823]] A General Framework to Enhance Fine-tuning-based LLM Unlearning(https://arxiv.org/abs/2502.17823)
• Keywords: generation
• Abstract: Unlearning has been proposed to remove copyrighted and privacy-sensitive data from Large Language Models (LLMs). Existing approaches primarily rely on fine-tuning-based methods, which can be categorized into gradient ascent-based (GA-based) and suppression-based methods. However, they often degrade model utility (the ability to respond to normal prompts). In this work, we aim to develop a general framework that enhances the utility of fine-tuning-based unlearning methods. To achieve this goal, we first investigate the common property between GA-based and suppression-based methods. We unveil that GA-based methods unlearn by distinguishing the target data (i.e., the data to be removed) and suppressing related generations, which is essentially the same strategy employed by suppression-based methods. Inspired by this finding, we introduce Gated Representation UNlearning (GRUN) which has two components: a soft gate function for distinguishing target data and a suppression module using Representation Fine-tuning (ReFT) to adjust representations rather than model parameters. Experiments show that GRUN significantly improves the unlearning and utility. Meanwhile, it is general for fine-tuning-based methods, efficient and promising for sequential unlearning.
• 摘要：已经提出了不学习以从大语言模型（LLMS）中删除受版权保护和隐私敏感的数据。现有方法主要依赖于基于微调的方法，这些方法可以归类为基于梯度上升（基于GA）和基于抑制的方法。但是，它们经常降低模型实用程序（对正常提示做出响应的能力）。在这项工作中，我们旨在开发一个一般框架，以增强基于微调的未学习方法的实用性。为了实现这一目标，我们首先研究基于GA的基于GA和基于抑制的方法之间的共同属性。我们通过区分目标数据（即要删除的数据）并抑制相关的世代来揭示基于GA的方法，这与基于抑制方法所采用的策略基本相同。受这一发现的启发，我们引入了封闭式表示（GRUN），它具有两个组成部分：使用表示数据的软门功能，用于区分目标数据和使用表示模块的抑制模块，以调整表示表示而不是模型参数。实验表明，Grun显着改善了学习和效用。同时，对于基于微调的方法来说，这是一般性的，有效且有望进行顺序学习。

Title: MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks

• Authors: Hyeonjeong Ha, Qiusi Zhan, Jeonghwan Kim, Dimitrios Bralios, Saikrishna Sanniboina, Nanyun Peng, Kai-wei Chang, Daniel Kang, Heng Ji
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.17832
• Pdf URL: https://arxiv.org/pdf/2502.17832
• Copy Paste: [[2502.17832]] MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks(https://arxiv.org/abs/2502.17832)
• Keywords: generation
• Abstract: Multimodal large language models (MLLMs) equipped with Retrieval Augmented Generation (RAG) leverage both their rich parametric knowledge and the dynamic, external knowledge to excel in tasks such as Question Answering. While RAG enhances MLLMs by grounding responses in query-relevant external knowledge, this reliance poses a critical yet underexplored safety risk: knowledge poisoning attacks, where misinformation or irrelevant knowledge is intentionally injected into external knowledge bases to manipulate model outputs to be incorrect and even harmful. To expose such vulnerabilities in multimodal RAG, we propose MM-PoisonRAG, a novel knowledge poisoning attack framework with two attack strategies: Localized Poisoning Attack (LPA), which injects query-specific misinformation in both text and images for targeted manipulation, and Globalized Poisoning Attack (GPA) to provide false guidance during MLLM generation to elicit nonsensical responses across all queries. We evaluate our attacks across multiple tasks, models, and access settings, demonstrating that LPA successfully manipulates the MLLM to generate attacker-controlled answers, with a success rate of up to 56% on MultiModalQA. Moreover, GPA completely disrupts model generation to 0% accuracy with just a single irrelevant knowledge injection. Our results highlight the urgent need for robust defenses against knowledge poisoning to safeguard multimodal RAG frameworks.
• 摘要：配备有检索增强生成（RAG）的多模式大语言模型（MLLM）都利用了他们丰富的参数知识和动态，外部知识在诸如问题回答之类的任务中表现出色。虽然RAG通过将与查询相关的外部知识的响应接地来增强MLLM，但这种依赖构成了一个关键而又毫无争议的安全风险：知识中毒攻击，其中错误的信息或无关紧要的知识被故意注入外部知识基础中，以操纵模型输出，甚至是有害的，甚至是有害的。 。为了在多模式抹布中暴露这种脆弱性，我们提出了MM-PoisonRag，这是一种新型的知识中毒攻击框架，具有两种攻击策略：局部中毒攻击（LPA），该攻击（LPA）在有针对性的操纵中注入了特定的文本和图像，以及全球化的中毒操纵图像攻击（GPA）在MLLM生成过程中提供虚假的指导，以在所有查询中引起荒谬的响应。我们评估了跨多个任务，模型和访问设置的攻击，表明LPA成功地操纵了MLLM以生成攻击者控制的答案，而多模态的成功率高达56％。此外，GPA仅通过一次不相关的知识注入将模型生成限制为0％的准确性。我们的结果强调了迫切需要防御知识中毒以保护多模式抹布框架的必要性。

Title: Task-Driven Semantic Quantization and Imitation Learning for Goal-Oriented Communications

• Authors: Yu-Chieh Chao, Yubei Chen, Weiwei Wang, Achintha Wijesinghe, Suchinthaka Wanninayaka, Songyang Zhang, Zhi Ding
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2502.17842
• Pdf URL: https://arxiv.org/pdf/2502.17842
• Copy Paste: [[2502.17842]] Task-Driven Semantic Quantization and Imitation Learning for Goal-Oriented Communications(https://arxiv.org/abs/2502.17842)
• Keywords: generation
• Abstract: Semantic communication marks a new paradigm shift from bit-wise data transmission to semantic information delivery for the purpose of bandwidth reduction. To more effectively carry out specialized downstream tasks at the receiver end, it is crucial to define the most critical semantic message in the data based on the task or goal-oriented features. In this work, we propose a novel goal-oriented communication (GO-COM) framework, namely Goal-Oriented Semantic Variational Autoencoder (GOS-VAE), by focusing on the extraction of the semantics vital to the downstream tasks. Specifically, we adopt a Vector Quantized Variational Autoencoder (VQ-VAE) to compress media data at the transmitter side. Instead of targeting the pixel-wise image data reconstruction, we measure the quality-of-service at the receiver end based on a pre-defined task-incentivized model. Moreover, to capture the relevant semantic features in the data reconstruction, imitation learning is adopted to measure the data regeneration quality in terms of goal-oriented semantics. Our experimental results demonstrate the power of imitation learning in characterizing goal-oriented semantics and bandwidth efficiency of our proposed GOS-VAE.
• 摘要：语义通信标志着从位的数据传输到语义信息传递的新范式转变，以减少带宽。要更有效地在接收器端执行专门的下游任务，根据任务或面向目标的功能来定义数据中最关键的语义消息至关重要。在这项工作中，我们提出了一个新颖的面向目标的通信（GO-COM）框架，即以目标为导向的语义变异自动编码器（GOS-VAE），专注于提取对下游任务至关重要的语义。具体而言，我们采用了量化的矢量变异自动编码器（VQ-VAE）来压缩发射器侧的媒体数据。我们没有针对像素图像数据重建，而是基于预定义的任务引起的模型来测量接收器端的服务质量。此外，为了捕获数据重建中相关的语义特征，采用模仿学习以根据目标语义来衡量数据再生质量。我们的实验结果表明，模仿学习在表征面向目标的语义和我们提出的GOS-VAE的带宽效率方面的力量。

Title: HRR: Hierarchical Retrospection Refinement for Generated Image Detection

• Authors: Peipei Yuan, Zijing Xie, Shuo Ye, Hong Chen, Yulong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.17862
• Pdf URL: https://arxiv.org/pdf/2502.17862
• Copy Paste: [[2502.17862]] HRR: Hierarchical Retrospection Refinement for Generated Image Detection(https://arxiv.org/abs/2502.17862)
• Keywords: generative
• Abstract: Generative artificial intelligence holds significant potential for abuse, and generative image detection has become a key focus of research. However, existing methods primarily focused on detecting a specific generative model and emphasizing the localization of synthetic regions, while neglecting the interference caused by image size and style on model learning. Our goal is to reach a fundamental conclusion: Is the image real or generated? To this end, we propose a diffusion model-based generative image detection framework termed Hierarchical Retrospection Refinement~(HRR). It designs a multi-scale style retrospection module that encourages the model to generate detailed and realistic multi-scale representations, while alleviating the learning biases introduced by dataset styles and generative models. Additionally, based on the principle of correntropy sparse additive machine, a feature refinement module is designed to reduce the impact of redundant features on learning and capture the intrinsic structure and patterns of the data, thereby improving the model's generalization ability. Extensive experiments demonstrate the HRR framework consistently delivers significant performance improvements, outperforming state-of-the-art methods in generated image detection task.
• 摘要：生成人工智能具有巨大的滥用潜力，而生成图像检测已成为研究的重点。但是，现有方法主要集中于检测特定的生成模型并强调合成区域的定位，同时忽略了图像大小和样式在模型学习上引起的干扰。我们的目标是得出一个基本的结论：图像是真实的还是产生的？为此，我们提出了一个基于扩散模型的生成图像检测框架，称为层次回溯细化〜（HRR）。它设计了一个多尺度样式回顾模块，该模块鼓励模型生成详细且现实的多尺度表示，同时减轻了数据集样式和生成模型引入的学习偏见。此外，基于Correntropy稀疏添加剂机的原理，功能改进模块旨在减少冗余特征对学习和捕获数据的内在结构和模式的影响，从而提高模型的概括能力。广泛的实验证明了HRR框架始终提供了显着的性能改进，在生成的图像检测任务中表现优于最先进的方法。

Title: ASurvey: Spatiotemporal Consistency in Video Generation

• Authors: Zhiyu Yin, Kehai Chen, Xuefeng Bai, Ruili Jiang, Juntao Li, Hongdong Li, Jin Liu, Yang Xiang, Jun Yu, Min Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.17863
• Pdf URL: https://arxiv.org/pdf/2502.17863
• Copy Paste: [[2502.17863]] ASurvey: Spatiotemporal Consistency in Video Generation(https://arxiv.org/abs/2502.17863)
• Keywords: generation
• Abstract: Video generation, by leveraging a dynamic visual generation method, pushes the boundaries of Artificial Intelligence Generated Content (AIGC). Video generation presents unique challenges beyond static image generation, requiring both high-quality individual frames and temporal coherence to maintain consistency across the spatiotemporal sequence. Recent works have aimed at addressing the spatiotemporal consistency issue in video generation, while few literature review has been organized from this perspective. This gap hinders a deeper understanding of the underlying mechanisms for high-quality video generation. In this survey, we systematically review the recent advances in video generation, covering five key aspects: foundation models, information representations, generation schemes, post-processing techniques, and evaluation metrics. We particularly focus on their contributions to maintaining spatiotemporal consistency. Finally, we discuss the future directions and challenges in this field, hoping to inspire further efforts to advance the development of video generation.
• 摘要：通过利用动态视觉生成方法，视频生成可以推动人工智能生成的内容的界限（AIGC）。视频生成提出了除静态图像生成以外的独特挑战，需要高质量的单个帧和时间连贯性，以保持整个时空序列的一致性。最近的著作旨在解决视频生成中时空的一致性问题，而从这个角度来看，很少有文学回顾。这一差距阻碍了对高质量视频生成的基本机制的更深入的了解。在这项调查中，我们系统地回顾了视频生成的最新进展，涵盖了五个关键方面：基础模型，信息表示，发电方案，后处理技术和评估指标。我们特别关注他们在保持时空一致性方面的贡献。最后，我们讨论了这一领域的未来方向和挑战，希望激发进一步的努力来推进视频生成的发展。

Title: Neural Graph Matching Improves Retrieval Augmented Generation in Molecular Machine Learning

• Authors: Runzhong Wang, Rui-Xi Wang, Mrunali Manjrekar, Connor W. Coley
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2502.17874
• Pdf URL: https://arxiv.org/pdf/2502.17874
• Copy Paste: [[2502.17874]] Neural Graph Matching Improves Retrieval Augmented Generation in Molecular Machine Learning(https://arxiv.org/abs/2502.17874)
• Keywords: generation
• Abstract: Molecular machine learning has gained popularity with the advancements of geometric deep learning. In parallel, retrieval-augmented generation has become a principled approach commonly used with language models. However, the optimal integration of retrieval augmentation into molecular machine learning remains unclear. Graph neural networks stand to benefit from clever matching to understand the structural alignment of retrieved molecules to a query molecule. Neural graph matching offers a compelling solution by explicitly modeling node and edge affinities between two structural graphs while employing a noise-robust, end-to-end neural network to learn affinity metrics. We apply this approach to mass spectrum simulation and introduce MARASON, a novel model that incorporates neural graph matching to enhance a fragmentation-based neural network. Experimental results highlight the effectiveness of our design, with MARASON achieving 28% top-1 accuracy, a substantial improvement over the non-retrieval state-of-the-art accuracy of 19%. Moreover, MARASON outperforms both naive retrieval-augmented generation methods and traditional graph matching approaches.
• 摘要：随着几何深度学习的发展，分子机器学习已获得了知名度。同时，检索演示的生成已成为一种常用于语言模型的原则方法。然而，将检索扩展到分子机器学习中的最佳整合尚不清楚。图神经网络将从巧妙的匹配中受益，以了解检索分子与查询分子的结构比对。神经图匹配通过在两个结构图之间明确建模节点和边缘亲和力，同时使用噪声刺激性，端到端的神经网络来学习亲和力指标，从而提供了令人信服的解决方案。我们将这种方法应用于质谱模拟，并引入Marason，这是一种新型模型，结合了神经图匹配以增强基于碎片的神经网络。实验结果突出了我们设计的有效性，Marason实现了28％的TOP-1准确性，比19％的非交换最新准确性的实质性提高了。此外，马拉森（Marason）的表现既优于幼稚的检索生成方法和传统的图形匹配方法。

Title: Integrating Boosted learning with Differential Evolution (DE) Optimizer: A Prediction of Groundwater Quality Risk Assessment in Odisha

• Authors: Sonalika Subudhi, Alok Kumar Pati, Sephali Bose, Subhasmita Sahoo, Avipsa Pattanaik, Biswa Mohan Acharya
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.17929
• Pdf URL: https://arxiv.org/pdf/2502.17929
• Copy Paste: [[2502.17929]] Integrating Boosted learning with Differential Evolution (DE) Optimizer: A Prediction of Groundwater Quality Risk Assessment in Odisha(https://arxiv.org/abs/2502.17929)
• Keywords: quality assessment
• Abstract: Groundwater is eventually undermined by human exercises, such as fast industrialization, urbanization, over-extraction, and contamination from agrarian and urban sources. From among the different contaminants, the presence of heavy metals like cadmium (Cd), chromium (Cr), arsenic (As), and lead (Pb) proves to have serious dangers when present in huge concentrations in groundwater. Long-term usage of these poisonous components may lead to neurological disorders, kidney failure and different sorts of cancer. To address these issues, this study developed a machine learning-based predictive model to evaluate the Groundwater Quality Index (GWQI) and identify the main contaminants which are affecting the water quality. It has been achieved with the help of a hybrid machine learning model i.e. LCBoost Fusion . The model has undergone several processes like data preprocessing, hyperparameter tuning using Differential Evolution (DE) optimization, and evaluation through cross-validation. The LCBoost Fusion model outperforms individual models (CatBoost and LightGBM), by achieving low RMSE (0.6829), MSE (0.5102), MAE (0.3147) and a high R$^2$ score of 0.9809. Feature importance analysis highlights Potassium (K), Fluoride (F) and Total Hardness (TH) as the most influential indicators of groundwater contamination. This research successfully demonstrates the application of machine learning in assessing groundwater quality risks in Odisha. The proposed LCBoost Fusion model offers a reliable and efficient approach for real-time groundwater monitoring and risk mitigation. These findings will help the environmental organizations and the policy makers to map out targeted places for sustainable groundwater management. Future work will focus on using remote sensing data and developing an interactive decision-making system for groundwater quality assessment.
• 摘要：最终受到人类锻炼的破坏，例如工业化，城市化，超额萃取以及农业和城市来源的污染。从不同的污染物中，在地下水中以巨大的浓度出现时，诸如镉（CD），铬（CR），砷（AS）和铅（PB）等重金属的存在被证明是严重的危险。这些有毒成分的长期使用可能导致神经系统疾病，肾衰竭和不同种类的癌症。为了解决这些问题，本研究开发了一种基于机器学习的预测模型，以评估地下水质量指数（GWQI）并确定影响水质的主要污染物。它是在混合机器学习模型（即LCBoost融合）的帮助下实现的。该模型经历了多个过程，例如数据预处理，使用差分进化（DE）优化的超参数调整以及通过交叉验证进行评估。 LCBoost Fusion模型通过实现低RMSE（0.6829），MSE（0.5102），MAE（0.3147）和高R $^2 $得分为0.9809的模型优于单个模型（CATBOOST和LIGHTGBM）。特征重要性分析突出了钾（K），氟化物（F）和总硬度（Th）是地下水污染的最有影响力的指标。这项研究成功地证明了机器学习在评估奥里萨邦地下水质量风险中的应用。拟议的LCBoost融合模型为实时地下水监测和降低风险提供了可靠，有效的方法。这些发现将有助于环保组织和政策制定者为可持续地下水管理制定目标地点。未来的工作将集中于使用遥感数据，并开发一种交互式决策系统进行地下水质量评估。

Title: Robust Polyp Detection and Diagnosis through Compositional Prompt-Guided Diffusion Models

• Authors: Jia Yu, Yan Zhu, Peiyao Fu, Tianyi Chen, Junbo Huang, Quanlin Li, Pinghong Zhou, Zhihua Wang, Fei Wu, Shuo Wang, Xian Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.17951
• Pdf URL: https://arxiv.org/pdf/2502.17951
• Copy Paste: [[2502.17951]] Robust Polyp Detection and Diagnosis through Compositional Prompt-Guided Diffusion Models(https://arxiv.org/abs/2502.17951)
• Keywords: generation
• Abstract: Colorectal cancer (CRC) is a significant global health concern, and early detection through screening plays a critical role in reducing mortality. While deep learning models have shown promise in improving polyp detection, classification, and segmentation, their generalization across diverse clinical environments, particularly with out-of-distribution (OOD) data, remains a challenge. Multi-center datasets like PolypGen have been developed to address these issues, but their collection is costly and time-consuming. Traditional data augmentation techniques provide limited variability, failing to capture the complexity of medical images. Diffusion models have emerged as a promising solution for generating synthetic polyp images, but the image generation process in current models mainly relies on segmentation masks as the condition, limiting their ability to capture the full clinical context. To overcome these limitations, we propose a Progressive Spectrum Diffusion Model (PSDM) that integrates diverse clinical annotations-such as segmentation masks, bounding boxes, and colonoscopy reports-by transforming them into compositional prompts. These prompts are organized into coarse and fine components, allowing the model to capture both broad spatial structures and fine details, generating clinically accurate synthetic images. By augmenting training data with PSDM-generated samples, our model significantly improves polyp detection, classification, and segmentation. For instance, on the PolypGen dataset, PSDM increases the F1 score by 2.12% and the mean average precision by 3.09%, demonstrating superior performance in OOD scenarios and enhanced generalization.
• 摘要：大肠癌（CRC）是全球重要的健康问题，通过筛查的早期检测在降低死亡率中起着至关重要的作用。尽管深度学习模型已经显示出有望改善息肉检测，分类和分割，但它们在各种临床环境中的概括（尤其是在分布式（OOD）数据）上的概括仍然是一个挑战。已经开发了像Polypgen这样的多中心数据集来解决这些问题，但是它们的收集既昂贵又耗时。传统的数据增强技术可提供有限的可变性，无法捕获医学图像的复杂性。扩散模型已成为生成合成息肉图像的有前途的解决方案，但是当前模型中的图像生成过程主要依赖于分割掩码作为条件，从而限制了它们捕获完整临床环境的能力。为了克服这些局限性，我们提出了一个进行性频谱扩散模型（PSDM），该模型（PSDM）整合了各种临床注释，例如分割面具，边界框和结肠镜检查，将它们转化为组成提示。这些提示被组织成粗糙和细小的组件，使模型可以捕获广泛的空间结构和细节，从而产生临床准确的合成图像。通过使用PSDM生成的样品增强培训数据，我们的模型可显着改善息肉检测，分类和分割。例如，在Polypgen数据集上，PSDM将F1分数提高2.12％，平均平均精度提高了3.09％，在OOD方案和增强的概括中表明了卓越的性能。

Title: Model-Free Adversarial Purification via Coarse-To-Fine Tensor Network Representation

• Authors: Guang Lin, Duc Thien Nguyen, Zerui Tao, Konstantinos Slavakis, Toshihisa Tanaka, Qibin Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.17972
• Pdf URL: https://arxiv.org/pdf/2502.17972
• Copy Paste: [[2502.17972]] Model-Free Adversarial Purification via Coarse-To-Fine Tensor Network Representation(https://arxiv.org/abs/2502.17972)
• Keywords: generative
• Abstract: Deep neural networks are known to be vulnerable to well-designed adversarial attacks. Although numerous defense strategies have been proposed, many are tailored to the specific attacks or tasks and often fail to generalize across diverse scenarios. In this paper, we propose Tensor Network Purification (TNP), a novel model-free adversarial purification method by a specially designed tensor network decomposition algorithm. TNP depends neither on the pre-trained generative model nor the specific dataset, resulting in strong robustness across diverse adversarial scenarios. To this end, the key challenge lies in relaxing Gaussian-noise assumptions of classical decompositions and accommodating the unknown distribution of adversarial perturbations. Unlike the low-rank representation of classical decompositions, TNP aims to reconstruct the unobserved clean examples from an adversarial example. Specifically, TNP leverages progressive downsampling and introduces a novel adversarial optimization objective to address the challenge of minimizing reconstruction error but without inadvertently restoring adversarial perturbations. Extensive experiments conducted on CIFAR-10, CIFAR-100, and ImageNet demonstrate that our method generalizes effectively across various norm threats, attack types, and tasks, providing a versatile and promising adversarial purification technique.
• 摘要：深度神经网络已知容易受到精心设计的对抗攻击。尽管已经提出了许多防御策略，但许多策略是针对特定攻击或任务量身定制的，并且通常无法跨越各种情况。在本文中，我们提出了Tensor网络纯化（TNP），这是一种通过特殊设计的张量网络分解算法的新型无模型对抗纯化方法。 TNP既不取决于预先训练的生成模型，也不取决于特定的数据集，从而在不同的对抗场景中产生了强大的鲁棒性。为此，关键挑战在于放松对经典分解的高斯噪声假设，并适应对抗性扰动的未知分布。与经典分解的低排名表示不同，TNP旨在从对抗性示例中重建未观察到的干净示例。具体而言，TNP利用渐进的缩减采样，并引入了一个新颖的对抗优化目标，以解决最小化重建误差的挑战，但没有无意中恢复对抗性扰动。在CIFAR-10，CIFAR-100和Imagenet上进行的广泛实验表明，我们的方法在各种规范威胁，攻击类型和任务中有效地概括了，提供了一种多功能且有希望的对抗性纯化技术。

Title: Broadening Discovery through Structural Models: Multimodal Combination of Local and Structural Properties for Predicting Chemical Features

• Authors: Nikolai Rekut, Alexey Orlov, Klea Ziu, Elizaveta Starykh, Martin Takac, Aleksandr Beznosikov
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.17986
• Pdf URL: https://arxiv.org/pdf/2502.17986
• Copy Paste: [[2502.17986]] Broadening Discovery through Structural Models: Multimodal Combination of Local and Structural Properties for Predicting Chemical Features(https://arxiv.org/abs/2502.17986)
• Keywords: generation
• Abstract: In recent years, machine learning has profoundly reshaped the field of chemistry, facilitating significant advancements across various applications, including the prediction of molecular properties and the generation of molecular structures. Language models and graph-based models are extensively utilized within this domain, consistently achieving state-of-the-art results across an array of tasks. However, the prevailing practice of representing chemical compounds in the SMILES format -- used by most datasets and many language models -- presents notable limitations as a training data format. In contrast, chemical fingerprints offer a more physically informed representation of compounds, thereby enhancing their suitability for model training. This study aims to develop a language model that is specifically trained on fingerprints. Furthermore, we introduce a bimodal architecture that integrates this language model with a graph model. Our proposed methodology synthesizes these approaches, utilizing RoBERTa as the language model and employing Graph Isomorphism Networks (GIN), Graph Convolutional Networks (GCN) and Graphormer as graph models. This integration results in a significant improvement in predictive performance compared to conventional strategies for tasks such as Quantitative Structure-Activity Relationship (QSAR) and the prediction of nuclear magnetic resonance (NMR) spectra, among others.
• 摘要：近年来，机器学习已深刻地重塑了化学领域，从而促进了各种应用的显着进步，包括对分子特性的预测和分子结构的产生。语言模型和基于图的模型在该域中广泛使用，始终在一系列任务中实现最新结果。但是，以微笑格式表示化学化合物的主要实践（大多数数据集和许多语言模型都使用）将显着的限制作为培训数据格式。相比之下，化学指纹提供了更明智的化合物表示，从而增强了其对模型训练的适用性。这项研究旨在开发一种专门针对指纹训练的语言模型。此外，我们介绍了将该语言模型与图形模型集成的双峰体系结构。我们提出的方法将这些方法综合了这些方法，利用罗伯塔作为语言模型，并采用图形同构网络（GIN），图形卷积网络（GCN）和图形机器作为图形模型。与定量结构 - 活性关系（QSAR）和核磁共振（NMR）光谱等的常规策略相比，这种整合导致预测性能的显着改善。

Title: ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents

• Authors: Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, Feng Zhao
• Subjects: cs.CV, cs.AI, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.18017
• Pdf URL: https://arxiv.org/pdf/2502.18017
• Copy Paste: [[2502.18017]] ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents(https://arxiv.org/abs/2502.18017)
• Keywords: generation
• Abstract: Understanding information from visually rich documents remains a significant challenge for traditional Retrieval-Augmented Generation (RAG) methods. Existing benchmarks predominantly focus on image-based question answering (QA), overlooking the fundamental challenges of efficient retrieval, comprehension, and reasoning within dense visual documents. To bridge this gap, we introduce ViDoSeek, a novel dataset designed to evaluate RAG performance on visually rich documents requiring complex reasoning. Based on it, we identify key limitations in current RAG approaches: (i) purely visual retrieval methods struggle to effectively integrate both textual and visual features, and (ii) previous approaches often allocate insufficient reasoning tokens, limiting their effectiveness. To address these challenges, we propose ViDoRAG, a novel multi-agent RAG framework tailored for complex reasoning across visual documents. ViDoRAG employs a Gaussian Mixture Model (GMM)-based hybrid strategy to effectively handle multi-modal retrieval. To further elicit the model's reasoning capabilities, we introduce an iterative agent workflow incorporating exploration, summarization, and reflection, providing a framework for investigating test-time scaling in RAG domains. Extensive experiments on ViDoSeek validate the effectiveness and generalization of our approach. Notably, ViDoRAG outperforms existing methods by over 10% on the competitive ViDoSeek benchmark.
• 摘要：从视觉上富裕的文档中了解信息仍然是传统检索型生成（RAG）方法的重大挑战。现有基准主要集中在基于图像的问题答案（QA）上，忽视了密集的视觉文档中有效检索，理解和推理的基本挑战。为了弥合这一差距，我们介绍了Vidoseek，这是一个新颖的数据集，旨在评估需要复杂推理的视觉丰富文档的抹布性能。基于它，我们确定了当前抹布方法中的关键局限性：（i）纯粹的视觉检索方法难以有效地整合文本和视觉特征，以及（ii）以前的方法通常分配不足的推理令牌，从而限制了它们的有效性。为了应对这些挑战，我们提出了Vidorag，这是一个新型的多代理RAG框架，该框架量身定制了跨视觉文档的复杂推理。 Vidorag采用高斯混合模型（GMM）的混合策略来有效处理多模式检索。为了进一步引起该模型的推理能力，我们引入了一个迭代代理工作流，其中包含了探索，摘要和反思，提供了一个框架，用于研究RAG域中的测试时间缩放。关于Vidoseek的广泛实验验证了我们方法的有效性和概括。值得注意的是，Vidorag在竞争性Vidoseek基准上优于现有方法超过10％。

Title: OpenFly: A Versatile Toolchain and Large-scale Benchmark for Aerial Vision-Language Navigation

• Authors: Yunpeng Gao, Chenhui Li, Zhongrui You, Junli Liu, Zhen Li, Pengan Chen, Qizhi Chen, Zhonghan Tang, Liansheng Wang, Penghui Yang, Yiwen Tang, Yuhang Tang, Shuai Liang, Songyi Zhu, Ziqin Xiong, Yifei Su, Xinyi Ye, Jianan Li, Yan Ding, Dong Wang, Zhigang Wang, Bin Zhao, Xuelong Li
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2502.18041
• Pdf URL: https://arxiv.org/pdf/2502.18041
• Copy Paste: [[2502.18041]] OpenFly: A Versatile Toolchain and Large-scale Benchmark for Aerial Vision-Language Navigation(https://arxiv.org/abs/2502.18041)
• Keywords: generation
• Abstract: Vision-Language Navigation (VLN) aims to guide agents through an environment by leveraging both language instructions and visual cues, playing a pivotal role in embodied AI. Indoor VLN has been extensively studied, whereas outdoor aerial VLN remains underexplored. The potential reason is that outdoor aerial view encompasses vast areas, making data collection more challenging, which results in a lack of benchmarks. To address this problem, we propose OpenFly, a platform comprising a versatile toolchain and large-scale benchmark for aerial VLN. Firstly, we develop a highly automated toolchain for data collection, enabling automatic point cloud acquisition, scene semantic segmentation, flight trajectory creation, and instruction generation. Secondly, based on the toolchain, we construct a large-scale aerial VLN dataset with 100k trajectories, covering diverse heights and lengths across 18 scenes. The corresponding visual data are generated using various rendering engines and advanced techniques, including Unreal Engine, GTA V, Google Earth, and 3D Gaussian Splatting (3D GS). All data exhibit high visual quality. Particularly, 3D GS supports real-to-sim rendering, further enhancing the realism of the dataset. Thirdly, we propose OpenFly-Agent, a keyframe-aware VLN model, which takes language instructions, current observations, and historical keyframes as input, and outputs flight actions directly. Extensive analyses and experiments are conducted, showcasing the superiority of our OpenFly platform and OpenFly-Agent. The toolchain, dataset, and codes will be open-sourced.
• 摘要：Vision语言导航（VLN）旨在通过利用语言说明和视觉提示，在体现的AI中发挥关键作用来指导代理商度过环境。室内VLN已经进行了广泛的研究，而户外空中VLN仍然没有被忽略。潜在的原因是，户外空中视图涵盖了广阔的区域，使数据收集更具挑战性，从而导致缺乏基准测试。为了解决这个问题，我们提出了OpenFly，该平台包括用于空中VLN的多功能工具链和大规模基准。首先，我们开发了一个高度自动化的工具链来收集数据，从而实现自动点云获取，场景语义细分，飞行轨迹创建和指令生成。其次，根据工具链，我们构建了一个具有100K轨迹的大型空中VLN数据集，涵盖了18个场景的各种高度和长度。相应的视觉数据是使用各种渲染引擎和高级技术生成的，包括虚幻引擎，GTA V，Google Earth和3D高斯分裂（3D GS）。所有数据均表现出较高的视觉质量。特别是，3D GS支持实际到SIM的渲染，进一步增强了数据集的现实主义。第三，我们建议将语言指令，当前观察和历史关键框架作为输入，并直接输出飞行操作，将OpenFly-Agent是一种关键框架VLN模型。进行了广泛的分析和实验，展示了我们的OpenFly平台和OpenFly-Agent的优越性。工具链，数据集和代码将被开源。

Title: Bayesian Optimization for Controlled Image Editing via LLMs

• Authors: Chengkun Cai, Haoliang Liu, Xu Zhao, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, Serge Belongie, Lei Li
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.18116
• Pdf URL: https://arxiv.org/pdf/2502.18116
• Copy Paste: [[2502.18116]] Bayesian Optimization for Controlled Image Editing via LLMs(https://arxiv.org/abs/2502.18116)
• Keywords: generation
• Abstract: In the rapidly evolving field of image generation, achieving precise control over generated content and maintaining semantic consistency remain significant limitations, particularly concerning grounding techniques and the necessity for model fine-tuning. To address these challenges, we propose BayesGenie, an off-the-shelf approach that integrates Large Language Models (LLMs) with Bayesian Optimization to facilitate precise and user-friendly image editing. Our method enables users to modify images through natural language descriptions without manual area marking, while preserving the original image's semantic integrity. Unlike existing techniques that require extensive pre-training or fine-tuning, our approach demonstrates remarkable adaptability across various LLMs through its model-agnostic design. BayesGenie employs an adapted Bayesian optimization strategy to automatically refine the inference process parameters, achieving high-precision image editing with minimal user intervention. Through extensive experiments across diverse scenarios, we demonstrate that our framework significantly outperforms existing methods in both editing accuracy and semantic preservation, as validated using different LLMs including Claude3 and GPT-4.
• 摘要：在图像产生的快速发展的领域中，实现对生成的内容并保持语义一致性的精确控制仍然是显着的局限性，尤其是关于接地技术以及模型微调的必要性。为了应对这些挑战，我们提出了贝叶斯根（Bayesgenie），这是一种现成的方法，将大型语言模型（LLMS）与贝叶斯优化集成在一起，以促进精确且用户友好的图像编辑。我们的方法使用户可以通过自然语言描述修改图像，而无需手动标记，同时保留原始图像的语义完整性。与需要广泛的预训练或微调的现有技术不同，我们的方法通过其模型 - 不合SNOSTIC设计在各种LLM上都表现出显着的适应性。 Bayesgenie采用改编的贝叶斯优化策略来自动完善推理过程参数，从而使用最少的用户干预来实现高精度图像编辑。通过跨不同场景的广泛实验，我们证明了我们的框架在编辑准确性和语义保存方面都显着优于现有方法，这是使用Claude3和GPT-4在内的不同LLM进行验证的。

Title: SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference

• Authors: Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, Jianfei Chen
• Subjects: cs.LG, cs.AI, cs.CV, cs.PF
• Abstract URL: https://arxiv.org/abs/2502.18137
• Pdf URL: https://arxiv.org/pdf/2502.18137
• Copy Paste: [[2502.18137]] SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference(https://arxiv.org/abs/2502.18137)
• Keywords: generation
• Abstract: An efficient attention implementation is essential for large models due to its quadratic time complexity. Fortunately, attention commonly exhibits sparsity, i.e., many values in the attention map are near zero, allowing for the omission of corresponding computations. Many studies have utilized the sparse pattern to accelerate attention. However, most existing works focus on optimizing attention within specific models by exploiting certain sparse patterns of the attention map. A universal sparse attention that guarantees both the speedup and end-to-end performance of diverse models remains elusive. In this paper, we propose SpargeAttn, a universal sparse and quantized attention for any model. Our method uses a two-stage online filter: in the first stage, we rapidly and accurately predict the attention map, enabling the skip of some matrix multiplications in attention. In the second stage, we design an online softmax-aware filter that incurs no extra overhead and further skips some matrix multiplications. Experiments show that our method significantly accelerates diverse models, including language, image, and video generation, without sacrificing end-to-end metrics. The codes are available at this https URL.
• 摘要：由于其二次时间的复杂性，有效的注意力实现对于大型模型至关重要。幸运的是，注意力通常表现出稀疏性，即注意图中的许多值接近零，从而允许省略相应的计算。许多研究利用稀疏模式来加速注意力。但是，大多数现有作品都集中在利用注意图的某些稀疏模式来优化特定模型中的注意力。普遍的稀疏关注可以确保各种模型的加速和端到端性能仍然难以捉摸。在本文中，我们提出了SpargeAttn，这是任何模型的通用稀疏和量化的关注。我们的方法使用了两个阶段的在线过滤器：在第一阶段，我们迅速准确地预测了注意力图，从而可以跳过一些矩阵乘法。在第二阶段，我们设计了一个在线软磁性过滤器，该过滤器不会造成额外的开销，并进一步跳过一些矩阵乘法。实验表明，我们的方法显着加速了不同的模型，包括语言，图像和视频生成，而无需牺牲端到端指标。这些代码可在此HTTPS URL上找到。

Title: Actively Inferring Optimal Measurement Sequences

• Authors: Catherine F. Higham, Paul Henderson, Roderick Murray-Smith
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.18142
• Pdf URL: https://arxiv.org/pdf/2502.18142
• Copy Paste: [[2502.18142]] Actively Inferring Optimal Measurement Sequences(https://arxiv.org/abs/2502.18142)
• Keywords: generative
• Abstract: Measurement of a physical quantity such as light intensity is an integral part of many reconstruction and decision scenarios but can be costly in terms of acquisition time, invasion of or damage to the environment and storage. Data minimisation and compliance with data protection laws is also an important consideration. Where there are a range of measurements that can be made, some may be more informative and compliant with the overall measurement objective than others. We develop an active sequential inference algorithm that uses the low dimensional representational latent space from a variational autoencoder (VAE) to choose which measurement to make next. Our aim is to recover high dimensional data by making as few measurements as possible. We adapt the VAE encoder to map partial data measurements on to the latent space of the complete data. The algorithm draws samples from this latent space and uses the VAE decoder to generate data conditional on the partial measurements. Estimated measurements are made on the generated data and fed back through the partial VAE encoder to the latent space where they can be evaluated prior to making a measurement. Starting from no measurements and a normal prior on the latent space, we consider alternative strategies for choosing the next measurement and updating the predictive posterior prior for the next step. The algorithm is illustrated using the Fashion MNIST dataset and a novel convolutional Hadamard pattern measurement basis. We see that useful patterns are chosen within 10 steps, leading to the convergence of the guiding generative images. Compared with using stochastic variational inference to infer the parameters of the posterior distribution for each generated data point individually, the partial VAE framework can efficiently process batches of generated data and obtains superior results with minimal measurements.
• 摘要：测量物理量（例如光强度）是许多重建和决策情景不可或缺的一部分，但就获取时间，入侵或对环境和存储的损害而言可能是昂贵的。数据最小化和遵守数据保护法也是一个重要的考虑因素。如果可以进行一系列测量，则有些可能比其他测量目标更有用，并且符合整体测量目标。我们开发了一种主动的顺序推理算法，该算法使用从变量自动编码器（VAE）中使用低维表示潜在空间来选择下一个测量值。我们的目的是通过进行尽可能少的测量来恢复高维数据。我们将VAE编码器调整为映射部分数据测量值，以适合完整数据的潜在空间。该算法从该潜在空间中绘制样本，并使用VAE解码器在部分测量中生成数据。对生成的数据进行了估计的测量，并通过部分VAE编码器回馈潜在空间，在进行测量之前可以对其进行评估。从没有测量和潜在空间上的正常先验开始，我们考虑选择下一个测量并更新下一步的预测后验。使用时尚MNIST数据集和新型的卷积Hadamard模式测量基础来说明该算法。我们看到有用的模式在10个步骤内选择，从而导致引导生成图像的收敛性。与使用随机变异推断分别推断每个生成的数据点的后验分布的参数相比，部分VAE框架可以有效地处理生成的数据的批处理，并获得最小测量的较高结果。

Title: Joint Reconstruction of Spatially-Coherent and Realistic Clothed Humans and Objects from a Single Image

• Authors: Ayushi Dutta, Marco Pesavento, Marco Volino, Adrian Hilton, Armin Mustafa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18150
• Pdf URL: https://arxiv.org/pdf/2502.18150
• Copy Paste: [[2502.18150]] Joint Reconstruction of Spatially-Coherent and Realistic Clothed Humans and Objects from a Single Image(https://arxiv.org/abs/2502.18150)
• Keywords: generative
• Abstract: Recent advances in human shape learning have focused on achieving accurate human reconstruction from single-view images. However, in the real world, humans share space with other objects. Reconstructing images with humans and objects is challenging due to the occlusions and lack of 3D spatial awareness, which leads to depth ambiguity in the reconstruction. Existing methods in monocular human-object reconstruction fail to capture intricate details of clothed human bodies and object surfaces due to their template-based nature. In this paper, we jointly reconstruct clothed humans and objects in a spatially coherent manner from single-view images, while addressing human-object occlusions. A novel attention-based neural implicit model is proposed that leverages image pixel alignment to retrieve high-quality details, and incorporates semantic features extracted from the human-object pose to enable 3D spatial awareness. A generative diffusion model is used to handle human-object occlusions. For training and evaluation, we introduce a synthetic dataset with rendered scenes of inter-occluded 3D human scans and diverse objects. Extensive evaluation on both synthetic and real datasets demonstrates the superior quality of proposed human-object reconstructions over competitive methods.
• 摘要：人类形态学习的最新进展重点是从单视图像中实现准确的人类重建。但是，在现实世界中，人类与其他物体共享空间。由于阻塞和缺乏3D空间意识，重建人类和物体的图像是具有挑战性的，这导致了重建的深度歧义。由于基于模板的性质，单身人类对象重建中的现有方法无法捕获穿衣服的人体和物体表面的复杂细节。在本文中，我们从单视图像中以空间连贯的方式共同重建了穿衣服的人和物体，同时解决了人类对象的闭合。提出了一个新型的基于注意力的神经隐式模型，它利用图像像素对齐来检索高质量的细节，并结合了从人体对象姿势提取的语义特征以启用3D空间意识。生成扩散模型用于处理人类对象的闭塞。为了进行培训和评估，我们介绍了一个合成数据集，其中包含相互封闭的3D人类扫描和不同物体的场景。对合成数据集和真实数据集的广泛评估证明了所提出的人类对象重建的质量优于竞争方法。

Title: CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification

• Authors: Mingkun Zhang, Keping Bi, Wei Chen, Jiafeng Guo, Xueqi Cheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.18176
• Pdf URL: https://arxiv.org/pdf/2502.18176
• Copy Paste: [[2502.18176]] CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification(https://arxiv.org/abs/2502.18176)
• Keywords: generation, generative
• Abstract: In this paper, we aim to build an adversarially robust zero-shot image classifier. We ground our work on CLIP, a vision-language pre-trained encoder model that can perform zero-shot classification by matching an image with text prompts ``a photo of a .''. Purification is the path we choose since it does not require adversarial training on specific attack types and thus can cope with any foreseen attacks. We then formulate purification risk as the KL divergence between the joint distributions of the purification process of denoising the adversarial samples and the attack process of adding perturbations to benign samples, through bidirectional Stochastic Differential Equations (SDEs). The final derived results inspire us to explore purification in the multi-modal latent space of CLIP. We propose two variants for our CLIPure approach: CLIPure-Diff which models the likelihood of images' latent vectors with the DiffusionPrior module in DaLLE-2 (modeling the generation process of CLIP's latent vectors), and CLIPure-Cos which models the likelihood with the cosine similarity between the embeddings of an image and ``a photo of a.''. As far as we know, CLIPure is the first purification method in multi-modal latent space and CLIPure-Cos is the first purification method that is not based on generative models, which substantially improves defense efficiency. We conducted extensive experiments on CIFAR-10, ImageNet, and 13 datasets that previous CLIP-based defense methods used for evaluating zero-shot classification robustness. Results show that CLIPure boosts the SOTA robustness by a large margin, e.g., from 71.7% to 91.1% on CIFAR10, from 59.6% to 72.6% on ImageNet, and 108% relative improvements of average robustness on the 13 datasets over previous SOTA. The code is available at this https URL.
• 摘要：在本文中，我们旨在构建一个对抗性稳健的零击图像分类器。我们将工作基于剪辑，这是一种视觉语言预训练的编码器模型，可以通过将图像与文本提示匹配``>''''的图像来执行零弹片分类。纯化是我们选择的路径，因为它不需要对特定攻击类型的对抗训练，因此可以应对任何预测的攻击。然后，我们提出纯化风险，因为纯化的纯化过程的关节分布之间的KL差异是通过双向随机微分方程（SDES）通过双向随机差异方程（SDES）向良性样品添加扰动的攻击过程。最终的衍生结果激发了我们在剪辑的多模式潜在空间中探索纯化。我们为我们的剪辑方法提出了两个变体：剪贴画 - 夹式 - 模拟图像的潜在向量的可能性与dalle-2中的扩散模块的潜在矢量（对剪辑的潜在载体的生成过程建模）和剪贴画，从而模拟了与可能性建模的可能性。图像的嵌入和``a。''的余弦相似性。据我们所知，剪贴画是多模式潜在空间中的第一种纯化方法，而夹子cos是第一种不基于生成模型的纯化方法，它显着提高了防御效率。我们在CIFAR-10，ImageNet和13个数据集上进行了广泛的实验，这些实验以前基于夹子的防御方法用于评估零弹药分类的鲁棒性。结果表明，夹子将SOTA鲁棒性提高了一个大幅度，例如，CIFAR10的固定性从39.7％降低到91.1％，ImageNet上的稳健性从59.6％升至72.6％，而先前SOTA的13个数据集的平均鲁棒性相对提高了108％。该代码可在此HTTPS URL上找到。

Title: Multi-Perspective Data Augmentation for Few-shot Object Detection

• Authors: Anh-Khoa Nguyen Vu, Quoc-Truong Truong, Vinh-Tiep Nguyen, Thanh Duc Ngo, Thanh-Toan Do, Tam V. Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18195
• Pdf URL: https://arxiv.org/pdf/2502.18195
• Copy Paste: [[2502.18195]] Multi-Perspective Data Augmentation for Few-shot Object Detection(https://arxiv.org/abs/2502.18195)
• Keywords: generation
• Abstract: Recent few-shot object detection (FSOD) methods have focused on augmenting synthetic samples for novel classes, show promising results to the rise of diffusion models. However, the diversity of such datasets is often limited in representativeness because they lack awareness of typical and hard samples, especially in the context of foreground and background relationships. To tackle this issue, we propose a Multi-Perspective Data Augmentation (MPAD) framework. In terms of foreground-foreground relationships, we propose in-context learning for object synthesis (ICOS) with bounding box adjustments to enhance the detail and spatial information of synthetic samples. Inspired by the large margin principle, support samples play a vital role in defining class boundaries. Therefore, we design a Harmonic Prompt Aggregation Scheduler (HPAS) to mix prompt embeddings at each time step of the generation process in diffusion models, producing hard novel samples. For foreground-background relationships, we introduce a Background Proposal method (BAP) to sample typical and hard backgrounds. Extensive experiments on multiple FSOD benchmarks demonstrate the effectiveness of our approach. Our framework significantly outperforms traditional methods, achieving an average increase of $17.5\%$ in nAP50 over the baseline on PASCAL VOC. Code is available at this https URL.
• 摘要：最近的几个射击对象检测（FSOD）方法集中在增加新型类别的合成样本上，这显示了扩散模型的兴起结果。但是，此类数据集的多样性通常在代表性上受到限制，因为它们缺乏对典型和硬样本的认识，尤其是在前景和背景关系的背景下。为了解决此问题，我们提出了一个多人数据增强（MPAD）框架。就前景 - 前景关系而言，我们建议对物体合成（ICO）进行围栏调整，以增强合成样本的细节和空间信息。受大幅度原则的启发，支持样本在定义阶级边界方面起着至关重要的作用。因此，我们设计了一个谐波提示聚合调度程序（HPA），以在扩散模型中的每一个时间步骤中混合提示嵌入，从而产生硬新颖的样品。对于前景 - 背景关系，我们引入了一种背景建议方法（BAP），以采样典型和硬背景。对多个FSOD基准测试的广泛实验证明了我们方法的有效性。我们的框架极大地胜过传统方法，在Pascal VOC上，NAP50的平均NAP50的平均增加了$ 17.5 \％。代码可在此HTTPS URL上找到。

Title: Training Consistency Models with Variational Noise Coupling

• Authors: Gianluigi Silvestri, Luca Ambrogioni, Chieh-Hsin Lai, Yuhta Takida, Yuki Mitsufuji
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18197
• Pdf URL: https://arxiv.org/pdf/2502.18197
• Copy Paste: [[2502.18197]] Training Consistency Models with Variational Noise Coupling(https://arxiv.org/abs/2502.18197)
• Keywords: generation, generative
• Abstract: Consistency Training (CT) has recently emerged as a promising alternative to diffusion models, achieving competitive performance in image generation tasks. However, non-distillation consistency training often suffers from high variance and instability, and analyzing and improving its training dynamics is an active area of research. In this work, we propose a novel CT training approach based on the Flow Matching framework. Our main contribution is a trained noise-coupling scheme inspired by the architecture of Variational Autoencoders (VAE). By training a data-dependent noise emission model implemented as an encoder architecture, our method can indirectly learn the geometry of the noise-to-data mapping, which is instead fixed by the choice of the forward process in classical CT. Empirical results across diverse image datasets show significant generative improvements, with our model outperforming baselines and achieving the state-of-the-art (SoTA) non-distillation CT FID on CIFAR-10, and attaining FID on par with SoTA on ImageNet at $64 \times 64$ resolution in 2-step generation. Our code is available at this https URL .
• 摘要：一致性训练（CT）最近成为扩散模型的有前途的替代方案，在图像生成任务中实现了竞争性能。但是，非降级一致性训练通常会遭受较高的差异和不稳定性的影响，并且分析和改善其训练动态是一个积极的研究领域。在这项工作中，我们根据流匹配框架提出了一种新型的CT训练方法。我们的主要贡献是受训练有素的噪声耦合方案，灵感来自各种自动编码器（VAE）的体系结构。通过训练作为编码器体系结构实现的数据依赖性噪声发射模型，我们的方法可以间接学习噪声到数据映射的几何形状，而是通过在经典CT中选择向前过程来固定的。各种图像数据集的经验结果表现出显着的生成改进，我们的模型的表现优于基准，并实现了最先进的（SOTA）非限制CT CT FID在CIFAR-10上，并且在Imagenet上与SOTA达到64美元的sota fid \ times 64 $分辨率为2步生成。我们的代码可在此HTTPS URL上找到。

Title: Synthesizing Consistent Novel Views via 3D Epipolar Attention without Re-Training

• Authors: Botao Ye, Sifei Liu, Xueting Li, Marc Pollefeys, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18219
• Pdf URL: https://arxiv.org/pdf/2502.18219
• Copy Paste: [[2502.18219]] Synthesizing Consistent Novel Views via 3D Epipolar Attention without Re-Training(https://arxiv.org/abs/2502.18219)
• Keywords: generation
• Abstract: Large diffusion models demonstrate remarkable zero-shot capabilities in novel view synthesis from a single image. However, these models often face challenges in maintaining consistency across novel and reference views. A crucial factor leading to this issue is the limited utilization of contextual information from reference views. Specifically, when there is an overlap in the viewing frustum between two views, it is essential to ensure that the corresponding regions maintain consistency in both geometry and appearance. This observation leads to a simple yet effective approach, where we propose to use epipolar geometry to locate and retrieve overlapping information from the input view. This information is then incorporated into the generation of target views, eliminating the need for training or fine-tuning, as the process requires no learnable parameters. Furthermore, to enhance the overall consistency of generated views, we extend the utilization of epipolar attention to a multi-view setting, allowing retrieval of overlapping information from the input view and other target views. Qualitative and quantitative experimental results demonstrate the effectiveness of our method in significantly improving the consistency of synthesized views without the need for any fine-tuning. Moreover, This enhancement also boosts the performance of downstream applications such as 3D reconstruction. The code is available at this https URL.
• 摘要：大型扩散模型在单个图像中表现出了新的视图合成中的显着零击功能。但是，这些模型通常在保持新颖和参考观点的一致性方面面临挑战。导致该问题的关键因素是从参考视图中对上下文信息的利用率有限。具体而言，当两种视图之间的视频中存在重叠时，必须确保相应的区域在几何和外观中保持一致性。该观察结果导致了一种简单而有效的方法，我们建议使用表现几何形状从输入视图找到和检索重叠的信息。然后将这些信息纳入目标视图的产生中，消除了对培训或微调的需求，因为该过程不需要可学习的参数。此外，为了增强生成视图的总体一致性，我们将外层注意力的利用扩展到多视图设置，从而可以从输入视图和其他目标视图中检索重叠的信息。定性和定量实验结果证明了我们方法在显着提高合成视图的一致性而无需进行任何微调的情况下的有效性。此外，这种增强还可以提高下游应用程序（例如3D重建）的性能。该代码可在此HTTPS URL上找到。

Title: Beyond the convexity assumption: Realistic tabular data generation under quantifier-free real linear constraints

• Authors: Mihaela Cătălina Stoian, Eleonora Giunchiglia
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.18237
• Pdf URL: https://arxiv.org/pdf/2502.18237
• Copy Paste: [[2502.18237]] Beyond the convexity assumption: Realistic tabular data generation under quantifier-free real linear constraints(https://arxiv.org/abs/2502.18237)
• Keywords: generation, generative
• Abstract: Synthetic tabular data generation has traditionally been a challenging problem due to the high complexity of the underlying distributions that characterise this type of data. Despite recent advances in deep generative models (DGMs), existing methods often fail to produce realistic datapoints that are well-aligned with available background knowledge. In this paper, we address this limitation by introducing Disjunctive Refinement Layer (DRL), a novel layer designed to enforce the alignment of generated data with the background knowledge specified in user-defined constraints. DRL is the first method able to automatically make deep learning models inherently compliant with constraints as expressive as quantifier-free linear formulas, which can define non-convex and even disconnected spaces. Our experimental analysis shows that DRL not only guarantees constraint satisfaction but also improves efficacy in downstream tasks. Notably, when applied to DGMs that frequently violate constraints, DRL eliminates violations entirely. Further, it improves performance metrics by up to 21.4% in F1-score and 20.9% in Area Under the ROC Curve, thus demonstrating its practical impact on data generation.
• 摘要：传统上，综合表格数据生成是一个具有挑战性的问题，这是由于表征这种类型数据的基础分布的高复杂性。尽管最近的深层生成模型（DGM）取得了进步，但现有方法通常无法生成与可用背景知识相结合的现实数据点。在本文中，我们通过引入分离的改进层（DRL）来解决此限制，这是一个新颖的层，旨在强制实施生成的数据与用户定义的约束中指定的背景知识的对齐。 DRL是第一种能够自动使深度学习模型固有地符合限制的方法，该模型具有像无量词的线性公式一样表达的，它可以定义非凸面甚至是断开的空间。我们的实验分析表明，DRL不仅保证了限制满意度，而且还提高了下游任务的功效。值得注意的是，当应用于经常违反约束的DGM时，DRL完全消除了违规行为。此外，它将F1评分的性能指标提高了21.4％，在ROC曲线下的面积为20.9％，从而证明了其对数据生成的实际影响。

Title: AMPO: Active Multi-Preference Optimization

• Authors: Taneesh Gupta, Rahul Madhavan, Xuchao Zhang, Chetan Bansal, Saravan Rajmohan
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.18293
• Pdf URL: https://arxiv.org/pdf/2502.18293
• Copy Paste: [[2502.18293]] AMPO: Active Multi-Preference Optimization(https://arxiv.org/abs/2502.18293)
• Keywords: generation
• Abstract: Multi-preference optimization enriches language-model alignment beyond pairwise preferences by contrasting entire sets of helpful and undesired responses, thereby enabling richer training signals for large language models. During self-play alignment, these models often produce numerous candidate answers per query, rendering it computationally infeasible to include all responses in the training objective. In this work, we propose $\textit{Active Multi-Preference Optimization}$ (AMPO), a novel approach that combines on-policy generation, a multi-preference group-contrastive loss, and active subset selection. Specifically, we score and embed large candidate pools of responses and then select a small, yet informative, subset that covers reward extremes and distinct semantic clusters for preference optimization. Our contrastive training scheme is capable of identifying not only the best and worst answers but also subtle, underexplored modes that are crucial for robust alignment. Theoretically, we provide guarantees for expected reward maximization using our active selection method, and empirically, AMPO achieves state-of-the-art results on $\textit{AlpacaEval}$ using Llama 8B.
• 摘要：通过对比整个有用和不希望的响应，多种优先优化可以超越成对偏好的语言模型对齐，从而为大型语言模型提供了更丰富的培训信号。在自我播放对准过程中，这些模型通常每个查询都会产生许多候选答案，从而使其在计算上不可行，以在训练目标中包含所有答案。在这项工作中，我们提出了$ \ textit {主动多优先优化} $（AMPO），这是一种新型方法，结合了policy生成，多首相群体对抗性损失和主动子集选择。具体而言，我们对响应的大量候选池进行了评分，然后选择一个小但有益的子集，该子集涵盖了奖励极端和独特的语义簇以进行优先优化。我们的对比训练计划不仅能够识别出最佳和最差的答案，而且还可以识别微妙的，毫无疑问的模式，这些模式对于稳健的一致性至关重要。从理论上讲，我们使用主动选择方法提供了预期奖励最大化的保证，并且从经验上，AMPO使用Llama 8B在$ \ textit {alpacaeval} $上实现了最新的结果。

Title: DeepCircuitX: A Comprehensive Repository-Level Dataset for RTL Code Understanding, Generation, and PPA Analysis

• Authors: Zeju Li, Changran Xu, Zhengyuan Shi, Zedong Peng, Yi Liu, Yunhao Zhou, Lingfeng Zhou, Chengyu Ma, Jianyuan Zhong, Xi Wang, Jieru Zhao, Zhufei Chu, Xiaoyan Yang, Qiang Xu
• Subjects: cs.LG, cs.PL
• Abstract URL: https://arxiv.org/abs/2502.18297
• Pdf URL: https://arxiv.org/pdf/2502.18297
• Copy Paste: [[2502.18297]] DeepCircuitX: A Comprehensive Repository-Level Dataset for RTL Code Understanding, Generation, and PPA Analysis(https://arxiv.org/abs/2502.18297)
• Keywords: generation
• Abstract: This paper introduces DeepCircuitX, a comprehensive repository-level dataset designed to advance RTL (Register Transfer Level) code understanding, generation, and power-performance-area (PPA) analysis. Unlike existing datasets that are limited to either file-level RTL code or physical layout data, DeepCircuitX provides a holistic, multilevel resource that spans repository, file, module, and block-level RTL code. This structure enables more nuanced training and evaluation of large language models (LLMs) for RTL-specific tasks. DeepCircuitX is enriched with Chain of Thought (CoT) annotations, offering detailed descriptions of functionality and structure at multiple levels. These annotations enhance its utility for a wide range of tasks, including RTL code understanding, generation, and completion. Additionally, the dataset includes synthesized netlists and PPA metrics, facilitating early-stage design exploration and enabling accurate PPA prediction directly from RTL code. We demonstrate the dataset's effectiveness on various LLMs finetuned with our dataset and confirm the quality with human evaluations. Our results highlight DeepCircuitX as a critical resource for advancing RTL-focused machine learning applications in hardware design this http URL data is available at this https URL.
• 摘要：本文介绍了DeepCircuitx，这是一个综合存储库级数据集，旨在推进RTL（寄存器传输级别）代码理解，生成和功率性能区域（PPA）分析。与仅限于文件级RTL代码或物理布局数据的现有数据集不同，DeepCircuitX提供了整体，多级资源，该资源跨越存储库，文件，模块和块级RTL代码。该结构可以针对特定于RTL的任务进行大型语言模型（LLM）的更多细微差别培训和评估。 DeepCircuitX充满了思想链（COT）注释，提供了多个级别功能和结构的详细描述。这些注释增强了其针对各种任务的实用性，包括RTL代码的理解，生成和完成。此外，数据集包括合成的网络名单和PPA指标，促进了早期设计探索，并直接从RTL代码直接启用了准确的PPA预测。我们证明了数据集对使用我们的数据集进行的各种LLM的有效性，并通过人类评估确认质量。我们的结果突出显示了DeepCircuitX是用于推进硬件设计中以RTL为重点的机器学习应用程序的关键资源。此HTTP URL数据可在此HTTPS URL上获得。

Title: Bayesian Computation in Deep Learning

• Authors: Wenlong Chen, Bolian Li, Ruqi Zhang, Yingzhen Li
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.18300
• Pdf URL: https://arxiv.org/pdf/2502.18300
• Copy Paste: [[2502.18300]] Bayesian Computation in Deep Learning(https://arxiv.org/abs/2502.18300)
• Keywords: generative
• Abstract: This review paper is intended for the 2nd edition of the Handbook of Markov chain Monte this http URL provide an introduction to approximate inference techniques as Bayesian computation methods applied to deep learning models. We organize the chapter by presenting popular computational methods for (1) Bayesian neural networks and (2) deep generative models, explaining their unique challenges in posterior inference as well as the solutions.
• 摘要：这篇评论论文旨在用于马尔可夫链蒙特手册的第二版，此HTTP URL提供了介绍近似推理技术的介绍，作为应用于深度学习模型的贝叶斯计算方法。我们通过为（1）贝叶斯神经网络和（2）深层生成模型提供流行的计算方法来组织本章，并解释了它们在后推理和解决方案中的独特挑战。

Title: LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representation

• Authors: Pengzhi Li, Pengfei Yu, Zide Liu, Wei He, Xuhao Pan, Xudong Rao, Tao Wei, Wei Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18302
• Pdf URL: https://arxiv.org/pdf/2502.18302
• Copy Paste: [[2502.18302]] LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representation(https://arxiv.org/abs/2502.18302)
• Keywords: generation
• Abstract: In this paper, we introduce LDGen, a novel method for integrating large language models (LLMs) into existing text-to-image diffusion models while minimizing computational demands. Traditional text encoders, such as CLIP and T5, exhibit limitations in multilingual processing, hindering image generation across diverse languages. We address these challenges by leveraging the advanced capabilities of LLMs. Our approach employs a language representation strategy that applies hierarchical caption optimization and human instruction techniques to derive precise semantic information,. Subsequently, we incorporate a lightweight adapter and a cross-modal refiner to facilitate efficient feature alignment and interaction between LLMs and image features. LDGen reduces training time and enables zero-shot multilingual image generation. Experimental results indicate that our method surpasses baseline models in both prompt adherence and image aesthetic quality, while seamlessly supporting multiple languages. Project page: this https URL.
• 摘要：在本文中，我们介绍了LDGEN，这是一种将大型语言模型（LLM）集成到现有文本图像扩散模型中的新方法，同时最大程度地减少了计算需求。传统的文本编码器，例如剪辑和T5，在多语言处理，阻碍跨不同语言的图像产生中显示出局限性。我们通过利用LLM的高级功能来应对这些挑战。我们的方法采用语言表示策略，该策略应用层次标题优化和人类教学技术来得出精确的语义信息。随后，我们结合了一个轻巧的适配器和跨模式炼油厂，以促进LLMS和图像特征之间有效的特征对齐和相互作用。 LDGEN减少了训练时间，并启用零拍的多语言图像生成。实验结果表明，我们的方法以迅速粘附和图像美学质量超过基线模型，同时无缝支持多种语言。项目页面：此HTTPS URL。

Title: Self-Supervised Data Generation for Precision Agriculture: Blending Simulated Environments with Real Imagery

• Authors: Leonardo Saraceni, Ionut Marian Motoi, Daniele Nardi, Thomas Alessandro Ciarfuglia
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2502.18320
• Pdf URL: https://arxiv.org/pdf/2502.18320
• Copy Paste: [[2502.18320]] Self-Supervised Data Generation for Precision Agriculture: Blending Simulated Environments with Real Imagery(https://arxiv.org/abs/2502.18320)
• Keywords: generation
• Abstract: In precision agriculture, the scarcity of labeled data and significant covariate shifts pose unique challenges for training machine learning models. This scarcity is particularly problematic due to the dynamic nature of the environment and the evolving appearance of agricultural subjects as living things. We propose a novel system for generating realistic synthetic data to address these challenges. Utilizing a vineyard simulator based on the Unity engine, our system employs a cut-and-paste technique with geometrical consistency considerations to produce accurate photo-realistic images and labels from synthetic environments to train detection algorithms. This approach generates diverse data samples across various viewpoints and lighting conditions. We demonstrate considerable performance improvements in training a state-of-the-art detector by applying our method to table grapes cultivation. The combination of techniques can be easily automated, an increasingly important consideration for adoption in agricultural practice.
• 摘要：在精确的农业中，标记的数据和重大协变量的稀缺性在训练机器学习模型中构成了独特的挑战。由于环境的动态性质以及农业主体作为生物的发展，这种稀缺性尤其有问题。我们提出了一个新型系统，以生成现实的合成数据来应对这些挑战。我们的系统利用基于统一发动机的葡萄园模拟器采用了带有几何一致性考虑的剪切技术，从而产生了来自合成环境的准确的光真实图像和标签，以训练检测算法。这种方法在各种观点和照明条件下生成各种数据样本。我们通过将我们的方法应用于葡萄种植来培训最先进的探测器，在培训最先进的探测器方面进行了可观的绩效提高。技术的组合可以很容易地自动化，这是在农业实践中采用的越来越重要的考虑因素。

Title: ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation

• Authors: Yifan Pu, Yiming Zhao, Zhicong Tang, Ruihong Yin, Haoxing Ye, Yuhui Yuan, Dong Chen, Jianmin Bao, Sirui Zhang, Yanbin Wang, Lin Liang, Lijuan Wang, Ji Li, Xiu Li, Zhouhui Lian, Gao Huang, Baining Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18364
• Pdf URL: https://arxiv.org/pdf/2502.18364
• Copy Paste: [[2502.18364]] ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation(https://arxiv.org/abs/2502.18364)
• Keywords: generation, generative
• Abstract: Multi-layer image generation is a fundamental task that enables users to isolate, select, and edit specific image layers, thereby revolutionizing interactions with generative models. In this paper, we introduce the Anonymous Region Transformer (ART), which facilitates the direct generation of variable multi-layer transparent images based on a global text prompt and an anonymous region layout. Inspired by Schema theory suggests that knowledge is organized in frameworks (schemas) that enable people to interpret and learn from new information by linking it to prior knowledge.}, this anonymous region layout allows the generative model to autonomously determine which set of visual tokens should align with which text tokens, which is in contrast to the previously dominant semantic layout for the image generation task. In addition, the layer-wise region crop mechanism, which only selects the visual tokens belonging to each anonymous region, significantly reduces attention computation costs and enables the efficient generation of images with numerous distinct layers (e.g., 50+). When compared to the full attention approach, our method is over 12 times faster and exhibits fewer layer conflicts. Furthermore, we propose a high-quality multi-layer transparent image autoencoder that supports the direct encoding and decoding of the transparency of variable multi-layer images in a joint manner. By enabling precise control and scalable layer generation, ART establishes a new paradigm for interactive content creation.
• 摘要：多层图像生成是一项基本任务，使用户能够隔离，选择和编辑特定的图像层，从而彻底改变了与生成模型的相互作用。在本文中，我们介绍了匿名区域变压器（ART），该变压器（ART）促进了基于全局文本提示和匿名区域布局的可变多层透明图像的直接生成。受模式理论启发的启发表明，知识是在框架（架构）中组织的，使人们能够通过将新信息链接到先验知识来解释和学习。}，这种匿名区域布局允许生成模型自主确定哪一组视觉令牌应该应确定与哪些文本令牌保持一致，这与图像生成任务的先前主导的语义布局形成鲜明对比。此外，仅选择属于每个匿名区域的视觉令牌的层裁剪机制可显着降低注意力计算成本，并可以有效地产生具有许多不同层的图像（例如50+）。与完全关注的方法相比，我们的方法的速度超过12倍，并且层冲突较少。此外，我们提出了一个高质量的多层透明图像自动编码器，该图像支持可变多层图像的透明度的直接编码和解码。通过启用精确的控制和可扩展的层产生，ART为交互式内容创建建立了一个新的范式。

Title: GHOST 2.0: generative high-fidelity one shot transfer of heads

• Authors: Alexander Groshev (1), Anastasiia Iashchenko (1), Pavel Paramonov (1), Denis Dimitrov (1 and 2), Andrey Kuznetsov (1 and 2) ((1) SberAI, (2) AIRI)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18417
• Pdf URL: https://arxiv.org/pdf/2502.18417
• Copy Paste: [[2502.18417]] GHOST 2.0: generative high-fidelity one shot transfer of heads(https://arxiv.org/abs/2502.18417)
• Keywords: generative
• Abstract: While the task of face swapping has recently gained attention in the research community, a related problem of head swapping remains largely unexplored. In addition to skin color transfer, head swap poses extra challenges, such as the need to preserve structural information of the whole head during synthesis and inpaint gaps between swapped head and background. In this paper, we address these concerns with GHOST 2.0, which consists of two problem-specific modules. First, we introduce enhanced Aligner model for head reenactment, which preserves identity information at multiple scales and is robust to extreme pose variations. Secondly, we use a Blender module that seamlessly integrates the reenacted head into the target background by transferring skin color and inpainting mismatched regions. Both modules outperform the baselines on the corresponding tasks, allowing to achieve state of the art results in head swapping. We also tackle complex cases, such as large difference in hair styles of source and target.
• 摘要：尽管面部交换的任务最近在研究界引起了人们的关注，但头部交换的相关问题仍未得到探索。除了肤色转移外，头交换还带来了额外的挑战，例如需要在合成过程中保留整个头部的结构信息以及交换头和背景之间的涂料差距。在本文中，我们用Ghost 2.0解决了这些问题，该问题由两个特定问题的模块组成。首先，我们引入了用于头部重新制定的增强型对齐器模型，该模型以多个尺度保留身份信息，并且对极端姿势变化是可靠的。其次，我们使用搅拌机模块，该模块通过转移肤色和介入不匹配的区域将重新成型的头部无缝整合到目标背景中。这两个模块在相应的任务上的基准都优于基线，从而实现了最新的状态，从而导致头部交换。我们还解决了复杂的病例，例如源和靶标的发型差异很大。

Title: Scalable Equilibrium Sampling with Sequential Boltzmann Generators

• Authors: Charlie B. Tan, Avishek Joey Bose, Chen Lin, Leon Klein, Michael M. Bronstein, Alexander Tong
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.18462
• Pdf URL: https://arxiv.org/pdf/2502.18462
• Copy Paste: [[2502.18462]] Scalable Equilibrium Sampling with Sequential Boltzmann Generators(https://arxiv.org/abs/2502.18462)
• Keywords: generation
• Abstract: Scalable sampling of molecular states in thermodynamic equilibrium is a long-standing challenge in statistical physics. Boltzmann generators tackle this problem by pairing powerful normalizing flows with importance sampling to obtain statistically independent samples under the target distribution. In this paper, we extend the Boltzmann generator framework and introduce Sequential Boltzmann generators (SBG) with two key improvements. The first is a highly efficient non-equivariant Transformer-based normalizing flow operating directly on all-atom Cartesian coordinates. In contrast to equivariant continuous flows of prior methods, we leverage exactly invertible non-equivariant architectures which are highly efficient both during sample generation and likelihood computation. As a result, this unlocks more sophisticated inference strategies beyond standard importance sampling. More precisely, as a second key improvement we perform inference-time scaling of flow samples using annealed Langevin dynamics which transports samples toward the target distribution leading to lower variance (annealed) importance weights which enable higher fidelity resampling with sequential Monte Carlo. SBG achieves state-of-the-art performance w.r.t. all metrics on molecular systems, demonstrating the first equilibrium sampling in Cartesian coordinates of tri, tetra, and hexapeptides that were so far intractable for prior Boltzmann generators.
• 摘要：热力学平衡中分子状态的可扩展采样是统计物理学的长期挑战。 Boltzmann发电机通过将强大的标准化流与重要性采样配对以获取目标分布下的统计独立样本来解决此问题。在本文中，我们扩展了Boltzmann Generator框架，并引入了Sequential Boltzmann Generator（SBG），并进行了两个关键改进。第一个是直接在全部原子笛卡尔坐标上运行的高效非等级变压器的高标准化流。与先前方法的等效连续流相比，我们利用准确的可逆非等级体系结构，这些体系结构在样本生成和可能性计算过程中都非常有效。结果，这解锁了超出标准重要性抽样的更复杂的推理策略。更确切地说，作为第二个键的改进，我们使用退火的langevin动力学对流程样品进行推理时间缩放，该动力学将样品传输到目标分布，从而导致较低的方差（退火）重要性权重，从而使较高的保真度与顺序蒙特卡洛进行重新采样。 SBG实现最先进的性能W.R.T.分子系统上的所有指标，都证明了三，四肽和六肽的笛卡尔坐标中的第一个平衡采样，这对于先前的Boltzmann发电机而言非常棘手。