2025-02-20

Title: Generative Topology Optimization: Exploring Diverse Solutions in Structural Design

• Authors: Andreas Radler, Eric Volkmann, Johannes Brandstetter, Arturs Berzins
• Subjects: cs.LG, cond-mat.mtrl-sci, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.13174
• Pdf URL: https://arxiv.org/pdf/2502.13174
• Copy Paste: [[2502.13174]] Generative Topology Optimization: Exploring Diverse Solutions in Structural Design(https://arxiv.org/abs/2502.13174)
• Keywords: generative
• Abstract: Topology optimization (TO) is a family of computational methods that derive near-optimal geometries from formal problem descriptions. Despite their success, established TO methods are limited to generating single solutions, restricting the exploration of alternative designs. To address this limitation, we introduce Generative Topology Optimization (GenTO) - a data-free method that trains a neural network to generate structurally compliant shapes and explores diverse solutions through an explicit diversity constraint. The network is trained with a solver-in-the-loop, optimizing the material distribution in each iteration. The trained model produces diverse shapes that closely adhere to the design requirements. We validate GenTO on 2D and 3D TO problems. Our results demonstrate that GenTO produces more diverse solutions than any prior method while maintaining near-optimality and being an order of magnitude faster due to inherent parallelism. These findings open new avenues for engineering and design, offering enhanced flexibility and innovation in structural optimization.
• 摘要：拓扑优化（TO）是一种计算方法家族，从形式的问题描述中得出了近乎最佳的几何形状。尽管成功，但建立的方法仅限于生成单个解决方案，从而限制了替代设计的探索。为了解决此限制，我们引入了生成拓扑优化（GENTO） - 一种无数据的方法，该方法训练神经网络以生成结构符合结构的形状，并通过明确的多样性约束来探索各种解决方案。该网络经过培训，可以在环路中进行求解，从而优化了每次迭代中的材料分布。训练有素的模型产生的形状多种多样，非常遵守设计要求。我们在2D和3D上验证Gento是否有问题。我们的结果表明，Gento产生的解决方案比任何先前的方法都多，同时维持近乎优势，并且由于固有的并行性而更快。这些发现为工程和设计开辟了新的途径，为结构优化提供了增强的灵活性和创新。

Title: Enhancing Machine Learning Performance through Intelligent Data Quality Assessment: An Unsupervised Data-centric Framework

• Authors: Manal Rahal, Bestoun S. Ahmed, Gergely Szabados, Torgny Fornstedt, Jorgen Samuelsson
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.13198
• Pdf URL: https://arxiv.org/pdf/2502.13198
• Copy Paste: [[2502.13198]] Enhancing Machine Learning Performance through Intelligent Data Quality Assessment: An Unsupervised Data-centric Framework(https://arxiv.org/abs/2502.13198)
• Keywords: quality assessment
• Abstract: Poor data quality limits the advantageous power of Machine Learning (ML) and weakens high-performing ML software systems. Nowadays, data are more prone to the risk of poor quality due to their increasing volume and complexity. Therefore, tedious and time-consuming work goes into data preparation and improvement before moving further in the ML pipeline. To address this challenge, we propose an intelligent data-centric evaluation framework that can identify high-quality data and improve the performance of an ML system. The proposed framework combines the curation of quality measurements and unsupervised learning to distinguish high- and low-quality data. The framework is designed to integrate flexible and general-purpose methods so that it is deployed in various domains and applications. To validate the outcomes of the designed framework, we implemented it in a real-world use case from the field of analytical chemistry, where it is tested on three datasets of anti-sense oligonucleotides. A domain expert is consulted to identify the relevant quality measurements and evaluate the outcomes of the framework. The results show that the quality-centric data evaluation framework identifies the characteristics of high-quality data that guide the conduct of efficient laboratory experiments and consequently improve the performance of the ML system.
• 摘要：差的数据质量限制了机器学习（ML）的优势能力，并削弱了高性能的ML软件系统。如今，由于数量和复杂性的增加，数据更容易容易出现质量差的风险。因此，在ML管道中进一步移动之前，乏味且耗时的工作将用于数据制备和改进。为了应对这一挑战，我们提出了一个以数据为中心的评估框架，该框架可以识别高质量的数据并改善ML系统的性能。提出的框架结合了质量测量和无监督学习的策划，以区分高质量和低质量数据。该框架旨在集成灵活和通用方法，以便将其部署在各种域和应用中。为了验证设计框架的结果，我们在分析化学领域的现实用例中实现了它，在该案例中，在三个抗Sensens寡核苷酸的数据集中对其进行了测试。咨询域专家以确定相关质量测量并评估框架的结果。结果表明，以质量为中心的数据评估框架确定了指导有效的实验室实验进行的高质量数据的特征，从而提高了ML系统的性能。

Title: Two Tickets are Better than One: Fair and Accurate Hiring Under Strategic LLM Manipulations

• Authors: Lee Cohen, Jack Hsieh, Connie Hong, Judy Hanwen Shen
• Subjects: cs.LG, cs.AI, cs.CY, cs.GT
• Abstract URL: https://arxiv.org/abs/2502.13221
• Pdf URL: https://arxiv.org/pdf/2502.13221
• Copy Paste: [[2502.13221]] Two Tickets are Better than One: Fair and Accurate Hiring Under Strategic LLM Manipulations(https://arxiv.org/abs/2502.13221)
• Keywords: generative
• Abstract: In an era of increasingly capable foundation models, job seekers are turning to generative AI tools to enhance their application materials. However, unequal access to and knowledge about generative AI tools can harm both employers and candidates by reducing the accuracy of hiring decisions and giving some candidates an unfair advantage. To address these challenges, we introduce a new variant of the strategic classification framework tailored to manipulations performed using large language models, accommodating varying levels of manipulations and stochastic outcomes. We propose a ``two-ticket'' scheme, where the hiring algorithm applies an additional manipulation to each submitted resume and considers this manipulated version together with the original submitted resume. We establish theoretical guarantees for this scheme, showing improvements for both the fairness and accuracy of hiring decisions when the true positive rate is maximized subject to a no false positives constraint. We further generalize this approach to an $n$-ticket scheme and prove that hiring outcomes converge to a fixed, group-independent decision, eliminating disparities arising from differential LLM access. Finally, we empirically validate our framework and the performance of our two-ticket scheme on real resumes using an open-source resume screening tool.
• 摘要：在一个越来越强大的基础模型的时代，求职者正在转向生成的AI工具来增强其应用材料。但是，通过降低雇用决策的准确性并使一些候选人具有不公平的优势，对生成AI工具的不平等访问和知识可能会损害雇主和候选人。为了应对这些挑战，我们介绍了针对使用大语言模型进行的操纵量身定制的战略分类框架的新变体，可容纳不同级别的操纵和随机结果。我们提出了一个``两票方案''计划，其中招聘算法对每个提交的简历进行了额外的操纵，并考虑了该操纵版本以及原始提交的简历。我们为该方案建立理论保证，显示出雇用决策的公平性和准确性的改进，而当真实的正率受到无误报约束而最大化的情况下。我们将这种方法进一步概括为$ n $门票方案，并证明雇用结果会融合到固定的，独立的决定，消除了差异LLM访问引起的差异。最后，我们使用开源简历筛选工具在实际简历上验证了我们的框架和两票方案的性能。

Title: Breaking the bonds of generative artificial intelligence by minimizing the maximum entropy

• Authors: Mattia Miotto, Lorenzo Monacelli
• Subjects: cs.LG, cond-mat.stat-mech, cs.IT
• Abstract URL: https://arxiv.org/abs/2502.13287
• Pdf URL: https://arxiv.org/pdf/2502.13287
• Copy Paste: [[2502.13287]] Breaking the bonds of generative artificial intelligence by minimizing the maximum entropy(https://arxiv.org/abs/2502.13287)
• Keywords: generation, generative
• Abstract: The emergence of generative artificial intelligence (GenAI), comprising large language models, text-to-image generators, and AI algorithms for medical drug and material design, had a transformative impact on society. However, despite an initial exponential growth surpassing Moore's law, progress is now plateauing, suggesting we are approaching the limits of current technology. Indeed, these models are notoriously data-hungry, prone to overfitting, and challenging to direct during the generative process, hampering their effective professional employment. To cope with these limitations, we propose a paradigm shift in GenAI by introducing an ab initio method based on the minimal maximum entropy principle. Our approach does not fit the data. Instead, it compresses information in the training set by finding a latent representation parameterized by arbitrary nonlinear functions, such as neural networks. The result is a general physics-driven model, which is data-efficient, resistant to overfitting, and flexible, permitting to control and influence the generative process. Benchmarking shows that our method outperforms variational autoencoders (VAEs) with similar neural architectures, particularly on undersampled datasets. We demonstrate the methods effectiveness in generating images, even with limited training data, and its unprecedented capability to customize the generation process a posteriori without the need of any fine-tuning or retraining.
• 摘要：生成人工智能（Genai）的出现，包括大型语言模型，文本到图像发生器以及用于医学药物和材料设计的AI算法，对社会产生了变革性的影响。但是，尽管最初的指数增长超过了摩尔定律，但现在的进步仍在平稳，这表明我们正在接近当前技术的局限性。确实，这些模型众所周知，渴望数据，容易过度拟合，并且在生成过程中指导的挑战，从而阻碍了其有效的专业工作。为了应对这些局限性，我们通过基于最小的最大熵原理引入AB启动方法提出了Genai的范式转移。我们的方法不符合数据。取而代之的是，它通过查找由任意非线性函数（例如神经网络）参数的潜在表示来压缩培训集中的信息。结果是一个通用物理驱动的模型，该模型具有数据效率，可抵抗过度拟合和灵活，可以控制和影响生成过程。基准测试表明，我们的方法优于具有类似神经体系结构的变异自动编码器（VAE），尤其是在不足的数据集上。即使训练数据有限，我们也证明了生成图像的有效性，以及其前所未有的能力，可以自定义生成过程，而无需进行任何微调或再培训。

Title: Geometry-Aware Diffusion Models for Multiview Scene Inpainting

• Authors: Ahmad Salimi, Tristan Aumentado-Armstrong, Marcus A. Brubaker, Konstantinos G. Derpanis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.13335
• Pdf URL: https://arxiv.org/pdf/2502.13335
• Copy Paste: [[2502.13335]] Geometry-Aware Diffusion Models for Multiview Scene Inpainting(https://arxiv.org/abs/2502.13335)
• Keywords: generative
• Abstract: In this paper, we focus on 3D scene inpainting, where parts of an input image set, captured from different viewpoints, are masked out. The main challenge lies in generating plausible image completions that are geometrically consistent across views. Most recent work addresses this challenge by combining generative models with a 3D radiance field to fuse information across viewpoints. However, a major drawback of these methods is that they often produce blurry images due to the fusion of inconsistent cross-view images. To avoid blurry inpaintings, we eschew the use of an explicit or implicit radiance field altogether and instead fuse cross-view information in a learned space. In particular, we introduce a geometry-aware conditional generative model, capable of inpainting multi-view consistent images based on both geometric and appearance cues from reference images. A key advantage of our approach over existing methods is its unique ability to inpaint masked scenes with a limited number of views (i.e., few-view inpainting), whereas previous methods require relatively large image sets for their 3D model fitting step. Empirically, we evaluate and compare our scene-centric inpainting method on two datasets, SPIn-NeRF and NeRFiller, which contain images captured at narrow and wide baselines, respectively, and achieve state-of-the-art 3D inpainting performance on both. Additionally, we demonstrate the efficacy of our approach in the few-view setting compared to prior methods.
• 摘要：在本文中，我们专注于3D场景介绍，其中从不同角度捕获的输入图像集的一部分被掩盖了。主要的挑战在于生成合理的图像完成，这些图像完成在各个视图上几乎是一致的。最新的工作通过将生成模型与3D辐射字段相结合以融合跨观点的信息来应对这一挑战。但是，这些方法的主要缺点是，由于不一致的跨视图图像融合，它们通常会产生模糊的图像。为了避免模糊的插图，我们避免完全使用显式或隐式辐射场，而是在学习空间中融合了跨视图的信息。特别是，我们引入了一个几何感知条件生成模型，该模型能够基于参考图像的几何和外观提示来介绍多视图一致的图像。我们方法比现有方法的关键优势是其独特的能力，可以用数量有限的视图（即少量视图介入）注入遮罩场景，而以前的方法需要相对较大的图像集来设置其3D模型拟合步骤。从经验上讲，我们在两个数据集上评估和比较了我们以场景为中心的镶嵌方法，即Spin-Nerf和Nerfiller，它们分别包含在狭窄和宽基线的图像，并在两者上实现最新的3D 3D介入性能。此外，我们证明了与先前方法相比，我们在几个视图设置中的方法的功效。

Title: Flow-based generative models as iterative algorithms in probability space

• Authors: Yao Xie, Xiuyuan Cheng
• Subjects: cs.LG, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.13394
• Pdf URL: https://arxiv.org/pdf/2502.13394
• Copy Paste: [[2502.13394]] Flow-based generative models as iterative algorithms in probability space(https://arxiv.org/abs/2502.13394)
• Keywords: generation, generative
• Abstract: Generative AI (GenAI) has revolutionized data-driven modeling by enabling the synthesis of high-dimensional data across various applications, including image generation, language modeling, biomedical signal processing, and anomaly detection. Flow-based generative models provide a powerful framework for capturing complex probability distributions, offering exact likelihood estimation, efficient sampling, and deterministic transformations between distributions. These models leverage invertible mappings governed by Ordinary Differential Equations (ODEs), enabling precise density estimation and likelihood evaluation. This tutorial presents an intuitive mathematical framework for flow-based generative models, formulating them as neural network-based representations of continuous probability densities. We explore key theoretical principles, including the Wasserstein metric, gradient flows, and density evolution governed by ODEs, to establish convergence guarantees and bridge empirical advancements with theoretical insights. By providing a rigorous yet accessible treatment, we aim to equip researchers and practitioners with the necessary tools to effectively apply flow-based generative models in signal processing and machine learning.
• 摘要：生成AI（Genai）通过在各种应用程序中促进高维数据的综合，包括图像产生，语言建模，生物医学信号处理和异常检测，彻底改变了数据驱动的建模。基于流量的生成模型为捕获复杂的概率分布提供了一个强大的框架，提供了精确的似然估计，有效的采样和分布之间的确定性转换。这些模型利用了由普通微分方程（ODE）控制的可逆映射，从而实现了精确的密度估计和似然评估。该教程为基于流量的生成模型提供了一个直观的数学框架，将它们作为基于神经网络的连续概率密度表示。我们探讨了关键的理论原理，包括Wasserstein度量，梯度流和受ODES管辖的密度进化，以建立与理论见解的融合保证和桥梁经验进步。通过提供严格但可访问的治疗方法，我们旨在为研究人员和从业人员提供必要的工具，以有效地将基于流量的生成模型应用于信号处理和机器学习中。

Title: Enhancing Chest X-ray Classification through Knowledge Injection in Cross-Modality Learning

• Authors: Yang Yan, Bingqing Yue, Qiaxuan Li, Man Huang, Jingyu Chen, Zhenzhong Lan
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.13447
• Pdf URL: https://arxiv.org/pdf/2502.13447
• Copy Paste: [[2502.13447]] Enhancing Chest X-ray Classification through Knowledge Injection in Cross-Modality Learning(https://arxiv.org/abs/2502.13447)
• Keywords: generation
• Abstract: The integration of artificial intelligence in medical imaging has shown tremendous potential, yet the relationship between pre-trained knowledge and performance in cross-modality learning remains unclear. This study investigates how explicitly injecting medical knowledge into the learning process affects the performance of cross-modality classification, focusing on Chest X-ray (CXR) images. We introduce a novel Set Theory-based knowledge injection framework that generates captions for CXR images with controllable knowledge granularity. Using this framework, we fine-tune CLIP model on captions with varying levels of medical information. We evaluate the model's performance through zero-shot classification on the CheXpert dataset, a benchmark for CXR classification. Our results demonstrate that injecting fine-grained medical knowledge substantially improves classification accuracy, achieving 72.5\% compared to 49.9\% when using human-generated captions. This highlights the crucial role of domain-specific knowledge in medical cross-modality learning. Furthermore, we explore the influence of knowledge density and the use of domain-specific Large Language Models (LLMs) for caption generation, finding that denser knowledge and specialized LLMs contribute to enhanced performance. This research advances medical image analysis by demonstrating the effectiveness of knowledge injection for improving automated CXR classification, paving the way for more accurate and reliable diagnostic tools.
• 摘要：人工智能在医学成像中的整合表现出了巨大的潜力，但是跨模式学习中预训练的知识与表现之间的关系尚不清楚。这项研究调查了如何将医学知识注入学习过程中如何影响跨模式分类的性能，重点是胸部X射线（CXR）图像。我们介绍了一个基于理论的新型知识注入框架，该框架为CXR图像生成具有可控知识粒度的标题。使用此框架，我们对具有不同级别的医疗信息的标题进行微调剪辑模型。我们通过在CHEXPERT数据集上的零摄像分类来评估模型的性能，CHEXPERT数据集是CXR分类的基准。我们的结果表明，使用人类生成的标题时，注射细颗粒的医学知识显着提高了分类准确性，相比之下，达到72.5％\％。这突出了特定于域知识在医学跨模式学习中的关键作用。此外，我们探讨了知识密度的影响以及特定于领域的大语言模型（LLM）来产生字幕的影响，发现较密集的知识和专业的LLM有助于增强性能。这项研究通过证明知识注入对改善自动化CXR分类的有效性，为更准确和可靠的诊断工具铺平道路来推进医学图像分析。

Title: Interleaved Gibbs Diffusion for Constrained Generation

• Authors: Gautham Govind Anil, Sachin Yadav, Dheeraj Nagaraj, Karthikeyan Shanmugam, Prateek Jain
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.13450
• Pdf URL: https://arxiv.org/pdf/2502.13450
• Copy Paste: [[2502.13450]] Interleaved Gibbs Diffusion for Constrained Generation(https://arxiv.org/abs/2502.13450)
• Keywords: generation, generative
• Abstract: We introduce Interleaved Gibbs Diffusion (IGD), a novel generative modeling framework for mixed continuous-discrete data, focusing on constrained generation problems. Prior works on discrete and continuous-discrete diffusion models assume factorized denoising distribution for fast generation, which can hinder the modeling of strong dependencies between random variables encountered in constrained generation. IGD moves beyond this by interleaving continuous and discrete denoising algorithms via a discrete time Gibbs sampling type Markov chain. IGD provides flexibility in the choice of denoisers, allows conditional generation via state-space doubling and inference time scaling via the ReDeNoise method. Empirical evaluations on three challenging tasks-solving 3-SAT, generating molecule structures, and generating layouts-demonstrate state-of-the-art performance. Notably, IGD achieves a 7% improvement on 3-SAT out of the box and achieves state-of-the-art results in molecule generation without relying on equivariant diffusion or domain-specific architectures. We explore a wide range of modeling, and interleaving strategies along with hyperparameters in each of these problems.
• 摘要：我们引入了交织的Gibbs扩散（IGD），这是一种用于混合连续数据的新型生成建模框架，重点是受约束的产生问题。先前在离散且连续的散差扩散模型上假设快速生成的分解分布分布，这可能会阻碍在受约束生成中遇到的随机变量之间的强依赖性建模。 IGD通过通过离散的时间Gibbs采样类型Markov链进行连续和离散的DeNo算法而超越了这一点。 IGD在选择Denoisiser的选择方面提供了灵活性，可以通过状态空间加倍和推理时间扩展通过再生方法来生成。对三个具有挑战性的解决3-SAT的经验评估，产生分子结构，并产生布局来证明最新性能。值得注意的是，IGD可以在3个SAT开箱即用的3个SAT上提高7％，并在不依赖于地位扩散或特定领域的体系结构的情况下实现最新的分子产生。我们探索了各种各样的建模，并在每个问题中都探索策略以及超参数。

Title: Enhancing Machine Learning Potentials through Transfer Learning across Chemical Elements

• Authors: Sebastien Röcken, Julija Zavadlav
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2502.13522
• Pdf URL: https://arxiv.org/pdf/2502.13522
• Copy Paste: [[2502.13522]] Enhancing Machine Learning Potentials through Transfer Learning across Chemical Elements(https://arxiv.org/abs/2502.13522)
• Keywords: generation
• Abstract: Machine Learning Potentials (MLPs) can enable simulations of ab initio accuracy at orders of magnitude lower computational cost. However, their effectiveness hinges on the availability of considerable datasets to ensure robust generalization across chemical space and thermodynamic conditions. The generation of such datasets can be labor-intensive, highlighting the need for innovative methods to train MLPs in data-scarce scenarios. Here, we introduce transfer learning of potential energy surfaces between chemically similar elements. Specifically, we leverage the trained MLP for silicon to initialize and expedite the training of an MLP for germanium. Utilizing classical force field and ab initio datasets, we demonstrate that transfer learning surpasses traditional training from scratch in force prediction, leading to more stable simulations and improved temperature transferability. These advantages become even more pronounced as the training dataset size decreases. The out-of-target property analysis shows that transfer learning leads to beneficial but sometimes adversarial effects. Our findings demonstrate that transfer learning across chemical elements is a promising technique for developing accurate and numerically stable MLPs, particularly in a data-scarce regime.
• 摘要：机器学习潜力（MLP）可以以较低的计算成本订单来模拟从头算的准确性。但是，它们的有效性取决于可用性数据集的可用性，以确保在化学空间和热力学条件上进行稳健的概括。这样的数据集的产生可能是劳动力密集的，强调了在数据筛选方案中训练MLP的创新方法。在这里，我们在化学相似的元素之间引入了势能表面的转移学习。具体来说，我们利用训练有素的MLP为硅来初始化和加快对锗的MLP培训。利用经典的力场和AB始于数据集，我们证明，转移学习超过了从头开始的传统训练，从而导致更稳定的模拟和改善的温度可传递性。随着训练数据集大小的减小，这些优势变得更加明显。目标外的属性分析表明，转移学习会导致有益但有时会产生对抗性影响。我们的发现表明，跨化学元素的转移学习是开发准确且数值稳定的MLP的有前途技术，尤其是在数据制度方案中。

Title: Are Large Language Models In-Context Graph Learners?

• Authors: Jintang Li, Ruofan Wu, Yuchang Zhu, Huizhe Zhang, Liang Chen, Zibin Zheng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.13562
• Pdf URL: https://arxiv.org/pdf/2502.13562
• Copy Paste: [[2502.13562]] Are Large Language Models In-Context Graph Learners?(https://arxiv.org/abs/2502.13562)
• Keywords: generation
• Abstract: Large language models (LLMs) have demonstrated remarkable in-context reasoning capabilities across a wide range of tasks, particularly with unstructured inputs such as language or images. However, LLMs struggle to handle structured data, such as graphs, due to their lack of understanding of non-Euclidean structures. As a result, without additional fine-tuning, their performance significantly lags behind that of graph neural networks (GNNs) in graph learning tasks. In this paper, we show that learning on graph data can be conceptualized as a retrieval-augmented generation (RAG) process, where specific instances (e.g., nodes or edges) act as queries, and the graph itself serves as the retrieved context. Building on this insight, we propose a series of RAG frameworks to enhance the in-context learning capabilities of LLMs for graph learning tasks. Comprehensive evaluations demonstrate that our proposed RAG frameworks significantly improve LLM performance on graph-based tasks, particularly in scenarios where a pretrained LLM must be used without modification or accessed via an API.
• 摘要：大型语言模型（LLMS）已在各种任务中表现出显着的上下文推理功能，尤其是在非结构化输入（例如语言或图像）的情况下。但是，由于缺乏对非欧几里得结构的了解，LLM努力处理结构化数据，例如图形。结果，在没有其他微调的情况下，它们的性能显着落后于图形学习任务中的图神经网络（GNN）。在本文中，我们表明，对图数据的学习可以概念化为检索增强的生成（RAG）过程，其中特定实例（例如节点或边缘）充当查询，并且图本身用作检索的上下文。在此洞察力的基础上，我们提出了一系列的抹布框架，以增强LLMS用于图形学习任务的文字学习能力。全面的评估表明，我们提出的抹布框架可显着提高基于图的任务的LLM性能，尤其是在必须使用未经修改或通过API访问的情况下使用的LLM的场景。

Title: Exploring Mutual Cross-Modal Attention for Context-Aware Human Affordance Generation

• Authors: Prasun Roy, Saumik Bhattacharya, Subhankar Ghosh, Umapada Pal, Michael Blumenstein
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2502.13637
• Pdf URL: https://arxiv.org/pdf/2502.13637
• Copy Paste: [[2502.13637]] Exploring Mutual Cross-Modal Attention for Context-Aware Human Affordance Generation(https://arxiv.org/abs/2502.13637)
• Keywords: generation
• Abstract: Human affordance learning investigates contextually relevant novel pose prediction such that the estimated pose represents a valid human action within the scene. While the task is fundamental to machine perception and automated interactive navigation agents, the exponentially large number of probable pose and action variations make the problem challenging and non-trivial. However, the existing datasets and methods for human affordance prediction in 2D scenes are significantly limited in the literature. In this paper, we propose a novel cross-attention mechanism to encode the scene context for affordance prediction by mutually attending spatial feature maps from two different modalities. The proposed method is disentangled among individual subtasks to efficiently reduce the problem complexity. First, we sample a probable location for a person within the scene using a variational autoencoder (VAE) conditioned on the global scene context encoding. Next, we predict a potential pose template from a set of existing human pose candidates using a classifier on the local context encoding around the predicted location. In the subsequent steps, we use two VAEs to sample the scale and deformation parameters for the predicted pose template by conditioning on the local context and template class. Our experiments show significant improvements over the previous baseline of human affordance injection into complex 2D scenes.
• 摘要：人类负担能力学习调查了上下文相关的新型姿势预测，因此估计的姿势代表了现场的有效人类行动。虽然该任务是机器感知和自动交互式导航代理的基础，但指数的可能的姿势和动作变化使问题具有挑战性和非平凡。但是，在文献中，2D场景中人类负担预测的现有数据集和方法受到了限制。在本文中，我们提出了一种新颖的跨注意机制，通过相互参与来自两种不同方式的空间特征图来编码现场上下文，以预测。所提出的方法在各个子任务之间分离，以有效地降低问题的复杂性。首先，我们使用在全局场景上下文编码的条件下，使用各种自动编码器（VAE）为场景中的一个人品尝了一个可能的位置。接下来，我们预测使用一组现有的人类姿势候选者的姿势模板使用分类器在围绕预测位置编码的本地上下文上进行。在随后的步骤中，我们使用两个VAE通过在本地上下文和模板类上进行调节来为预测的姿势模板采样规模和变形参数。我们的实验表明，与先前的人类负担能力注入基线的重大改善为复杂的2D场景。

Title: Reverse Markov Learning: Multi-Step Generative Models for Complex Distributions

• Authors: Xinwei Shen, Nicolai Meinshausen, Tong Zhang
• Subjects: cs.LG, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.13747
• Pdf URL: https://arxiv.org/pdf/2502.13747
• Copy Paste: [[2502.13747]] Reverse Markov Learning: Multi-Step Generative Models for Complex Distributions(https://arxiv.org/abs/2502.13747)
• Keywords: generative
• Abstract: Learning complex distributions is a fundamental challenge in contemporary applications. Generative models, such as diffusion models, have demonstrated remarkable success in overcoming many limitations of traditional statistical methods. Shen and Meinshausen (2024) introduced engression, a generative approach based on scoring rules that maps noise (and covariates, if available) directly to data. While effective, engression struggles with highly complex distributions, such as those encountered in image data. In this work, we extend engression to improve its capability in learning complex distributions. We propose a framework that defines a general forward process transitioning from the target distribution to a known distribution (e.g., Gaussian) and then learns a reverse Markov process using multiple engression models. This reverse process reconstructs the target distribution step by step. Our approach supports general forward processes, allows for dimension reduction, and naturally discretizes the generative process. As a special case, when using a diffusion-based forward process, our framework offers a method to discretize the training and inference of diffusion models efficiently. Empirical evaluations on simulated and climate data validate our theoretical insights, demonstrating the effectiveness of our approach in capturing complex distributions.
• 摘要：学习复杂分布是当代应用中的一个基本挑战。生成模型（例如扩散模型）在克服传统统计方法的许多局限性方面取得了显着的成功。 Shen and Meinshausen（2024）引入了Engression，这是一种基于评分规则的生成方法，将噪声（以及协变量（如果有））直接映射到数据。虽然有效，但Engression与高度复杂的分布（例如图像数据中遇到的分布）进行了斗争。在这项工作中，我们扩展了工程，以提高其学习复杂分布的能力。我们提出了一个框架，该框架定义了从目标分布到已知分布（例如高斯），然后使用多重工程模型学习反向Markov过程的框架。此反向过程逐步重建目标分布。我们的方法支持一般的前向过程，允许降低维度，并自然地分散生成过程。作为一种特殊情况，当使用基于扩散的远期过程时，我们的框架提供了一种有效地离散扩散模型的训练和推断的方法。对模拟和气候数据的经验评估验证了我们的理论见解，证明了我们方法在捕获复杂分布方面的有效性。

Title: RobustX: Robust Counterfactual Explanations Made Easy

• Authors: Junqi Jiang, Luca Marzari, Aaryan Purohit, Francesco Leofante
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.13751
• Pdf URL: https://arxiv.org/pdf/2502.13751
• Copy Paste: [[2502.13751]] RobustX: Robust Counterfactual Explanations Made Easy(https://arxiv.org/abs/2502.13751)
• Keywords: generation
• Abstract: The increasing use of Machine Learning (ML) models to aid decision-making in high-stakes industries demands explainability to facilitate trust. Counterfactual Explanations (CEs) are ideally suited for this, as they can offer insights into the predictions of an ML model by illustrating how changes in its input data may lead to different outcomes. However, for CEs to realise their explanatory potential, significant challenges remain in ensuring their robustness under slight changes in the scenario being explained. Despite the widespread recognition of CEs' robustness as a fundamental requirement, a lack of standardised tools and benchmarks hinders a comprehensive and effective comparison of robust CE generation methods. In this paper, we introduce RobustX, an open-source Python library implementing a collection of CE generation and evaluation methods, with a focus on the robustness property. RobustX provides interfaces to several existing methods from the literature, enabling streamlined access to state-of-the-art techniques. The library is also easily extensible, allowing fast prototyping of novel robust CE generation and evaluation methods.
• 摘要：机器学习（ML）模型的越来越多的使用来帮助高风险行业的决策，需要解释性促进信任。反事实解释（CES）非常适合这一点，因为它们可以通过说明其输入数据的变化如何导致不同结果来提供对ML模型预测的见解。但是，为了使CES意识到其解释潜力，在确保其稳健性在所解释的情况下，确保其稳健性仍然存在。尽管人们对CES的鲁棒性是基本要求的广泛认识，但缺乏标准化工具和基准却阻碍了对强大CE生成方法的全面和有效比较。在本文中，我们介绍了一个开源的Python库Robustx，该图书馆实施了CE发电和评估方法的集合，重点是鲁棒性属性。 Robustx为文献提供了几种现有方法的接口，从而可以简化地访问最先进的技术。该库也很容易扩展，从而可以快速原型制作新颖的强大CE生成和评估方法。

Title: An Overall Real-Time Mechanism for Classification and Quality Evaluation of Rice

• Authors: Wanke Xia, Ruxin Peng, Haoqi Chu, Xinlei Zhu, Zhiyu Yang, Yaojun Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.13764
• Pdf URL: https://arxiv.org/pdf/2502.13764
• Copy Paste: [[2502.13764]] An Overall Real-Time Mechanism for Classification and Quality Evaluation of Rice(https://arxiv.org/abs/2502.13764)
• Keywords: quality assessment
• Abstract: Rice is one of the most widely cultivated crops globally and has been developed into numerous varieties. The quality of rice during cultivation is primarily determined by its cultivar and characteristics. Traditionally, rice classification and quality assessment rely on manual visual inspection, a process that is both time-consuming and prone to errors. However, with advancements in machine vision technology, automating rice classification and quality evaluation based on its cultivar and characteristics has become increasingly feasible, enhancing both accuracy and efficiency. This study proposes a real-time evaluation mechanism for comprehensive rice grain assessment, integrating a one-stage object detection approach, a deep convolutional neural network, and traditional machine learning techniques. The proposed framework enables rice variety identification, grain completeness grading, and grain chalkiness evaluation. The rice grain dataset used in this study comprises approximately 20,000 images from six widely cultivated rice varieties in China. Experimental results demonstrate that the proposed mechanism achieves a mean average precision (mAP) of 99.14% in the object detection task and an accuracy of 97.89% in the classification task. Furthermore, the framework attains an average accuracy of 97.56% in grain completeness grading within the same rice variety, contributing to an effective quality evaluation system.
• 摘要：大米是全球种植最广泛的农作物之一，已发展为多种品种。栽培过程中大米的质量主要取决于其品种和特征。传统上，水稻分类和质量评估依赖于手动视觉检查，这一过程既耗时又容易出现错误。但是，随着机器视觉技术的进步，基于其品种和特征的水稻分类和质量评估的发展变得越来越可行，从而提高了准确性和效率。这项研究提出了一种实时评估机制，用于全面的水稻谷物评估，整合了一种阶段对象检测方法，深层卷积神经网络和传统的机器学习技术。提出的框架可以使水稻品种识别，谷物完整性分级和谷物粉笔评估。这项研究中使用的水稻谷物数据集包括来自中国六个广泛种植的水稻品种的大约20,000张图像。实验结果表明，所提出的机制在对象检测任务中达到了平均平均精度（MAP）为99.14％，分类任务的准确性为97.89％。此外，该框架的平均准确度在同一大米中的谷物完整性分级为97.56％，这有助于有效的质量评估系统。

Title: From Correctness to Comprehension: AI Agents for Personalized Error Diagnosis in Education

• Authors: Yi-Fan Zhang, Hang Li, Dingjie Song, Lichao Sun, Tianlong Xu, Qingsong Wen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.13789
• Pdf URL: https://arxiv.org/pdf/2502.13789
• Copy Paste: [[2502.13789]] From Correctness to Comprehension: AI Agents for Personalized Error Diagnosis in Education(https://arxiv.org/abs/2502.13789)
• Keywords: generation
• Abstract: Large Language Models (LLMs), such as GPT-4, have demonstrated impressive mathematical reasoning capabilities, achieving near-perfect performance on benchmarks like GSM8K. However, their application in personalized education remains limited due to an overemphasis on correctness over error diagnosis and feedback generation. Current models fail to provide meaningful insights into the causes of student mistakes, limiting their utility in educational contexts. To address these challenges, we present three key contributions. First, we introduce \textbf{MathCCS} (Mathematical Classification and Constructive Suggestions), a multi-modal benchmark designed for systematic error analysis and tailored feedback. MathCCS includes real-world problems, expert-annotated error categories, and longitudinal student data. Evaluations of state-of-the-art models, including \textit{Qwen2-VL}, \textit{LLaVA-OV}, \textit{Claude-3.5-Sonnet} and \textit{GPT-4o}, reveal that none achieved classification accuracy above 30\% or generated high-quality suggestions (average scores below 4/10), highlighting a significant gap from human-level performance. Second, we develop a sequential error analysis framework that leverages historical data to track trends and improve diagnostic precision. Finally, we propose a multi-agent collaborative framework that combines a Time Series Agent for historical analysis and an MLLM Agent for real-time refinement, enhancing error classification and feedback generation. Together, these contributions provide a robust platform for advancing personalized education, bridging the gap between current AI capabilities and the demands of real-world teaching.
• 摘要：大型语言模型（LLMS），例如GPT-4，已经显示出令人印象深刻的数学推理能力，在GSM8K等基准上实现了几乎完美的性能。但是，由于对错误诊断和反馈产生的正确性，他们在个性化教育中的应用仍然受到限制。当前的模型无法提供对学生错误原因的有意义的见解，从而限制了他们在教育环境中的效用。为了应对这些挑战，我们提出了三个关键贡献。首先，我们介绍\ textbf {Mathccs}（数学分类和建设性建议），这是一种用于系统错误分析和量身定制反馈的多模式基准测试。 MATHCC包括现实世界中的问题，专家注册的错误类别和纵向学生数据。对最新模型的评估，包括\ textit {qwen2-vl}，\ textit {llava-ov}，\ textit {claude-3.5-sonnet}和\ textit {gpt-4o}分类精度以上高于30 \％或产生的高质量建议（平均分数低于4/10），突出了从人类水平的表现。其次，我们开发了一个顺序错误分析框架，该框架利用历史数据来跟踪趋势并提高诊断精度。最后，我们提出了一个多代理协作框架，该框架结合了一个用于历史分析的时间序列代理和用于实时改进的MLLM代理，从而增强了错误分类和反馈生成。这些贡献共同为推进个性化的教育提供了一个强大的平台，弥合了当前的AI功能与现实世界教学的需求之间的差距。

Title: MagicGeo: Training-Free Text-Guided Geometric Diagram Generation

• Authors: Junxiao Wang, Ting Zhang, Heng Yu, Jingdong Wang, Hua Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.13855
• Pdf URL: https://arxiv.org/pdf/2502.13855
• Copy Paste: [[2502.13855]] MagicGeo: Training-Free Text-Guided Geometric Diagram Generation(https://arxiv.org/abs/2502.13855)
• Keywords: generation
• Abstract: Geometric diagrams are critical in conveying mathematical and scientific concepts, yet traditional diagram generation methods are often manual and resource-intensive. While text-to-image generation has made strides in photorealistic imagery, creating accurate geometric diagrams remains a challenge due to the need for precise spatial relationships and the scarcity of geometry-specific datasets. This paper presents MagicGeo, a training-free framework for generating geometric diagrams from textual descriptions. MagicGeo formulates the diagram generation process as a coordinate optimization problem, ensuring geometric correctness through a formal language solver, and then employs coordinate-aware generation. The framework leverages the strong language translation capability of large language models, while formal mathematical solving ensures geometric correctness. We further introduce MagicGeoBench, a benchmark dataset of 220 geometric diagram descriptions, and demonstrate that MagicGeo outperforms current methods in both qualitative and quantitative evaluations. This work provides a scalable, accurate solution for automated diagram generation, with significant implications for educational and academic applications.
• 摘要：几何图对于传达数学和科学概念至关重要，但是传统图的生成方法通常是手动和资源密集型的。尽管文本到图像的生成已经在影像图中取得了长足进步，但由于需要精确的空间关系和几何特异性数据集的稀缺性，创建准确的几何图仍然是一个挑战。本文介绍了MagicGeo，这是一个无训练的框架，用于从文本描述中生成几何图。 MagicGeo将图生成过程作为坐标优化问题提出，从而确保通过正式语言求解器的几何正确性，然后采用坐标感知生成。该框架利用大型语言模型的强大语言翻译能力，而正式的数学解决方案可确保几何正确性。我们进一步介绍了MagicGeObench，这是220个几何图描述的基准数据集，并证明MagicGeo在定性和定量评估中都优于当前方法。这项工作为自动图生成提供了可扩展，准确的解决方案，对教育和学术应用产生了重大影响。

Title: Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis

• Authors: Jiahao Gai, Hao (Mark)Chen, Zhican Wang, Hongyu Zhou, Wanru Zhao, Nicholas Lane, Hongxiang Fan
• Subjects: cs.LG, cs.AR, cs.SE
• Abstract URL: https://arxiv.org/abs/2502.13921
• Pdf URL: https://arxiv.org/pdf/2502.13921
• Copy Paste: [[2502.13921]] Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis(https://arxiv.org/abs/2502.13921)
• Keywords: generation
• Abstract: Recent advances in code generation have illuminated the potential of employing large language models (LLMs) for general-purpose programming languages such as Python and C++, opening new opportunities for automating software development and enhancing programmer productivity. The potential of LLMs in software programming has sparked significant interest in exploring automated hardware generation and automation. Although preliminary endeavors have been made to adopt LLMs in generating hardware description languages (HDLs), several challenges persist in this direction. First, the volume of available HDL training data is substantially smaller compared to that for software programming languages. Second, the pre-trained LLMs, mainly tailored for software code, tend to produce HDL designs that are more error-prone. Third, the generation of HDL requires a significantly higher number of tokens compared to software programming, leading to inefficiencies in cost and energy consumption. To tackle these challenges, this paper explores leveraging LLMs to generate High-Level Synthesis (HLS)-based hardware design. Although code generation for domain-specific programming languages is not new in the literature, we aim to provide experimental results, insights, benchmarks, and evaluation infrastructure to investigate the suitability of HLS over low-level HDLs for LLM-assisted hardware design generation. To achieve this, we first finetune pre-trained models for HLS-based hardware generation, using a collected dataset with text prompts and corresponding reference HLS designs. An LLM-assisted framework is then proposed to automate end-to-end hardware code generation, which also investigates the impact of chain-of-thought and feedback loops promoting techniques on HLS-design generation. Limited by the timeframe of this research, we plan to evaluate more advanced reasoning models in the future.
• 摘要：代码生成的最新进展阐明了使用大型语言模型（LLM）来用于通用编程语言（例如Python和C ++），这为自动化软件开发和增强程序员生产率开放了新的机会。 LLM在软件编程中的潜力引发了人们对探索自动硬件生成和自动化的重大兴趣。尽管已经做出了初步的努力来采用LLM在生成硬件说明语言（HDLS）中，但在这个方向上仍然存在一些挑战。首先，与软件编程语言相比，可用HDL培训数据的数量大大较小。其次，主要针对软件代码量身定制的预训练的LLM倾向于生成更容易出错的HDL设计。第三，与软件编程相比，HDL的产生需要更高的代币数量，从而导致成本和能源消耗的效率低下。为了应对这些挑战，本文探讨了利用LLM生成基于高级合成（HLS）的硬件设计。尽管文献中针对特定领域的编程语言的代码生成并不新鲜，但我们旨在提供实验结果，见解，基准和评估基础架构，以研究HLS对LLM HDLS对LLM辅助硬件设计生成的适用性。为了实现这一目标，我们首先使用带有文本提示和相应参考HLS设计的数据集进行了基于HLS的硬件生成的预训练模型。然后提出了一个LLM辅助框架以自动化端到端硬件代码的生成，该框架还研究了促进技术链和反馈循环对HLS设计生成的影响。受这项研究的时间范围的限制，我们计划将来评估更高级的推理模型。

Title: Image compositing is all you need for data augmentation

• Authors: Ang Jia Ning Shermaine, Michalis Lazarou, Tania Stathaki
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.13936
• Pdf URL: https://arxiv.org/pdf/2502.13936
• Copy Paste: [[2502.13936]] Image compositing is all you need for data augmentation(https://arxiv.org/abs/2502.13936)
• Keywords: generative
• Abstract: This paper investigates the impact of various data augmentation techniques on the performance of object detection models. Specifically, we explore classical augmentation methods, image compositing, and advanced generative models such as Stable Diffusion XL and ControlNet. The objective of this work is to enhance model robustness and improve detection accuracy, particularly when working with limited annotated data. Using YOLOv8, we fine-tune the model on a custom dataset consisting of commercial and military aircraft, applying different augmentation strategies. Our experiments show that image compositing offers the highest improvement in detection performance, as measured by precision, recall, and mean Average Precision (mAP@0.50). Other methods, including Stable Diffusion XL and ControlNet, also demonstrate significant gains, highlighting the potential of advanced data augmentation techniques for object detection tasks. The results underline the importance of dataset diversity and augmentation in achieving better generalization and performance in real-world applications. Future work will explore the integration of semi-supervised learning methods and further optimizations to enhance model performance across larger and more complex datasets.
• 摘要：本文研究了各种数据增强技术对对象检测模型性能的影响。具体而言，我们探索经典的增强方法，图像合成和高级生成模型，例如稳定的扩散XL和ControlNet。这项工作的目的是增强模型鲁棒性并提高检测准确性，尤其是在使用有限的注释数据时。使用Yolov8，我们将模型调整在由商用和军用飞机组成的自定义数据集上，采用不同的增强策略。我们的实验表明，图像合成提供了检测性能的最大改善，如精确，召回和平均平均精度（map@0.50）。其他方法，包括稳定的扩散XL和ControlNet，也显示出显着的收益，突出了对象检测任务的高级数据增强技术的潜力。结果强调了数据集多样性和增强在实现现实应用程序中更好地概括和性能方面的重要性。未来的工作将探讨半监督学习方法的集成以及进一步的优化，以增强较大且更复杂的数据集的模型性能。

Title: IP-Composer: Semantic Composition of Visual Concepts

• Authors: Sara Dorfman, Dana Cohen-Bar, Rinon Gal, Daniel Cohen-Or
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2502.13951
• Pdf URL: https://arxiv.org/pdf/2502.13951
• Copy Paste: [[2502.13951]] IP-Composer: Semantic Composition of Visual Concepts(https://arxiv.org/abs/2502.13951)
• Keywords: generation
• Abstract: Content creators often draw inspiration from multiple visual sources, combining distinct elements to craft new compositions. Modern computational approaches now aim to emulate this fundamental creative process. Although recent diffusion models excel at text-guided compositional synthesis, text as a medium often lacks precise control over visual details. Image-based composition approaches can capture more nuanced features, but existing methods are typically limited in the range of concepts they can capture, and require expensive training procedures or specialized data. We present IP-Composer, a novel training-free approach for compositional image generation that leverages multiple image references simultaneously, while using natural language to describe the concept to be extracted from each image. Our method builds on IP-Adapter, which synthesizes novel images conditioned on an input image's CLIP embedding. We extend this approach to multiple visual inputs by crafting composite embeddings, stitched from the projections of multiple input images onto concept-specific CLIP-subspaces identified through text. Through comprehensive evaluation, we show that our approach enables more precise control over a larger range of visual concept compositions.
• 摘要：内容创作者经常从多个视觉来源中汲取灵感，结合不同的元素来制作新的作品。现代计算方法现在旨在模仿这一基本创造过程。尽管最近的扩散模型在文本指导的组成合成方面表现出色，但文本作为一种媒介通常缺乏对视觉细节的精确控制。基于图像的构图方法可以捕获更多细微的特征，但是现有方法通常在它们可以捕获的概念范围内受到限制，并且需要昂贵的培训程序或专业数据。我们提出了IP-Composer，这是一种用于组成图像生成的新型无训练方法，同时利用多个图像参考，同时使用自然语言描述从每个图像中提取的概念。我们的方法建立在IP-ADAPTER上，该IP适应器合成了以输入图像的剪辑嵌入为条件的新型图像。我们通过制作复合嵌入将这种​​方法扩展到多个视觉输入，从多个输入图像的投影缝合到通过文本标识的特定于概念的剪贴夹。通过全面的评估，我们表明我们的方法可以更精确地控制更大的视觉概念组成。

Title: FlexTok: Resampling Images into 1D Token Sequences of Flexible Length

• Authors: Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oğuzhan Fatih Kar, Elmira Amirloo, Alaaeldin El-Nouby, Amir Zamir, Afshin Dehghan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.13967
• Pdf URL: https://arxiv.org/pdf/2502.13967
• Copy Paste: [[2502.13967]] FlexTok: Resampling Images into 1D Token Sequences of Flexible Length(https://arxiv.org/abs/2502.13967)
• Keywords: generation
• Abstract: Image tokenization has enabled major advances in autoregressive image generation by providing compressed, discrete representations that are more efficient to process than raw pixels. While traditional approaches use 2D grid tokenization, recent methods like TiTok have shown that 1D tokenization can achieve high generation quality by eliminating grid redundancies. However, these methods typically use a fixed number of tokens and thus cannot adapt to an image's inherent complexity. We introduce FlexTok, a tokenizer that projects 2D images into variable-length, ordered 1D token sequences. For example, a 256x256 image can be resampled into anywhere from 1 to 256 discrete tokens, hierarchically and semantically compressing its information. By training a rectified flow model as the decoder and using nested dropout, FlexTok produces plausible reconstructions regardless of the chosen token sequence length. We evaluate our approach in an autoregressive generation setting using a simple GPT-style Transformer. On ImageNet, this approach achieves an FID<2 across 8 to 128 tokens, outperforming TiTok and matching state-of-the-art methods with far fewer tokens. We further extend the model to support to text-conditioned image generation and examine how FlexTok relates to traditional 2D tokenization. A key finding is that FlexTok enables next-token prediction to describe images in a coarse-to-fine "visual vocabulary", and that the number of tokens to generate depends on the complexity of the generation task.
• 摘要：图像令牌化已通过提供比原始像素更有效处理的压缩，离散表示，从而实现了自回归图像生成的重大进展。尽管传统方法使用2D网格令牌化，但诸如Titok之类的最新方法表明，1D令牌化可以通过消除网格冗余来实现高生成质量。但是，这些方法通常使用固定数量的令牌，因此无法适应图像的固有复杂性。我们介绍了flextok，这是一个将2D图像投射到可变长度的，下令1D令牌序列的代币。例如，可以将256x256图像重新采样到1到256个离散令牌，从层次和语义上压缩其信息。通过训练一个整流的流程模型作为解码器并使用嵌套辍学器，flextok会产生合理的重建，而不论所选的令牌序列长度如何。我们使用简单的GPT风格的变压器在自回归生成设置中评估我们的方法。在ImageNet上，这种方法在8至128个令牌上实现了FID <2，表现优于Titok，并且具有较少令牌的最新方法。我们进一步扩展了该模型以支持文本条件的图像生成，并研究Flextok与传统2D令牌化的关系。一个关键的发现是，弗莱克斯托克（Flextok）使下一步的预测能够在粗到最新的“视觉词汇”中描述图像，并且要生成的代币数量取决于生成任务的复杂性。