2025-02-03

Title: VLMaterial: Procedural Material Generation with Large Vision-Language Models

• Authors: Beichen Li, Rundi Wu, Armando Solar-Lezama, Changxi Zheng, Liang Shi, Bernd Bickel, Wojciech Matusik
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.18623
• Pdf URL: https://arxiv.org/pdf/2501.18623
• Copy Paste: [[2501.18623]] VLMaterial: Procedural Material Generation with Large Vision-Language Models(https://arxiv.org/abs/2501.18623)
• Keywords: generation
• Abstract: Procedural materials, represented as functional node graphs, are ubiquitous in computer graphics for photorealistic material appearance design. They allow users to perform intuitive and precise editing to achieve desired visual appearances. However, creating a procedural material given an input image requires professional knowledge and significant effort. In this work, we leverage the ability to convert procedural materials into standard Python programs and fine-tune a large pre-trained vision-language model (VLM) to generate such programs from input images. To enable effective fine-tuning, we also contribute an open-source procedural material dataset and propose to perform program-level augmentation by prompting another pre-trained large language model (LLM). Through extensive evaluation, we show that our method outperforms previous methods on both synthetic and real-world examples.
• 摘要：程序化材质以功能节点图的形式表示，在计算机图形学中随处可见，用于逼真的材质外观设计。它们允许用户执行直观而精确的编辑，以实现所需的视觉外观。但是，根据输入图像创建程序化材质需要专业知识和大量工作。在这项工作中，我们利用将程序化材质转换为标准 Python 程序的功能，并微调大型预训练视觉语言模型 (VLM)，以从输入图像生成此类程序。为了实现有效的微调，我们还贡献了一个开源程序化材质数据集，并建议通过提示另一个预训练大型语言模型 (LLM) 来执行程序级增强。通过广泛的评估，我们表明我们的方法在合成和真实世界示例上都优于以前的方法。

Title: DebiasPI: Inference-time Debiasing by Prompt Iteration of a Text-to-Image Generative Model

• Authors: Sarah Bonna, Yu-Cheng Huang, Ekaterina Novozhilova, Sejin Paik, Zhengyang Shan, Michelle Yilin Feng, Ge Gao, Yonish Tayal, Rushil Kulkarni, Jialin Yu, Nupur Divekar, Deepti Ghadiyaram, Derry Wijaya, Margrit Betke
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.18642
• Pdf URL: https://arxiv.org/pdf/2501.18642
• Copy Paste: [[2501.18642]] DebiasPI: Inference-time Debiasing by Prompt Iteration of a Text-to-Image Generative Model(https://arxiv.org/abs/2501.18642)
• Keywords: generation, generative
• Abstract: Ethical intervention prompting has emerged as a tool to counter demographic biases of text-to-image generative AI models. Existing solutions either require to retrain the model or struggle to generate images that reflect desired distributions on gender and race. We propose an inference-time process called DebiasPI for Debiasing-by-Prompt-Iteration that provides prompt intervention by enabling the user to control the distributions of individuals' demographic attributes in image generation. DebiasPI keeps track of which attributes have been generated either by probing the internal state of the model or by using external attribute classifiers. Its control loop guides the text-to-image model to select not yet sufficiently represented attributes, With DebiasPI, we were able to create images with equal representations of race and gender that visualize challenging concepts of news headlines. We also experimented with the attributes age, body type, profession, and skin tone, and measured how attributes change when our intervention prompt targets the distribution of an unrelated attribute type. We found, for example, if the text-to-image model is asked to balance racial representation, gender representation improves but the skin tone becomes less diverse. Attempts to cover a wide range of skin colors with various intervention prompts showed that the model struggles to generate the palest skin tones. We conducted various ablation studies, in which we removed DebiasPI's attribute control, that reveal the model's propensity to generate young, male characters. It sometimes visualized career success by generating two-panel images with a pre-success dark-skinned person becoming light-skinned with success, or switching gender from pre-success female to post-success male, thus further motivating ethical intervention prompting with DebiasPI.
• 摘要：道德干预提示已成为一种应对文本到图像生成 AI 模型的人口统计学偏见的工具。现有的解决方案要么需要重新训练模型，要么难以生成反映性别和种族所需分布的图像。我们提出了一种称为 DebiasPI 的推理时间流程，用于通过提示迭代进行去偏，该流程通过允许用户控制图像生成中个人人口统计属性的分布来提供及时干预。DebiasPI 通过探测模型的内部状态或使用外部属性分类器来跟踪已生成的属性。它的控制循环引导文本到图像模型选择尚未充分表示的属性，借助 DebiasPI，我们能够创建具有同等种族和性别表示的图像，以可视化新闻标题的挑战性概念。我们还尝试了年龄、体型、职业和肤色等属性，并测量了当我们的干预提示针对不相关属性类型的分布时属性如何变化。例如，我们发现，如果要求文本转图像模型平衡种族代表性，性别代表性会有所改善，但肤色会变得不那么多样化。尝试使用各种干预提示覆盖各种肤色表明，该模型很难生成最苍白的肤色。我们进行了各种消融研究，在这些研究中，我们删除了 DebiasPI 的属性控制，结果揭示了该模型倾向于生成年轻的男性角色。它有时会通过生成双面板图像来可视化职业成功，其中成功前的黑皮肤人成功后皮肤变白，或者将性别从成功前的女性转变为成功后的男性，从而进一步激发使用 DebiasPI 进行道德干预提示。

Title: BARNN: A Bayesian Autoregressive and Recurrent Neural Network

• Authors: Dario Coscia, Max Welling, Nicola Demo, Gianluigi Rozza
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18665
• Pdf URL: https://arxiv.org/pdf/2501.18665
• Copy Paste: [[2501.18665]] BARNN: A Bayesian Autoregressive and Recurrent Neural Network(https://arxiv.org/abs/2501.18665)
• Keywords: generation
• Abstract: Autoregressive and recurrent networks have achieved remarkable progress across various fields, from weather forecasting to molecular generation and Large Language Models. Despite their strong predictive capabilities, these models lack a rigorous framework for addressing uncertainty, which is key in scientific applications such as PDE solving, molecular generation and Machine Learning Force Fields. To address this shortcoming we present BARNN: a variational Bayesian Autoregressive and Recurrent Neural Network. BARNNs aim to provide a principled way to turn any autoregressive or recurrent model into its Bayesian version. BARNN is based on the variational dropout method, allowing to apply it to large recurrent neural networks as well. We also introduce a temporal version of the "Variational Mixtures of Posteriors" prior (tVAMP-prior) to make Bayesian inference efficient and well-calibrated. Extensive experiments on PDE modelling and molecular generation demonstrate that BARNN not only achieves comparable or superior accuracy compared to existing methods, but also excels in uncertainty quantification and modelling long-range dependencies.
• 摘要：自回归和循环网络在从天气预报到分子生成和大型语言模型等各个领域都取得了显著进展。尽管这些模型具有强大的预测能力，但它们缺乏解决不确定性的严格框架，而不确定性是 PDE 求解、分子生成和机器学习力场等科学应用的关键。为了解决这一缺点，我们提出了 BARNN：一种变分贝叶斯自回归和循环神经网络。BARNN 旨在提供一种将任何自回归或循环模型转变为其贝叶斯版本的原则性方法。BARNN 基于变分 dropout 方法，允许将其应用于大型循环神经网络。我们还引入了“后验变分混合”先验 (tVAMP-prior) 的时间版本，以使贝叶斯推理高效且校准良好。对 PDE 建模和分子生成的大量实验表明，BARNN 不仅实现了与现有方法相当或更高的精度，而且在量化不确定性和建模长距离依赖性方面也表现出色。

Title: Regularized second-order optimization of tensor-network Born machines

• Authors: Matan Ben-Dov, Jing Chen
• Subjects: cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2501.18691
• Pdf URL: https://arxiv.org/pdf/2501.18691
• Copy Paste: [[2501.18691]] Regularized second-order optimization of tensor-network Born machines(https://arxiv.org/abs/2501.18691)
• Keywords: generative
• Abstract: Tensor-network Born machines (TNBMs) are quantum-inspired generative models for learning data distributions. Using tensor-network contraction and optimization techniques, the model learns an efficient representation of the target distribution, capable of capturing complex correlations with a compact parameterization. Despite their promise, the optimization of TNBMs presents several challenges. A key bottleneck of TNBMs is the logarithmic nature of the loss function that is commonly used for this problem. The single-tensor logarithmic optimization problem cannot be solved analytically, necessitating an iterative approach that slows down convergence and increases the risk of getting trapped in one of many non-optimal local minima. In this paper, we present an improved second-order optimization technique for TNBM training, which significantly enhances convergence rates and the quality of the optimized model. Our method employs a modified Newton's method on the manifold of normalized states, incorporating regularization of the loss landscape to mitigate local minima issues. We demonstrate the effectiveness of our approach by training a one-dimensional matrix product state (MPS) on both discrete and continuous datasets, showcasing its advantages in terms of stability, efficiency, and generalization.
• 摘要：张量网络生成机 (TNBM) 是受量子启发的生成模型，用于学习数据分布。使用张量网络收缩和优化技术，该模型可以学习目标分布的有效表示，能够通过紧凑的参数化捕获复杂的相关性。尽管 TNBM 前景光明，但其优化仍面临一些挑战。TNBM 的一个关键瓶颈是通常用于此问题的损失函数的对数性质。单张量对数优化问题无法通过分析解决，因此需要采用迭代方法，这会减慢收敛速度并增加陷入众多非最优局部最小值之一的风险。在本文中，我们提出了一种改进的二阶优化技术用于 TNBM 训练，可显著提高收敛速度和优化模型的质量。我们的方法在归一化状态流形上采用改进的牛顿法，结合损失景观的正则化来缓解局部最小值问题。我们通过在离散和连续数据集上训练一维矩阵乘积状态（MPS）来证明我们方法的有效性，展示了其在稳定性、效率和泛化方面的优势。

Title: Human Re-ID Meets LVLMs: What can we expect?

• Authors: Kailash Hambarde, Pranita Samale, Hugo Proença
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18698
• Pdf URL: https://arxiv.org/pdf/2501.18698
• Copy Paste: [[2501.18698]] Human Re-ID Meets LVLMs: What can we expect?(https://arxiv.org/abs/2501.18698)
• Keywords: generation
• Abstract: Large vision-language models (LVLMs) have been regarded as a breakthrough advance in an astoundingly variety of tasks, from content generation to virtual assistants and multimodal search or retrieval. However, for many of these applications, the performance of these methods has been widely criticized, particularly when compared with state-of-the-art methods and technologies in each specific domain. In this work, we compare the performance of the leading large vision-language models in the human re-identification task, using as baseline the performance attained by state-of-the-art AI models specifically designed for this problem. We compare the results due to ChatGPT-4o, Gemini-2.0-Flash, Claude 3.5 Sonnet, and Qwen-VL-Max to a baseline ReID PersonViT model, using the well-known Market1501 dataset. Our evaluation pipeline includes the dataset curation, prompt engineering, and metric selection to assess the models' performance. Results are analyzed from many different perspectives: similarity scores, classification accuracy, and classification metrics, including precision, recall, F1 score, and area under curve (AUC). Our results confirm the strengths of LVLMs, but also their severe limitations that often lead to catastrophic answers and should be the scope of further research. As a concluding remark, we speculate about some further research that should fuse traditional and LVLMs to combine the strengths from both families of techniques and achieve solid improvements in performance.
• 摘要：大型视觉语言模型 (LVLM) 被视为从内容生成到虚拟助手以及多模态搜索或检索等各种任务的突破性进展。然而，对于许多此类应用，这些方法的性能受到了广泛批评，尤其是与每个特定领域中最先进的方法和技术相比时。在这项工作中，我们比较了领先的大型视觉语言模型在人类重新识别任务中的表现，以专门为该问题设计的最先进的 AI 模型所取得的性能为基准。我们使用著名的 Market1501 数据集，将 ChatGPT-4o、Gemini-2.0-Flash、Claude 3.5 Sonnet 和 Qwen-VL-Max 的结果与基准 ReID PersonViT 模型进行比较。我们的评估流程包括数据集管理、快速工程和指标选择，以评估模型的性能。从许多不同的角度分析结果：相似度得分、分类准确度和分类指标，包括精度、召回率、F1 得分和曲线下面积 (AUC)。我们的结果证实了 LVLM 的优势，但也证实了它们的严重局限性，这些局限性往往会导致灾难性的答案，应该成为进一步研究的范围。最后，我们推测一些进一步的研究应该融合传统和 LVLM，以结合两种技术的优势并实现性能的稳步提升。

Title: Strong and Controllable 3D Motion Generation

• Authors: Canxuan Gang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18726
• Pdf URL: https://arxiv.org/pdf/2501.18726
• Copy Paste: [[2501.18726]] Strong and Controllable 3D Motion Generation(https://arxiv.org/abs/2501.18726)
• Keywords: generation, generative
• Abstract: Human motion generation is a significant pursuit in generative computer vision with widespread applications in film-making, video games, AR/VR, and human-robot interaction. Current methods mainly utilize either diffusion-based generative models or autoregressive models for text-to-motion generation. However, they face two significant challenges: (1) The generation process is time-consuming, posing a major obstacle for real-time applications such as gaming, robot manipulation, and other online settings. (2) These methods typically learn a relative motion representation guided by text, making it difficult to generate motion sequences with precise joint-level control. These challenges significantly hinder progress and limit the real-world application of human motion generation techniques. To address this gap, we propose a simple yet effective architecture consisting of two key components. Firstly, we aim to improve hardware efficiency and computational complexity in transformer-based diffusion models for human motion generation. By customizing flash linear attention, we can optimize these models specifically for generating human motion efficiently. Furthermore, we will customize the consistency model in the motion latent space to further accelerate motion generation. Secondly, we introduce Motion ControlNet, which enables more precise joint-level control of human motion compared to previous text-to-motion generation methods. These contributions represent a significant advancement for text-to-motion generation, bringing it closer to real-world applications.
• 摘要：人体运动生成是生成式计算机视觉领域的一个重要研究方向，广泛应用于电影制作、视频游戏、AR/VR 和人机交互。当前的方法主要利用基于扩散的生成模型或自回归模型进行文本到运动的生成。然而，它们面临两个重大挑战：（1）生成过程耗时，对游戏、机器人操控和其他在线设置等实时应用构成重大障碍。（2）这些方法通常学习由文本引导的相对运动表示，因此很难生成具有精确关节级控制的运动序列。这些挑战严重阻碍了人体运动生成技术的发展，并限制了其在现实世界中的应用。为了解决这一问题，我们提出了一个由两个关键组件组成的简单而有效的架构。首先，我们的目标是提高基于 Transformer 的扩散模型的硬件效率和计算复杂度，以生成人体运动。通过定制 flash 线性注意力，我们可以专门优化这些模型，以高效地生成人体运动。此外，我们将定制运动潜在空间中的一致性模型，以进一步加速运动生成。其次，我们引入了 Motion ControlNet，与之前的文本转动作生成方法相比，它能够更精确地控制人体运动的关节级。这些贡献代表了文本转动作生成的重大进步，使其更接近现实世界的应用。

Title: Evaluating Spoken Language as a Biomarker for Automated Screening of Cognitive Impairment

• Authors: Maria R. Lima, Alexander Capstick, Fatemeh Geranmayeh, Ramin Nilforooshan, Maja Matarić, Ravi Vaidyanathan, Payam Barnaghi
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18731
• Pdf URL: https://arxiv.org/pdf/2501.18731
• Copy Paste: [[2501.18731]] Evaluating Spoken Language as a Biomarker for Automated Screening of Cognitive Impairment(https://arxiv.org/abs/2501.18731)
• Keywords: generation
• Abstract: Timely and accurate assessment of cognitive impairment is a major unmet need in populations at risk. Alterations in speech and language can be early predictors of Alzheimer's disease and related dementias (ADRD) before clinical signs of neurodegeneration. Voice biomarkers offer a scalable and non-invasive solution for automated screening. However, the clinical applicability of machine learning (ML) remains limited by challenges in generalisability, interpretability, and access to patient data to train clinically applicable predictive models. Using DementiaBank recordings (N=291, 64% female), we evaluated ML techniques for ADRD screening and severity prediction from spoken language. We validated model generalisability with pilot data collected in-residence from older adults (N=22, 59% female). Risk stratification and linguistic feature importance analysis enhanced the interpretability and clinical utility of predictions. For ADRD classification, a Random Forest applied to lexical features achieved a mean sensitivity of 69.4% (95% confidence interval (CI) = 66.4-72.5) and specificity of 83.3% (78.0-88.7). On real-world pilot data, this model achieved a mean sensitivity of 70.0% (58.0-82.0) and specificity of 52.5% (39.3-65.7). For severity prediction using Mini-Mental State Examination (MMSE) scores, a Random Forest Regressor achieved a mean absolute MMSE error of 3.7 (3.7-3.8), with comparable performance of 3.3 (3.1-3.5) on pilot data. Linguistic features associated with higher ADRD risk included increased use of pronouns and adverbs, greater disfluency, reduced analytical thinking, lower lexical diversity and fewer words reflecting a psychological state of completion. Our interpretable predictive modelling offers a novel approach for in-home integration with conversational AI to monitor cognitive health and triage higher-risk individuals, enabling earlier detection and intervention.
• 摘要：及时准确地评估认知障碍是高危人群的主要未满足需求。在出现神经退行性疾病的临床症状之前，言语和语言的改变可以作为阿尔茨海默病和相关痴呆症 (ADRD) 的早期预测指标。语音生物标记为自动筛查提供了一种可扩展且非侵入性的解决方案。然而，机器学习 (ML) 的临床适用性仍然受到通用性、可解释性和获取患者数据以训练临床适用的预测模型方面的挑战的限制。使用 DementiaBank 录音（N=291，64% 为女性），我们评估了 ML 技术在 ADRD 筛查和口语严重程度预测中的应用。我们通过从老年人（N=22，59% 为女性）的住院试验数据验证了模型的通用性。风险分层和语言特征重要性分析增强了预测的可解释性和临床效用。对于 ADRD 分类，应用于词汇特征的随机森林实现了 69.4% 的平均敏感度（95% 置信区间 (CI) = 66.4-72.5）和 83.3% (78.0-88.7) 的特异性。在现实世界的试点数据上，该模型实现了 70.0% (58.0-82.0) 的平均敏感度和 52.5% (39.3-65.7) 的特异性。对于使用简易精神状态检查 (MMSE) 分数进行严重程度预测，随机森林回归器实现了 3.7 (3.7-3.8) 的平均绝对 MMSE 误差，在试点数据上的表现为 3.3 (3.1-3.5)。与较高 ADRD 风险相关的语言特征包括代词和副词的使用增加、不流畅性增加、分析性思维减少、词汇多样性降低以及反映心理完成状态的词语减少。我们的可解释预测模型提供了一种新方法，可以将对话式人工智能融入家庭中以监测认知健康并分类高风险个体，从而实现更早的发现和干预。

Title: Synthetic Data Generation for Augmenting Small Samples

• Authors: Dan Liu, Samer El Kababji, Nicholas Mitsakakis, Lisa Pilgram, Thomas Walters, Mark Clemons, Greg Pond, Alaa El-Hussuna, Khaled El Emam
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18741
• Pdf URL: https://arxiv.org/pdf/2501.18741
• Copy Paste: [[2501.18741]] Synthetic Data Generation for Augmenting Small Samples(https://arxiv.org/abs/2501.18741)
• Keywords: generation, generative
• Abstract: Small datasets are common in health research. However, the generalization performance of machine learning models is suboptimal when the training datasets are small. To address this, data augmentation is one solution. Augmentation increases sample size and is seen as a form of regularization that increases the diversity of small datasets, leading them to perform better on unseen data. We found that augmentation improves prognostic performance for datasets that: have fewer observations, with smaller baseline AUC, have higher cardinality categorical variables, and have more balanced outcome variables. No specific generative model consistently outperformed the others. We developed a decision support model that can be used to inform analysts if augmentation would be useful. For seven small application datasets, augmenting the existing data results in an increase in AUC between 4.31% (AUC from 0.71 to 0.75) and 43.23% (AUC from 0.51 to 0.73), with an average 15.55% relative improvement, demonstrating the nontrivial impact of augmentation on small datasets (p=0.0078). Augmentation AUC was higher than resampling only AUC (p=0.016). The diversity of augmented datasets was higher than the diversity of resampled datasets (p=0.046).
• 摘要：小型数据集在健康研究中很常见。然而，当训练数据集较小时，机器学习模型的泛化性能并不理想。为了解决这个问题，数据增强是一种解决方案。增强可以增加样本量，并被视为一种正则化形式，可以增加小型数据集的多样性，从而使它们在看不见的数据上表现更好。我们发现增强可以提高以下数据集的预后性能：观察次数较少、基线 AUC 较小、基数分类变量较高、结果变量更平衡。没有特定的生成模型始终优于其他模型。我们开发了一个决策支持模型，可用于告知分析师增强是否有用。对于七个小型应用数据集，增强现有数据可使 AUC 增加 4.31%（AUC 从 0.71 增加到 0.75）和 ​​43.23%（AUC 从 0.51 增加到 0.73），平均相对改善 15.55%，表明增强对小型数据集的影响非同小可（p=0.0078）。增强 AUC 高于仅重采样 AUC（p=0.016）。增强数据集的多样性高于重采样数据集的多样性（p=0.046）。

Title: INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation

• Authors: Jian Hu, Zixu Cheng, Shaogang Gong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18753
• Pdf URL: https://arxiv.org/pdf/2501.18753
• Copy Paste: [[2501.18753]] INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation(https://arxiv.org/abs/2501.18753)
• Keywords: generation
• Abstract: Task-generic promptable image segmentation aims to achieve segmentation of diverse samples under a single task description by utilizing only one task-generic prompt. Current methods leverage the generalization capabilities of Vision-Language Models (VLMs) to infer instance-specific prompts from these task-generic prompts in order to guide the segmentation process. However, when VLMs struggle to generalise to some image instances, predicting instance-specific prompts becomes poor. To solve this problem, we introduce \textbf{I}nstance-specific \textbf{N}egative Mining for \textbf{T}ask-Generic Promptable Segmentation (\textbf{INT}). The key idea of INT is to adaptively reduce the influence of irrelevant (negative) prior knowledge whilst to increase the use the most plausible prior knowledge, selected by negative mining with higher contrast, in order to optimise instance-specific prompts generation. Specifically, INT consists of two components: (1) instance-specific prompt generation, which progressively fliters out incorrect information in prompt generation; (2) semantic mask generation, which ensures each image instance segmentation matches correctly the semantics of the instance-specific prompts. INT is validated on six datasets, including camouflaged objects and medical images, demonstrating its effectiveness, robustness and scalability.
• 摘要：任务通用提示式图像分割旨在通过仅使用一个任务通用提示来实现单一任务描述下不同样本的分割。当前方法利用视觉语言模型 (VLM) 的泛化能力，从这些任务通用提示中推断出特定于实例的提示，以指导分割过程。然而，当 VLM 难以推广到某些图像实例时，预测特定于实例的提示会变得很差。为了解决这个问题，我们引入了 \textbf{I} 实例特定 \textbf{N} 负挖掘，用于 \textbf{T}ask-Generic 提示式分割 (\textbf{INT})。INT 的关键思想是自适应地减少不相关（负）先验知识的影响，同时增加使用最合理的先验知识，通过具有更高对比度的负挖掘选择，以优化特定于实例的提示生成。具体来说，INT 由两个部分组成：（1）实例特定提示生成，逐步过滤提示生成中的错误信息；（2）语义掩码生成，确保每个图像实例分割与实例特定提示的语义正确匹配。INT 在六个数据集上进行了验证，包括伪装对象和医学图像，证明了其有效性、稳健性和可扩展性。

Title: Probabilistic Joint Recovery Method for CO$_2$ Plume Monitoring

• Authors: Zijun Deng, Rafael Orozco, Abhinav Prakash Gahlot, Felix J. Herrmann
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2501.18761
• Pdf URL: https://arxiv.org/pdf/2501.18761
• Copy Paste: [[2501.18761]] Probabilistic Joint Recovery Method for CO$_2$ Plume Monitoring(https://arxiv.org/abs/2501.18761)
• Keywords: generative
• Abstract: Reducing CO$_2$ emissions is crucial to mitigating climate change. Carbon Capture and Storage (CCS) is one of the few technologies capable of achieving net-negative CO$_2$ emissions. However, predicting fluid flow patterns in CCS remains challenging due to uncertainties in CO$_2$ plume dynamics and reservoir properties. Building on existing seismic imaging methods like the Joint Recovery Method (JRM), which lacks uncertainty quantification, we propose the Probabilistic Joint Recovery Method (pJRM). By estimating posterior distributions across surveys using a shared generative model, pJRM provides uncertainty information to improve risk assessment in CCS projects.
• 摘要：减少二氧化碳排放对于缓解气候变化至关重要。碳捕获与储存 (CCS) 是少数能够实现二氧化碳净负排放的技术之一。然而，由于二氧化碳羽流动力学和储层特性的不确定性，预测 CCS 中的流体流动模式仍然具有挑战性。基于现有的地震成像方法（如缺乏不确定性量化的联合恢复方法 (JRM)），我们提出了概率联合恢复方法 (pJRM)。通过使用共享生成模型估计调查中的后验分布，pJRM 提供了不确定性信息，以改进 CCS 项目的风险评估。

Title: Navigating the Fragrance space Via Graph Generative Models And Predicting Odors

• Authors: Mrityunjay Sharma, Sarabeshwar Balaji, Pinaki Saha, Ritesh Kumar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18777
• Pdf URL: https://arxiv.org/pdf/2501.18777
• Copy Paste: [[2501.18777]] Navigating the Fragrance space Via Graph Generative Models And Predicting Odors(https://arxiv.org/abs/2501.18777)
• Keywords: generation, generative
• Abstract: We explore a suite of generative modelling techniques to efficiently navigate and explore the complex landscapes of odor and the broader chemical space. Unlike traditional approaches, we not only generate molecules but also predict the odor likeliness with ROC AUC score of 0.97 and assign probable odor labels. We correlate odor likeliness with physicochemical features of molecules using machine learning techniques and leverage SHAP (SHapley Additive exPlanations) to demonstrate the interpretability of the function. The whole process involves four key stages: molecule generation, stringent sanitization checks for molecular validity, fragrance likeliness screening and odor prediction of the generated molecules. By making our code and trained models publicly accessible, we aim to facilitate broader adoption of our research across applications in fragrance discovery and olfactory research.
• 摘要：我们探索了一套生成建模技术，以有效地导航和探索复杂的气味景观和更广泛的化学空间。与传统方法不同，我们不仅生成分子，还预测气味可能性（ROC AUC 得分为 0.97）并分配可能的气味标签。我们使用机器学习技术将气味可能性与分子的物理化学特征相关联，并利用 SHAP（SHapley Additive exPlanations）来证明该函数的可解释性。整个过程涉及四个关键阶段：分子生成、严格的分子有效性消毒检查、香味可能性筛选和生成分子的气味预测。通过公开我们的代码和训练模型，我们旨在促进我们的研究在香味发现和嗅觉研究领域的更广泛应用。

Title: Compositional Generalization Requires More Than Disentangled Representations

• Authors: Qiyao Liang, Daoyuan Qian, Liu Ziyin, Ila Fiete
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18797
• Pdf URL: https://arxiv.org/pdf/2501.18797
• Copy Paste: [[2501.18797]] Compositional Generalization Requires More Than Disentangled Representations(https://arxiv.org/abs/2501.18797)
• Keywords: generation, generative
• Abstract: Composition-the ability to generate myriad variations from finite means-is believed to underlie powerful generalization. However, compositional generalization remains a key challenge for deep learning. A widely held assumption is that learning disentangled (factorized) representations naturally supports this kind of extrapolation. Yet, empirical results are mixed, with many generative models failing to recognize and compose factors to generate out-of-distribution (OOD) samples. In this work, we investigate a controlled 2D Gaussian "bump" generation task, demonstrating that standard generative architectures fail in OOD regions when training with partial data, even when supplied with fully disentangled $(x, y)$ coordinates, re-entangling them through subsequent layers. By examining the model's learned kernels and manifold geometry, we show that this failure reflects a "memorization" strategy for generation through the superposition of training data rather than by combining the true factorized features. We show that models forced-through architectural modifications with regularization or curated training data-to create disentangled representations in the full-dimensional representational (pixel) space can be highly data-efficient and effective at learning to compose in OOD regions. These findings underscore that bottlenecks with factorized/disentangled representations in an abstract representation are insufficient: the model must actively maintain or induce factorization directly in the representational space in order to achieve robust compositional generalization.
• 摘要：组合（从有限均值生成无数变体的能力）被认为是强大泛化的基础。然而，组合泛化仍然是深度学习面临的一个关键挑战。一个普遍持有的假设是，学习解开（分解）的表示自然支持这种推断。然而，经验结果好坏参半，许多生成模型无法识别和组合因子以生成分布外（OOD）样本。在这项工作中，我们研究了一个受控的二维高斯“碰撞”生成任务，表明标准生成架构在使用部分数据进行训练时会在 OOD 区域失败，即使提供完全解开的 $(x, y)$ 坐标，也会通过后续层重新纠缠它们。通过检查模型的学习内核和流形几何，我们表明这种失败反映了通过叠加训练数据而不是组合真正的分解特征来生成的“记忆”策略。我们表明，通过正则化或精心策划的训练数据强制修改架构，在全维表征（像素）空间中创建解缠结表征，可以高度高效地学习在 OOD 区域中进行组合。这些发现强调，抽象表征中因式分解/解缠结表征的瓶颈是不够的：模型必须在表征空间中直接主动维持或诱导因式分解，才能实现稳健的组合泛化。

Title: Every Image Listens, Every Image Dances: Music-Driven Image Animation

• Authors: Zhikang Dong, Weituo Hao, Ju-Chiang Wang, Peng Zhang, Pawel Polak
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18801
• Pdf URL: https://arxiv.org/pdf/2501.18801
• Copy Paste: [[2501.18801]] Every Image Listens, Every Image Dances: Music-Driven Image Animation(https://arxiv.org/abs/2501.18801)
• Keywords: generation
• Abstract: Image animation has become a promising area in multimodal research, with a focus on generating videos from reference images. While prior work has largely emphasized generic video generation guided by text, music-driven dance video generation remains underexplored. In this paper, we introduce MuseDance, an innovative end-to-end model that animates reference images using both music and text inputs. This dual input enables MuseDance to generate personalized videos that follow text descriptions and synchronize character movements with the music. Unlike existing approaches, MuseDance eliminates the need for complex motion guidance inputs, such as pose or depth sequences, making flexible and creative video generation accessible to users of all expertise levels. To advance research in this field, we present a new multimodal dataset comprising 2,904 dance videos with corresponding background music and text descriptions. Our approach leverages diffusion-based methods to achieve robust generalization, precise control, and temporal consistency, setting a new baseline for the music-driven image animation task.
• 摘要：图像动画已成为多模态研究中一个有前途的领域，其重点是从参考图像生成视频。虽然先前的研究主要强调由文本引导的通用视频生成，但音乐驱动的舞蹈视频生成仍未得到充分探索。在本文中，我们介绍了 MuseDance，这是一种创新的端到端模型，它使用音乐和文本输入为参考图像制作动画。这种双重输入使 MuseDance 能够生成遵循文本描述并将角色动作与音乐同步的个性化视频。与现有方法不同，MuseDance 消除了对复杂运动指导输入（例如姿势或深度序列）的需求，使所有专业水平的用户都可以灵活而富有创意地生成视频。为了推动该领域的研究，我们提出了一个新的多模态数据集，其中包含 2,904 个舞蹈视频以及相应的背景音乐和文本描述。我们的方法利用基于扩散的方法来实现强大的泛化、精确控制和时间一致性，为音乐驱动的图像动画任务设定了新的基线。

Title: Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion

• Authors: Vitor Guizilini, Muhammad Zubair Irshad, Dian Chen, Greg Shakhnarovich, Rares Ambrus
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18804
• Pdf URL: https://arxiv.org/pdf/2501.18804
• Copy Paste: [[2501.18804]] Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion(https://arxiv.org/abs/2501.18804)
• Keywords: generation
• Abstract: Current methods for 3D scene reconstruction from sparse posed images employ intermediate 3D representations such as neural fields, voxel grids, or 3D Gaussians, to achieve multi-view consistent scene appearance and geometry. In this paper we introduce MVGD, a diffusion-based architecture capable of direct pixel-level generation of images and depth maps from novel viewpoints, given an arbitrary number of input views. Our method uses raymap conditioning to both augment visual features with spatial information from different viewpoints, as well as to guide the generation of images and depth maps from novel views. A key aspect of our approach is the multi-task generation of images and depth maps, using learnable task embeddings to guide the diffusion process towards specific modalities. We train this model on a collection of more than 60 million multi-view samples from publicly available datasets, and propose techniques to enable efficient and consistent learning in such diverse conditions. We also propose a novel strategy that enables the efficient training of larger models by incrementally fine-tuning smaller ones, with promising scaling behavior. Through extensive experiments, we report state-of-the-art results in multiple novel view synthesis benchmarks, as well as multi-view stereo and video depth estimation.
• 摘要：当前，从稀疏姿势图像进行 3D 场景重建的方法采用中间 3D 表示，例如神经场、体素网格或 3D 高斯，以实现多视图一致的场景外观和几何形状。在本文中，我们介绍了 MVGD，这是一种基于扩散的架构，能够在给定任意数量的输入视图的情况下，从新视点直接生成像素级图像和深度图。我们的方法使用射线图条件，既可以用来自不同视点的空间信息来增强视觉特征，又可以指导从新视图生成图像和深度图。我们方法的一个关键方面是多任务生成图像和深度图，使用可学习的任务嵌入来引导扩散过程朝向特定模态。我们在来自公开数据集的 6000 多万个多视图样本集合上训练该模型，并提出了在如此多样化的条件下实现高效一致学习的技术。我们还提出了一种新颖的策略，通过逐步微调较小的模型来实现对较大模型的高效训练，并具有良好的扩展行为。通过大量实验，我们报告了多个新颖的视图合成基准以及多视图立体和视频深度估计的最新结果。

Title: Project-and-Fuse: Improving RGB-D Semantic Segmentation via Graph Convolution Networks

• Authors: Xiaoyan Jiang, Bohan Wang, Xinlong Wan, Zhi Zhou, Hamido Fujita
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18851
• Pdf URL: https://arxiv.org/pdf/2501.18851
• Copy Paste: [[2501.18851]] Project-and-Fuse: Improving RGB-D Semantic Segmentation via Graph Convolution Networks(https://arxiv.org/abs/2501.18851)
• Keywords: generation
• Abstract: Most existing RGB-D semantic segmentation methods focus on the feature level fusion, including complex cross-modality and cross-scale fusion modules. However, these methods may cause misalignment problem in the feature fusion process and counter-intuitive patches in the segmentation results. Inspired by the popular pixel-node-pixel pipeline, we propose to 1) fuse features from two modalities in a late fusion style, during which the geometric feature injection is guided by texture feature prior; 2) employ Graph Neural Networks (GNNs) on the fused feature to alleviate the emergence of irregular patches by inferring patch relationship. At the 3D feature extraction stage, we argue that traditional CNNs are not efficient enough for depth maps. So, we encode depth map into normal map, after which CNNs can easily extract object surface this http URL projection matrix generation stage, we find the existence of Biased-Assignment and Ambiguous-Locality issues in the original pipeline. Therefore, we propose to 1) adopt the Kullback-Leibler Loss to ensure no missing important pixel features, which can be viewed as hard pixel mining process; 2) connect regions that are close to each other in the Euclidean space as well as in the semantic space with larger edge weights so that location informations can been considered. Extensive experiments on two public datasets, NYU-DepthV2 and SUN RGB-D, have shown that our approach can consistently boost the performance of RGB-D semantic segmentation task.
• 摘要：现有的RGB-D语义分割方法大多侧重于特征级融合，包括复杂的跨模态和跨尺度融合模块。然而，这些方法可能导致特征融合过程中的错位问题和分割结果中出现违反直觉的斑块。受到流行的像素-节点-像素流程的启发，我们提出1）以后期融合方式融合两种模态的特征，在此过程中，几何特征注入由纹理特征先验引导；2）在融合特征上使用图神经网络（GNN），通过推断斑块关系来缓解不规则斑块的出现。在3D特征提取阶段，我们认为传统的CNN对于深度图来说效率不够高。因此，我们将深度图编码为法线图，之后CNN可以轻松提取物体表面。在这个http URL投影矩阵生成阶段，我们发现原始流程中存在Biased-Assignment和Ambiguous-Locality问题。因此，我们提出：1）采用 Kullback-Leibler Loss 来确保不遗漏重要的像素特征，这可以看作是硬像素挖掘过程；2）使用较大的边权重连接欧几里得空间和语义空间中彼此接近的区域，以便可以考虑位置信息。在两个公共数据集 NYU-DepthV2 和 SUN RGB-D 上进行的大量实验表明，我们的方法可以持续提高 RGB-D 语义分割任务的性能。

Title: BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning

• Authors: Han Zhong, Yutong Yin, Shenao Zhang, Xiaojun Xu, Yuanxin Liu, Yifei Zuo, Zhihan Liu, Boyi Liu, Sirui Zheng, Hongyi Guo, Liwei Wang, Mingyi Hong, Zhaoran Wang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18858
• Pdf URL: https://arxiv.org/pdf/2501.18858
• Copy Paste: [[2501.18858]] BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning(https://arxiv.org/abs/2501.18858)
• Keywords: generation
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, yet generating reliable reasoning processes remains a significant challenge. We present a unified probabilistic framework that formalizes LLM reasoning through a novel graphical model incorporating latent thinking processes and evaluation signals. Within this framework, we introduce the Bootstrapping Reinforced Thinking Process (BRiTE) algorithm, which works in two steps. First, it generates high-quality rationales by approximating the optimal thinking process through reinforcement learning, using a novel reward shaping mechanism. Second, it enhances the base LLM by maximizing the joint probability of rationale generation with respect to the model's parameters. Theoretically, we demonstrate BRiTE's convergence at a rate of $1/T$ with $T$ representing the number of iterations. Empirical evaluations on math and coding benchmarks demonstrate that our approach consistently improves performance across different base models without requiring human-annotated thinking processes. In addition, BRiTE demonstrates superior performance compared to existing algorithms that bootstrap thinking processes use alternative methods such as rejection sampling, and can even match or exceed the results achieved through supervised fine-tuning with human-annotated data.
• 摘要：大型语言模型 (LLM) 在复杂的推理任务中表现出了卓越的能力，但生成可靠的推理过程仍然是一项重大挑战。我们提出了一个统一的概率框架，该框架通过结合潜在思维过程和评估信号的新型图形模型将 LLM 推理形式化。在这个框架内，我们引入了引导强化思维过程 (BRiTE) 算法，该算法分两步工作。首先，它通过强化学习近似最佳思维过程，使用一种新颖的奖励塑造机制，生成高质量的理由。其次，它通过最大化与模型参数相关的理由生成的联合概率来增强基础 LLM。从理论上讲，我们证明了 BRiTE 的收敛速度为 $1/T$，其中 $T$ 代表迭代次数。对数学和编码基准的实证评估表明，我们的方法可以持续提高不同基础模型的性能，而无需人工注释的思维过程。此外，BRiTE 与现有的引导思维过程使用拒绝抽样等替代方法的算法相比表现出了更优异的性能，甚至可以达到或超过通过使用人工注释数据进行监督微调所取得的结果。

Title: REG: Rectified Gradient Guidance for Conditional Diffusion Models

• Authors: Zhengqi Gao, Kaiwen Zha, Tianyuan Zhang, Zihui Xue, Duane S. Boning
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18865
• Pdf URL: https://arxiv.org/pdf/2501.18865
• Copy Paste: [[2501.18865]] REG: Rectified Gradient Guidance for Conditional Diffusion Models(https://arxiv.org/abs/2501.18865)
• Keywords: generation
• Abstract: Guidance techniques are simple yet effective for improving conditional generation in diffusion models. Albeit their empirical success, the practical implementation of guidance diverges significantly from its theoretical motivation. In this paper, we reconcile this discrepancy by replacing the scaled marginal distribution target, which we prove theoretically invalid, with a valid scaled joint distribution objective. Additionally, we show that the established guidance implementations are approximations to the intractable optimal solution under no future foresight constraint. Building on these theoretical insights, we propose rectified gradient guidance (REG), a versatile enhancement designed to boost the performance of existing guidance methods. Experiments on 1D and 2D demonstrate that REG provides a better approximation to the optimal solution than prior guidance techniques, validating the proposed theoretical framework. Extensive experiments on class-conditional ImageNet and text-to-image generation tasks show that incorporating REG consistently improves FID and Inception/CLIP scores across various settings compared to its absence.
• 摘要：引导技术简单但有效，可改善扩散模型中的条件生成。尽管它们在经验上取得了成功，但实际实施的引导与其理论动机存在很大差异。在本文中，我们通过用有效的缩放联合分布目标替换我们证明理论上无效的缩放边际分布目标来协调这种差异。此外，我们表明，在没有未来预见约束的情况下，已建立的引导实现是对难以解决的最优解的近似。基于这些理论见解，我们提出了修正梯度引导 (REG)，这是一种多功能增强功能，旨在提高现有引导方法的性能。在 1D 和 2D 上的实验表明，与之前的引导技术相比，REG 提供了对最优解的更好的近似，从而验证了所提出的理论框架。在类条件 ImageNet 和文本到图像生成任务上进行的大量实验表明，与没有 REG 相比，加入 REG 可在各种设置中持续提高 FID 和 Inception/CLIP 分数。

Title: Neural SDEs as a Unified Approach to Continuous-Domain Sequence Modeling

• Authors: Macheng Shen, Chen Cheng
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18871
• Pdf URL: https://arxiv.org/pdf/2501.18871
• Copy Paste: [[2501.18871]] Neural SDEs as a Unified Approach to Continuous-Domain Sequence Modeling(https://arxiv.org/abs/2501.18871)
• Keywords: generative
• Abstract: Inspired by the ubiquitous use of differential equations to model continuous dynamics across diverse scientific and engineering domains, we propose a novel and intuitive approach to continuous sequence modeling. Our method interprets time-series data as \textit{discrete samples from an underlying continuous dynamical system}, and models its time evolution using Neural Stochastic Differential Equation (Neural SDE), where both the flow (drift) and diffusion terms are parameterized by neural networks. We derive a principled maximum likelihood objective and a \textit{simulation-free} scheme for efficient training of our Neural SDE model. We demonstrate the versatility of our approach through experiments on sequence modeling tasks across both embodied and generative AI. Notably, to the best of our knowledge, this is the first work to show that SDE-based continuous-time modeling also excels in such complex scenarios, and we hope that our work opens up new avenues for research of SDE models in high-dimensional and temporally intricate domains.
• 摘要：受微分方程在各种科学和工程领域中广泛用于建模连续动力学的启发，我们提出了一种新颖且直观的连续序列建模方法。我们的方法将时间序列数据解释为来自底层连续动态系统的离散样本，并使用神经随机微分方程 (Neural SDE) 对其时间演化进行建模，其中流动 (漂移) 和扩散项均由神经网络参数化。我们推导出一个原则性最大似然目标和一个无需模拟的方案，以高效训练我们的神经 SDE 模型。我们通过对具身 AI 和生成 AI 的序列建模任务进行实验，证明了我们方法的多功能性。值得注意的是，据我们所知，这是首次表明基于 SDE 的连续时间建模在如此复杂的场景中也表现出色，我们希望我们的工作为在高维和时间复杂领域研究 SDE 模型开辟新的途径。

Title: Best Policy Learning from Trajectory Preference Feedback

• Authors: Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Zheng Wen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18873
• Pdf URL: https://arxiv.org/pdf/2501.18873
• Copy Paste: [[2501.18873]] Best Policy Learning from Trajectory Preference Feedback(https://arxiv.org/abs/2501.18873)
• Keywords: generative
• Abstract: We address the problem of best policy identification in preference-based reinforcement learning (PbRL), where learning occurs from noisy binary preferences over trajectory pairs rather than explicit numerical rewards. This approach is useful for post-training optimization of generative AI models during multi-turn user interactions, where preference feedback is more robust than handcrafted reward models. In this setting, learning is driven by both an offline preference dataset -- collected from a rater of unknown 'competence' -- and online data collected with pure exploration. Since offline datasets may exhibit out-of-distribution (OOD) biases, principled online data collection is necessary. To address this, we propose Posterior Sampling for Preference Learning ($\mathsf{PSPL}$), a novel algorithm inspired by Top-Two Thompson Sampling, that maintains independent posteriors over the true reward model and transition dynamics. We provide the first theoretical guarantees for PbRL in this setting, establishing an upper bound on the simple Bayesian regret of $\mathsf{PSPL}$. Since the exact algorithm can be computationally impractical, we also provide an approximate version that outperforms existing baselines.
• 摘要：我们解决了基于偏好的强化学习 (PbRL) 中最佳策略识别的问题，其中学习发生在轨迹对的嘈杂二元偏好中，而不是明确的数值奖励中。这种方法对于在多轮用户交互期间生成 AI 模型的训练后优化非常有用，其中偏好反馈比手工制作的奖励模型更稳健。在这种情况下，学习由离线偏好数据集（从未知“能力”的评估者那里收集）和通过纯探索收集的在线数据驱动。由于离线数据集可能表现出分布外 (OOD) 偏差，因此有原则的在线数据收集是必要的。为了解决这个问题，我们提出了偏好学习的后验抽样 ($\mathsf{PSPL}$)，这是一种受 Top-Two Thompson Sampling 启发的新算法，它保持了真实奖励模型和过渡动态的独立后验。我们为这种设置中的 PbRL 提供了第一个理论保证，为简单的贝叶斯遗憾 $\mathsf{PSPL}$ 建立了上限。由于精确算法在计算上不切实际，我们还提供了一个优于现有基线的近似版本。

Title: Distorting Embedding Space for Safety: A Defense Mechanism for Adversarially Robust Diffusion Models

• Authors: Jaesin Ahn, Heechul Jung
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18877
• Pdf URL: https://arxiv.org/pdf/2501.18877
• Copy Paste: [[2501.18877]] Distorting Embedding Space for Safety: A Defense Mechanism for Adversarially Robust Diffusion Models(https://arxiv.org/abs/2501.18877)
• Keywords: generation
• Abstract: Text-to-image diffusion models show remarkable generation performance following text prompts, but risk generating Not Safe For Work (NSFW) contents from unsafe prompts. Existing approaches, such as prompt filtering or concept unlearning, fail to defend against adversarial attacks while maintaining benign image quality. In this paper, we propose a novel approach called Distorting Embedding Space (DES), a text encoder-based defense mechanism that effectively tackles these issues through innovative embedding space control. DES transforms unsafe embeddings, extracted from a text encoder using unsafe prompts, toward carefully calculated safe embedding regions to prevent unsafe contents generation, while reproducing the original safe embeddings. DES also neutralizes the nudity embedding, extracted using prompt ``nudity", by aligning it with neutral embedding to enhance robustness against adversarial attacks. These methods ensure both robust defense and high-quality image generation. Additionally, DES can be adopted in a plug-and-play manner and requires zero inference overhead, facilitating its deployment. Extensive experiments on diverse attack types, including black-box and white-box scenarios, demonstrate DES's state-of-the-art performance in both defense capability and benign image generation quality. Our model is available at this https URL.
• 摘要：文本到图像的扩散模型在文本提示后表现出卓越的生成性能，但存在从不安全提示生成不适合工作 (NSFW) 内容的风险。现有方法（例如提示过滤或概念反学习）无法在保持良好图像质量的同时抵御对抗性攻击。在本文中，我们提出了一种称为扭曲嵌入空间 (DES) 的新方法，这是一种基于文本编码器的防御机制，可通过创新的嵌入空间控制有效解决这些问题。DES 将使用不安全提示从文本编码器中提取的不安全嵌入转换为精心计算的安全嵌入区域，以防止生成不安全内容，同时重现原始的安全嵌入。 DES 还通过将使用提示“裸体”提取的裸体嵌入与中性嵌入对齐来中和裸体嵌入，以增强对对抗性攻击的鲁棒性。这些方法既确保了强大的防御能力，又确保了高质量的图像生成。此外，DES 可以即插即用的方式采用，并且不需要任何推理开销，从而方便其部署。对各种攻击类型（包括黑盒和白盒场景）的大量实验证明了 DES 在防御能力和良性图像生成质量方面的最先进性能。我们的模型可在此 https URL 上找到。

Title: RLS3: RL-Based Synthetic Sample Selection to Enhance Spatial Reasoning in Vision-Language Models for Indoor Autonomous Perception

• Authors: Joshua R. Waite, Md. Zahid Hasan, Qisai Liu, Zhanhong Jiang, Chinmay Hegde, Soumik Sarkar
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18880
• Pdf URL: https://arxiv.org/pdf/2501.18880
• Copy Paste: [[2501.18880]] RLS3: RL-Based Synthetic Sample Selection to Enhance Spatial Reasoning in Vision-Language Models for Indoor Autonomous Perception(https://arxiv.org/abs/2501.18880)
• Keywords: generation
• Abstract: Vision-language model (VLM) fine-tuning for application-specific visual grounding based on natural language instructions has become one of the most popular approaches for learning-enabled autonomous systems. However, such fine-tuning relies heavily on high-quality datasets to achieve successful performance in various downstream tasks. Additionally, VLMs often encounter limitations due to insufficient and imbalanced fine-tuning data. To address these issues, we propose a new generalizable framework to improve VLM fine-tuning by integrating it with a reinforcement learning (RL) agent. Our method utilizes the RL agent to manipulate objects within an indoor setting to create synthetic data for fine-tuning to address certain vulnerabilities of the VLM. Specifically, we use the performance of the VLM to provide feedback to the RL agent to generate informative data that efficiently fine-tune the VLM over the targeted task (e.g. spatial reasoning). The key contribution of this work is developing a framework where the RL agent serves as an informative data sampling tool and assists the VLM in order to enhance performance and address task-specific vulnerabilities. By targeting the data sampling process to address the weaknesses of the VLM, we can effectively train a more context-aware model. In addition, generating synthetic data allows us to have precise control over each scene and generate granular ground truth captions. Our results show that the proposed data generation approach improves the spatial reasoning performance of VLMs, which demonstrates the benefits of using RL-guided data generation in vision-language tasks.
• 摘要：基于自然语言指令的视觉语言模型 (VLM) 微调，用于特定应用的视觉基础，已成为学习型自主系统最流行的方法之一。然而，这种微调在很大程度上依赖于高质量的数据集，才能在各种下游任务中取得成功。此外，由于微调数据不足和不平衡，VLM 经常会遇到限制。为了解决这些问题，我们提出了一个新的可通用框架，通过将 VLM 微调与强化学习 (RL) 代理相结合来改进 VLM 微调。我们的方法利用 RL 代理在室内环境中操纵物体，以创建合成数据进行微调，以解决 VLM 的某些漏洞。具体来说，我们使用 VLM 的性能向 RL 代理提供反馈，以生成信息数据，从而有效地针对目标任务（例如空间推理）微调 VLM。这项工作的主要贡献是开发一个框架，其中 RL 代理充当信息数据采样工具并协助 VLM 以提高性能并解决特定于任务的漏洞。通过针对数据采样过程来解决 VLM 的弱点，我们可以有效地训练更具情境感知能力的模型。此外，生成合成数据使我们能够精确控制每个场景并生成细粒度的地面真实字幕。我们的结果表明，所提出的数据生成方法提高了 VLM 的空间推理性能，这证明了在视觉语言任务中使用 RL 引导的数据生成的好处。

Title: GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling

• Authors: Pinxin Liu, Luchuan Song, Junhua Huang, Chenliang Xu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.18898
• Pdf URL: https://arxiv.org/pdf/2501.18898
• Copy Paste: [[2501.18898]] GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling(https://arxiv.org/abs/2501.18898)
• Keywords: generation
• Abstract: Controlling human gestures based on speech signals presents a significant challenge in computer vision. While existing works did preliminary studies of generating holistic co-speech gesture from speech, the spatial interaction of each body region during the speech remains barely explored. This leads to wield body part interactions given the speech signal. Furthermore, the slow generation speed limits the construction of real-world digital avatars. To resolve these problems, we propose \textbf{GestureLSM}, a Latent Shortcut based approach for Co-Speech Gesture Generation with spatial-temporal modeling. We tokenize various body regions and explicitly model their interactions with spatial and temporal attention. To achieve real-time gesture generations, we exam the denoising patterns and design an effective time distribution to speed up sampling while improve the generation quality for shortcut model. Extensive quantitative and qualitative experiments demonstrate the effectiveness of GestureLSM, showcasing its potential for various applications in the development of digital humans and embodied agents. Project Page: this https URL
• 摘要：基于语音信号控制人类手势是计算机视觉领域的一大挑战。虽然现有研究对从语音生成整体语音手势进行了初步研究，但语音过程中每个身体部位的空间交互仍未得到深入探索。这导致在给定语音信号的情况下无法进行身体部位交互。此外，缓慢的生成速度限制了现实世界数字化身的构建。为了解决这些问题，我们提出了 \textbf{GestureLSM}，这是一种基于潜在快捷方式的语音手势生成方法，具有时空建模。我们对各个身体部位进行标记，并使用空间和时间注意力明确建模它们的相互作用。为了实现实时手势生成，我们检查了去噪模式并设计了有效的时间分布以加快采样速度，同时提高了快捷方式模型的生成质量。大量的定量和定性实验证明了 GestureLSM 的有效性，展示了其在数字人类和具身代理开发中各种应用的潜力。项目页面：此 https URL

Title: Adaptive Prompt: Unlocking the Power of Visual Prompt Tuning

• Authors: Minh Le, Anh Nguyen, Huy Nguyen, Chau Nguyen, Nhat Ho
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18936
• Pdf URL: https://arxiv.org/pdf/2501.18936
• Copy Paste: [[2501.18936]] Adaptive Prompt: Unlocking the Power of Visual Prompt Tuning(https://arxiv.org/abs/2501.18936)
• Keywords: generation
• Abstract: Visual Prompt Tuning (VPT) has recently emerged as a powerful method for adapting pre-trained vision models to downstream tasks. By introducing learnable prompt tokens as task-specific instructions, VPT effectively guides pre-trained transformer models with minimal overhead. Despite its empirical success, a comprehensive theoretical understanding of VPT remains an active area of research. Building on recent insights into the connection between mixture of experts and prompt-based approaches, we identify a key limitation in VPT: the restricted functional expressiveness in prompt formulation. To address this limitation, we propose Visual Adaptive Prompt Tuning (VAPT), a new generation of prompts that redefines prompts as adaptive functions of the input. Our theoretical analysis shows that this simple yet intuitive approach achieves optimal sample efficiency. Empirical results on VTAB-1K and FGVC further demonstrate VAPT's effectiveness, with performance gains of 7.34% and 1.04% over fully fine-tuning baselines, respectively. Notably, VAPT also surpasses VPT by a substantial margin while using fewer parameters. These results highlight both the effectiveness and efficiency of our method and pave the way for future research to explore the potential of adaptive prompts.
• 摘要：视觉提示调优 (VPT) 最近成为一种将预训练视觉模型适应下游任务的强大方法。通过引入可学习的提示标记作为任务特定指令，VPT 可以以最小的开销有效地指导预训练的 Transformer 模型。尽管在实证上取得了成功，但对 VPT 的全面理论理解仍然是一个活跃的研究领域。基于最近对专家混合和基于提示的方法之间联系的见解，我们发现了 VPT 的一个关键限制：提示制定中的功能表达能力受限。为了解决这一限制，我们提出了视觉自适应提示调优 (VAPT)，这是一种新一代提示，它将提示重新定义为输入的自适应函数。我们的理论分析表明，这种简单而直观的方法实现了最佳的样本效率。VTAB-1K 和 FGVC 上的实证结果进一步证明了 VAPT 的有效性，与完全微调基线相比，性能分别提高了 7.34% 和 1.04%。值得注意的是，VAPT 在使用更少参数的情况下也大幅超越了 VPT。这些结果突出了我们方法的有效性和效率，并为未来探索自适应提示的潜力的研究铺平了道路。

Title: TV-Dialogue: Crafting Theme-Aware Video Dialogues with Immersive Interaction

• Authors: Sai Wang, Fan Ma, Xinyi Li, Hehe Fan, Yu Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18940
• Pdf URL: https://arxiv.org/pdf/2501.18940
• Copy Paste: [[2501.18940]] TV-Dialogue: Crafting Theme-Aware Video Dialogues with Immersive Interaction(https://arxiv.org/abs/2501.18940)
• Keywords: generation
• Abstract: Recent advancements in LLMs have accelerated the development of dialogue generation across text and images, yet video-based dialogue generation remains underexplored and presents unique challenges. In this paper, we introduce Theme-aware Video Dialogue Crafting (TVDC), a novel task aimed at generating new dialogues that align with video content and adhere to user-specified themes. We propose TV-Dialogue, a novel multi-modal agent framework that ensures both theme alignment (i.e., the dialogue revolves around the theme) and visual consistency (i.e., the dialogue matches the emotions and behaviors of characters in the video) by enabling real-time immersive interactions among video characters, thereby accurately understanding the video content and generating new dialogue that aligns with the given themes. To assess the generated dialogues, we present a multi-granularity evaluation benchmark with high accuracy, interpretability and reliability, demonstrating the effectiveness of TV-Dialogue on self-collected dataset over directly using existing LLMs. Extensive experiments reveal that TV-Dialogue can generate dialogues for videos of any length and any theme in a zero-shot manner without training. Our findings underscore the potential of TV-Dialogue for various applications, such as video re-creation, film dubbing and its use in downstream multimodal tasks.
• 摘要：LLM 的最新进展加速了跨文本和图像对话生成的发展，但基于视频的对话生成仍未得到充分探索并带来了独特的挑战。在本文中，我们介绍了主题感知视频对话制作 (TVDC)，这是一项新颖的任务，旨在生成与视频内容一致并遵循用户指定主题的新对话。我们提出了 TV-Dialogue，这是一种新颖的多模态代理框架，通过实现视频角色之间的实时沉浸式交互，确保主题一致（即对话围绕主题展开）和视觉一致性（即对话与视频中角色的情绪和行为相匹配），从而准确理解视频内容并生成与给定主题一致的新对话。为了评估生成的对话，我们提出了一个多粒度评估基准，该基准具有高精度、可解释性和可靠性，证明了 TV-Dialogue 在自收集数据集上的有效性，优于直接使用现有的 LLM。大量实验表明，TV-Dialogue 无需训练即可以零样本方式为任意长度和任意主题的视频生成对话。我们的研究结果强调了 TV-Dialogue 在各种应用中的潜力，例如视频重制、电影配音及其在下游多模态任务中的应用。

Title: Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them

• Authors: Anh Bui, Trang Vu, Long Vuong, Trung Le, Paul Montague, Tamas Abraham, Junae Kim, Dinh Phung
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18950
• Pdf URL: https://arxiv.org/pdf/2501.18950
• Copy Paste: [[2501.18950]] Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them(https://arxiv.org/abs/2501.18950)
• Keywords: generation
• Abstract: Concept erasure has emerged as a promising technique for mitigating the risk of harmful content generation in diffusion models by selectively unlearning undesirable concepts. The common principle of previous works to remove a specific concept is to map it to a fixed generic concept, such as a neutral concept or just an empty text prompt. In this paper, we demonstrate that this fixed-target strategy is suboptimal, as it fails to account for the impact of erasing one concept on the others. To address this limitation, we model the concept space as a graph and empirically analyze the effects of erasing one concept on the remaining concepts. Our analysis uncovers intriguing geometric properties of the concept space, where the influence of erasing a concept is confined to a local region. Building on this insight, we propose the Adaptive Guided Erasure (AGE) method, which \emph{dynamically} selects optimal target concepts tailored to each undesirable concept, minimizing unintended side effects. Experimental results show that AGE significantly outperforms state-of-the-art erasure methods on preserving unrelated concepts while maintaining effective erasure performance. Our code is published at {this https URL}.
• 摘要：概念擦除已成为一种有前途的技术，通过选择性地忘记不良概念来减轻传播模型中有害内容生成的风险。先前研究删除特定概念的共同原则是将其映射到固定的通用概念，例如中性概念或空文本提示。在本文中，我们证明这种固定目标策略不是最优的，因为它无法解释擦除一个概念对其他概念的影响。为了解决这一限制，我们将概念空间建模为图形，并实证分析擦除一个概念对剩余概念的影响。我们的分析揭示了概念空间有趣的几何特性，其中擦除概念的影响仅限于局部区域。基于这一见解，我们提出了自适应引导擦除 (AGE) 方法，该方法 \emph{动态} 选择针对每个不良概念的最佳目标概念，从而最大限度地减少意外副作用。实验结果表明，AGE 在保留不相关概念方面的表现明显优于最先进的擦除方法，同时保持了有效的擦除性能。我们的代码发布在{此 https URL}。

Title: LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models

• Authors: Shenghao Fu, Qize Yang, Qijie Mo, Junkai Yan, Xihan Wei, Jingke Meng, Xiaohua Xie, Wei-Shi Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18954
• Pdf URL: https://arxiv.org/pdf/2501.18954
• Copy Paste: [[2501.18954]] LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models(https://arxiv.org/abs/2501.18954)
• Keywords: generation
• Abstract: Recent open-vocabulary detectors achieve promising performance with abundant region-level annotated data. In this work, we show that an open-vocabulary detector co-training with a large language model by generating image-level detailed captions for each image can further improve performance. To achieve the goal, we first collect a dataset, GroundingCap-1M, wherein each image is accompanied by associated grounding labels and an image-level detailed caption. With this dataset, we finetune an open-vocabulary detector with training objectives including a standard grounding loss and a caption generation loss. We take advantage of a large language model to generate both region-level short captions for each region of interest and image-level long captions for the whole image. Under the supervision of the large language model, the resulting detector, LLMDet, outperforms the baseline by a clear margin, enjoying superior open-vocabulary ability. Further, we show that the improved LLMDet can in turn build a stronger large multi-modal model, achieving mutual benefits. The code, model, and dataset is available at this https URL.
• 摘要：最近的开放词汇检测器凭借丰富的区域级注释数据取得了令人鼓舞的性能。在这项工作中，我们表明，通过为每幅图像生成图像级详细字幕，开放词汇检测器与大型语言模型共同训练可以进一步提高性能。为了实现这一目标，我们首先收集一个数据集 GroundingCap-1M，其中每幅图像都附有相关的基础标签和图像级详细字幕。利用这个数据集，我们对开放词汇检测器进行了微调，训练目标包括标准基础损失和字幕生成损失。我们利用大型语言模型为每个感兴趣的区域生成区域级短字幕，为整个图像生成图像级长字幕。在大型语言模型的监督下，生成的检测器 LLMDet 的表现明显优于基线，具有卓越的开放词汇能力。此外，我们表明改进的 LLMDet 可以反过来构建更强大的大型多模态模型，实现互利共赢。代码、模型和数据集可在此 https URL 上获得。

Title: Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Boostrapping

• Authors: Pu Yang, Yunzhen Feng, Ziyuan Chen, Yuhang Wu, Zhuoyuan Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18962
• Pdf URL: https://arxiv.org/pdf/2501.18962
• Copy Paste: [[2501.18962]] Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Boostrapping(https://arxiv.org/abs/2501.18962)
• Keywords: generation
• Abstract: Modern foundation models often undergo iterative ``bootstrapping'' in their post-training phase: a model generates synthetic data, an external verifier filters out low-quality samples, and the high-quality subset is used for further fine-tuning. Over multiple iterations, the model's performance improves--raising a crucial question: how should the total budget on generation and training be allocated across iterations to maximize final performance? In this work, we develop a theoretical framework to analyze budget allocation strategies. Specifically, we show that constant policies fail to converge with high probability, while increasing policies--particularly exponential growth policies--exhibit significant theoretical advantages. Experiments on image denoising with diffusion probabilistic models and math reasoning with large language models show that both exponential and polynomial growth policies consistently outperform constant policies, with exponential policies often providing more stable performance.
• 摘要：现代基础模型通常在训练后阶段经历迭代“引导”：模型生成合成数据，外部验证器过滤掉低质量样本，并使用高质量子集进行进一步微调。经过多次迭代，模型的性能得到提高——这引出了一个关键问题：应该如何在迭代中分配生成和训练的总预算以最大化最终性能？在这项工作中，我们开发了一个理论框架来分析预算分配策略。具体而言，我们表明恒定策略无法以高概率收敛，而增加策略——尤其是指数增长策略——表现出显着的理论优势。使用扩散概率模型进行图像去噪和使用大型语言模型进行数学推理的实验表明，指数和多项式增长策略都始终优于恒定策略，而指数策略通常提供更稳定的性能。

Title: BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics

• Authors: Yuxuan Liu, Jingmin Sun, Hayden Schaeffer
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2501.18972
• Pdf URL: https://arxiv.org/pdf/2501.18972
• Copy Paste: [[2501.18972]] BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics(https://arxiv.org/abs/2501.18972)
• Keywords: generation
• Abstract: We introduce BCAT, a PDE foundation model designed for autoregressive prediction of solutions to two dimensional fluid dynamics problems. Our approach uses a block causal transformer architecture to model next frame predictions, leveraging previous frames as contextual priors rather than relying solely on sub-frames or pixel-based inputs commonly used in image generation methods. This block causal framework more effectively captures the spatial dependencies inherent in nonlinear spatiotemporal dynamics and physical phenomena. In an ablation study, next frame prediction demonstrated a 2.9x accuracy improvement over next token prediction. BCAT is trained on a diverse range of fluid dynamics datasets, including incompressible and compressible Navier-Stokes equations across various geometries and parameter regimes, as well as the shallow-water equations. The model's performance was evaluated on 6 distinct downstream prediction tasks and tested on about 8K trajectories to measure robustness on a variety of fluid dynamics simulations. BCAT achieved an average relative error of 1.92% across all evaluation tasks, outperforming prior approaches on standard benchmarks.
• 摘要：我们引入了 BCAT，这是一种 PDE 基础模型，旨在对二维流体动力学问题的解决方案进行自回归预测。我们的方法使用块因果变换器架构来模拟下一帧预测，利用前一帧作为上下文先验，而不是仅仅依赖于图像生成方法中常用的子帧或基于像素的输入。这个块因果框架更有效地捕捉了非线性时空动力学和物理现象中固有的空间依赖性。在一项消融研究中，下一帧预测比下一个标记预测的准确率提高了 2.9 倍。BCAT 在各种流体动力学数据集上进行训练，包括跨各种几何和参数范围的不可压缩和可压缩 Navier-Stokes 方程，以及浅水方程。该模型的性能在 6 个不同的下游预测任务上进行了评估，并在大约 8K 条轨迹上进行了测试，以测量在各种流体动力学模拟中的稳健性。 BCAT 在所有评估任务中实现了 1.92% 的平均相对误差，在标准基准上优于之前的方法。

Title: OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation

• Authors: Yuchen Lin, Chenguo Lin, Jianjin Xu, Yadong Mu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18982
• Pdf URL: https://arxiv.org/pdf/2501.18982
• Copy Paste: [[2501.18982]] OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation(https://arxiv.org/abs/2501.18982)
• Keywords: generation
• Abstract: Recently, significant advancements have been made in the reconstruction and generation of 3D assets, including static cases and those with physical interactions. To recover the physical properties of 3D assets, existing methods typically assume that all materials belong to a specific predefined category (e.g., elasticity). However, such assumptions ignore the complex composition of multiple heterogeneous objects in real scenarios and tend to render less physically plausible animation given a wider range of objects. We propose OmniPhysGS for synthesizing a physics-based 3D dynamic scene composed of more general objects. A key design of OmniPhysGS is treating each 3D asset as a collection of constitutive 3D Gaussians. For each Gaussian, its physical material is represented by an ensemble of 12 physical domain-expert sub-models (rubber, metal, honey, water, etc.), which greatly enhances the flexibility of the proposed model. In the implementation, we define a scene by user-specified prompts and supervise the estimation of material weighting factors via a pretrained video diffusion model. Comprehensive experiments demonstrate that OmniPhysGS achieves more general and realistic physical dynamics across a broader spectrum of materials, including elastic, viscoelastic, plastic, and fluid substances, as well as interactions between different materials. Our method surpasses existing methods by approximately 3% to 16% in metrics of visual quality and text alignment.
• 摘要：最近，在 3D 资产的重建和生成方面取得了重大进展，包括静态情况和具有物理相互作用的情况。为了恢复 3D 资产的物理属性，现有方法通常假设所有材料都属于特定的预定义类别（例如弹性）。然而，这种假设忽略了现实场景中多个异构对象的复杂组成，并且在给定更广泛对象的情况下往往会呈现不太符合物理原理的动画。我们提出了 OmniPhysGS 来合成由更一般对象组成的基于物理的 3D 动态场景。OmniPhysGS 的一个关键设计是将每个 3D 资产视为组成 3D 高斯的集合。对于每个高斯，其物理材料由 12 个物理领域专家子模型（橡胶、金属、蜂蜜、水等）的集合表示，这大大增强了所提模型的灵活性。在实现过程中，我们根据用户指定的提示定义场景，并通过预训练的视频扩散模型监督材料权重因子的估计。综合实验表明，OmniPhysGS 可以在更广泛的材料范围内实现更通用、更逼真的物理动力学，包括弹性、粘弹性、塑性和流体物质，以及不同材料之间的相互作用。在视觉质量和文本对齐指标方面，我们的方法比现有方法高出约 3% 至 16%。

Title: Visual Autoregressive Modeling for Image Super-Resolution

• Authors: Yunpeng Qu, Kun Yuan, Jinhua Hao, Kai Zhao, Qizhi Xie, Ming Sun, Chao Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18993
• Pdf URL: https://arxiv.org/pdf/2501.18993
• Copy Paste: [[2501.18993]] Visual Autoregressive Modeling for Image Super-Resolution(https://arxiv.org/abs/2501.18993)
• Keywords: super-resolution, generation, generative
• Abstract: Image Super-Resolution (ISR) has seen significant progress with the introduction of remarkable generative models. However, challenges such as the trade-off issues between fidelity and realism, as well as computational complexity, have also posed limitations on their application. Building upon the tremendous success of autoregressive models in the language domain, we propose \textbf{VARSR}, a novel visual autoregressive modeling for ISR framework with the form of next-scale prediction. To effectively integrate and preserve semantic information in low-resolution images, we propose using prefix tokens to incorporate the condition. Scale-aligned Rotary Positional Encodings are introduced to capture spatial structures and the diffusion refiner is utilized for modeling quantization residual loss to achieve pixel-level fidelity. Image-based Classifier-free Guidance is proposed to guide the generation of more realistic images. Furthermore, we collect large-scale data and design a training process to obtain robust generative priors. Quantitative and qualitative results show that VARSR is capable of generating high-fidelity and high-realism images with more efficiency than diffusion-based methods. Our codes will be released at this https URL.
• 摘要：随着卓越的生成模型的引入，图像超分辨率 (ISR) 取得了重大进展。然而，诸如保真度和真实度之间的权衡问题以及计算复杂性等挑战也对其应用造成了限制。基于语言领域自回归模型的巨大成功，我们提出了 \textbf{VARSR}，一种具有下一尺度预测形式的 ISR 框架的新型视觉自回归建模。为了有效地整合和保留低分辨率图像中的语义信息，我们建议使用前缀标记来合并该条件。引入尺度对齐的旋转位置编码来捕获空间结构，并利用扩散细化器对量化残差损失进行建模以实现像素级保真度。提出了基于图像的无分类器指导来指导生成更逼真的图像。此外，我们收集大规模数据并设计训练过程以获得稳健的生成先验。定量和定性结果表明，VARSR 能够比基于扩散的方法更高效地生成高保真度和高真实感的图像。我们的代码将在此 https URL 上发布。

Title: Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities

• Authors: Arjun Krishna, Erick Galinkin, Leon Derczynski, Jeffrey Martin
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.19012
• Pdf URL: https://arxiv.org/pdf/2501.19012
• Copy Paste: [[2501.19012]] Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities(https://arxiv.org/abs/2501.19012)
• Keywords: generation
• Abstract: Large Language Models (LLMs) have become an essential tool in the programmer's toolkit, but their tendency to hallucinate code can be used by malicious actors to introduce vulnerabilities to broad swathes of the software supply chain. In this work, we analyze package hallucination behaviour in LLMs across popular programming languages examining both existing package references and fictional dependencies. By analyzing this package hallucination behaviour we find potential attacks and suggest defensive strategies to defend against these attacks. We discover that package hallucination rate is predicated not only on model choice, but also programming language, model size, and specificity of the coding task request. The Pareto optimality boundary between code generation performance and package hallucination is sparsely populated, suggesting that coding models are not being optimized for secure code. Additionally, we find an inverse correlation between package hallucination rate and the HumanEval coding benchmark, offering a heuristic for evaluating the propensity of a model to hallucinate packages. Our metrics, findings and analyses provide a base for future models, securing AI-assisted software development workflows against package supply chain attacks.
• 摘要：大型语言模型 (LLM) 已成为程序员工具包中必不可少的工具，但它们产生幻觉代码的倾向可能被恶意行为者利用，从而给软件供应链的广泛领域带来漏洞。在这项工作中，我们分析了流行编程语言中 LLM 中的包幻觉行为，检查了现有的包引用和虚构的依赖关系。通过分析这种包幻觉行为，我们发现了潜在的攻击并提出了防御策略来防御这些攻击。我们发现包幻觉率不仅取决于模型选择，还取决于编程语言、模型大小和编码任务请求的特殊性。代码生成性能和包幻觉之间的帕累托最优边界稀疏分布，这表明编码模型并未针对安全代码进行优化。此外，我们发现包幻觉率与 HumanEval 编码基准之间存在反比相关性，这为评估模型产生幻觉包的倾向提供了一种启发式方法。我们的指标、发现和分析为未来模型提供了基础，确保人工智能辅助软件开发工作流程免受包裹供应链攻击。

Title: XRF V2: A Dataset for Action Summarization with Wi-Fi Signals, and IMUs in Phones, Watches, Earbuds, and Glasses

• Authors: Bo Lan, Pei Li, Jiaxi Yin, Yunpeng Song, Ge Wang, Han Ding, Jinsong Han, Fei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19034
• Pdf URL: https://arxiv.org/pdf/2501.19034
• Copy Paste: [[2501.19034]] XRF V2: A Dataset for Action Summarization with Wi-Fi Signals, and IMUs in Phones, Watches, Earbuds, and Glasses(https://arxiv.org/abs/2501.19034)
• Keywords: generation
• Abstract: Human Action Recognition (HAR) plays a crucial role in applications such as health monitoring, smart home automation, and human-computer interaction. While HAR has been extensively studied, action summarization, which involves identifying and summarizing continuous actions, remains an emerging task. This paper introduces the novel XRF V2 dataset, designed for indoor daily activity Temporal Action Localization (TAL) and action summarization. XRF V2 integrates multimodal data from Wi-Fi signals, IMU sensors (smartphones, smartwatches, headphones, and smart glasses), and synchronized video recordings, offering a diverse collection of indoor activities from 16 volunteers across three distinct environments. To tackle TAL and action summarization, we propose the XRFMamba neural network, which excels at capturing long-term dependencies in untrimmed sensory sequences and outperforms state-of-the-art methods, such as ActionFormer and WiFiTAD. We envision XRF V2 as a valuable resource for advancing research in human action localization, action forecasting, pose estimation, multimodal foundation models pre-training, synthetic data generation, and more.
• 摘要：人体动作识别 (HAR) 在健康监测、智能家居自动化和人机交互等应用中起着至关重要的作用。虽然 HAR 已经得到了广泛的研究，但动作总结（涉及识别和总结连续动作）仍然是一项新兴任务。本文介绍了新颖的 XRF V2 数据集，该数据集专为室内日常活动时间动作定位 (TAL) 和动作总结而设计。XRF V2 集成了来自 Wi-Fi 信号、IMU 传感器（智能手机、智能手表、耳机和智能眼镜）和同步视频录制的多模态数据，提供了来自三个不同环境中的 16 名志愿者的丰富室内活动集合。为了解决 TAL 和动作总结问题，我们提出了 XRFMamba 神经网络，它擅长捕捉未修剪的传感序列中的长期依赖关系，并且优于最先进的方法，例如 ActionFormer 和 WiFiTAD。我们将 XRF V2 视为推动人类动作定位、动作预测、姿势估计、多模态基础模型预训练、合成数据生成等领域研究的宝贵资源。

Title: Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models

• Authors: Ruiyu Wang, Yu Yuan, Shizhao Sun, Jiang Bian
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19054
• Pdf URL: https://arxiv.org/pdf/2501.19054
• Copy Paste: [[2501.19054]] Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models(https://arxiv.org/abs/2501.19054)
• Keywords: generation
• Abstract: Creating Computer-Aided Design (CAD) models requires significant expertise and effort. Text-to-CAD, which converts textual descriptions into CAD parametric sequences, is crucial in streamlining this process. Recent studies have utilized ground-truth parametric sequences, known as sequential signals, as supervision to achieve this goal. However, CAD models are inherently multimodal, comprising parametric sequences and corresponding rendered visual objects. Besides,the rendering process from parametric sequences to visual objects is many-to-one. Therefore, both sequential and visual signals are critical for effective training. In this work, we introduce CADFusion, a framework that uses Large Language Models (LLMs) as the backbone and alternates between two training stages: the sequential learning (SL) stage and the visual feedback (VF) stage. In the SL stage, we train LLMs using ground-truth parametric sequences, enabling the generation of logically coherent parametric sequences. In the VF stage, we reward parametric sequences that render into visually preferred objects and penalize those that do not, allowing LLMs to learn how rendered visual objects are perceived and evaluated. These two stages alternate throughout the training, ensuring balanced learning and preserving benefits of both signals. Experiments demonstrate that CADFusion significantly improves performance, both qualitatively and quantitatively.
• 摘要：创建计算机辅助设计 (CAD) 模型需要大量的专业知识和努力。文本转 CAD 可将文本描述转换为 CAD 参数序列，这对于简化此过程至关重要。最近的研究利用地面实况参数序列（称为序列信号）作为监督来实现这一目标。然而，CAD 模型本质上是多模态的，包括参数序列和相应的渲染视觉对象。此外，从参数序列到视觉对象的渲染过程是多对一的。因此，序列和视觉信号对于有效训练都至关重要。在这项工作中，我们引入了 CADFusion，这是一个使用大型语言模型 (LLM) 作为骨干并在两个训练阶段之间交替的框架：顺序学习 (SL) 阶段和视觉反馈 (VF) 阶段。在 SL 阶段，我们使用地面实况参数序列训练 LLM，从而生成逻辑上连贯的参数序列。在 VF 阶段，我们奖励那些渲染成视觉偏好对象的参数序列，惩罚那些没有渲染成视觉偏好对象的参数序列，从而让 LLM 学习如何感知和评估渲染的视觉对象。这两个阶段在整个训练过程中交替进行，确保平衡学习并保留两种信号的优势。实验表明，CADFusion 显著提高了性能，无论是质量还是数量。

Title: Concept Steerers: Leveraging K-Sparse Autoencoders for Controllable Generations

• Authors: Dahye Kim, Deepti Ghadiyaram
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19066
• Pdf URL: https://arxiv.org/pdf/2501.19066
• Copy Paste: [[2501.19066]] Concept Steerers: Leveraging K-Sparse Autoencoders for Controllable Generations(https://arxiv.org/abs/2501.19066)
• Keywords: generation, generative
• Abstract: Despite the remarkable progress in text-to-image generative models, they are prone to adversarial attacks and inadvertently generate unsafe, unethical content. Existing approaches often rely on fine-tuning models to remove specific concepts, which is computationally expensive, lack scalability, and/or compromise generation quality. In this work, we propose a novel framework leveraging k-sparse autoencoders (k-SAEs) to enable efficient and interpretable concept manipulation in diffusion models. Specifically, we first identify interpretable monosemantic concepts in the latent space of text embeddings and leverage them to precisely steer the generation away or towards a given concept (e.g., nudity) or to introduce a new concept (e.g., photographic style). Through extensive experiments, we demonstrate that our approach is very simple, requires no retraining of the base model nor LoRA adapters, does not compromise the generation quality, and is robust to adversarial prompt manipulations. Our method yields an improvement of $\mathbf{20.01\%}$ in unsafe concept removal, is effective in style manipulation, and is $\mathbf{\sim5}$x faster than current state-of-the-art.
• 摘要：尽管文本到图像的生成模型取得了显著进展，但它们很容易受到对抗性攻击，并无意中生成不安全、不道德的内容。现有方法通常依赖于微调模型来删除特定概念，这在计算上很昂贵、缺乏可扩展性和/或损害生成质量。在这项工作中，我们提出了一个利用 k 稀疏自动编码器 (k-SAE) 的新框架，以在扩散模型中实现高效且可解释的概念操作。具体来说，我们首先在文本嵌入的潜在空间中识别可解释的单义概念，并利用它们精确地引导生成远离或朝向给定概念（例如裸体）或引入新概念（例如摄影风格）。通过大量实验，我们证明了我们的方法非常简单，不需要重新训练基础模型或 LoRA 适配器，不会损害生成质量，并且对对抗性提示操作具有鲁棒性。我们的方法在删除不安全概念方面取得了$\mathbf{20.01\%}$的改进，在风格操纵方面也很有效，并且比目前最先进的方法快$\mathbf{\sim5}$x。

Title: MotionPCM: Real-Time Motion Synthesis with Phased Consistency Model

• Authors: Lei Jiang, Ye Wei, Hao Ni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19083
• Pdf URL: https://arxiv.org/pdf/2501.19083
• Copy Paste: [[2501.19083]] MotionPCM: Real-Time Motion Synthesis with Phased Consistency Model(https://arxiv.org/abs/2501.19083)
• Keywords: generative
• Abstract: Diffusion models have become a popular choice for human motion synthesis due to their powerful generative capabilities. However, their high computational complexity and large sampling steps pose challenges for real-time applications. Fortunately, the Consistency Model (CM) provides a solution to greatly reduce the number of sampling steps from hundreds to a few, typically fewer than four, significantly accelerating the synthesis of diffusion models. However, its application to text-conditioned human motion synthesis in latent space remains challenging. In this paper, we introduce \textbf{MotionPCM}, a phased consistency model-based approach designed to improve the quality and efficiency of real-time motion synthesis in latent space.
• 摘要：扩散模型因其强大的生成能力而成为人体运动合成的热门选择。然而，它们的高计算复杂度和较大的采样步骤对实时应用构成了挑战。幸运的是，一致性模型 (CM) 提供了一种解决方案，可以将采样步骤的数量从数百个大大减少到几个，通常少于四个，从而显著加快了扩散模型的合成。然而，将其应用于潜在空间中的文本条件人体运动合成仍然具有挑战性。在本文中，我们介绍了 \textbf{MotionPCM}，这是一种基于分阶段一致性模型的方法，旨在提高潜在空间中实时运动合成的质量和效率。

Title: Ambient Denoising Diffusion Generative Adversarial Networks for Establishing Stochastic Object Models from Noisy Image Data

• Authors: Xichen Xu, Wentao Chen, Weimin Zhou
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.19094
• Pdf URL: https://arxiv.org/pdf/2501.19094
• Copy Paste: [[2501.19094]] Ambient Denoising Diffusion Generative Adversarial Networks for Establishing Stochastic Object Models from Noisy Image Data(https://arxiv.org/abs/2501.19094)
• Keywords: generation, generative
• Abstract: It is widely accepted that medical imaging systems should be objectively assessed via task-based image quality (IQ) measures that ideally account for all sources of randomness in the measured image data, including the variation in the ensemble of objects to be imaged. Stochastic object models (SOMs) that can randomly draw samples from the object distribution can be employed to characterize object variability. To establish realistic SOMs for task-based IQ analysis, it is desirable to employ experimental image data. However, experimental image data acquired from medical imaging systems are subject to measurement noise. Previous work investigated the ability of deep generative models (DGMs) that employ an augmented generative adversarial network (GAN), AmbientGAN, for establishing SOMs from noisy measured image data. Recently, denoising diffusion models (DDMs) have emerged as a leading DGM for image synthesis and can produce superior image quality than GANs. However, original DDMs possess a slow image-generation process because of the Gaussian assumption in the denoising steps. More recently, denoising diffusion GAN (DDGAN) was proposed to permit fast image generation while maintain high generated image quality that is comparable to the original DDMs. In this work, we propose an augmented DDGAN architecture, Ambient DDGAN (ADDGAN), for learning SOMs from noisy image data. Numerical studies that consider clinical computed tomography (CT) images and digital breast tomosynthesis (DBT) images are conducted. The ability of the proposed ADDGAN to learn realistic SOMs from noisy image data is demonstrated. It has been shown that the ADDGAN significantly outperforms the advanced AmbientGAN models for synthesizing high resolution medical images with complex textures.
• 摘要：人们普遍认为，医学成像系统应通过基于任务的图像质量 (IQ) 指标进行客观评估，理想情况下，该指标应考虑测量图像数据中的所有随机性来源，包括要成像的物体整体的变化。可以使用能够从物体分布中随机抽取样本的随机物体模型 (SOM) 来表征物体的变异性。为了建立用于基于任务的 IQ 分析的逼真的 SOM，最好使用实验图像数据。然而，从医学成像系统获取的实验图像数据容易受到测量噪声的影响。先前的研究调查了使用增强型生成对抗网络 (GAN) AmbientGAN 的深度生成模型 (DGM) 从嘈杂的测量图像数据中建立 SOM 的能力。最近，去噪扩散模型 (DDM) 已成为图像合成的领先 DGM，并且可以产生比 GAN 更好的图像质量。然而，由于去噪步骤中的高斯假设，原始 DDM 的图像生成过程很慢。最近，提出了去噪扩散 GAN (DDGAN)，以允许快速生成图像，同时保持与原始 DDM 相当的高生成图像质量。在这项工作中，我们提出了一种增强型 DDGAN 架构 Ambient DDGAN (ADDGAN)，用于从嘈杂的图像数据中学习 SOM。进行了考虑临床计算机断层扫描 (CT) 图像和数字乳房断层合成 (DBT) 图像的数值研究。证明了所提出的 ADDGAN 从嘈杂的图像数据中学习逼真的 SOM 的能力。事实证明，ADDGAN 在合成具有复杂纹理的高分辨率医学图像方面明显优于先进的 AmbientGAN 模型。

Title: A theoretical framework for overfitting in energy-based modeling

• Authors: Giovanni Catania, Aurélien Decelle, Cyril Furtlehner, Beatriz Seoane
• Subjects: cs.LG, cond-mat.dis-nn, cond-mat.stat-mech
• Abstract URL: https://arxiv.org/abs/2501.19158
• Pdf URL: https://arxiv.org/pdf/2501.19158
• Copy Paste: [[2501.19158]] A theoretical framework for overfitting in energy-based modeling(https://arxiv.org/abs/2501.19158)
• Keywords: generative
• Abstract: We investigate the impact of limited data on training pairwise energy-based models for inverse problems aimed at identifying interaction networks. Utilizing the Gaussian model as testbed, we dissect training trajectories across the eigenbasis of the coupling matrix, exploiting the independent evolution of eigenmodes and revealing that the learning timescales are tied to the spectral decomposition of the empirical covariance matrix. We see that optimal points for early stopping arise from the interplay between these timescales and the initial conditions of training. Moreover, we show that finite data corrections can be accurately modeled through asymptotic random matrix theory calculations and provide the counterpart of generalized cross-validation in the energy based model context. Our analytical framework extends to binary-variable maximum-entropy pairwise models with minimal variations. These findings offer strategies to control overfitting in discrete-variable models through empirical shrinkage corrections, improving the management of overfitting in energy-based generative models.
• 摘要：我们研究了有限数据对训练成对能量模型的影响，该模型用于识别交互网络的逆问题。利用高斯模型作为试验台，我们剖析了耦合矩阵特征基上的训练轨迹，利用了特征模式的独立演化，并揭示了学习时间尺度与经验协方差矩阵的谱分解相关。我们看到，早期停止的最佳点来自这些时间尺度与训练初始条件之间的相互作用。此外，我们表明，有限数据校正可以通过渐近随机矩阵理论计算准确建模，并在基于能量的模型环境中提供广义交叉验证的对应物。我们的分析框架扩展到具有最小变化的二元变量最大熵成对模型。这些发现提供了通过经验收缩校正来控制离散变量模型中过度拟合的策略，从而改善了基于能量的生成模型中过度拟合的管理。

Title: PSyDUCK: Training-Free Steganography for Latent Diffusion

• Authors: Georgia Channing, Aqib Mahfuz, Mark van der Wilk, Philip Torr, Fabio Pizzati, Christian Schroeder de Witt
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.19172
• Pdf URL: https://arxiv.org/pdf/2501.19172
• Copy Paste: [[2501.19172]] PSyDUCK: Training-Free Steganography for Latent Diffusion(https://arxiv.org/abs/2501.19172)
• Keywords: generative
• Abstract: Recent advances in AI-generated steganography highlight its potential for safeguarding the privacy of vulnerable democratic actors, including aid workers, journalists, and whistleblowers operating in oppressive regimes. In this work, we address current limitations and establish the foundations for large-throughput generative steganography. We introduce a novel approach that enables secure and efficient steganography within latent diffusion models. We show empirically that our methods perform well across a variety of open-source latent diffusion models, particularly in generative image and video tasks.
• 摘要：人工智能生成的隐写术的最新进展凸显了其在保护弱势民主行为者隐私方面的潜力，这些行为者包括援助工作者、记者和在压迫性政权下运作的告密者。在这项工作中，我们解决了当前的局限性并为大吞吐量生成隐写术奠定了基础。我们引入了一种新颖的方法，可以在潜在扩散模型中实现安全高效的隐写术。我们通过经验证明，我们的方法在各种开源潜在扩散模型中表现良好，特别是在生成图像和视频任务中。

Title: A Comunication Framework for Compositional Generation

• Authors: Rafael Elberg, Mircea Petrache, Denis Parra
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19182
• Pdf URL: https://arxiv.org/pdf/2501.19182
• Copy Paste: [[2501.19182]] A Comunication Framework for Compositional Generation(https://arxiv.org/abs/2501.19182)
• Keywords: generation, generative
• Abstract: Compositionality and compositional generalization--the ability to understand novel combinations of known concepts--are central characteristics of human language and are hypothesized to be essential for human cognition. In machine learning, the emergence of this property has been studied in a communication game setting, where independent agents (a sender and a receiver) converge to a shared encoding policy from a set of states to a space of discrete messages, where the receiver can correctly reconstruct the states observed by the sender using only the sender's messages. The use of communication games in generation tasks is still largely unexplored, with recent methods for compositional generation focusing mainly on the use of supervised guidance (either through class labels or text). In this work, we take the first steps to fill this gap, and we present a self-supervised generative communication game-based framework for creating compositional encodings in learned representations from pre-trained encoder-decoder models. In an Iterated Learning (IL) protocol involving a sender and a receiver, we apply alternating pressures for compression and diversity of encoded discrete messages, so that the protocol converges to an efficient but unambiguous encoding. Approximate message entropy regularization is used to favor compositional encodings. Our framework is based on rigorous justifications and proofs of defining and balancing the concepts of Eficiency, Unambiguity and Non-Holisticity in encoding. We test our method on the compositional image dataset Shapes3D, demonstrating robust performance in both reconstruction and compositionality metrics, surpassing other tested discrete message frameworks.
• 摘要：组合性和组合泛化（理解已知概念的新组合的能力）是人类语言的核心特征，并且被认为对人类认知至关重要。在机器学习中，这种特性的出现已在通信游戏环境中得到研究，其中独立代理（发送者和接收者）从一组状态收敛到离散消息空间的共享编码策略，其中接收者可以仅使用发送者的消息正确地重建发送者观察到的状态。通信游戏在生成任务中的使用仍在很大程度上尚未得到探索，最近的组合生成方法主要侧重于使用监督指导（通过类标签或文本）。在这项工作中，我们迈出了填补这一空白的第一步，并提出了一个基于自监督的生成通信游戏框架，用于在从预训练的编码器-解码器模型中学习到的表示中创建组合编码。在涉及发送者和接收者的迭代学习 (IL) 协议中，我们交替施加压力以压缩和多样化编码的离散消息，以便协议收敛到高效但无歧义的编码。近似消息熵正则化用于支持组合编码。我们的框架基于严格的论证和证明，定义和平衡编码中的效率、无歧义和非整体性概念。我们在组合图像数据集 Shapes3D 上测试了我们的方法，在重建和组合性指标方面都表现出稳健的性能，超越了其他经过测试的离散消息框架。

Title: A Variational Perspective on Generative Protein Fitness Optimization

• Authors: Lea Bogensperger, Dominik Narnhofer, Ahmed Allam, Konrad Schindler, Michael Krauthammer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19200
• Pdf URL: https://arxiv.org/pdf/2501.19200
• Copy Paste: [[2501.19200]] A Variational Perspective on Generative Protein Fitness Optimization(https://arxiv.org/abs/2501.19200)
• Keywords: generative
• Abstract: The goal of protein fitness optimization is to discover new protein variants with enhanced fitness for a given use. The vast search space and the sparsely populated fitness landscape, along with the discrete nature of protein sequences, pose significant challenges when trying to determine the gradient towards configurations with higher fitness. We introduce Variational Latent Generative Protein Optimization (VLGPO), a variational perspective on fitness optimization. Our method embeds protein sequences in a continuous latent space to enable efficient sampling from the fitness distribution and combines a (learned) flow matching prior over sequence mutations with a fitness predictor to guide optimization towards sequences with high fitness. VLGPO achieves state-of-the-art results on two different protein benchmarks of varying complexity. Moreover, the variational design with explicit prior and likelihood functions offers a flexible plug-and-play framework that can be easily customized to suit various protein design tasks.
• 摘要：蛋白质适应度优化的目标是发现针对特定用途具有增强适应度的新蛋白质变体。巨大的搜索空间和稀疏的适应度景观，以及蛋白质序列的离散性质，在尝试确定具有更高适应度的配置的梯度时带来了重大挑战。我们引入了变分潜在生成蛋白质优化 (VLGPO)，这是适应度优化的变分视角。我们的方法将蛋白质序列嵌入连续潜在空间中，以便从适应度分布中进行有效采样，并将 (学习的) 流匹配先验与序列突变相结合，并与适应度预测器相结合，以指导优化具有高适应度的序列。VLGPO 在两个不同复杂度的蛋白质基准上取得了最先进的结果。此外，具有显式先验和似然函数的变分设计提供了一个灵活的即插即用框架，可以轻松定制以适应各种蛋白质设计任务。

Title: Hourly Short Term Load Forecasting for Residential Buildings and Energy Communities

• Authors: Aleksei Kychkin, Georgios C. Chasparis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19234
• Pdf URL: https://arxiv.org/pdf/2501.19234
• Copy Paste: [[2501.19234]] Hourly Short Term Load Forecasting for Residential Buildings and Energy Communities(https://arxiv.org/abs/2501.19234)
• Keywords: generation
• Abstract: Electricity load consumption may be extremely complex in terms of profile patterns, as it depends on a wide range of human factors, and it is often correlated with several exogenous factors, such as the availability of renewable energy and the weather conditions. The first goal of this paper is to investigate the performance of a large selection of different types of forecasting models in predicting the electricity load consumption within the short time horizon of a day or few hours ahead. Such forecasts may be rather useful for the energy management of individual residential buildings or small energy communities. In particular, we introduce persistence models, standard auto-regressive-based machine learning models, and more advanced deep learning models. The second goal of this paper is to introduce two alternative modeling approaches that are simpler in structure while they take into account domain specific knowledge, as compared to the previously mentioned black-box modeling techniques. In particular, we consider the persistence-based auto-regressive model (PAR) and the seasonal persistence-based regressive model (SPR), priorly introduced by the authors. In this paper, we specifically tailor these models to accommodate the generation of hourly forecasts. The introduced models and the induced comparative analysis extend prior work of the authors which was restricted to day-ahead forecasts. We observed a 15-30% increase in the prediction accuracy of the newly introduced hourly-based forecasting models over existing approaches.
• 摘要：电力负荷消耗在剖面模式方面可能极其复杂，因为它取决于多种人为因素，并且通常与多种外生因素相关，例如可再生能源的可用性和天气条件。本文的第一个目标是研究大量不同类型的预测模型在预测一天或几小时后的短时间范围内的电力负荷消耗方面的表现。这种预测可能对单个住宅建筑或小型能源社区的能源管理非常有用。特别是，我们介绍了持久性模型、基于标准自回归的机器学习模型和更先进的深度学习模型。本文的第二个目标是介绍两种替代建模方法，与前面提到的黑箱建模技术相比，它们结构更简单，同时考虑到特定领域的知识。特别是，我们考虑了作者之前介绍的基于持久性的自回归模型 (PAR) 和基于季节性持久性的回归模型 (SPR)。在本文中，我们专门定制了这些模型以适应每小时预测的生成。引入的模型和所进行的比较分析扩展了作者之前仅限于日前预测的工作。我们观察到，与现有方法相比，新引入的每小时预测模型的预测准确率提高了 15-30%。

Title: Accelerating Diffusion Transformer via Error-Optimized Cache

• Authors: Junxiang Qiu, Shuo Wang, Jinda Lu, Lin Liu, Houcheng Jiang, Yanbin Hao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19243
• Pdf URL: https://arxiv.org/pdf/2501.19243
• Copy Paste: [[2501.19243]] Accelerating Diffusion Transformer via Error-Optimized Cache(https://arxiv.org/abs/2501.19243)
• Keywords: generation
• Abstract: Diffusion Transformer (DiT) is a crucial method for content generation. However, it needs a lot of time to sample. Many studies have attempted to use caching to reduce the time consumption of sampling. Existing caching methods accelerate generation by reusing DiT features from the previous time step and skipping calculations in the next, but they tend to locate and cache low-error modules without focusing on reducing caching-induced errors, resulting in a sharp decline in generated content quality when increasing caching intensity. To solve this problem, we propose the Error-Optimized Cache (EOC). This method introduces three key improvements: (1) Prior knowledge extraction: Extract and process the caching differences; (2) A judgment method for cache optimization: Determine whether certain caching steps need to be optimized; (3) Cache optimization: reduce caching errors. Experiments show that this algorithm significantly reduces the error accumulation caused by caching (especially over-caching). On the ImageNet dataset, without significantly increasing the computational burden, this method improves the quality of the generated images under the over-caching, rule-based, and training-based methods. Specifically, the Fréchet Inception Distance (FID) values are improved as follows: from 6.857 to 5.821, from 3.870 to 3.692 and form 3.539 to 3.451 respectively.
• 摘要：扩散变换器（DiT）是内容生成的重要方法，然而其需要大量的时间进行采样，许多研究尝试利用缓存来减少采样的时间消耗。现有的缓存方法通过重用前一时间步的 DiT 特征、在下一时间步跳过计算来加速生成，但是它们倾向于定位和缓存低错误率的模块而没有注重减少缓存导致的错误，导致在增加缓存强度时生成内容质量急剧下降。针对该问题，我们提出了错误优化缓存（EOC）。该方法引入了三个关键的改进：（1）先验知识提取：提取并处理缓存差异；（2）缓存优化的判断方法：判断某些缓存步骤是否需要优化；（3）缓存优化：减少缓存错误。实验表明，该算法明显减少了缓存（尤其是过度缓存）导致的错误积累。在ImageNet数据集上，该方法在不显著增加计算负担的情况下，提升了过缓存、基于规则和基于训练方法下生成图像的质量，具体而言，Fréchet Inception Distance (FID)值提升如下：从6.857提升到5.821，从3.870提升到3.692，从3.539提升到3.451。

Title: Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search

• Authors: Yuta Oshima, Masahiro Suzuki, Yutaka Matsuo, Hiroki Furuta
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19252
• Pdf URL: https://arxiv.org/pdf/2501.19252
• Copy Paste: [[2501.19252]] Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search(https://arxiv.org/abs/2501.19252)
• Keywords: generation
• Abstract: The remarkable progress in text-to-video diffusion models enables photorealistic generations, although the contents of the generated video often include unnatural movement or deformation, reverse playback, and motionless scenes. Recently, an alignment problem has attracted huge attention, where we steer the output of diffusion models based on some quantity on the goodness of the content. Because there is a large room for improvement of perceptual quality along the frame direction, we should address which metrics we should optimize and how we can optimize them in the video generation. In this paper, we propose diffusion latent beam search with lookahead estimator, which can select better diffusion latent to maximize a given alignment reward, at inference time. We then point out that the improvement of perceptual video quality considering the alignment to prompts requires reward calibration by weighting existing metrics. When evaluating outputs by using vision language models as a proxy of humans, many previous metrics to quantify the naturalness of video do not always correlate with evaluation and also depend on the degree of dynamic descriptions in evaluation prompts. We demonstrate that our method improves the perceptual quality based on the calibrated reward, without model parameter update, and outputs the best generation compared to greedy search and best-of-N sampling. We provide practical guidelines on which axes, among search budget, lookahead steps for reward estimate, and denoising steps, in the reverse diffusion process, we should allocate the inference-time computation.
• 摘要：文本到视频扩散模型的显著进步使得照片级逼真的生成成为可能，尽管生成的视频内容通常包括不自然的运动或变形、反向播放和静止场景。最近，对齐问题引起了极大的关注，我们根据内容优劣的某些量来控制扩散模型的输出。由于沿帧方向的感知质量有很大的改进空间，因此我们应该解决在视频生成中应该优化哪些指标以及如何优化它们。在本文中，我们提出了带有前瞻估计器的扩散潜伏波束搜索，它可以在推理时选择更好的扩散潜伏以最大化给定的对齐奖励。然后我们指出，考虑到与提示对齐的感知视频质量的改进需要通过加权现有指标来进行奖励校准。当使用视觉语言模型作为人类的代理来评估输出时，许多以前量化视频自然度的指标并不总是与评估相关，而且还取决于评估提示中的动态描述程度。我们证明了我们的方法基于校准的奖励提高了感知质量，无需更新模型参数，并且与贪婪搜索和 N 中最佳采样相比，输出了最佳生成。我们提供了实用指南，说明在反向扩散过程中，在搜索预算、奖励估计的前瞻步骤和去噪步骤中，我们应该分配推理时间计算的哪些轴。

Title: Application of Generative Adversarial Network (GAN) for Synthetic Training Data Creation to improve performance of ANN Classifier for extracting Built-Up pixels from Landsat Satellite Imagery

• Authors: Amritendu Mukherjee, Dipanwita Sinha Mukherjee, Parthasarathy Ramachandran
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19283
• Pdf URL: https://arxiv.org/pdf/2501.19283
• Copy Paste: [[2501.19283]] Application of Generative Adversarial Network (GAN) for Synthetic Training Data Creation to improve performance of ANN Classifier for extracting Built-Up pixels from Landsat Satellite Imagery(https://arxiv.org/abs/2501.19283)
• Keywords: generative
• Abstract: Training a neural network for pixel based classification task using low resolution Landsat images is difficult as the size of the training data is usually small due to less number of available pixels that represent a single class without any mixing with other classes. Due to this scarcity of training data, neural network may not be able to attain expected level of accuracy. This limitation could be overcome using a generative network that aims to generate synthetic data having the same distribution as the sample data with which it is trained. In this work, we have proposed a methodology for improving the performance of ANN classifier to identify built-up pixels in the Landsat$7$ image with the help of developing a simple GAN architecture that could generate synthetic training pixels when trained using original set of sample built-up pixels. To ensure that the marginal and joint distributions of all the bands corresponding to the generated and original set of pixels are indistinguishable, non-parametric Kolmogorov Smirnov Test and Ball Divergence based Equality of Distributions Test have been performed respectively. It has been observed that the overall accuracy and kappa coefficient of the ANN model for built-up classification have continuously improved from $0.9331$ to $0.9983$ and $0.8277$ to $0.9958$ respectively, with the inclusion of generated sets of built-up pixels to the original one.
• 摘要：使用低分辨率 Landsat 图像训练神经网络进行基于像素的分类任务很困难，因为训练数据的大小通常很小，因为表示单个类别而不与其他类别混合的可用像素数量较少。由于训练数据稀缺，神经网络可能无法达到预期的准确度。可以使用生成网络来克服这一限制，该网络旨在生成具有与其训练的样本数据相同分布的合成数据。在这项工作中，我们提出了一种改进 ANN 分类器性能的方法，以借助开发一个简单的 GAN 架构来识别 Landsat$7$ 图像中的组合像素，该架构可以在使用原始样本组合像素集进行训练时生成合成训练像素。为了确保与生成的和原始像素集相对应的所有波段的边际分布和联合分布无法区分，分别进行了非参数 Kolmogorov Smirnov 检验和基于 Ball Divergence 的分布相等性检验。观察发现，随着生成的建筑像素集加入到原始建筑像素集，ANN 模型对建筑分类的总体准确度和 kappa 系数分别从 0.9331 提高到 0.9983 和从 0.8277 提高到 0.9958。

Title: Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs

• Authors: James Flemings, Haosheng Gan, Hongyi Li, Meisam Razaviyayn, Murali Annavaram
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19287
• Pdf URL: https://arxiv.org/pdf/2501.19287
• Copy Paste: [[2501.19287]] Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs(https://arxiv.org/abs/2501.19287)
• Keywords: generation
• Abstract: In-context learning (ICL) has shown promising improvement in downstream task adaptation of LLMs by augmenting prompts with relevant input-output examples (demonstrations). However, the ICL demonstrations can contain privacy-sensitive information, which can be leaked and/or regurgitated by the LLM output. Differential Privacy (DP), a widely adopted privacy safeguard, has emerged to mitigate this privacy leakage, with recent work demonstrating strong privacy-utility tradeoffs in classification tasks for ICL. However, generation tasks for ICL are challenging due to the high-dimensional output space of open-ended generation. To this end, we propose $\texttt{dps-mozo}$, Differentially Private Sampling by Mixing One-shot with Zero-shot Outputs, a decoding framework that generates DP text by sampling from the product of multiple one-shot outputs mixed with a zero-shot output. This mixing effectively reduces the amount of information that can be leaked by each demonstration. By utilizing the inherent randomness in sampling from the mixed distributions, we can achieve DP without adding noise, thereby improving the privacy-utility tradeoff. Our experimental evaluations show $\texttt{dps-mozo}$ can achieve a strong privacy guarantee, $\epsilon=2$, with minimal utility degradation compared to non-private few-shot learning, $\textbf{0.3}$% ROUGE-L F1 score decrease on the SAMSum dataset with Gemma 2 2B.
• 摘要：上下文学习 (ICL) 通过使用相关输入输出示例 (演示) 增强提示，在 LLM 的下游任务适应方面表现出良好的改善效果。但是，ICL 演示可能包含隐私敏感信息，这些信息可能会被 LLM 输出泄露和/或反刍。差分隐私 (DP) 是一种广泛采用的隐私保护措施，可以缓解这种隐私泄露，最近的研究表明，ICL 在分类任务中具有强大的隐私效用权衡。然而，由于开放式生成的高维输出空间，ICL 的生成任务具有挑战性。为此，我们提出了 $\texttt{dps-mozo}$，即通过混合一次性输出和零次输出进行差分隐私采样，这是一种解码框架，通过从多个一次性输出与零次输出的混合乘积中采样来生成 DP 文本。这种混合有效地减少了每次演示可能泄露的信息量。通过利用混合分布采样的固有随机性，我们可以在不增加噪声的情况下实现 DP，从而改善隐私效用权衡。我们的实验评估表明，与非隐私小样本学习相比，$\texttt{dps-mozo}$ 可以实现强大的隐私保证，$\epsilon=2$，同时效用下降最小，在具有 Gemma 2 2B 的 SAMSum 数据集上，ROUGE-L F1 得分下降了 $\textbf{0.3}$%。

Title: Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment

• Authors: Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, Jonas Kohler
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19309
• Pdf URL: https://arxiv.org/pdf/2501.19309
• Copy Paste: [[2501.19309]] Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment(https://arxiv.org/abs/2501.19309)
• Keywords: generation
• Abstract: The performance of large language models (LLMs) is closely linked to their underlying size, leading to ever-growing networks and hence slower inference. Speculative decoding has been proposed as a technique to accelerate autoregressive generation, leveraging a fast draft model to propose candidate tokens, which are then verified in parallel based on their likelihood under the target model. While this approach guarantees to reproduce the target output, it incurs a substantial penalty: many high-quality draft tokens are rejected, even when they represent objectively valid continuations. Indeed, we show that even powerful draft models such as GPT-4o, as well as human text cannot achieve high acceptance rates under the standard verification scheme. This severely limits the speedup potential of current speculative decoding methods, as an early rejection becomes overwhelmingly likely when solely relying on alignment of draft and target. We thus ask the following question: Can we adapt verification to recognize correct, but non-aligned replies? To this end, we draw inspiration from the LLM-as-a-judge framework, which demonstrated that LLMs are able to rate answers in a versatile way. We carefully design a dataset to elicit the same capability in the target model by training a compact module on top of the embeddings to produce ``judgements" of the current continuation. We showcase our strategy on the Llama-3.1 family, where our 8b/405B-Judge achieves a speedup of 9x over Llama-405B, while maintaining its quality on a large range of benchmarks. These benefits remain present even in optimized inference frameworks, where our method reaches up to 141 tokens/s for 8B/70B-Judge and 129 tokens/s for 8B/405B on 2 and 8 H100s respectively.
• 摘要：大型语言模型 (LLM) 的性能与其底层规模密切相关，这导致网络不断增长，因此推理速度变慢。推测解码已被提出作为一种加速自回归生成的技术，利用快速草稿模型提出候选标记，然后根据它们在目标模型下的可能性并行验证这些标记。虽然这种方法保证重现目标输出，但它会带来很大的损失：许多高质量的草稿标记被拒绝，即使它们代表客观有效的延续。事实上，我们表明，即使是 GPT-4o 等强大的草稿模型以及人类文本，在标准验证方案下也无法实现高接受率。这严重限制了当前推测解码方法的加速潜力，因为当仅依靠草稿和目标的对齐时，早期拒绝的可能性变得非常大。因此，我们提出以下问题：我们能否调整验证以识别正确但不对齐的答复？为此，我们从 LLM-as-a-judge 框架中汲取灵感，该框架表明 LLM 能够以多种方式对答案进行评分。我们精心设计了一个数据集，通过在嵌入之上训练一个紧凑模块来产生当前延续的“判断”，从而在目标模型中引出相同的功能。我们在 Llama-3.1 系列上展示了我们的策略，其中我们的 8b/405B-Judge 比 Llama-405B 的速度提高了 9 倍，同时在大量基准测试中保持了其质量。即使在优化的推理框架中，这些优势仍然存在，我们的方法在 2 个和 8 个 H100 上分别达到 8B/70B-Judge 的 141 个 token/s 和 8B/405B 的 129 个 token/s。

Title: Consistent Video Colorization via Palette Guidance

• Authors: Han Wang, Yuang Zhang, Yuhong Zhang, Lingxiao Lu, Li Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19331
• Pdf URL: https://arxiv.org/pdf/2501.19331
• Copy Paste: [[2501.19331]] Consistent Video Colorization via Palette Guidance(https://arxiv.org/abs/2501.19331)
• Keywords: restoration, generation, generative
• Abstract: Colorization is a traditional computer vision task and it plays an important role in many time-consuming tasks, such as old film restoration. Existing methods suffer from unsaturated color and temporally inconsistency. In this paper, we propose a novel pipeline to overcome the challenges. We regard the colorization task as a generative task and introduce Stable Video Diffusion (SVD) as our base model. We design a palette-based color guider to assist the model in generating vivid and consistent colors. The color context introduced by the palette not only provides guidance for color generation, but also enhances the stability of the generated colors through a unified color context across multiple sequences. Experiments demonstrate that the proposed method can provide vivid and stable colors for videos, surpassing previous methods.
• 摘要：着色是一项传统的计算机视觉任务，它在许多耗时的任务中起着重要作用，例如旧电影修复。现有的方法存在色彩不饱和和时间不一致的问题。在本文中，我们提出了一种新颖的流程来克服这些挑战。我们将着色任务视为生成任务，并引入稳定视频扩散（SVD）作为我们的基础模型。我们设计了一个基于调色板的色彩指南来帮助模型生成鲜艳一致的色彩。调色板引入的色彩上下文不仅为色彩生成提供指导，而且通过跨多个序列的统一色彩上下文增强了生成色彩的稳定性。实验表明，所提出的方法可以为视频提供鲜艳而稳定的色彩，超越了以前的方法。

Title: CoSTI: Consistency Models for (a faster) Spatio-Temporal Imputation

• Authors: Javier Solís-García, Belén Vega-Márquez, Juan A. Nepomuceno, Isabel A. Nepomuceno-Chamorro
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19364
• Pdf URL: https://arxiv.org/pdf/2501.19364
• Copy Paste: [[2501.19364]] CoSTI: Consistency Models for (a faster) Spatio-Temporal Imputation(https://arxiv.org/abs/2501.19364)
• Keywords: generative
• Abstract: Multivariate Time Series Imputation (MTSI) is crucial for many applications, such as healthcare monitoring and traffic management, where incomplete data can compromise decision-making. Existing state-of-the-art methods, like Denoising Diffusion Probabilistic Models (DDPMs), achieve high imputation accuracy; however, they suffer from significant computational costs and are notably time-consuming due to their iterative nature. In this work, we propose CoSTI, an innovative adaptation of Consistency Models (CMs) for the MTSI domain. CoSTI employs Consistency Training to achieve comparable imputation quality to DDPMs while drastically reducing inference times, making it more suitable for real-time applications. We evaluate CoSTI across multiple datasets and missing data scenarios, demonstrating up to a 98% reduction in imputation time with performance on par with diffusion-based models. This work bridges the gap between efficiency and accuracy in generative imputation tasks, providing a scalable solution for handling missing data in critical spatio-temporal systems.
• 摘要：多元时间序列插补 (MTSI) 对于许多应用至关重要，例如医疗监测和交通管理，其中不完整的数据会影响决策。现有的先进方法，如去噪扩散概率模型 (DDPM)，实现了高插补精度；然而，由于它们的迭代性质，它们的计算成本很高，而且非常耗时。在这项工作中，我们提出了 CoSTI，这是针对 MTSI 领域的一致性模型 (CM) 的创新改编。CoSTI 采用一致性训练来实现与 DDPM 相当的插补质量，同时大幅减少推理时间，使其更适合实时应用。我们在多个数据集和缺失数据场景中评估了 CoSTI，结果表明插补时间减少了 98%，性能与基于扩散的模型相当。这项工作弥合了生成插补任务中效率和准确性之间的差距，为处理关键时空系统中的缺失数据提供了可扩展的解决方案。