2025-01-06

Title: LS-GAN: Human Motion Synthesis with Latent-space GANs

• Authors: Avinash Amballa, Gayathri Akkinapalli, Vinitra Muralikrishnan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01449
• Pdf URL: https://arxiv.org/pdf/2501.01449
• Copy Paste: [[2501.01449]] LS-GAN: Human Motion Synthesis with Latent-space GANs(https://arxiv.org/abs/2501.01449)
• Keywords: generative
• Abstract: Human motion synthesis conditioned on textual input has gained significant attention in recent years due to its potential applications in various domains such as gaming, film production, and virtual reality. Conditioned Motion synthesis takes a text input and outputs a 3D motion corresponding to the text. While previous works have explored motion synthesis using raw motion data and latent space representations with diffusion models, these approaches often suffer from high training and inference times. In this paper, we introduce a novel framework that utilizes Generative Adversarial Networks (GANs) in the latent space to enable faster training and inference while achieving results comparable to those of the state-of-the-art diffusion methods. We perform experiments on the HumanML3D, HumanAct12 benchmarks and demonstrate that a remarkably simple GAN in the latent space achieves a FID of 0.482 with more than 91% in FLOPs reduction compared to latent diffusion model. Our work opens up new possibilities for efficient and high-quality motion synthesis using latent space GANs.
• 摘要：近年来，以文本输入为条件的人体运动合成因其在游戏、电影制作和虚拟现实等各个领域的潜在应用而备受关注。条件运动合成以文本为输入，输出与文本相对应的 3D 运动。虽然之前的研究已经探索了使用原始运动数据和潜在空间表示与扩散模型进行运动合成，但这些方法通常存在训练和推理时间较长的问题。在本文中，我们介绍了一种新颖的框架，该框架利用潜在空间中的生成对抗网络 (GAN) 来实现更快的训练和推理，同时实现与最先进的扩散方法相当的结果。我们在 HumanML3D、HumanAct12 基准上进行了实验，并证明与潜在扩散模型相比，潜在空间中非常简单的 GAN 实现了 0.482 的 FID，FLOP 减少了 91% 以上。我们的工作为使用潜在空间 GAN 进行高效、高质量的运动合成开辟了新的可能性。

Title: AI-Enabled Operations at Fermi Complex: Multivariate Time Series Prediction for Outage Prediction and Diagnosis

• Authors: Milan Jain, Burcu O. Mutlu, Caleb Stam, Jan Strube, Brian A. Schupbach, Jason M. St. John, William A. Pellico
• Subjects: cs.LG, cs.AI, cs.ET, eess.SP
• Abstract URL: https://arxiv.org/abs/2501.01509
• Pdf URL: https://arxiv.org/pdf/2501.01509
• Copy Paste: [[2501.01509]] AI-Enabled Operations at Fermi Complex: Multivariate Time Series Prediction for Outage Prediction and Diagnosis(https://arxiv.org/abs/2501.01509)
• Keywords: restoration
• Abstract: The Main Control Room of the Fermilab accelerator complex continuously gathers extensive time-series data from thousands of sensors monitoring the beam. However, unplanned events such as trips or voltage fluctuations often result in beam outages, causing operational downtime. This downtime not only consumes operator effort in diagnosing and addressing the issue but also leads to unnecessary energy consumption by idle machines awaiting beam restoration. The current threshold-based alarm system is reactive and faces challenges including frequent false alarms and inconsistent outage-cause labeling. To address these limitations, we propose an AI-enabled framework that leverages predictive analytics and automated labeling. Using data from $2,703$ Linac devices and $80$ operator-labeled outages, we evaluate state-of-the-art deep learning architectures, including recurrent, attention-based, and linear models, for beam outage prediction. Additionally, we assess a Random Forest-based labeling system for providing consistent, confidence-scored outage annotations. Our findings highlight the strengths and weaknesses of these architectures for beam outage prediction and identify critical gaps that must be addressed to fully harness AI for transitioning downtime handling from reactive to predictive, ultimately reducing downtime and improving decision-making in accelerator management.
• 摘要：费米实验室加速器综合体的主控制室不断从数千个监测光束的传感器收集大量时间序列数据。然而，意外事件（例如跳闸或电压波动）通常会导致光束中断，从而导致运行停机。这种停机不仅消耗了操作员诊断和解决问题的精力，还导致等待光束恢复的闲置机器产生不必要的能源消耗。当前基于阈值的警报系统是被动的，面临着包括频繁误报和不一致的中断原因标记等挑战。为了解决这些限制，我们提出了一个利用预测分析和自动标记的 AI 框架。使用来自 $2,703$ 台直线加速器设备和 $80$ 台操作员标记的中断的数据，我们评估了最先进的深度学习架构，包括循环、基于注意力和线性模型，用于光束中断预测。此外，我们还评估了基于随机森林的标记系统，以提供一致的、置信度评分的中断注释。我们的研究结果突出了这些架构在光束中断预测方面的优势和劣势，并确定了必须解决的关键差距，以充分利用人工智能将停机处理从被动转变为预测，最终减少停机时间并改善加速器管理的决策。

Title: Explainable Brain Age Gap Prediction in Neurodegenerative Conditions using coVariance Neural Networks

• Authors: Saurabh Sihag, Gonzalo Mateos, Alejandro Ribeiro
• Subjects: cs.LG, eess.SP, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2501.01510
• Pdf URL: https://arxiv.org/pdf/2501.01510
• Copy Paste: [[2501.01510]] Explainable Brain Age Gap Prediction in Neurodegenerative Conditions using coVariance Neural Networks(https://arxiv.org/abs/2501.01510)
• Keywords: generation, generative
• Abstract: Brain age is the estimate of biological age derived from neuroimaging datasets using machine learning algorithms. Increasing \textit{brain age gap} characterized by an elevated brain age relative to the chronological age can reflect increased vulnerability to neurodegeneration and cognitive decline. Hence, brain age gap is a promising biomarker for monitoring brain health. However, black-box machine learning approaches to brain age gap prediction have limited practical utility. Recent studies on coVariance neural networks (VNN) have proposed a relatively transparent deep learning pipeline for neuroimaging data analyses, which possesses two key features: (i) inherent \textit{anatomically interpretablity} of derived biomarkers; and (ii) a methodologically interpretable perspective based on \textit{linkage with eigenvectors of anatomic covariance matrix}. In this paper, we apply the VNN-based approach to study brain age gap using cortical thickness features for various prevalent neurodegenerative conditions. Our results reveal distinct anatomic patterns for brain age gap in Alzheimer's disease, frontotemporal dementia, and atypical Parkinsonian disorders. Furthermore, we demonstrate that the distinct anatomic patterns of brain age gap are linked with the differences in how VNN leverages the eigenspectrum of the anatomic covariance matrix, thus lending explainability to the reported results.
• 摘要：脑年龄是使用机器学习算法从神经影像数据集得出的生物年龄估计值。脑年龄差距的扩大（相对于实际年龄脑年龄增大）可以反映出对神经退行性疾病和认知能力下降的脆弱性增加。因此，脑年龄差距是监测脑健康的有前途的生物标志物。然而，用于预测脑年龄差距的黑箱机器学习方法的实际用途有限。最近对协方差神经网络 (VNN) 的研究提出了一种相对透明的深度学习流程，用于神经影像数据分析，它具有两个关键特征：(i) 衍生生物标志物的固有 \textit{解剖学可解释性}；(ii) 基于 \textit{与解剖协方差矩阵的特征向量的链接} 的方法学可解释视角。在本文中，我们应用基于 VNN 的方法，使用皮质厚度特征研究各种常见神经退行性疾病的脑年龄差距。我们的研究结果揭示了阿尔茨海默病、额颞叶痴呆和非典型帕金森病中大脑年龄差距的不同解剖模式。此外，我们还证明了大脑年龄差距的不同解剖模式与 VNN 如何利用解剖协方差矩阵的特征谱的差异有关，从而为报告的结果提供了可解释性。

Title: BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery

• Authors: Kanishk Gandhi, Michael Y. Li, Lyle Goodyear, Louise Li, Aditi Bhaskar, Mohammed Zaman, Noah D. Goodman
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01540
• Pdf URL: https://arxiv.org/pdf/2501.01540
• Copy Paste: [[2501.01540]] BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery(https://arxiv.org/abs/2501.01540)
• Keywords: generative
• Abstract: Understanding the world and explaining it with scientific theories is a central aspiration of artificial intelligence research. Proposing theories, designing experiments to test them, and then revising them based on data are fundamental to scientific discovery. Despite the significant promise of LLM-based scientific agents, no benchmarks systematically test LLM's ability to propose scientific models, collect experimental data, and revise them in light of new data. We introduce BoxingGym, a benchmark with 10 environments for systematically evaluating both experimental design (e.g. collecting data to test a scientific theory) and model discovery (e.g. proposing and revising scientific theories). To enable tractable and quantitative evaluation, we implement each environment as a generative probabilistic model with which a scientific agent can run interactive experiments. These probabilistic models are drawn from various real-world scientific domains ranging from psychology to ecology. To quantitatively evaluate a scientific agent's ability to collect informative experimental data, we compute the expected information gain (EIG), an information-theoretic quantity which measures how much an experiment reduces uncertainty about the parameters of a generative model. A good scientific theory is a concise and predictive explanation. Therefore, to quantitatively evaluate model discovery, we ask a scientific agent to explain their model and then assess whether this explanation enables another scientific agent to make reliable predictions about this environment. In addition to this explanation-based evaluation, we compute standard model evaluation metrics such as prediction errors. We find that current LLMs, such as GPT-4o, struggle with both experimental design and model discovery. We find that augmenting the LLM-based agent with an explicit statistical model does not reliably improve these results.
• 摘要：理解世界并用科学理论解释世界是人工智能研究的核心愿望。提出理论、设计实验来测试它们，然后根据数据进行修改，这些是科学发现的基础。尽管基于 LLM 的科学代理前景广阔，但没有基准系统地测试 LLM 提出科学模型、收集实验数据并根据新数据修改它们的能力。我们推出了 BoxingGym，这是一个具有 10 个环境的基准，用于系统地评估实验设计（例如收集数据来测试科学理论）和模型发现（例如提出和修改科学理论）。为了实现易于处理和定量的评估，我们将每个环境实现为生成概率模型，科学代理可以使用该模型运行交互式实验。这些概率模型来自从心理学到生态学的各种现实世界科学领域。为了定量评估科学代理收集信息性实验数据的能力，我们计算了预期信息增益 (EIG)，这是一个信息论量，用于衡量实验在多大程度上减少了生成模型参数的不确定性。好的科学理论是简明扼要且具有预测性的解释。因此，为了定量评估模型发现，我们要求科学代理解释他们的模型，然后评估这种解释是否使另一个科学代理能够对此环境做出可靠的预测。除了这种基于解释的评估之外，我们还计算标准模型评估指标，例如预测误差。我们发现当前的 LLM（例如 GPT-4o）在实验设计和模型发现方面都存在困难。我们发现，用显式统计模型增强基于 LLM 的代理并不能可靠地改善这些结果。

Title: Predicting the Performance of Black-box LLMs through Self-Queries

• Authors: Dylan Sam, Marc Finzi, J. Zico Kolter
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01558
• Pdf URL: https://arxiv.org/pdf/2501.01558
• Copy Paste: [[2501.01558]] Predicting the Performance of Black-box LLMs through Self-Queries(https://arxiv.org/abs/2501.01558)
• Keywords: generation
• Abstract: As large language models (LLMs) are increasingly relied on in AI systems, predicting when they make mistakes is crucial. While a great deal of work in the field uses internal representations to interpret model behavior, these representations are inaccessible when given solely black-box access through an API. In this paper, we extract features of LLMs in a black-box manner by using follow-up prompts and taking the probabilities of different responses as representations to train reliable predictors of model behavior. We demonstrate that training a linear model on these low-dimensional representations produces reliable and generalizable predictors of model performance at the instance level (e.g., if a particular generation correctly answers a question). Remarkably, these can often outperform white-box linear predictors that operate over a model's hidden state or the full distribution over its vocabulary. In addition, we demonstrate that these extracted features can be used to evaluate more nuanced aspects of a language model's state. For instance, they can be used to distinguish between a clean version of GPT-4o-mini and a version that has been influenced via an adversarial system prompt that answers question-answering tasks incorrectly or introduces bugs into generated code. Furthermore, they can reliably distinguish between different model architectures and sizes, enabling the detection of misrepresented models provided through an API (e.g., identifying if GPT-3.5 is supplied instead of GPT-4o-mini).
• 摘要：随着人工智能系统越来越依赖大型语言模型 (LLM)，预测它们何时出错至关重要。虽然该领域的大量工作使用内部表示来解释模型行为，但当仅通过 API 提供黑盒访问时，这些表示是无法访问的。在本文中，我们以黑盒方式提取 LLM 的特征，使用后续提示并将不同响应的概率作为表示来训练可靠的模型行为预测器。我们证明，在这些低维表示上训练线性模型会在实例级别产生可靠且可推广的模型性能预测器（例如，如果特定代正确回答了问题）。值得注意的是，这些通常可以胜过对模型的隐藏状态或其词汇表的完整分布进行操作的白盒线性预测器。此外，我们证明这些提取的特征可用于评估语言模型状态的更细微方面。例如，它们可用于区分 GPT-4o-mini 的纯净版本和受到对抗系统提示影响的版本，后者会错误地回答问答任务或在生成的代码中引入错误。此外，它们还可以可靠地区分不同的模型架构和大小，从而能够检测通过 API 提供的错误表示模型（例如，识别提供的是 GPT-3.5 还是 GPT-4o-mini）。

Title: Multivariate Time Series Anomaly Detection using DiffGAN Model

• Authors: Guangqiang Wu, Fu Zhang
• Subjects: cs.LG, math.ST
• Abstract URL: https://arxiv.org/abs/2501.01591
• Pdf URL: https://arxiv.org/pdf/2501.01591
• Copy Paste: [[2501.01591]] Multivariate Time Series Anomaly Detection using DiffGAN Model(https://arxiv.org/abs/2501.01591)
• Keywords: generation, generative
• Abstract: In recent years, some researchers have applied diffusion models to multivariate time series anomaly detection. The partial diffusion strategy, which depends on the diffusion steps, is commonly used for anomaly detection in these models. However, different diffusion steps have an impact on the reconstruction of the original data, thereby impacting the effectiveness of anomaly detection. To address this issue, we propose a novel method named DiffGAN, which adds a generative adversarial network component to the denoiser of diffusion model. This addition allows for the simultaneous generation of noisy data and prediction of diffusion steps. Compared to multiple state-of-the-art reconstruction models, experimental results demonstrate that DiffGAN achieves superior performance in anomaly detection.
• 摘要：近年来，一些研究者将扩散模型应用于多变量时间序列异常检测。这些模型通常采用依赖于扩散步骤的部分扩散策略进行异常检测。然而，不同的扩散步骤对原始数据的重构会产生影响，从而影响异常检测的效果。针对这一问题，我们提出了一种新方法DiffGAN，该方法在扩散模型的去噪器中添加了一个生成对抗网络组件。这一添加允许同时生成噪声数据和预测扩散步骤。与多种最先进的重构模型相比，实验结果表明DiffGAN 在异常检测中取得了优异的性能。

Title: Adaptive Homophily Clustering: A Structure Homophily Graph Learning with Adaptive Filter for Hyperspectral Image

• Authors: Yao Ding, Weijie Kang, Aitao Yang, Zhili Zhang, Junyang Zhao, Jie Feng, Danfeng Hong, Qinhe Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01595
• Pdf URL: https://arxiv.org/pdf/2501.01595
• Copy Paste: [[2501.01595]] Adaptive Homophily Clustering: A Structure Homophily Graph Learning with Adaptive Filter for Hyperspectral Image(https://arxiv.org/abs/2501.01595)
• Keywords: generation
• Abstract: Hyperspectral image (HSI) clustering has been a fundamental but challenging task with zero training labels. Currently, some deep graph clustering methods have been successfully explored for HSI due to their outstanding performance in effective spatial structural information encoding. Nevertheless, insufficient structural information utilization, poor feature presentation ability, and weak graph update capability limit their performance. Thus, in this paper, a homophily structure graph learning with an adaptive filter clustering method (AHSGC) for HSI is proposed. Specifically, homogeneous region generation is first developed for HSI processing and constructing the original graph. Afterward, an adaptive filter graph encoder is designed to adaptively capture the high and low frequency features on the graph for subsequence processing. Then, a graph embedding clustering self-training decoder is developed with KL Divergence, with which the pseudo-label is generated for network training. Meanwhile, homophily-enhanced structure learning is introduced to update the graph according to the clustering task, in which the orient correlation estimation is adopted to estimate the node connection, and graph edge sparsification is designed to adjust the edges in the graph dynamically. Finally, a joint network optimization is introduced to achieve network self-training and update the graph. The K-means is adopted to express the latent features. Extensive experiments and repeated comparative analysis have verified that our AHSGC contains high clustering accuracy, low computational complexity, and strong robustness. The code source will be available at this https URL.
• 摘要：高光谱图像 (HSI) 聚类是一项基本但具有挑战性的任务，因为它没有训练标签。目前，一些深度图聚类方法已经成功地应用于 HSI，因为它们在有效的空间结构信息编码方面表现出色。然而，结构信息利用不足、特征呈现能力差和图更新能力弱限制了它们的性能。因此，本文提出了一种用于 HSI 的同质结构图学习和自适应过滤器聚类方法 (AHSGC)。具体而言，首先开发同质区域生成以用于 HSI 处理和构建原始图。然后，设计一个自适应过滤器图编码器来自适应地捕获图上的高频和低频特征以进行子序列处理。然后，开发了一个具有 KL 散度的图嵌入聚类自训练解码器，使用该解码器生成伪标签用于网络训练。同时，引入同质增强结构学习来根据聚类任务更新图，其中采用方向相关估计来估计节点连接，并设计图边稀疏化来动态调整图中的边。最后，引入联合网络优化，实现网络自训练和更新图。采用 K-means 来表达潜在特征。大量实验和反复比较分析证明我们的 AHSGC 具有较高的聚类精度、较低的计算复杂度和较强的鲁棒性。代码源将在此 https URL 上提供。

Title: Few-shot Implicit Function Generation via Equivariance

• Authors: Suizhi Huang, Xingyi Yang, Hongtao Lu, Xinchao Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01601
• Pdf URL: https://arxiv.org/pdf/2501.01601
• Copy Paste: [[2501.01601]] Few-shot Implicit Function Generation via Equivariance(https://arxiv.org/abs/2501.01601)
• Keywords: generation
• Abstract: Implicit Neural Representations (INRs) have emerged as a powerful framework for representing continuous signals. However, generating diverse INR weights remains challenging due to limited training data. We introduce Few-shot Implicit Function Generation, a new problem setup that aims to generate diverse yet functionally consistent INR weights from only a few examples. This is challenging because even for the same signal, the optimal INRs can vary significantly depending on their initializations. To tackle this, we propose EquiGen, a framework that can generate new INRs from limited data. The core idea is that functionally similar networks can be transformed into one another through weight permutations, forming an equivariance group. By projecting these weights into an equivariant latent space, we enable diverse generation within these groups, even with few examples. EquiGen implements this through an equivariant encoder trained via contrastive learning and smooth augmentation, an equivariance-guided diffusion process, and controlled perturbations in the equivariant subspace. Experiments on 2D image and 3D shape INR datasets demonstrate that our approach effectively generates diverse INR weights while preserving their functional properties in few-shot scenarios.
• 摘要：隐式神经表征 (INR) 已成为表示连续信号的强大框架。然而，由于训练数据有限，生成多样化的 INR 权重仍然具有挑战性。我们引入了少样本隐式函数生成，这是一种新的问题设置，旨在仅从几个示例中生成多样化但功能一致的 INR 权重。这很有挑战性，因为即使对于相同的信号，最佳 INR 也会因其初始化而有很大差异。为了解决这个问题，我们提出了 EquiGen，这是一个可以从有限数据中生成新 INR 的框架。核心思想是功能相似的网络可以通过权重排列相互转换，形成一个等方差组。通过将这些权重投影到等方差潜在空间中，我们能够在这些组中实现多样化生成，即使只有很少的示例。EquiGen 通过经过对比学习和平滑增强训练的等方差编码器、等方差引导的扩散过程和等方差子空间中的受控扰动来实现这一点。在 2D 图像和 3D 形状 INR 数据集上的实验表明，我们的方法能够在小样本场景中有效地生成不同的 INR 权重，同时保留其功能特性。

Title: A Probabilistic Model for Node Classification in Directed Graphs

• Authors: Diego Huerta, Gerardo Arizmendi
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.01630
• Pdf URL: https://arxiv.org/pdf/2501.01630
• Copy Paste: [[2501.01630]] A Probabilistic Model for Node Classification in Directed Graphs(https://arxiv.org/abs/2501.01630)
• Keywords: generative
• Abstract: In this work, we present a probabilistic model for directed graphs where nodes have attributes and labels. This model serves as a generative classifier capable of predicting the labels of unseen nodes using either maximum likelihood or maximum a posteriori estimations. The predictions made by this model are highly interpretable, contrasting with some common methods for node classification, such as graph neural networks. We applied the model to two datasets, demonstrating predictive performance that is competitive with, and even superior to, state-of-the-art methods. One of the datasets considered is adapted from the Math Genealogy Project, which has not previously been utilized for this purpose. Consequently, we evaluated several classification algorithms on this dataset to compare the performance of our model and provide benchmarks for this new resource.
• 摘要：在本研究中，我们提出了一个有向图的概率模型，其中节点具有属性和标签。该模型用作生成分类器，能够使用最大似然或最大后验估计来预测未见节点的标签。与一些常见的节点分类方法（例如图神经网络）相比，该模型的预测具有高度可解释性。我们将该模型应用于两个数据集，结果显示其预测性能可与最先进的方法相媲美，甚至优于最先进的方法。其中一个数据集改编自数学谱系项目，该项目以前从未用于此目的。因此，我们评估了该数据集上的几种分类算法，以比较我们模型的性能并为这种新资源提供基准。

Title: ACE: Anti-Editing Concept Erasure in Text-to-Image Models

• Authors: Zihao Wang, Yuxiang Wei, Fan Li, Renjing Pei, Hang Xu, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01633
• Pdf URL: https://arxiv.org/pdf/2501.01633
• Copy Paste: [[2501.01633]] ACE: Anti-Editing Concept Erasure in Text-to-Image Models(https://arxiv.org/abs/2501.01633)
• Keywords: generation
• Abstract: Recent advance in text-to-image diffusion models have significantly facilitated the generation of high-quality images, but also raising concerns about the illegal creation of harmful content, such as copyrighted images. Existing concept erasure methods achieve superior results in preventing the production of erased concept from prompts, but typically perform poorly in preventing undesired editing. To address this issue, we propose an Anti-Editing Concept Erasure (ACE) method, which not only erases the target concept during generation but also filters out it during editing. Specifically, we propose to inject the erasure guidance into both conditional and the unconditional noise prediction, enabling the model to effectively prevent the creation of erasure concepts during both editing and generation. Furthermore, a stochastic correction guidance is introduced during training to address the erosion of unrelated concepts. We conducted erasure editing experiments with representative editing methods (i.e., LEDITS++ and MasaCtrl) to erase IP characters, and the results indicate that our ACE effectively filters out target concepts in both types of edits. Additional experiments on erasing explicit concepts and artistic styles further demonstrate that our ACE performs favorably against state-of-the-art methods. Our code will be publicly available at this https URL.
• 摘要：文本到图像传播模型的最新进展极大地促进了高质量图像的生成，但也引发了对非法创建有害内容（例如受版权保护的图像）的担忧。现有的概念擦除方法在防止提示产生被擦除的概念方面取得了优异的效果，但在防止不必要的编辑方面通常表现不佳。为了解决这个问题，我们提出了一种反编辑概念擦除 (ACE) 方法，它不仅在生成过程中擦除目标概念，而且在编辑过程中将其过滤掉。具体来说，我们建议将擦除指导注入条件和非条件噪声预测中，使模型能够有效防止在编辑和生成过程中产生擦除概念。此外，在训练期间引入了随机校正指导来解决不相关概念的侵蚀问题。我们使用代表性编辑方法（即 LEDITS++ 和 MasaCtrl）进行了擦除编辑实验以擦除 IP 字符，结果表明我们的 ACE 有效地过滤掉了两种编辑类型的目标概念。在消除明确概念和艺术风格方面进行的额外实验进一步表明，我们的 ACE 表现优于最先进的方法。我们的代码将在此 https URL 上公开提供。

Title: Uncertainty and Energy based Loss Guided Semi-Supervised Semantic Segmentation

• Authors: Rini Smita Thakur, Vinod K. Kurmi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01640
• Pdf URL: https://arxiv.org/pdf/2501.01640
• Copy Paste: [[2501.01640]] Uncertainty and Energy based Loss Guided Semi-Supervised Semantic Segmentation(https://arxiv.org/abs/2501.01640)
• Keywords: generative
• Abstract: Semi-supervised (SS) semantic segmentation exploits both labeled and unlabeled images to overcome tedious and costly pixel-level annotation problems. Pseudolabel supervision is one of the core approaches of training networks with both pseudo labels and ground-truth labels. This work uses aleatoric or data uncertainty and energy based modeling in intersection-union pseudo supervised this http URL aleatoric uncertainty is modeling the inherent noise variations of the data in a network with two predictive branches. The per-pixel variance parameter obtained from the network gives a quantitative idea about the data uncertainty. Moreover, energy-based loss realizes the potential of generative modeling on the downstream SS segmentation task. The aleatoric and energy loss are applied in conjunction with pseudo-intersection labels, pseudo-union labels, and ground-truth on the respective network branch. The comparative analysis with state-of-the-art methods has shown improvement in performance metrics.
• 摘要：半监督 (SS) 语义分割利用标记和未标记的图像来克服繁琐且昂贵的像素级注释问题。伪标签监督是使用伪标签和地面实况标签训练网络的核心方法之一。这项工作在交叉并集伪监督中使用随机或数据不确定性和基于能量的建模，此 http URL 随机不确定性正在使用两个预测分支对网络中数据的固有噪声变化进行建模。从网络获得的每像素方差参数给出了数据不确定性的定量概念。此外，基于能量的损失实现了下游 SS 分割任务上生成建模的潜力。随机和能量损失与伪交叉标签、伪并集标签和地面实况结合应用于相应的网络分支。与最先进方法的比较分析表明性能指标有所改善。

Title: AVATAR: Adversarial Autoencoders with Autoregressive Refinement for Time Series Generation

• Authors: MohammadReza EskandariNasab, Shah Muhammad Hamdi, Soukaina Filali Boubrahimi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01649
• Pdf URL: https://arxiv.org/pdf/2501.01649
• Copy Paste: [[2501.01649]] AVATAR: Adversarial Autoencoders with Autoregressive Refinement for Time Series Generation(https://arxiv.org/abs/2501.01649)
• Keywords: generation
• Abstract: Data augmentation can significantly enhance the performance of machine learning tasks by addressing data scarcity and improving generalization. However, generating time series data presents unique challenges. A model must not only learn a probability distribution that reflects the real data distribution but also capture the conditional distribution at each time step to preserve the inherent temporal dependencies. To address these challenges, we introduce AVATAR, a framework that combines Adversarial Autoencoders (AAE) with Autoregressive Learning to achieve both objectives. Specifically, our technique integrates the autoencoder with a supervisor and introduces a novel supervised loss to assist the decoder in learning the temporal dynamics of time series data. Additionally, we propose another innovative loss function, termed distribution loss, to guide the encoder in more efficiently aligning the aggregated posterior of the autoencoder's latent representation with a prior Gaussian distribution. Furthermore, our framework employs a joint training mechanism to simultaneously train all networks using a combined loss, thereby fulfilling the dual objectives of time series generation. We evaluate our technique across a variety of time series datasets with diverse characteristics. Our experiments demonstrate significant improvements in both the quality and practical utility of the generated data, as assessed by various qualitative and quantitative metrics.
• 摘要：数据增强可以通过解决数据稀缺性和提高泛化能力来显著提高机器学习任务的性能。然而，生成时间序列数据带来了独特的挑战。模型不仅必须学习反映真实数据分布的概率分布，还必须捕获每个时间步骤的条件分布以保留固有的时间依赖性。为了应对这些挑战，我们引入了 AVATAR，这是一个将对抗性自动编码器 (AAE) 与自回归学习相结合的框架，以实现这两个目标。具体来说，我们的技术将自动编码器与监督器集成在一起，并引入了一种新颖的监督损失，以帮助解码器学习时间序列数据的时间动态。此外，我们提出了另一种创新的损失函数，称为分布损失，以指导编码器更有效地将自动编码器潜在表示的聚合后验与先验高斯分布对齐。此外，我们的框架采用联合训练机制，使用组合损失同时训练所有网络，从而实现时间序列生成的双重目标。我们在具有不同特征的各种时间序列数据集上评估了我们的技术。我们的实验表明，通过各种定性和定量指标的评估，生成的数据的质量和实用性都有了显著的提高。

Title: AR4D: Autoregressive 4D Generation from Monocular Videos

• Authors: Hanxin Zhu, Tianyu He, Xiqian Yu, Junliang Guo, Zhibo Chen, Jiang Bian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01722
• Pdf URL: https://arxiv.org/pdf/2501.01722
• Copy Paste: [[2501.01722]] AR4D: Autoregressive 4D Generation from Monocular Videos(https://arxiv.org/abs/2501.01722)
• Keywords: generation, generative
• Abstract: Recent advancements in generative models have ignited substantial interest in dynamic 3D content creation (\ie, 4D generation). Existing approaches primarily rely on Score Distillation Sampling (SDS) to infer novel-view videos, typically leading to issues such as limited diversity, spatial-temporal inconsistency and poor prompt alignment, due to the inherent randomness of SDS. To tackle these problems, we propose AR4D, a novel paradigm for SDS-free 4D generation. Specifically, our paradigm consists of three stages. To begin with, for a monocular video that is either generated or captured, we first utilize pre-trained expert models to create a 3D representation of the first frame, which is further fine-tuned to serve as the canonical space. Subsequently, motivated by the fact that videos happen naturally in an autoregressive manner, we propose to generate each frame's 3D representation based on its previous frame's representation, as this autoregressive generation manner can facilitate more accurate geometry and motion estimation. Meanwhile, to prevent overfitting during this process, we introduce a progressive view sampling strategy, utilizing priors from pre-trained large-scale 3D reconstruction models. To avoid appearance drift introduced by autoregressive generation, we further incorporate a refinement stage based on a global deformation field and the geometry of each frame's 3D representation. Extensive experiments have demonstrated that AR4D can achieve state-of-the-art 4D generation without SDS, delivering greater diversity, improved spatial-temporal consistency, and better alignment with input prompts.
• 摘要：生成模型的最新进展激发了人们对动态 3D 内容创建（即 4D 生成）的极大兴趣。现有方法主要依靠分数蒸馏采样 (SDS) 来推断新视角视频，由于 SDS 固有的随机性，通常会导致诸如多样性有限、时空不一致和即时对齐不佳等问题。为了解决这些问题，我们提出了 AR4D，一种无 SDS 4D 生成的新范式。具体来说，我们的范式包括三个阶段。首先，对于生成或捕获的单目视频，我们首先利用预先训练的专家模型来创建第一帧的 3D 表示，然后进一步微调以用作规范空间。随后，受视频以自回归方式自然发生这一事实的启发，我们建议根据其前一帧的表示来生成每帧的 3D 表示，因为这种自回归生成方式可以促进更准确的几何和运动估计。同时，为了防止在此过程中出现过度拟合，我们引入了一种渐进式视图采样策略，利用预先训练的大规模 3D 重建模型的先验知识。为了避免自回归生成引入的外观漂移，我们进一步加入了一个基于全局变形场和每帧 3D 表示几何形状的细化阶段。大量实验表明，AR4D 可以在没有 SDS 的情况下实现最先进的 4D 生成，提供更大的多样性、更好的时空一致性以及与输入提示更好的对齐。

Title: IGAF: Incremental Guided Attention Fusion for Depth Super-Resolution

• Authors: Athanasios Tragakis, Chaitanya Kaul, Kevin J. Mitchell, Hang Dai, Roderick Murray-Smith, Daniele Faccio
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01723
• Pdf URL: https://arxiv.org/pdf/2501.01723
• Copy Paste: [[2501.01723]] IGAF: Incremental Guided Attention Fusion for Depth Super-Resolution(https://arxiv.org/abs/2501.01723)
• Keywords: super-resolution
• Abstract: Accurate depth estimation is crucial for many fields, including robotics, navigation, and medical imaging. However, conventional depth sensors often produce low-resolution (LR) depth maps, making detailed scene perception challenging. To address this, enhancing LR depth maps to high-resolution (HR) ones has become essential, guided by HR-structured inputs like RGB or grayscale images. We propose a novel sensor fusion methodology for guided depth super-resolution (GDSR), a technique that combines LR depth maps with HR images to estimate detailed HR depth maps. Our key contribution is the Incremental guided attention fusion (IGAF) module, which effectively learns to fuse features from RGB images and LR depth maps, producing accurate HR depth maps. Using IGAF, we build a robust super-resolution model and evaluate it on multiple benchmark datasets. Our model achieves state-of-the-art results compared to all baseline models on the NYU v2 dataset for $\times 4$, $\times 8$, and $\times 16$ upsampling. It also outperforms all baselines in a zero-shot setting on the Middlebury, Lu, and RGB-D-D datasets. Code, environments, and models are available on GitHub.
• 摘要：准确的深度估计对于许多领域都至关重要，包括机器人技术、导航和医学成像。然而，传统的深度传感器通常会产生低分辨率 (LR) 深度图，这使得详细的场景感知具有挑战性。为了解决这个问题，将 LR 深度图增强为高分辨率 (HR) 深度图变得至关重要，并以 RGB 或灰度图像等 HR 结构化输入为指导。我们提出了一种用于引导深度超分辨率 (GDSR) 的新型传感器融合方法，该技术将 LR 深度图与 HR 图像相结合以估计详细的 HR 深度图。我们的主要贡献是增量引导注意力融合 (IGAF) 模块，它可以有效地学习融合 RGB 图像和 LR 深度图中的特征，从而生成准确的 HR 深度图。使用 IGAF，我们构建了一个强大的超分辨率模型并在多个基准数据集上对其进行了评估。与 NYU v2 数据集上的所有基线模型相比，我们的模型在 $\times 4$、$\times 8$ 和 $\times 16$ 上采样方面实现了最先进的结果。它还在 Middlebury、Lu 和 RGB-D-D 数据集的零样本设置中超越了所有基线。代码、环境和模型可在 GitHub 上找到。

Title: Multi-modal classification of forest biodiversity potential from 2D orthophotos and 3D airborne laser scanning point clouds

• Authors: Simon B. Jensen, Stefan Oehmcke, Andreas Møgelmose, Meysam Madadi, Christian Igel, Sergio Escalera, Thomas B. Moeslund
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01728
• Pdf URL: https://arxiv.org/pdf/2501.01728
• Copy Paste: [[2501.01728]] Multi-modal classification of forest biodiversity potential from 2D orthophotos and 3D airborne laser scanning point clouds(https://arxiv.org/abs/2501.01728)
• Keywords: quality assessment
• Abstract: Accurate assessment of forest biodiversity is crucial for ecosystem management and conservation. While traditional field surveys provide high-quality assessments, they are labor-intensive and spatially limited. This study investigates whether deep learning-based fusion of close-range sensing data from 2D orthophotos (12.5 cm resolution) and 3D airborne laser scanning (ALS) point clouds (8 points/m^2) can enhance biodiversity assessment. We introduce the BioVista dataset, comprising 44.378 paired samples of orthophotos and ALS point clouds from temperate forests in Denmark, designed to explore multi-modal fusion approaches for biodiversity potential classification. Using deep neural networks (ResNet for orthophotos and PointVector for ALS point clouds), we investigate each data modality's ability to assess forest biodiversity potential, achieving mean accuracies of 69.4% and 72.8%, respectively. We explore two fusion approaches: a confidence-based ensemble method and a feature-level concatenation strategy, with the latter achieving a mean accuracy of 75.5%. Our results demonstrate that spectral information from orthophotos and structural information from ALS point clouds effectively complement each other in forest biodiversity assessment.
• 摘要：准确评估森林生物多样性对于生态系统管理和保护至关重要。虽然传统的实地调查可以提供高质量的评估，但它们是劳动密集型的并且空间有限。本研究调查了基于深度学习的 2D 正射影像（12.5 厘米分辨率）和 3D 机载激光扫描 (ALS) 点云（8 点/平方米）近距离传感数据融合是否可以增强生物多样性评估。我们引入了 BioVista 数据集，该数据集包含来自丹麦温带森林的 44.378 个正射影像和 ALS 点云配对样本，旨在探索用于生物多样性潜力分类的多模态融合方法。使用深度神经网络（正射影像的 ResNet 和 ALS 点云的 PointVector），我们调查了每种数据模态评估森林生物多样性潜力的能力，平均准确率分别达到 69.4% 和 72.8%。我们探索了两种融合方法：基于置信度的集成方法和特征级连接策略，后者的平均准确率达到 75.5%。我们的结果表明，正射影像的光谱信息和 ALS 点云的结构信息在森林生物多样性评估中可以有效互补。

Title: Adverse Weather Conditions Augmentation of LiDAR Scenes with Latent Diffusion Models

• Authors: Andrea Matteazzi, Pascal Colling, Michael Arnold, Dietmar Tutsch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01761
• Pdf URL: https://arxiv.org/pdf/2501.01761
• Copy Paste: [[2501.01761]] Adverse Weather Conditions Augmentation of LiDAR Scenes with Latent Diffusion Models(https://arxiv.org/abs/2501.01761)
• Keywords: generative
• Abstract: LiDAR scenes constitute a fundamental source for several autonomous driving applications. Despite the existence of several datasets, scenes from adverse weather conditions are rarely available. This limits the robustness of downstream machine learning models, and restrains the reliability of autonomous driving systems in particular locations and seasons. Collecting feature-diverse scenes under adverse weather conditions is challenging due to seasonal limitations. Generative models are therefore essentials, especially for generating adverse weather conditions for specific driving scenarios. In our work, we propose a latent diffusion process constituted by autoencoder and latent diffusion models. Moreover, we leverage the clear condition LiDAR scenes with a postprocessing step to improve the realism of the generated adverse weather condition scenes.
• 摘要：LiDAR 场景是多种自动驾驶应用的基本来源。尽管存在多个数据集，但恶劣天气条件下的场景很少可用。这限制了下游机器学习模型的稳健性，并限制了特定地点和季节的自动驾驶系统的可靠性。由于季节限制，在恶劣天气条件下收集特征多样的场景具有挑战性。因此，生成模型至关重要，尤其是对于为特定驾驶场景生成恶劣天气条件。在我们的工作中，我们提出了一种由自动编码器和潜在扩散模型组成的潜在扩散过程。此外，我们利用清晰的 LiDAR 场景和后处理步骤来提高生成的恶劣天气条件场景的真实感。

Title: Can Synthetic Data be Fair and Private? A Comparative Study of Synthetic Data Generation and Fairness Algorithms

• Authors: Qinyi Liu, Oscar Deho, Farhad Vadiee, Mohammad Khalil, Srecko Joksimovic, George Siemens
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.01785
• Pdf URL: https://arxiv.org/pdf/2501.01785
• Copy Paste: [[2501.01785]] Can Synthetic Data be Fair and Private? A Comparative Study of Synthetic Data Generation and Fairness Algorithms(https://arxiv.org/abs/2501.01785)
• Keywords: generation
• Abstract: The increasing use of machine learning in learning analytics (LA) has raised significant concerns around algorithmic fairness and privacy. Synthetic data has emerged as a dual-purpose tool, enhancing privacy and improving fairness in LA models. However, prior research suggests an inverse relationship between fairness and privacy, making it challenging to optimize both. This study investigates which synthetic data generators can best balance privacy and fairness, and whether pre-processing fairness algorithms, typically applied to real datasets, are effective on synthetic data. Our results highlight that the DEbiasing CAusal Fairness (DECAF) algorithm achieves the best balance between privacy and fairness. However, DECAF suffers in utility, as reflected in its predictive accuracy. Notably, we found that applying pre-processing fairness algorithms to synthetic data improves fairness even more than when applied to real data. These findings suggest that combining synthetic data generation with fairness pre-processing offers a promising approach to creating fairer LA models.
• 摘要：机器学习在学习分析 (LA) 中的应用日益广泛，这引发了人们对算法公平性和隐私性的重大担忧。合成数据已成为一种双重用途的工具，可增强隐私并提高 LA 模型中的公平性。然而，先前的研究表明公平性和隐私之间存在反比关系，因此很难同时优化两者。本研究调查了哪些合成数据生成器能够最好地平衡隐私和公平性，以及通常应用于真实数据集的预处理公平性算法是否对合成数据有效。我们的结果强调，DEbiasing CAusal Fairness (DECAF) 算法在隐私和公平性之间实现了最佳平衡。然而，DECAF 的实用性较差，这反映在其预测准确性上。值得注意的是，我们发现将预处理公平性算法应用于合成数据比应用于真实数据更能提高公平性。这些发现表明，将合成数据生成与公平性预处理相结合为创建更公平的 LA 模型提供了一种有前途的方法。

Title: Ingredients: Blending Custom Photos with Video Diffusion Transformers

• Authors: Zhengcong Fei, Debang Li, Di Qiu, Changqian Yu, Mingyuan Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01790
• Pdf URL: https://arxiv.org/pdf/2501.01790
• Copy Paste: [[2501.01790]] Ingredients: Blending Custom Photos with Video Diffusion Transformers(https://arxiv.org/abs/2501.01790)
• Keywords: generative
• Abstract: This paper presents a powerful framework to customize video creations by incorporating multiple specific identity (ID) photos, with video diffusion Transformers, referred to as \texttt{Ingredients}. Generally, our method consists of three primary modules: (\textbf{i}) a facial extractor that captures versatile and precise facial features for each human ID from both global and local perspectives; (\textbf{ii}) a multi-scale projector that maps face embeddings into the contextual space of image query in video diffusion transformers; (\textbf{iii}) an ID router that dynamically combines and allocates multiple ID embedding to the corresponding space-time regions. Leveraging a meticulously curated text-video dataset and a multi-stage training protocol, \texttt{Ingredients} demonstrates superior performance in turning custom photos into dynamic and personalized video content. Qualitative evaluations highlight the advantages of proposed method, positioning it as a significant advancement toward more effective generative video control tools in Transformer-based architecture, compared to existing methods. The data, code, and model weights are publicly available at: \url{this https URL}.
• 摘要：本文提出了一个强大的框架，通过结合多个特定身份 (ID) 照片和视频扩散 Transformers（称为 \texttt{Ingredients}）来定制视频创作。通常，我们的方法由三个主要模块组成：(\textbf{i}) 面部提取器，从全局和局部角度为每个人类 ID 捕获多功能和精确的面部特征；(\textbf{ii}) 多尺度投影仪，将面部嵌入映射到视频扩散 Transformers 中图像查询的上下文空间中；(\textbf{iii}) ID 路由器，动态组合并将多个 ID 嵌入分配到相应的时空区域。利用精心策划的文本视频数据集和多阶段训练协议，\texttt{Ingredients} 在将自定义照片转换为动态和个性化的视频内容方面表现出色。定性评估突出了所提方法的优势，与现有方法相比，将其定位为朝着基于 Transformer 的架构中更有效的生成视频控制工具迈出的重大进步。数据、代码和模型权重可在以下网址公开获取：\url{此 https URL}。

Title: Creating Artificial Students that Never Existed: Leveraging Large Language Models and CTGANs for Synthetic Data Generation

• Authors: Mohammad Khalil, Farhad Vadiee, Ronas Shakya, Qinyi Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01793
• Pdf URL: https://arxiv.org/pdf/2501.01793
• Copy Paste: [[2501.01793]] Creating Artificial Students that Never Existed: Leveraging Large Language Models and CTGANs for Synthetic Data Generation(https://arxiv.org/abs/2501.01793)
• Keywords: generation, generative
• Abstract: In this study, we explore the growing potential of AI and deep learning technologies, particularly Generative Adversarial Networks (GANs) and Large Language Models (LLMs), for generating synthetic tabular data. Access to quality students data is critical for advancing learning analytics, but privacy concerns and stricter data protection regulations worldwide limit their availability and usage. Synthetic data offers a promising alternative. We investigate whether synthetic data can be leveraged to create artificial students for serving learning analytics models. Using the popular GAN model CTGAN and three LLMs- GPT2, DistilGPT2, and DialoGPT, we generate synthetic tabular student data. Our results demonstrate the strong potential of these methods to produce high-quality synthetic datasets that resemble real students data. To validate our findings, we apply a comprehensive set of utility evaluation metrics to assess the statistical and predictive performance of the synthetic data and compare the different generator models used, specially the performance of LLMs. Our study aims to provide the learning analytics community with valuable insights into the use of synthetic data, laying the groundwork for expanding the field methodological toolbox with new innovative approaches for learning analytics data generation.
• 摘要：在本研究中，我们探索了人工智能和深度学习技术，特别是生成对抗网络 (GAN) 和大型语言模型 (LLM)，用于生成合成表格数据的增长潜力。获取高质量的学生数据对于推进学习分析至关重要，但隐私问题和全球更严格的数据保护法规限制了这些数据的可用性和使用。合成数据提供了一种有前途的替代方案。我们研究是否可以利用合成数据来创建人工智能学生以服务于学习分析模型。使用流行的 GAN 模型 CTGAN 和三个 LLM——GPT2、DistilGPT2 和 DialoGPT，我们生成合成表格学生数据。我们的结果证明了这些方法在生成与真实学生数据相似的高质量合成数据集方面具有强大的潜力。为了验证我们的发现，我们应用了一套全面的效用评估指标来评估合成数据的统计和预测性能，并比较了所使用的不同生成器模型，特别是 LLM 的性能。我们的研究旨在为学习分析社区提供有关使用合成数据的宝贵见解，为通过学习分析数据生成的新创新方法扩展领域方法工具箱奠定基础。

Title: JoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing

• Authors: Qili Wang, Dajiang Wu, Zihang Xu, Junshi Huang, Jun Lv
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01798
• Pdf URL: https://arxiv.org/pdf/2501.01798
• Copy Paste: [[2501.01798]] JoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing(https://arxiv.org/abs/2501.01798)
• Keywords: generation
• Abstract: Significant progress has been made in talking-face video generation research; however, precise lip-audio synchronization and high visual quality remain challenging in editing lip shapes based on input audio. This paper introduces JoyGen, a novel two-stage framework for talking-face generation, comprising audio-driven lip motion generation and visual appearance synthesis. In the first stage, a 3D reconstruction model and an audio2motion model predict identity and expression coefficients respectively. Next, by integrating audio features with a facial depth map, we provide comprehensive supervision for precise lip-audio synchronization in facial generation. Additionally, we constructed a Chinese talking-face dataset containing 130 hours of high-quality video. JoyGen is trained on the open-source HDTF dataset and our curated dataset. Experimental results demonstrate superior lip-audio synchronization and visual quality achieved by our method.
• 摘要：说话人脸视频生成研究已经取得了重大进展；然而，在基于输入音频编辑唇形时，精确的唇音同步和高视觉质量仍然具有挑战性。本文介绍了一种新颖的两阶段说话人脸生成框架 JoyGen，包括音频驱动的唇部运动生成和视觉外观合成。在第一阶段，3D 重建模型和音频运动模型分别预测身份和表情系数。接下来，通过将音频特征与面部深度图相结合，我们为面部生成中的精确唇音同步提供全面监督。此外，我们构建了一个包含 130 小时高质量视频的中文说话人脸数据集。JoyGen 在开源 HDTF 数据集和我们精选的数据集上进行训练。实验结果表明，我们的方法实现了卓越的唇音同步和视觉质量。

Title: MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animation

• Authors: Huaize Liu, Wenzhang Sun, Donglin Di, Shibo Sun, Jiahui Yang, Changqing Zou, Hujun Bao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01808
• Pdf URL: https://arxiv.org/pdf/2501.01808
• Copy Paste: [[2501.01808]] MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animation(https://arxiv.org/abs/2501.01808)
• Keywords: generation
• Abstract: The generation of talking avatars has achieved significant advancements in precise audio synchronization. However, crafting lifelike talking head videos requires capturing a broad spectrum of emotions and subtle facial expressions. Current methods face fundamental challenges: a)the absence of frameworks for modeling single basic emotional expressions, which restricts the generation of complex emotions such as compound emotions; b)the lack of comprehensive datasets rich in human emotional expressions, which limits the potential of models. To address these challenges, we propose the following innovations: 1)the Mixture of Emotion Experts (MoEE) model, which decouples six fundamental emotions to enable the precise synthesis of both singular and compound emotional states; 2)the DH-FaceEmoVid-150 dataset, specifically curated to include six prevalent human emotional expressions as well as four types of compound emotions, thereby expanding the training potential of emotion-driven models. Furthermore, to enhance the flexibility of emotion control, we propose an emotion-to-latents module that leverages multimodal inputs, aligning diverse control signals-such as audio, text, and labels-to ensure more varied control inputs as well as the ability to control emotions using audio alone. Through extensive quantitative and qualitative evaluations, we demonstrate that the MoEE framework, in conjunction with the DH-FaceEmoVid-150 dataset, excels in generating complex emotional expressions and nuanced facial details, setting a new benchmark in the field. These datasets will be publicly released.
• 摘要：会说话的虚拟人物的生成在精确的音频同步方面取得了重大进步。然而，制作栩栩如生的会说话的头部视频需要捕捉广泛的情绪和微妙的面部表情。当前的方法面临着根本性的挑战：a）缺乏用于建模单一基本情绪表达的框架，这限制了复合情绪等复杂情绪的生成；b）缺乏富含人类情绪表达的综合数据集，这限制了模型的潜力。为了应对这些挑战，我们提出了以下创新：1）情绪专家混合 (MoEE) 模型，该模型将六种基本情绪分离，以实现单一情绪状态和复合情绪状态的精确合成；2）DH-FaceEmoVid-150 数据集，专门包括六种常见的人类情绪表达以及四种复合情绪，从而扩展了情绪驱动模型的训练潜力。此外，为了提高情绪控制的灵活性，我们提出了一个情绪到潜意识模块，该模块利用多模态输入，协调各种控制信号（例如音频、文本和标签），以确保更多样化的控制输入以及仅使用音频控制情绪的能力。通过广泛的定量和定性评估，我们证明了 MoEE 框架与 DH-FaceEmoVid-150 数据集相结合，在生成复杂的情绪表情和细微的面部细节方面表现出色，为该领域树立了新的标杆。这些数据集将公开发布。

Title: MoColl: Agent-Based Specific and General Model Collaboration for Image Captioning

• Authors: Pu Yang, Bin Dong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01834
• Pdf URL: https://arxiv.org/pdf/2501.01834
• Copy Paste: [[2501.01834]] MoColl: Agent-Based Specific and General Model Collaboration for Image Captioning(https://arxiv.org/abs/2501.01834)
• Keywords: generation
• Abstract: Image captioning is a critical task at the intersection of computer vision and natural language processing, with wide-ranging applications across various domains. For complex tasks such as diagnostic report generation, deep learning models require not only domain-specific image-caption datasets but also the incorporation of relevant general knowledge to provide contextual accuracy. Existing approaches exhibit inherent limitations: specialized models excel in capturing domain-specific details but lack generalization, while vision-language models (VLMs) built on large language models (LLMs) leverage general knowledge but struggle with domain-specific adaptation. To address these limitations, this paper proposes a novel agent-enhanced model collaboration framework, which we called \textbf{MoColl}, designed to effectively integrate domain-specific and general knowledge. Specifically, our approach is to decompose complex image captioning tasks into a series of interconnected question-answer subtasks. A trainable visual question answering (VQA) model is employed as a specialized tool to focus on domain-specific visual analysis, answering task-specific questions based on image content. Concurrently, an LLM-based agent with general knowledge formulates these questions and synthesizes the resulting question-answer pairs into coherent captions. Beyond its role in leveraging the VQA model, the agent further guides its training to enhance its domain-specific capabilities. Experimental results on radiology report generation validate the effectiveness of the proposed framework, demonstrating significant improvements in the quality of generated reports.
• 摘要：图像字幕制作是计算机视觉和自然语言处理交叉领域的一项关键任务，在各个领域都有广泛的应用。对于诊断报告生成等复杂任务，深度学习模型不仅需要特定领域的图像字幕数据集，还需要结合相关的一般知识来提供上下文准确性。现有方法表现出固有的局限性：专门的模型擅长捕捉特定领域的细节，但缺乏泛化能力，而建立在大型语言模型 (LLM) 上的视觉语言模型 (VLM) 利用一般知识，但在特定领域的适应性方面却举步维艰。为了解决这些限制，本文提出了一种新颖的代理增强模型协作框架，我们称之为 \textbf{MoColl}，旨在有效地整合领域特定知识和一般知识。具体来说，我们的方法是将复杂的图像字幕制作任务分解为一系列相互关联的问答子任务。可训练的视觉问答 (VQA) 模型被用作一种专门的工具，专注于特定领域的视觉分析，根据图像内容回答特定任务的问题。同时，具有一般知识的 LLM 代理会制定这些问题，并将生成的问答对合成为连贯的字幕。除了利用 VQA 模型的作用外，代理还会进一步指导其训练以增强其特定领域的能力。放射学报告生成的实验结果验证了所提框架的有效性，表明生成的报告质量显著提高。

Title: Learning from Ambiguous Data with Hard Labels

• Authors: Zeke Xie, Zheng He, Nan Lu, Lichen Bai, Bao Li, Shuo Yang, Mingming Sun, Ping Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01844
• Pdf URL: https://arxiv.org/pdf/2501.01844
• Copy Paste: [[2501.01844]] Learning from Ambiguous Data with Hard Labels(https://arxiv.org/abs/2501.01844)
• Keywords: generation
• Abstract: Real-world data often contains intrinsic ambiguity that the common single-hard-label annotation paradigm ignores. Standard training using ambiguous data with these hard labels may produce overly confident models and thus leading to poor generalization. In this paper, we propose a novel framework called Quantized Label Learning (QLL) to alleviate this issue. First, we formulate QLL as learning from (very) ambiguous data with hard labels: ideally, each ambiguous instance should be associated with a ground-truth soft-label distribution describing its corresponding probabilistic weight in each class, however, this is usually not accessible; in practice, we can only observe a quantized label, i.e., a hard label sampled (quantized) from the corresponding ground-truth soft-label distribution, of each instance, which can be seen as a biased approximation of the ground-truth soft-label. Second, we propose a Class-wise Positive-Unlabeled (CPU) risk estimator that allows us to train accurate classifiers from only ambiguous data with quantized labels. Third, to simulate ambiguous datasets with quantized labels in the real world, we design a mixing-based ambiguous data generation procedure for empirical evaluation. Experiments demonstrate that our CPU method can significantly improve model generalization performance and outperform the baselines.
• 摘要：现实世界的数据通常包含常见的单硬标签注释范式所忽略的内在模糊性。使用具有这些硬标签的模糊数据进行标准训练可能会产生过于自信的模型，从而导致泛化能力差。在本文中，我们提出了一种称为量化标签学习 (QLL) 的新框架来缓解这一问题。首先，我们将 QLL 表述为从具有硬标签的 (非常) 模糊数据中学习：理想情况下，每个模糊实例都应与一个描述其在每个类中对应概率权重的真实软标签分布相关联，然而，这通常是无法实现的；在实践中，我们只能观察到每个实例的量化标签，即从相应的真实软标签分布中采样 (量化) 的硬标签，这可以看作是真实软标签的有偏近似。其次，我们提出了一个逐类正无标签 (CPU) 风险估计器，它使我们能够仅从具有量化标签的模糊数据中训练准确的分类器。第三，为了模拟现实世界中带有量化标签的模糊数据集，我们设计了一个基于混合的模糊数据生成程序进行实证评估。实验表明，我们的 CPU 方法可以显著提高模型泛化性能并超越基线。

Title: Transformer-Driven Inverse Problem Transform for Fast Blind Hyperspectral Image Dehazing

• Authors: Po-Wei Tang, Chia-Hsiang Lin, Yangrui Liu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.01924
• Pdf URL: https://arxiv.org/pdf/2501.01924
• Copy Paste: [[2501.01924]] Transformer-Driven Inverse Problem Transform for Fast Blind Hyperspectral Image Dehazing(https://arxiv.org/abs/2501.01924)
• Keywords: super-resolution
• Abstract: Hyperspectral dehazing (HyDHZ) has become a crucial signal processing technology to facilitate the subsequent identification and classification tasks, as the airborne visible/infrared imaging spectrometer (AVIRIS) data portal reports a massive portion of haze-corrupted areas in typical hyperspectral remote sensing images. The idea of inverse problem transform (IPT) has been proposed in recent remote sensing literature in order to reformulate a hardly tractable inverse problem (e.g., HyDHZ) into a relatively simple one. Considering the emerging spectral super-resolution (SSR) technique, which spectrally upsamples multispectral data to hyperspectral data, we aim to solve the challenging HyDHZ problem by reformulating it as an SSR problem. Roughly speaking, the proposed algorithm first automatically selects some uncorrupted/informative spectral bands, from which SSR is applied to spectrally upsample the selected bands in the feature space, thereby obtaining a clean hyperspectral image (HSI). The clean HSI is then further refined by a deep transformer network to obtain the final dehazed HSI, where a global attention mechanism is designed to capture nonlocal information. There are very few HyDHZ works in existing literature, and this article introduces the powerful spatial-spectral transformer into HyDHZ for the first time. Remarkably, the proposed transformer-driven IPT-based HyDHZ (T2HyDHZ) is a blind algorithm without requiring the user to manually select the corrupted region. Extensive experiments demonstrate the superiority of T2HyDHZ with less color distortion.
• 摘要：高光谱去雾 (HyDHZ) 已成为一种重要的信号处理技术，有助于后续的识别和分类任务，因为机载可见光/红外成像光谱仪 (AVIRIS) 数据门户报告了典型的高光谱遥感图像中大量被雾霾破坏的区域。最近的遥感文献中提出了逆问题变换 (IPT) 的概念，以便将难以处理的逆问题 (例如 HyDHZ) 重新表述为相对简单的问题。考虑到新兴的光谱超分辨率 (SSR) 技术，该技术将多光谱数据光谱上采样为高光谱数据，我们旨在通过将具有挑战性的 HyDHZ 问题重新表述为 SSR 问题来解决它。粗略地说，所提出的算法首先自动选择一些未损坏/信息丰富的光谱带，从中应用 SSR 对特征空间中的选定波段进行光谱上采样，从而获得干净的高光谱图像 (HSI)。然后，通过深度变换器网络进一步细化干净的 HSI，以获得最终的去雾 HSI，其中设计了全局注意机制来捕获非局部信息。现有文献中关于 HyDHZ 的研究很少，本文首次将强大的空间光谱变换器引入 HyDHZ。值得注意的是，所提出的基于变换器驱动的 IPT 的 HyDHZ (T2HyDHZ) 是一种盲算法，无需用户手动选择损坏区域。大量实验证明了 T2HyDHZ 的优越性，色彩失真更少。

Title: Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding

• Authors: Jiaming Li, Jiacheng Zhang, Zequn Jie, Lin Ma, Guanbin Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01926
• Pdf URL: https://arxiv.org/pdf/2501.01926
• Copy Paste: [[2501.01926]] Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding(https://arxiv.org/abs/2501.01926)
• Keywords: generation
• Abstract: Large vision-language models (LVLMs) have shown remarkable capabilities in visual-language understanding for downstream multi-modal tasks. Despite their success, LVLMs still suffer from generating hallucinations in complex generation tasks, leading to inconsistencies between visual inputs and generated content. To address this issue, some approaches have introduced inference-time interventions, such as contrastive decoding and attention rectification, to reduce overreliance on language priors. However, these approaches overlook hallucinations stemming from spurious inter-modality correlations. In this paper, we propose an Inter-Modality Correlation Calibration Decoding (IMCCD) method to mitigate hallucinations in LVLMs in a training-free manner. In this method, we design a Cross-Modal Value-Enhanced Decoding(CMVED) module to alleviate hallucination by a novel contrastive decoding mechanism. During the estimation of distorted distribution, CMVED masks the value vectors associated with significant cross-modal attention weights, which address both uni-modality overreliance and misleading inter-modality correlations. Additionally, a Content-Driven Attention Refinement(CDAR) module refines cross-modal attention weights, guiding LVLMs to focus on important visual content. Experimental results on diverse hallucination benchmarks validate the superiority of our method over existing state-of-the-art techniques in reducing hallucinations in LVLM text generation. Our code will be available at this https URL.
• 摘要：大型视觉语言模型 (LVLM) 在下游多模态任务的视觉语言理解方面表现出了卓越的能力。尽管取得了成功，但 LVLM 仍然存在于复杂的生成任务中产生幻觉的问题，导致视觉输入和生成内容之间不一致。为了解决这个问题，一些方法引入了推理时间干预，例如对比解码和注意力纠正，以减少对语言先验的过度依赖。然而，这些方法忽视了由虚假的模态间相关性引起的幻觉。在本文中，我们提出了一种模态间相关性校准解码 (IMCCD) 方法，以无需训练的方式减轻 LVLM 中的幻觉。在这种方法中，我们设计了一个跨模态值增强解码 (CMVED) 模块，通过一种新颖的对比解码机制来缓解幻觉。在估计扭曲分布的过程中，CMVED 会掩盖与显著的跨模态注意权重相关的值向量，从而解决单模态过度依赖和误导性模态间相关性问题。此外，内容驱动的注意力细化 (CDAR) 模块会细化跨模态注意权重，引导 LVLM 专注于重要的视觉内容。在各种幻觉基准上进行的实验结果验证了我们的方法在减少 LVLM 文本生成中的幻觉方面优于现有的最先进技术。我们的代码将在此 https URL 上提供。

Title: MADGEN -- Mass-Spec attends to De Novo Molecular generation

• Authors: Yinkai Wang, Xiaohui Chen, Liping Liu, Soha Hassoun
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01950
• Pdf URL: https://arxiv.org/pdf/2501.01950
• Copy Paste: [[2501.01950]] MADGEN -- Mass-Spec attends to De Novo Molecular generation(https://arxiv.org/abs/2501.01950)
• Keywords: generation, generative
• Abstract: The annotation (assigning structural chemical identities) of MS/MS spectra remains a significant challenge due to the enormous molecular diversity in biological samples and the limited scope of reference databases. Currently, the vast majority of spectral measurements remain in the "dark chemical space" without structural annotations. To improve annotation, we propose MADGEN (Mass-spec Attends to De Novo Molecular GENeration), a scaffold-based method for de novo molecular structure generation guided by mass spectrometry data. MADGEN operates in two stages: scaffold retrieval and spectra-conditioned molecular generation starting with the scaffold. In the first stage, given an MS/MS spectrum, we formulate scaffold retrieval as a ranking problem and employ contrastive learning to align mass spectra with candidate molecular scaffolds. In the second stage, starting from the retrieved scaffold, we employ the MS/MS spectrum to guide an attention-based generative model to generate the final molecule. Our approach constrains the molecular generation search space, reducing its complexity and improving generation accuracy. We evaluate MADGEN on three datasets (NIST23, CANOPUS, and MassSpecGym) and evaluate MADGEN's performance with a predictive scaffold retriever and with an oracle retriever. We demonstrate the effectiveness of using attention to integrate spectral information throughout the generation process to achieve strong results with the oracle retriever.
• 摘要：由于生物样本中分子多样性巨大，参考数据库范围有限，MS/MS 光谱的注释（分配结构化学身份）仍然是一项重大挑战。目前，绝大多数光谱测量仍处于没有结构注释的“暗化学空间”中。为了改进注释，我们提出了 MADGEN（质谱关注从头分子生成），这是一种基于支架的方法，用于由质谱数据引导的从头分子结构生成。MADGEN 分为两个阶段：支架检索和从支架开始的光谱条件分子生成。在第一阶段，给定一个 MS/MS 光谱，我们将支架检索公式化为排序问题，并使用对比学习将质谱与候选分子支架对齐。在第二阶段，从检索到的支架开始，我们使用 MS/MS 光谱来指导基于注意力的生成模型来生成最终分子。我们的方法限制了分子生成搜索空间，降低了其复杂性并提高了生成准确性。我们在三个数据集（NIST23、CANO​​PUS 和 MassSpecGym）上评估了 MADGEN，并使用预测支架检索器和 oracle 检索器评估了 MADGEN 的性能。我们证明了使用注意力在整个生成过程中整合光谱信息的有效性，从而使用 oracle 检索器取得了出色的结果。