2025-01-28

Title: Measuring and Mitigating Hallucinations in Vision-Language Dataset Generation for Remote Sensing

• Authors: Madeline Anderson, Miriam Cha, William T. Freeman, J. Taylor Perron, Nathaniel Maidel, Kerri Cahoy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14905
• Pdf URL: https://arxiv.org/pdf/2501.14905
• Copy Paste: [[2501.14905]] Measuring and Mitigating Hallucinations in Vision-Language Dataset Generation for Remote Sensing(https://arxiv.org/abs/2501.14905)
• Keywords: generation
• Abstract: Vision language models have achieved impressive results across various fields. However, adoption in remote sensing remains limited, largely due to the scarcity of paired image-text data. To bridge this gap, synthetic caption generation has gained interest, traditionally relying on rule-based methods that use metadata or bounding boxes. While these approaches provide some description, they often lack the depth needed to capture complex wide-area scenes. Large language models (LLMs) offer a promising alternative for generating more descriptive captions, yet they can produce generic outputs and are prone to hallucination. In this paper, we propose a new method to enhance vision-language datasets for remote sensing by integrating maps as external data sources, enabling the generation of detailed, context-rich captions. Additionally, we present methods to measure and mitigate hallucinations in LLM-generated text. We introduce fMoW-mm, a multimodal dataset incorporating satellite imagery, maps, metadata, and text annotations. We demonstrate its effectiveness for automatic target recognition in few-shot settings, achieving superior performance compared to other vision-language remote sensing datasets.
• 摘要：视觉语言模型在各个领域都取得了令人瞩目的成果。然而，它在遥感领域的应用仍然有限，这主要是由于成对的图像文本数据稀缺。为了弥补这一差距，合成字幕生成引起了人们的兴趣，传统上依赖于使用元数据或边界框的基于规则的方法。虽然这些方法提供了一些描述，但它们往往缺乏捕捉复杂广域场景所需的深度。大型语言模型 (LLM) 为生成更具描述性的字幕提供了一种有前途的替代方案，但它们可以产生通用输出并且容易产生幻觉。在本文中，我们提出了一种新方法，通过将地图集成为外部数据源来增强遥感的视觉语言数据集，从而能够生成详细、上下文丰富的字幕。此外，我们还提出了测量和减轻 LLM 生成的文本中幻觉的方法。我们介绍了 fMoW-mm，这是一个包含卫星图像、地图、元数据和文本注释的多模态数据集。我们证明了它在小样本设置中自动目标识别的有效性，与其他视觉语言遥感数据集相比，其性能更出色。

Title: E-Gen: Leveraging E-Graphs to Improve Continuous Representations of Symbolic Expressions

• Authors: Hongbo Zheng, Suyuan Wang, Neeraj Gangwar, Nickvash Kani
• Subjects: cs.LG, cs.CL, cs.SC
• Abstract URL: https://arxiv.org/abs/2501.14951
• Pdf URL: https://arxiv.org/pdf/2501.14951
• Copy Paste: [[2501.14951]] E-Gen: Leveraging E-Graphs to Improve Continuous Representations of Symbolic Expressions(https://arxiv.org/abs/2501.14951)
• Keywords: generation
• Abstract: As vector representations have been pivotal in advancing natural language processing (NLP), some prior research has concentrated on creating embedding techniques for mathematical expressions by leveraging mathematically equivalent expressions. While effective, these methods are limited by the training data. In this work, we propose augmenting prior algorithms with larger synthetic dataset, using a novel e-graph-based generation scheme. This new mathematical dataset generation scheme, E-Gen, improves upon prior dataset-generation schemes that are limited in size and operator types. We use this dataset to compare embedding models trained with two methods: (1) training the model to generate mathematically equivalent expressions, and (2) training the model using contrastive learning to group mathematically equivalent expressions explicitly. We evaluate the embeddings generated by these methods against prior work on both in-distribution and out-of-distribution language processing tasks. Finally, we compare the performance of our embedding scheme against state-of-the-art large language models and demonstrate that embedding-based language processing methods perform better than LLMs on several tasks, demonstrating the necessity of optimizing embedding methods for the mathematical data modality.
• 摘要：由于向量表示在推进自然语言处理 (NLP) 方面发挥着关键作用，一些先前的研究集中在通过利用数学上等价的表达式来创建数学表达式的嵌入技术。这些方法虽然有效，但受到训练数据的限制。在这项工作中，我们建议使用一种新的基于 e-graph 的生成方案，用更大的合成数据集来增强先前的算法。这种新的数学数据集生成方案 E-Gen 改进了先前在大小和运算符类型上有限的数据集生成方案。我们使用该数据集比较用两种方法训练的嵌入模型：(1) 训练模型以生成数学上等价的表达式，以及 (2) 使用对比学习训练模型以明确对数学上等价的表达式进行分组。我们根据分布内和分布外语言处理任务的先前工作来评估这些方法生成的嵌入。最后，我们将我们的嵌入方案的性能与最先进的大型语言模型进行了比较，并证明基于嵌入的语言处理方法在多个任务上的表现优于 LLM，证明了针对数学数据模态优化嵌入方法的必要性。

Title: GreenAuto: An Automated Platform for Sustainable AI Model Design on Edge Devices

• Authors: Xiaolong Tu, Dawei Chen, Kyungtae Han, Onur Altintas, Haoxin Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14995
• Pdf URL: https://arxiv.org/pdf/2501.14995
• Copy Paste: [[2501.14995]] GreenAuto: An Automated Platform for Sustainable AI Model Design on Edge Devices(https://arxiv.org/abs/2501.14995)
• Keywords: generation
• Abstract: We present GreenAuto, an end-to-end automated platform designed for sustainable AI model exploration, generation, deployment, and evaluation. GreenAuto employs a Pareto front-based search method within an expanded neural architecture search (NAS) space, guided by gradient descent to optimize model exploration. Pre-trained kernel-level energy predictors estimate energy consumption across all models, providing a global view that directs the search toward more sustainable solutions. By automating performance measurements and iteratively refining the search process, GreenAuto demonstrates the efficient identification of sustainable AI models without the need for human intervention.
• 摘要：我们推出了 GreenAuto，这是一个端到端自动化平台，专为可持续 AI 模型探索、生成、部署和评估而设计。GreenAuto 在扩展的神经架构搜索 (NAS) 空间内采用基于帕累托前沿的搜索方法，以梯度下降为指导来优化模型探索。预先训练的内核级能量预测器可估算所有模型的能耗，提供全局视图，引导搜索寻找更可持续的解决方案。通过自动化性能测量和迭代优化搜索过程，GreenAuto 展示了无需人工干预即可高效识别可持续 AI 模型。

Title: HuGDiffusion: Generalizable Single-Image Human Rendering via 3D Gaussian Diffusion

• Authors: Yingzhi Tang, Qijian Zhang, Junhui Hou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15008
• Pdf URL: https://arxiv.org/pdf/2501.15008
• Copy Paste: [[2501.15008]] HuGDiffusion: Generalizable Single-Image Human Rendering via 3D Gaussian Diffusion(https://arxiv.org/abs/2501.15008)
• Keywords: generation
• Abstract: We present HuGDiffusion, a generalizable 3D Gaussian splatting (3DGS) learning pipeline to achieve novel view synthesis (NVS) of human characters from single-view input images. Existing approaches typically require monocular videos or calibrated multi-view images as inputs, whose applicability could be weakened in real-world scenarios with arbitrary and/or unknown camera poses. In this paper, we aim to generate the set of 3DGS attributes via a diffusion-based framework conditioned on human priors extracted from a single image. Specifically, we begin with carefully integrated human-centric feature extraction procedures to deduce informative conditioning signals. Based on our empirical observations that jointly learning the whole 3DGS attributes is challenging to optimize, we design a multi-stage generation strategy to obtain different types of 3DGS attributes. To facilitate the training process, we investigate constructing proxy ground-truth 3D Gaussian attributes as high-quality attribute-level supervision signals. Through extensive experiments, our HuGDiffusion shows significant performance improvements over the state-of-the-art methods. Our code will be made publicly available.
• 摘要：我们提出了 HuGDiffusion，这是一种可推广的 3D 高斯分层 (3DGS) 学习管道，用于从单视图输入图像实现人物角色的新型视图合成 (NVS)。现有方法通常需要单目视频或校准的多视图图像作为输入，在具有任意和/或未知相机姿势的现实场景中，其适用性可能会减弱。在本文中，我们旨在通过基于扩散的框架生成 3DGS 属性集，该框架以从单个图像中提取的人类先验为条件。具体来说，我们首先从精心整合的以人为中心的特征提取程序开始，以推断出信息丰富的条件信号。根据我们的经验观察，联合学习整个 3DGS 属性很难优化，我们设计了一种多阶段生成策略来获得不同类型的 3DGS 属性。为了促进训练过程，我们研究构建代理地面实况 3D 高斯属性作为高质量属性级监督信号。通过大量实验，我们的 HuGDiffusion 显示出比最先进的方法显着的性能改进。我们的代码将会公开。

Title: Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities

• Authors: Shounak Datta, Dhanasekar Sundararaman
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.15046
• Pdf URL: https://arxiv.org/pdf/2501.15046
• Copy Paste: [[2501.15046]] Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities(https://arxiv.org/abs/2501.15046)
• Keywords: generation
• Abstract: Despite their impressive performance on multi-modal tasks, large vision-language models (LVLMs) tend to suffer from hallucinations. An important type is object hallucination, where LVLMs generate objects that are inconsistent with the images shown to the model. Existing works typically attempt to quantify object hallucinations by detecting and measuring the fraction of hallucinated objects in generated captions. Additionally, more recent work also measures object hallucinations by directly querying the LVLM with binary questions about the presence of likely hallucinated objects based on object statistics like top-k frequent objects and top-k co-occurring objects. In this paper, we present Context-Aware Object Similarities (CAOS), a novel approach for evaluating object hallucination in LVLMs using object statistics as well as the generated captions. CAOS uniquely integrates object statistics with semantic relationships between objects in captions and ground-truth data. Moreover, existing approaches usually only detect and measure hallucinations belonging to a predetermined set of in-domain objects (typically the set of all ground-truth objects for the training dataset) and ignore generated objects that are not part of this set, leading to under-evaluation. To address this, we further employ language model--based object recognition to detect potentially out-of-domain hallucinated objects and use an ensemble of LVLMs for verifying the presence of such objects in the query image. CAOS also examines the sequential dynamics of object generation, shedding light on how the order of object appearance influences hallucinations, and employs word embedding models to analyze the semantic reasons behind hallucinations. CAOS aims to offer a nuanced understanding of the hallucination tendencies of LVLMs by providing a systematic framework to identify and interpret object hallucinations.
• 摘要：尽管大型视觉语言模型 (LVLM) 在多模态任务上的表现令人印象深刻，但它们往往会产生幻觉。一种重要的幻觉类型是物体幻觉，LVLM 生成的物体与模型所见的图像不一致。现有研究通常试图通过检测和测量生成的字幕中幻觉物体的比例来量化物体幻觉。此外，最近的研究还通过直接向 LVLM 查询二元问题来测量物体幻觉，该问题基于物体统计数据（如前 k 个频繁物体和前 k 个共现物体）来确定可能产生幻觉的物体的存在。在本文中，我们提出了上下文感知物体相似性 (CAOS)，这是一种使用物体统计数据以及生成的字幕来评估 LVLM 中物体幻觉的新方法。CAOS 独特地将物体统计数据与字幕和地面实况数据中物体之间的语义关系相结合。此外，现有方法通常仅检测和测量属于预定域内对象集（通常是训练数据集的所有地面实况对象集）的幻觉，而忽略不属于该集的生成对象，从而导致评估不足。为了解决这个问题，我们进一步采用基于语言模型的对象识别来检测可能超出域外的幻觉对象，并使用 LVLM 集合来验证查询图像中是否存在此类对象。CAOS 还研究了对象生成的顺序动态，揭示了对象出现的顺序如何影响幻觉，并使用词嵌入模型来分析幻觉背后的语义原因。CAOS 旨在通过提供识别和解释对象幻觉的系统框架，提供对 LVLM 幻觉倾向的细致入微的理解。

Title: Exploring the impact of Optimised Hyperparameters on Bi-LSTM-based Contextual Anomaly Detector

• Authors: Aafan Ahmad Toor, Jia-Chun Lin, Ernst Gunnar Gran
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.15053
• Pdf URL: https://arxiv.org/pdf/2501.15053
• Copy Paste: [[2501.15053]] Exploring the impact of Optimised Hyperparameters on Bi-LSTM-based Contextual Anomaly Detector(https://arxiv.org/abs/2501.15053)
• Keywords: generation
• Abstract: The exponential growth in the usage of Internet of Things in daily life has caused immense increase in the generation of time series data. Smart homes is one such domain where bulk of data is being generated and anomaly detection is one of the many challenges addressed by researchers in recent years. Contextual anomaly is a kind of anomaly that may show deviation from the normal pattern like point or sequence anomalies, but it also requires prior knowledge about the data domain and the actions that caused the deviation. Recent studies based on Recurrent Neural Networks (RNN) have demonstrated strong performance in anomaly detection. This study explores the impact of automatically tuned hyperparamteres on Unsupervised Online Contextual Anomaly Detection (UoCAD) approach by proposing UoCAD with Optimised Hyperparamnters (UoCAD-OH). UoCAD-OH conducts hyperparameter optimisation on Bi-LSTM model in an offline phase and uses the fine-tuned hyperparameters to detect anomalies during the online phase. The experiments involve evaluating the proposed framework on two smart home air quality datasets containing contextual anomalies. The evaluation metrics used are Precision, Recall, and F1 score.
• 摘要：物联网在日常生活中的使用呈指数级增长，导致时间序列数据的生成量大幅增加。智能家居就是这样一个产生大量数据的领域，异常检测是近年来研究人员面临的众多挑战之一。上下文异常是一种可能偏离正常模式的异常，如点异常或序列异常，但它也需要关于数据域和导致偏差的操作的先验知识。最近基于循环神经网络 (RNN) 的研究在异常检测方面表现出色。本研究通过提出具有优化超参数的 UoCAD（UoCAD-OH），探讨了自动调整超参数对无监督在线上下文异常检测 (UoCAD) 方法的影响。UoCAD-OH 在离线阶段对 Bi-LSTM 模型进行超参数优化，并使用微调后的超参数在在线阶段检测异常。实验涉及在两个包含上下文异常的智能家居空气质量数据集上评估所提出的框架。使用的评估指标是准确率、召回率和 F1 分数。

Title: KETA: Kinematic-Phrases-Enhanced Text-to-Motion Generation via Fine-grained Alignment

• Authors: Yu Jiang, Yixing Chen, Xingyang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15058
• Pdf URL: https://arxiv.org/pdf/2501.15058
• Copy Paste: [[2501.15058]] KETA: Kinematic-Phrases-Enhanced Text-to-Motion Generation via Fine-grained Alignment(https://arxiv.org/abs/2501.15058)
• Keywords: generation
• Abstract: Motion synthesis plays a vital role in various fields of artificial intelligence. Among the various conditions of motion generation, text can describe motion details elaborately and is easy to acquire, making text-to-motion(T2M) generation important. State-of-the-art T2M techniques mainly leverage diffusion models to generate motions with text prompts as guidance, tackling the many-to-many nature of T2M tasks. However, existing T2M approaches face challenges, given the gap between the natural language domain and the physical domain, making it difficult to generate motions fully consistent with the texts. We leverage kinematic phrases(KP), an intermediate representation that bridges these two modalities, to solve this. Our proposed method, KETA, decomposes the given text into several decomposed texts via a language model. It trains an aligner to align decomposed texts with the KP segments extracted from the generated motions. Thus, it's possible to restrict the behaviors for diffusion-based T2M models. During the training stage, we deploy the text-KP alignment loss as an auxiliary goal to supervise the models. During the inference stage, we refine our generated motions for multiple rounds in our decoder structure, where we compute the text-KP distance as the guidance signal in each new round. Experiments demonstrate that KETA achieves up to 1.19x, 2.34x better R precision and FID value on both backbones of the base model, motion diffusion model. Compared to a wide range of T2M generation models. KETA achieves either the best or the second-best performance.
• 摘要：动作合成在人工智能的各个领域都发挥着重要作用。在各种动作生成条件中，文本可以精细地描述动作细节并且易于获取，因此文本到动作 (T2M) 生成非常重要。最先进的 T2M 技术主要利用扩散模型以文本提示为指导生成动作，以解决 T2M 任务的多对多性质。然而，鉴于自然语言领域和物理领域之间的差距，现有的 T2M 方法面临挑战，难以生成与文本完全一致的动作。我们利用运动短语 (KP)（一种连接这两种模态的中间表示）来解决这个问题。我们提出的方法 KETA 通过语言模型将给定的文本分解为几个分解的文本。它训练一个对齐器来将分解的文本与从生成的动作中提取的 KP 段对齐。因此，可以限制基于扩散的 T2M 模型的行为。在训练阶段，我们部署文本-KP 对齐损失作为辅助目标来监督模型。在推理阶段，我们在解码器结构中对生成的动作进行多轮优化，其中我们计算文本-KP 距离作为每个新轮次的指导信号。实验表明，KETA 在基础模型、运动扩散模型的两个主干上实现了高达 1.19 倍、2.34 倍的 R 精度和 FID 值。与各种 T2M 生成模型相比，KETA 实现了最佳或第二好的表现。

Title: CFT-RAG: An Entity Tree Based Retrieval Augmented Generation Algorithm With Cuckoo Filter

• Authors: Zihang Li, Yangdong Ruan, Wenjun Liu, Zhengyang Wang, Tong Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.15098
• Pdf URL: https://arxiv.org/pdf/2501.15098
• Copy Paste: [[2501.15098]] CFT-RAG: An Entity Tree Based Retrieval Augmented Generation Algorithm With Cuckoo Filter(https://arxiv.org/abs/2501.15098)
• Keywords: generation, generative
• Abstract: Although retrieval-augmented generation(RAG) significantly improves generation quality by retrieving external knowledge bases and integrating generated content, it faces computational efficiency bottlenecks, particularly in knowledge retrieval tasks involving hierarchical structures for Tree-RAG. This paper proposes a Tree-RAG acceleration method based on the improved Cuckoo Filter, which optimizes entity localization during the retrieval process to achieve significant performance improvements. Tree-RAG effectively organizes entities through the introduction of a hierarchical tree structure, while the Cuckoo Filter serves as an efficient data structure that supports rapid membership queries and dynamic updates. The experiment results demonstrate that our method is much faster than naive Tree-RAG while maintaining high levels of generative quality. When the number of trees is large, our method is hundreds of times faster than naive Tree-RAG. Our work is available at this https URL.
• 摘要：虽然检索增强生成 (RAG) 通过检索外部知识库和集成生成内容显著提高了生成质量，但它面临计算效率瓶颈，特别是在涉及 Tree-RAG 层次结构的知识检索任务中。本文提出了一种基于改进的 Cuckoo Filter 的 Tree-RAG 加速方法，该方法优化了检索过程中的实体定位，从而显著提高了性能。Tree-RAG 通过引入层次树结构有效地组织实体，而 Cuckoo Filter 是一种高效的数据结构，支持快速成员查询和动态更新。实验结果表明，我们的方法在保持高水平生成质量的同时比朴素的 Tree-RAG 快得多。当树的数量很大时，我们的方法比朴素的 Tree-RAG 快数百倍。我们的工作可以在这个 https URL 上找到。

Title: SpikSSD: Better Extraction and Fusion for Object Detection with Spiking Neuron Networks

• Authors: Yimeng Fan, Chagsong Liu, Mingyang Li, Wei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15151
• Pdf URL: https://arxiv.org/pdf/2501.15151
• Copy Paste: [[2501.15151]] SpikSSD: Better Extraction and Fusion for Object Detection with Spiking Neuron Networks(https://arxiv.org/abs/2501.15151)
• Keywords: generation
• Abstract: As the third generation of neural networks, Spiking Neural Networks (SNNs) have gained widespread attention due to their low energy consumption and biological interpretability. Recently, SNNs have made considerable advancements in computer vision. However, efficiently conducting feature extraction and fusion under the spiking characteristics of SNNs for object detection remains a pressing challenge. To address this problem, we propose the SpikSSD, a novel Spiking Single Shot Multibox Detector. Specifically, we design a full-spiking backbone network, MDS-ResNet, which effectively adjusts the membrane synaptic input distribution at each layer, achieving better spiking feature extraction. Additionally, for spiking feature fusion, we introduce the Spiking Bi-direction Fusion Module (SBFM), which for the first time realizes bi-direction fusion of spiking features, enhancing the multi-scale detection capability of the model. Experimental results show that SpikSSD achieves 40.8\% mAP on the GEN1 dataset, 76.3\% and 52.4\% mAP@0.5 on VOC 2007 and COCO 2017 datasets respectively with the lowest firing rate, outperforming existing SNN-based approaches at ultralow energy consumption. This work sets a new benchmark for future research in SNN-based object detection. Our code is publicly available in this https URL.
• 摘要：作为第三代神经网络，脉冲神经网络 (SNN) 因其能耗低、生物可解释等特点受到了广泛关注。近年来，SNN 在计算机视觉领域取得了长足的进步。然而，如何在 SNN 的脉冲特性下高效地进行特征提取与融合以进行目标检测仍然是一个迫切的挑战。针对这一问题，我们提出了一种新型的脉冲单次多框检测器 SpikSSD。具体而言，我们设计了一个全脉冲主干网络 MDS-ResNet，有效地调整了各层的膜突触输入分布，实现了更好的脉冲特征提取。此外，对于脉冲特征融合，我们引入了脉冲双向融合模块 (SBFM)，首次实现了脉冲特征的双向融合，增强了模型的多尺度检测能力。实验结果表明，SpikSSD 在 GEN1 数据集上实现了 40.8% 的 mAP，在 VOC 2007 和 COCO 2017 数据集上分别实现了 76.3% 和 52.4% 的 mAP@0.5，并且具有最低的触发率，在超低能耗下优于现有的基于 SNN 的方法。这项工作为基于 SNN 的物体检测的未来研究树立了新的标杆。我们的代码在此 https URL 中公开提供。

Title: Enhancing Intent Understanding for Ambiguous Prompts through Human-Machine Co-Adaptation

• Authors: Yangfan He, Jianhui Wang, Kun Li, Yijin Wang, Li Sun, Jun Yin, Miao Zhang, Xueqian Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15167
• Pdf URL: https://arxiv.org/pdf/2501.15167
• Copy Paste: [[2501.15167]] Enhancing Intent Understanding for Ambiguous Prompts through Human-Machine Co-Adaptation(https://arxiv.org/abs/2501.15167)
• Keywords: generation
• Abstract: Modern image generation systems can produce high-quality visuals, yet user prompts often contain ambiguities, requiring multiple revisions. Existing methods struggle to address the nuanced needs of non-expert users. We propose Visual Co-Adaptation (VCA), a novel framework that iteratively refines prompts and aligns generated images with user preferences. VCA employs a fine-tuned language model with reinforcement learning and multi-turn dialogues for prompt disambiguation. Key components include the Incremental Context-Enhanced Dialogue Block for interactive clarification, the Semantic Exploration and Disambiguation Module (SESD) leveraging Retrieval-Augmented Generation (RAG) and CLIP scoring, and the Pixel Precision and Consistency Optimization Module (PPCO) for refining image details using Proximal Policy Optimization (PPO). A human-in-the-loop feedback mechanism further improves performance. Experiments show that VCA surpasses models like DALL-E 3 and Stable Diffusion, reducing dialogue rounds to 4.3, achieving a CLIP score of 0.92, and enhancing user satisfaction to 4.73/5. Additionally, we introduce a novel multi-round dialogue dataset with prompt-image pairs and user intent annotations.
• 摘要：现代图像生成系统可以生成高质量的视觉效果，但用户提示通常包含歧义，需要多次修改。现有方法难以满足非专家用户的细微需求。我们提出了视觉协同适应 (VCA)，这是一种新颖的框架，可以迭代地细化提示并使生成的图像与用户偏好保持一致。VCA 采用经过微调的语言模型，结合强化学习和多轮对话来消除提示歧义。关键组件包括用于交互式澄清的增量上下文增强对话块、利用检索增强生成 (RAG) 和 CLIP 评分的语义探索和消歧模块 (SESD)，以及使用近端策略优化 (PPO) 细化图像细节的像素精度和一致性优化模块 (PPCO)。人机反馈机制进一步提高了性能。实验表明，VCA 超越了 DALL-E 3 和 Stable Diffusion 等模型，将对话轮数减少到 4.3，CLIP 得分达到 0.92，用户满意度提高到 4.73/5。此外，我们还引入了一个新颖的多轮对话数据集，其中包含提示图像对和用户意图注释。

Title: Uni-Sign: Toward Unified Sign Language Understanding at Scale

• Authors: Zecheng Li, Wengang Zhou, Weichao Zhao, Kepeng Wu, Hezhen Hu, Houqiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15187
• Pdf URL: https://arxiv.org/pdf/2501.15187
• Copy Paste: [[2501.15187]] Uni-Sign: Toward Unified Sign Language Understanding at Scale(https://arxiv.org/abs/2501.15187)
• Keywords: generative
• Abstract: Sign language pre-training has gained increasing attention for its ability to enhance performance across various sign language understanding (SLU) tasks. However, existing methods often suffer from a gap between pre-training and fine-tuning, leading to suboptimal results. To address this, we propose \modelname, a unified pre-training framework that eliminates the gap between pre-training and downstream SLU tasks through a large-scale generative pre-training strategy and a novel fine-tuning paradigm. First, we introduce CSL-News, a large-scale Chinese Sign Language (CSL) dataset containing 1,985 hours of video paired with textual annotations, which enables effective large-scale pre-training. Second, \modelname unifies SLU tasks by treating downstream tasks as a single sign language translation (SLT) task during fine-tuning, ensuring seamless knowledge transfer between pre-training and fine-tuning. Furthermore, we incorporate a prior-guided fusion (PGF) module and a score-aware sampling strategy to efficiently fuse pose and RGB information, addressing keypoint inaccuracies and improving computational efficiency. Extensive experiments across multiple SLU benchmarks demonstrate that \modelname achieves state-of-the-art performance across multiple downstream SLU tasks. Dataset and code are available at \url{this http URL}.
• 摘要：手语预训练因其能够提高各种手语理解 (SLU) 任务的性能而受到越来越多的关注。然而，现有方法往往存在预训练和微调之间的差距，导致结果不理想。为了解决这个问题，我们提出了 \modelname，这是一个统一的预训练框架，它通过大规模生成预训练策略和新颖的微调范式消除了预训练和下游 SLU 任务之间的差距。首先，我们引入了 CSL-News，这是一个包含 1,985 小时视频和文本注释的大型中国手语 (CSL) 数据集，可实现有效的大规模预训练。其次，\modelname 通过在微调期间将下游任务视为单个手语翻译 (SLT) 任务来统一 SLU 任务，确保预训练和微调之间的无缝知识转移。此外，我们结合了先验引导融合 (PGF) 模块和分数感知采样策略，以高效融合姿势和 RGB 信息，解决关键点不准确问题并提高计算效率。在多个 SLU 基准测试中开展的大量实验表明，\modelname 在多个下游 SLU 任务中实现了最先进的性能。数据集和代码可在 \url{此 http URL} 上找到。

Title: Generalizable Deepfake Detection via Effective Local-Global Feature Extraction

• Authors: Jiazhen Yan, Ziqiang Li, Ziwen He, Zhangjie Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15253
• Pdf URL: https://arxiv.org/pdf/2501.15253
• Copy Paste: [[2501.15253]] Generalizable Deepfake Detection via Effective Local-Global Feature Extraction(https://arxiv.org/abs/2501.15253)
• Keywords: generation, generative
• Abstract: The rapid advancement of GANs and diffusion models has led to the generation of increasingly realistic fake images, posing significant hidden dangers and threats to society. Consequently, deepfake detection has become a pressing issue in today's world. While some existing methods focus on forgery features from either a local or global perspective, they often overlook the complementary nature of these features. Other approaches attempt to incorporate both local and global features but rely on simplistic strategies, such as cropping, which fail to capture the intricate relationships between local features. To address these limitations, we propose a novel method that effectively combines local spatial-frequency domain features with global frequency domain information, capturing detailed and holistic forgery traces. Specifically, our method uses Discrete Wavelet Transform (DWT) and sliding windows to tile forged features and leverages attention mechanisms to extract local spatial-frequency domain information. Simultaneously, the phase component of the Fast Fourier Transform (FFT) is integrated with attention mechanisms to extract global frequency domain information, complementing the local features and ensuring the integrity of forgery detection. Comprehensive evaluations on open-world datasets generated by 34 distinct generative models demonstrate a significant improvement of 2.9% over existing state-of-the-art methods.
• 摘要：GAN 和扩散模型的快速发展导致生成越来越逼真的假图像，对社会构成了重大隐患和威胁。因此，深度伪造检测已成为当今世界的一个紧迫问题。虽然一些现有的方法侧重于从局部或全局角度的伪造特征，但它们往往忽略了这些特征的互补性。其他方法试图结合局部和全局特征，但依赖于裁剪等简单的策略，无法捕捉局部特征之间的复杂关系。为了解决这些限制，我们提出了一种新方法，有效地将局部空间频域特征与全局频域信息相结合，捕获详细而整体的伪造痕迹。具体而言，我们的方法使用离散小波变换 (DWT) 和滑动窗口来平铺伪造特征，并利用注意机制提取局部空间频域信息。同时，快速傅里叶变换 (FFT) 的相位分量与注意机制相结合，以提取全局频域信息，补充局部特征并确保伪造检测的完整性。对 34 种不同生成模型生成的开放世界数据集进行综合评估表明，与现有的最先进方法相比，显著提高了 2.9%。

Title: Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink

• Authors: Yining Wang, Mi Zhang, Junjie Sun, Chenyue Wang, Min Yang, Hui Xue, Jialing Tao, Ranjie Duan, Jiexi Liu
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15269
• Pdf URL: https://arxiv.org/pdf/2501.15269
• Copy Paste: [[2501.15269]] Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink(https://arxiv.org/abs/2501.15269)
• Keywords: generation
• Abstract: Fusing visual understanding into language generation, Multi-modal Large Language Models (MLLMs) are revolutionizing visual-language applications. Yet, these models are often plagued by the hallucination problem, which involves generating inaccurate objects, attributes, and relationships that do not match the visual content. In this work, we delve into the internal attention mechanisms of MLLMs to reveal the underlying causes of hallucination, exposing the inherent vulnerabilities in the instruction-tuning process. We propose a novel hallucination attack against MLLMs that exploits attention sink behaviors to trigger hallucinated content with minimal image-text relevance, posing a significant threat to critical downstream applications. Distinguished from previous adversarial methods that rely on fixed patterns, our approach generates dynamic, effective, and highly transferable visual adversarial inputs, without sacrificing the quality of model responses. Comprehensive experiments on 6 prominent MLLMs demonstrate the efficacy of our attack in compromising black-box MLLMs even with extensive mitigating mechanisms, as well as the promising results against cutting-edge commercial APIs, such as GPT-4o and Gemini 1.5. Our code is available at this https URL.
• 摘要：多模态大型语言模型 (MLLM) 将视觉理解与语言生成融合在一起，正在彻底改变视觉语言应用。然而，这些模型经常受到幻觉问题的困扰，即生成与视觉内容不匹配的不准确对象、属性和关系。在这项工作中，我们深入研究了 MLLM 的内部注意力机制，以揭示幻觉的根本原因，暴露了指令调整过程中固有的漏洞。我们提出了一种针对 MLLM 的新型幻觉攻击，利用注意力下沉行为触发具有最小图像文本相关性的幻觉内容，对关键的下游应用构成重大威胁。与以前依赖固定模式的对抗方法不同，我们的方法可以生成动态、有效且高度可转移的视觉对抗输入，而不会牺牲模型响应的质量。对 6 个著名 MLLM 进行的全面实验证明了我们的攻击在攻陷黑盒 MLLM 方面非常有效，即使存在广泛的缓解机制，并且针对 GPT-4o 和 Gemini 1.5 等尖端商业 API 也取得了令人鼓舞的结果。我们的代码可在此 https URL 上找到。

Title: Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data

• Authors: Jiajie Li, Brian R Quaranto, Chenhui Xu, Ishan Mishra, Ruiyang Qin, Dancheng Liu, Peter C W Kim, Jinjun Xiong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15326
• Pdf URL: https://arxiv.org/pdf/2501.15326
• Copy Paste: [[2501.15326]] Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data(https://arxiv.org/abs/2501.15326)
• Keywords: generation
• Abstract: We present RASO, a foundation model designed to Recognize Any Surgical Object, offering robust open-set recognition capabilities across a broad range of surgical procedures and object classes, in both surgical images and videos. RASO leverages a novel weakly-supervised learning framework that generates tag-image-text pairs automatically from large-scale unannotated surgical lecture videos, significantly reducing the need for manual annotations. Our scalable data generation pipeline gatherers to 2,200 surgical procedures and produces 3.6 million tag annotations across 2,066 unique surgical tags. Our experiments show that RASO achieves improvements of 2.9 mAP, 4.5 mAP, 10.6 mAP, and 7.2 mAP on four standard surgical benchmarks respectively in zero-shot settings, and surpasses state-of-the-art models in supervised surgical action recognition tasks. We will open-source our code, model, and dataset to facilitate further research.
• 摘要：我们提出了 RASO，这是一种旨在识别任何手术对象的基础模型，可在手术图像和视频中为广泛的手术程序和对象类别提供强大的开放集识别功能。RASO 利用一种新颖的弱监督学习框架，可从大规模未注释的手术讲座视频中自动生成标签-图像-文本对，从而大大减少了对手动注释的需求。我们的可扩展数据生成管道收集了 2,200 个手术程序，并在 2,066 个独特的手术标签中生成了 360 万个标签注释。我们的实验表明，RASO 在零样本设置中分别在四个标准手术基准上实现了 2.9 mAP、4.5 mAP、10.6 mAP 和 7.2 mAP 的改进，并在监督手术动作识别任务中超越了最先进的模型。我们将开源代码、模型和数据集以促进进一步研究。

Title: Development and Application of Self-Supervised Machine Learning for Smoke Plume and Active Fire Identification from the FIREX-AQ Datasets

• Authors: Nicholas LaHaye, Anistasija Easley, Kyongsik Yun, Huikyo Lee, Erik Linstead, Michael J. Garay, Olga V. Kalashnikova
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15343
• Pdf URL: https://arxiv.org/pdf/2501.15343
• Copy Paste: [[2501.15343]] Development and Application of Self-Supervised Machine Learning for Smoke Plume and Active Fire Identification from the FIREX-AQ Datasets(https://arxiv.org/abs/2501.15343)
• Keywords: generation
• Abstract: Fire Influence on Regional to Global Environments and Air Quality (FIREX-AQ) was a field campaign aimed at better understanding the impact of wildfires and agricultural fires on air quality and climate. The FIREX-AQ campaign took place in August 2019 and involved two aircraft and multiple coordinated satellite observations. This study applied and evaluated a self-supervised machine learning (ML) method for the active fire and smoke plume identification and tracking in the satellite and sub-orbital remote sensing datasets collected during the campaign. Our unique methodology combines remote sensing observations with different spatial and spectral resolutions. The demonstrated approach successfully differentiates fire pixels and smoke plumes from background imagery, enabling the generation of a per-instrument smoke and fire mask product, as well as smoke and fire masks created from the fusion of selected data from independent instruments. This ML approach has a potential to enhance operational wildfire monitoring systems and improve decision-making in air quality management through fast smoke plume identification12 and tracking and could improve climate impact studies through fusion data from independent instruments.
• 摘要：火灾对区域乃至全球环境和空气质量的影响 (FIREX-AQ) 是一项实地活动，旨在更好地了解野火和农业火灾对空气质量和气候的影响。FIREX-AQ 活动于 2019 年 8 月开展，涉及两架飞机和多个协调卫星观测。本研究应用并评估了一种自监督机器学习 (ML) 方法，用于在活动期间收集的卫星和亚轨道遥感数据集中识别和跟踪主动火灾和烟羽。我们独特的方法结合了具有不同空间和光谱分辨率的遥感观测。所展示的方法成功地将火灾像素和烟羽与背景图像区分开来，从而能够生成每个仪器的烟雾和防火面罩产品，以及通过融合来自独立仪器的选定数据创建的烟雾和防火面罩。这种 ML 方法有可能通过快速识别 12 和跟踪烟羽来增强运营中的野火监测系统并改善空气质量管理的决策，并可以通过融合来自独立仪器的数据来改进气候影响研究。

Title: Federated Class-Incremental Learning: A Hybrid Approach Using Latent Exemplars and Data-Free Techniques to Address Local and Global Forgetting

• Authors: Milad Khademi Nori, Il-Min Kim, Guanghui (Richard)Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.15356
• Pdf URL: https://arxiv.org/pdf/2501.15356
• Copy Paste: [[2501.15356]] Federated Class-Incremental Learning: A Hybrid Approach Using Latent Exemplars and Data-Free Techniques to Address Local and Global Forgetting(https://arxiv.org/abs/2501.15356)
• Keywords: generation
• Abstract: Federated Class-Incremental Learning (FCIL) refers to a scenario where a dynamically changing number of clients collaboratively learn an ever-increasing number of incoming tasks. FCIL is known to suffer from local forgetting due to class imbalance at each client and global forgetting due to class imbalance across clients. We develop a mathematical framework for FCIL that formulates local and global forgetting. Then, we propose an approach called Hybrid Rehearsal (HR), which utilizes latent exemplars and data-free techniques to address local and global forgetting, respectively. HR employs a customized autoencoder designed for both data classification and the generation of synthetic data. To determine the embeddings of new tasks for all clients in the latent space of the encoder, the server uses the Lennard-Jones Potential formulations. Meanwhile, at the clients, the decoder decodes the stored low-dimensional latent space exemplars back to the high-dimensional input space, used to address local forgetting. To overcome global forgetting, the decoder generates synthetic data. Furthermore, our mathematical framework proves that our proposed approach HR can, in principle, tackle the two local and global forgetting challenges. In practice, extensive experiments demonstrate that while preserving privacy, our proposed approach outperforms the state-of-the-art baselines on multiple FCIL benchmarks with low compute and memory footprints.
• 摘要：联邦类增量学习 (FCIL) 是指动态变化的客户端数量协作学习不断增加的传入任务的场景。众所周知，FCIL 会因每个客户端的类别不平衡而遭受局部遗忘，并因客户端之间的类别不平衡而遭受全局遗忘。我们为 FCIL 开发了一个数学框架，用于公式化局部和全局遗忘。然后，我们提出了一种称为混合排练 (HR) 的方法，该方法分别利用潜在样本和无数据技术来解决局部和全局遗忘问题。HR 采用定制的自动编码器，专为数据分类和合成数据生成而设计。为了确定编码器潜在空间中所有客户端的新任务的嵌入，服务器使用 Lennard-Jones 势公式。同时，在客户端，解码器将存储的低维潜在空间样本解码回高维输入空间，用于解决局部遗忘问题。为了克服全局遗忘，解码器生成合成数据。此外，我们的数学框架证明了我们提出的 HR 方法原则上可以解决局部和全局遗忘问题。在实践中，大量实验表明，在保护隐私的同时，我们提出的方法在多个 FCIL 基准测试中的表现优于最先进的基线，并且计算和内存占用较低。

Title: MetaOcc: Surround-View 4D Radar and Camera Fusion Framework for 3D Occupancy Prediction with Dual Training Strategies

• Authors: Long Yang, Lianqing Zheng, Wenjin Ai, Minghao Liu, Sen Li, Qunshu Lin, Shengyu Yan, Jie Bai, Zhixiong Ma, Xichan Zhu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.15384
• Pdf URL: https://arxiv.org/pdf/2501.15384
• Copy Paste: [[2501.15384]] MetaOcc: Surround-View 4D Radar and Camera Fusion Framework for 3D Occupancy Prediction with Dual Training Strategies(https://arxiv.org/abs/2501.15384)
• Keywords: generation
• Abstract: 3D occupancy prediction is crucial for autonomous driving perception. Fusion of 4D radar and camera provides a potential solution of robust occupancy prediction on serve weather with least cost. How to achieve effective multi-modal feature fusion and reduce annotation costs remains significant challenges. In this work, we propose MetaOcc, a novel multi-modal occupancy prediction framework that fuses surround-view cameras and 4D radar for comprehensive environmental perception. We first design a height self-attention module for effective 3D feature extraction from sparse radar points. Then, a local-global fusion mechanism is proposed to adaptively capture modality contributions while handling spatio-temporal misalignments. Temporal alignment and fusion module is employed to further aggregate historical feature. Furthermore, we develop a semi-supervised training procedure leveraging open-set segmentor and geometric constraints for pseudo-label generation, enabling robust perception with limited annotations. Extensive experiments on OmniHD-Scenes dataset demonstrate that MetaOcc achieves state-of-the-art performance, surpassing previous methods by significant margins. Notably, as the first semi-supervised 4D radar and camera fusion-based occupancy prediction approach, MetaOcc maintains 92.5% of the fully-supervised performance while using only 50% of ground truth annotations, establishing a new benchmark for multi-modal 3D occupancy prediction. Code and data are available at this https URL.
• 摘要：3D 占用预测对于自动驾驶感知至关重要。4D 雷达和摄像头的融合为在恶劣天气下以最低成本进行稳健的占用预测提供了潜在的解决方案。如何实现有效的多模态特征融合并降低注释成本仍然是重大挑战。在这项工作中，我们提出了 MetaOcc，这是一种新颖的多模态占用预测框架，它融合了环视摄像头和 4D 雷达以实现全面的环境感知。我们首先设计了一个高度自注意模块，用于从稀疏雷达点有效地提取 3D 特征。然后，提出了一种局部-全局融合机制，以自适应地捕获模态贡献，同时处理时空错位。时间对齐和融合模块用于进一步聚合历史特征。此外，我们开发了一种半监督训练程序，利用开放集分割器和几何约束来生成伪标签，从而在有限的注释下实现稳健的感知。在 OmniHD-Scenes 数据集上进行的大量实验表明，MetaOcc 实现了最先进的性能，远远超过了以前的方法。值得注意的是，作为第一个基于半监督 4D 雷达和摄像头融合的占用预测方法，MetaOcc 仅使用 50% 的地面真实注释即可保持全监督性能的 92.5%，为多模态 3D 占用预测树立了新的标杆。代码和数据可在此 https URL 上获取。

Title: Doracamom: Joint 3D Detection and Occupancy Prediction with Multi-view 4D Radars and Cameras for Omnidirectional Perception

• Authors: Lianqing Zheng, Jianan Liu, Runwei Guan, Long Yang, Shouyi Lu, Yuanzhe Li, Xiaokai Bai, Jie Bai, Zhixiong Ma, Hui-Liang Shen, Xichan Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15394
• Pdf URL: https://arxiv.org/pdf/2501.15394
• Copy Paste: [[2501.15394]] Doracamom: Joint 3D Detection and Occupancy Prediction with Multi-view 4D Radars and Cameras for Omnidirectional Perception(https://arxiv.org/abs/2501.15394)
• Keywords: generation
• Abstract: 3D object detection and occupancy prediction are critical tasks in autonomous driving, attracting significant attention. Despite the potential of recent vision-based methods, they encounter challenges under adverse conditions. Thus, integrating cameras with next-generation 4D imaging radar to achieve unified multi-task perception is highly significant, though research in this domain remains limited. In this paper, we propose Doracamom, the first framework that fuses multi-view cameras and 4D radar for joint 3D object detection and semantic occupancy prediction, enabling comprehensive environmental perception. Specifically, we introduce a novel Coarse Voxel Queries Generator that integrates geometric priors from 4D radar with semantic features from images to initialize voxel queries, establishing a robust foundation for subsequent Transformer-based refinement. To leverage temporal information, we design a Dual-Branch Temporal Encoder that processes multi-modal temporal features in parallel across BEV and voxel spaces, enabling comprehensive spatio-temporal representation learning. Furthermore, we propose a Cross-Modal BEV-Voxel Fusion module that adaptively fuses complementary features through attention mechanisms while employing auxiliary tasks to enhance feature quality. Extensive experiments on the OmniHD-Scenes, View-of-Delft (VoD), and TJ4DRadSet datasets demonstrate that Doracamom achieves state-of-the-art performance in both tasks, establishing new benchmarks for multi-modal 3D perception. Code and models will be publicly available.
• 摘要：3D 物体检测和占用预测是自动驾驶中的关键任务，引起了广泛关注。尽管最近的基于视觉的方法具有潜力，但它们在恶劣条件下仍面临挑战。因此，将摄像头与下一代 4D 成像雷达集成以实现统一的多任务感知具有重要意义，尽管该领域的研究仍然有限。在本文中，我们提出了 Doracamom，这是第一个融合多视角摄像头和 4D 雷达的框架，用于联合 3D 物体检测和语义占用预测，从而实现全面的环境感知。具体来说，我们引入了一种新颖的粗体素查询生成器，它将来自 4D 雷达的几何先验与来自图像的语义特征相结合以初始化体素查询，为后续基于 Transformer 的细化奠定了坚实的基础。为了利用时间信息，我们设计了一个双分支时间编码器，它可以跨 BEV 和体素空间并行处理多模态时间特征，从而实现全面的时空表示学习。此外，我们提出了一个跨模态 BEV-Voxel 融合模块，该模块通过注意力机制自适应地融合互补特征，同时使用辅助任务来提高特征质量。在 OmniHD-Scenes、View-of-Delft (VoD) 和 TJ4DRadSet 数据集上进行的大量实验表明，Doracamom 在两个任务中都实现了最先进的性能，为多模态 3D 感知建立了新的基准。代码和模型将公开提供。

Title: Visual Generation Without Guidance

• Authors: Huayu Chen, Kai Jiang, Kaiwen Zheng, Jianfei Chen, Hang Su, Jun Zhu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.15420
• Pdf URL: https://arxiv.org/pdf/2501.15420
• Copy Paste: [[2501.15420]] Visual Generation Without Guidance(https://arxiv.org/abs/2501.15420)
• Keywords: generation, generative
• Abstract: Classifier-Free Guidance (CFG) has been a default technique in various visual generative models, yet it requires inference from both conditional and unconditional models during sampling. We propose to build visual models that are free from guided sampling. The resulting algorithm, Guidance-Free Training (GFT), matches the performance of CFG while reducing sampling to a single model, halving the computational cost. Unlike previous distillation-based approaches that rely on pretrained CFG networks, GFT enables training directly from scratch. GFT is simple to implement. It retains the same maximum likelihood objective as CFG and differs mainly in the parameterization of conditional models. Implementing GFT requires only minimal modifications to existing codebases, as most design choices and hyperparameters are directly inherited from CFG. Our extensive experiments across five distinct visual models demonstrate the effectiveness and versatility of GFT. Across domains of diffusion, autoregressive, and masked-prediction modeling, GFT consistently achieves comparable or even lower FID scores, with similar diversity-fidelity trade-offs compared with CFG baselines, all while being guidance-free. Code will be available at this https URL.
• 摘要：无分类器引导 (CFG) 一直是各种视觉生成模型中的默认技术，但它在采样期间需要从条件模型和非条件模型进行推理。我们建议构建不受引导采样影响的视觉模型。由此产生的算法无引导训练 (GFT) 可与 CFG 的性能相匹配，同时将采样减少到单个模型，从而将计算成本减半。与以前依赖于预训练 CFG 网络的基于蒸馏的方法不同，GFT 可以直接从头开始训练。GFT 易于实现。它保留了与 CFG 相同的最大似然目标，主要区别在于条件模型的参数化。实现 GFT 只需要对现有代码库进行最少的修改，因为大多数设计选择和超参数都直接继承自 CFG。我们在五种不同的视觉模型上进行的大量实验证明了 GFT 的有效性和多功能性。在扩散、自回归和掩蔽预测建模领域，GFT 始终能够实现相当甚至更低的 FID 分数，与 CFG 基线相比，其多样性-保真度权衡相似，同时无需指导。代码将在此 https URL 上提供。

Title: Dfilled: Repurposing Edge-Enhancing Diffusion for Guided DSM Void Filling

• Authors: Daniel Panangian, Ksenia Bittner
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15440
• Pdf URL: https://arxiv.org/pdf/2501.15440
• Copy Paste: [[2501.15440]] Dfilled: Repurposing Edge-Enhancing Diffusion for Guided DSM Void Filling(https://arxiv.org/abs/2501.15440)
• Keywords: super-resolution
• Abstract: Digital Surface Models (DSMs) are essential for accurately representing Earth's topography in geospatial analyses. DSMs capture detailed elevations of natural and manmade features, crucial for applications like urban planning, vegetation studies, and 3D reconstruction. However, DSMs derived from stereo satellite imagery often contain voids or missing data due to occlusions, shadows, and lowsignal areas. Previous studies have primarily focused on void filling for digital elevation models (DEMs) and Digital Terrain Models (DTMs), employing methods such as inverse distance weighting (IDW), kriging, and spline interpolation. While effective for simpler terrains, these approaches often fail to handle the intricate structures present in DSMs. To overcome these limitations, we introduce Dfilled, a guided DSM void filling method that leverages optical remote sensing images through edge-enhancing diffusion. Dfilled repurposes deep anisotropic diffusion models, which originally designed for super-resolution tasks, to inpaint DSMs. Additionally, we utilize Perlin noise to create inpainting masks that mimic natural void patterns in DSMs. Experimental evaluations demonstrate that Dfilled surpasses traditional interpolation methods and deep learning approaches in DSM void filling tasks. Both quantitative and qualitative assessments highlight the method's ability to manage complex features and deliver accurate, visually coherent results.
• 摘要：数字表面模型 (DSM) 对于在地理空间分析中准确表示地球地形至关重要。DSM 可捕捉自然和人造特征的详细高程，这对于城市规划、植被研究和 3D 重建等应用至关重要。然而，从立体卫星图像中得到的 DSM 通常包含空隙或缺失数据，这是由于遮挡、阴影和低信号区域造成的。先前的研究主要集中在数字高程模型 (DEM) 和数字地形模型 (DTM) 的空隙填充，采用的方法包括反距离加权 (IDW)、克里金法和样条插值。虽然这些方法对于较简单的地形有效，但它们通常无法处理 DSM 中存在的复杂结构。为了克服这些限制，我们引入了 Dfilled，这是一种引导式 DSM 空隙填充方法，它通过边缘增强扩散利用光学遥感图像。Dfilled 重新利用最初为超分辨率任务设计的深度各向异性扩散模型来修复 DSM。此外，我们利用 Perlin 噪声创建修复蒙版，以模仿 DSM 中的自然空隙模式。实验评估表明，Dfilled 在 DSM 空隙填充任务中超越了传统插值方法和深度学习方法。定量和定性评估都强调了该方法管理复杂特征并提供准确、视觉连贯的结果的能力。

Title: InfoBFR: Real-World Blind Face Restoration via Information Bottleneck

• Authors: Nan Gao, Jia Li, Huaibo Huang, Ke Shang, Ran He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15443
• Pdf URL: https://arxiv.org/pdf/2501.15443
• Copy Paste: [[2501.15443]] InfoBFR: Real-World Blind Face Restoration via Information Bottleneck(https://arxiv.org/abs/2501.15443)
• Keywords: restoration
• Abstract: Blind face restoration (BFR) is a highly challenging problem due to the uncertainty of data degradation patterns. Current BFR methods have realized certain restored productions but with inherent neural degradations that limit real-world generalization in complicated scenarios. In this paper, we propose a plug-and-play framework InfoBFR to tackle neural degradations, e.g., prior bias, topological distortion, textural distortion, and artifact residues, which achieves high-generalization face restoration in diverse wild and heterogeneous scenes. Specifically, based on the results from pre-trained BFR models, InfoBFR considers information compression using manifold information bottleneck (MIB) and information compensation with efficient diffusion LoRA to conduct information optimization. InfoBFR effectively synthesizes high-fidelity faces without attribute and identity distortions. Comprehensive experimental results demonstrate the superiority of InfoBFR over state-of-the-art GAN-based and diffusion-based BFR methods, with around 70ms consumption, 16M trainable parameters, and nearly 85% BFR-boosting. It is promising that InfoBFR will be the first plug-and-play restorer universally employed by diverse BFR models to conquer neural degradations.
• 摘要：由于数据退化模式的不确定性，盲人脸恢复 (BFR) 是一个非常具有挑战性的问题。当前的 BFR 方法已经实现了某些恢复产品，但固有的神经退化限制了复杂场景中现实世界的泛化。在本文中，我们提出了一个即插即用的框架 InfoBFR 来解决神经退化问题，例如先验偏差、拓扑失真、纹理失真和伪影残留，该框架在各种野生和异构场景中实现了高泛化的人脸恢复。具体而言，基于预先训练的 BFR 模型的结果，InfoBFR 考虑使用流形信息瓶颈 (MIB) 进行信息压缩，并使用高效扩散 LoRA 进行信息补偿以进行信息优化。InfoBFR 有效地合成了高保真人脸，而没有属性和身份失真。全面的实验结果表明，InfoBFR 优于最先进的基于 GAN 和基于扩散的 BFR 方法，消耗时间约为 70 毫秒，可训练参数为 16M，BFR 提升率接近 85%。有望的是，InfoBFR将成为第一个被各种BFR模型普遍采用以克服神经退化的即插即用修复器。

Title: StochSync: Stochastic Diffusion Synchronization for Image Generation in Arbitrary Spaces

• Authors: Kyeongmin Yeo, Jaihoon Kim, Minhyuk Sung
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.15445
• Pdf URL: https://arxiv.org/pdf/2501.15445
• Copy Paste: [[2501.15445]] StochSync: Stochastic Diffusion Synchronization for Image Generation in Arbitrary Spaces(https://arxiv.org/abs/2501.15445)
• Keywords: generation
• Abstract: We propose a zero-shot method for generating images in arbitrary spaces (e.g., a sphere for 360° panoramas and a mesh surface for texture) using a pretrained image diffusion model. The zero-shot generation of various visual content using a pretrained image diffusion model has been explored mainly in two directions. First, Diffusion Synchronization-performing reverse diffusion processes jointly across different projected spaces while synchronizing them in the target space-generates high-quality outputs when enough conditioning is provided, but it struggles in its absence. Second, Score Distillation Sampling-gradually updating the target space data through gradient descent-results in better coherence but often lacks detail. In this paper, we reveal for the first time the interconnection between these two methods while highlighting their differences. To this end, we propose StochSync, a novel approach that combines the strengths of both, enabling effective performance with weak conditioning. Our experiments demonstrate that StochSync provides the best performance in 360° panorama generation (where image conditioning is not given), outperforming previous finetuning-based methods, and also delivers comparable results in 3D mesh texturing (where depth conditioning is provided) with previous methods.
• 摘要：我们提出了一种零样本方法，使用预训练的图像扩散模型在任意空间（例如，360°全景图的球体和纹理的网格表面）中生成图像。使用预训练的图像扩散模型对各种视觉内容进行零样本生成主要在两个方向上进行了探索。首先，扩散同步 - 在不同的投影空间中联合执行反向扩散过程，同时在目标空间中同步它们 - 在提供足够的条件时会生成高质量的输出，但如果没有条件则会很吃力。其次，分数蒸馏采样 - 通过梯度下降逐步更新目标空间数据 - 可以产生更好的连贯性但通常缺乏细节。在本文中，我们首次揭示了这两种方法之间的相互联系，同时强调了它们之间的差异。为此，我们提出了 StochSync，这是一种结合两者优势的新方法，可在弱条件下实现有效的性能。我们的实验表明，StochSync 在 360° 全景图生成（未提供图像调节）中提供了最佳性能，优于以前基于微调的方法，并且在 3D 网格纹理（提供深度调节）中提供了与以前方法相当的结果。

Title: SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsity

• Authors: Zichen Fan, Steve Dai, Rangharajan Venkatesan, Dennis Sylvester, Brucek Khailany
• Subjects: cs.CV, cs.AI, cs.AR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.15448
• Pdf URL: https://arxiv.org/pdf/2501.15448
• Copy Paste: [[2501.15448]] SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsity(https://arxiv.org/abs/2501.15448)
• Keywords: generation
• Abstract: Diffusion models have gained significant popularity in image generation tasks. However, generating high-quality content remains notably slow because it requires running model inference over many time steps. To accelerate these models, we propose to aggressively quantize both weights and activations, while simultaneously promoting significant activation sparsity. We further observe that the stated sparsity pattern varies among different channels and evolves across time steps. To support this quantization and sparsity scheme, we present a novel diffusion model accelerator featuring a heterogeneous mixed-precision dense-sparse architecture, channel-last address mapping, and a time-step-aware sparsity detector for efficient handling of the sparsity pattern. Our 4-bit quantization technique demonstrates superior generation quality compared to existing 4-bit methods. Our custom accelerator achieves 6.91x speed-up and 51.5% energy reduction compared to traditional dense accelerators.
• 摘要：扩散模型在图像生成任务中已获得广泛欢迎。然而，生成高质量内容的速度仍然非常慢，因为它需要在多个时间步骤上运行模型推理。为了加速这些模型，我们建议积极量化权重和激活，同时促进显着的激活稀疏性。我们进一步观察到，所述稀疏模式在不同通道之间有所不同，并随时间步骤而演变。为了支持这种量化和稀疏方案，我们提出了一种新颖的扩散模型加速器，该加速器具有异构混合精度密集稀疏架构、通道最后地址映射和时间步感知稀疏检测器，可有效处理稀疏模式。与现有的 4 位方法相比，我们的 4 位量化技术表现出卓越的生成质量。与传统的密集加速器相比，我们的定制加速器实现了 6.91 倍的速度提升和 51.5% 的能耗降低。

Title: CD-Lamba: Boosting Remote Sensing Change Detection via a Cross-Temporal Locally Adaptive State Space Model

• Authors: Zhenkai Wu, Xiaowen Ma, Rongrong Lian, Kai Zheng, Mengting Ma, Wei Zhang, Siyang Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15455
• Pdf URL: https://arxiv.org/pdf/2501.15455
• Copy Paste: [[2501.15455]] CD-Lamba: Boosting Remote Sensing Change Detection via a Cross-Temporal Locally Adaptive State Space Model(https://arxiv.org/abs/2501.15455)
• Keywords: generation
• Abstract: Mamba, with its advantages of global perception and linear complexity, has been widely applied to identify changes of the target regions within the remote sensing (RS) images captured under complex scenarios and varied conditions. However, existing remote sensing change detection (RSCD) approaches based on Mamba frequently struggle to effectively perceive the inherent locality of change regions as they direct flatten and scan RS images (i.e., the features of the same region of changes are not distributed continuously within the sequence but are mixed with features from other regions throughout the sequence). In this paper, we propose a novel locally adaptive SSM-based approach, termed CD-Lamba, which effectively enhances the locality of change detection while maintaining global perception. Specifically, our CD-Lamba includes a Locally Adaptive State-Space Scan (LASS) strategy for locality enhancement, a Cross-Temporal State-Space Scan (CTSS) strategy for bi-temporal feature fusion, and a Window Shifting and Perception (WSP) mechanism to enhance interactions across segmented windows. These strategies are integrated into a multi-scale Cross-Temporal Locally Adaptive State-Space Scan (CT-LASS) module to effectively highlight changes and refine changes' representations feature generation. CD-Lamba significantly enhances local-global spatio-temporal interactions in bi-temporal images, offering improved performance in RSCD tasks. Extensive experimental results show that CD-Lamba achieves state-of-the-art performance on four benchmark datasets with a satisfactory efficiency-accuracy trade-off. Our code is publicly available at this https URL.
• 摘要：Mamba 具有全局感知和线性复杂度的优势，已被广泛应用于识别复杂场景和多变条件下拍摄的遥感 (RS) 图像中目标区域的变化。然而，现有的基于 Mamba 的遥感变化检测 (RSCD) 方法经常难以有效感知变化区域的固有局部性，因为它们直接展平和扫描 RS 图像（即，同一变化区域的特征在序列内不是连续分布的，而是与整个序列中其他区域的特征混合在一起）。在本文中，我们提出了一种新的基于 SSM 的局部自适应方法，称为 CD-Lamba，它在保持全局感知的同时有效地增强了变化检测的局部性。具体来说，我们的 CD-Lamba 包括用于局部性增强的局部自适应状态空间扫描 (LASS) 策略、用于双时间特征融合的跨时间状态空间扫描 (CTSS) 策略以及用于增强分段窗口间交互的窗口移位和感知 (WSP) 机制。这些策略被集成到多尺度跨时间局部自适应状态空间扫描 (CT-LASS) 模块中，以有效突出变化并改进变化的表示特征生成。CD-Lamba 显著增强了双时间图像中的局部-全局时空交互，从而提高了 RSCD 任务的性能。大量实验结果表明，CD-Lamba 在四个基准数据集上实现了最佳性能，并实现了令人满意的效率-准确度权衡。我们的代码在此 https URL 上公开提供。

Title: Domain Adaptation from Generated Multi-Weather Images for Unsupervised Maritime Object Classification

• Authors: Dan Song, Shumeng Huo, Wenhui Li, Lanjun Wang, Chao Xue, An-An Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15503
• Pdf URL: https://arxiv.org/pdf/2501.15503
• Copy Paste: [[2501.15503]] Domain Adaptation from Generated Multi-Weather Images for Unsupervised Maritime Object Classification(https://arxiv.org/abs/2501.15503)
• Keywords: generative
• Abstract: The classification and recognition of maritime objects are crucial for enhancing maritime safety, monitoring, and intelligent sea environment prediction. However, existing unsupervised methods for maritime object classification often struggle with the long-tail data distributions in both object categories and weather conditions. In this paper, we construct a dataset named AIMO produced by large-scale generative models with diverse weather conditions and balanced object categories, and collect a dataset named RMO with real-world images where long-tail issue exists. We propose a novel domain adaptation approach that leverages AIMO (source domain) to address the problem of limited labeled data, unbalanced distribution and domain shift in RMO (target domain), and enhance the generalization of source features with the Vision-Language Models such as CLIP. Experimental results shows that the proposed method significantly improves the classification accuracy, particularly for samples within rare object categories and weather conditions. Datasets and codes will be publicly available at this https URL.
• 摘要：海上物体的分类与识别对于加强海上安全、监测和智能海洋环境预测至关重要。然而，现有的无监督海上物体分类方法往往难以应对物体类别和天气条件下的长尾数据分布。在本文中，我们构建了一个由具有多样化天气条件和平衡物体类别的大规模生成模型生成的数据集 AIMO，并收集了一个包含存在长尾问题的真实世界图像的数据集 RMO。我们提出了一种新颖的领域自适应方法，利用 AIMO（源域）来解决 RMO（目标域）中标记数据有限、分布不均衡和领域偏移的问题，并使用 CLIP 等视觉语言模型增强源特征的泛化。实验结果表明，所提出的方法显著提高了分类准确率，特别是对于稀有物体类别和天气条件下的样本。数据集和代码将在此 https URL 上公开提供。

Title: Universal Image Restoration Pre-training via Degradation Classification

• Authors: JiaKui Hu, Lujia Jin, Zhengjian Yao, Yanye Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15510
• Pdf URL: https://arxiv.org/pdf/2501.15510
• Copy Paste: [[2501.15510]] Universal Image Restoration Pre-training via Degradation Classification(https://arxiv.org/abs/2501.15510)
• Keywords: restoration
• Abstract: This paper proposes the Degradation Classification Pre-Training (DCPT), which enables models to learn how to classify the degradation type of input images for universal image restoration pre-training. Unlike the existing self-supervised pre-training methods, DCPT utilizes the degradation type of the input image as an extremely weak supervision, which can be effortlessly obtained, even intrinsic in all image restoration datasets. DCPT comprises two primary stages. Initially, image features are extracted from the encoder. Subsequently, a lightweight decoder, such as ResNet18, is leveraged to classify the degradation type of the input image solely based on the features extracted in the first stage, without utilizing the input image. The encoder is pre-trained with a straightforward yet potent DCPT, which is used to address universal image restoration and achieve outstanding performance. Following DCPT, both convolutional neural networks (CNNs) and transformers demonstrate performance improvements, with gains of up to 2.55 dB in the 10D all-in-one restoration task and 6.53 dB in the mixed degradation scenarios. Moreover, previous self-supervised pretraining methods, such as masked image modeling, discard the decoder after pre-training, while our DCPT utilizes the pre-trained parameters more effectively. This superiority arises from the degradation classifier acquired during DCPT, which facilitates transfer learning between models of identical architecture trained on diverse degradation types. Source code and models are available at this https URL.
• 摘要：本文提出了退化分类预训练 (DCPT)，使模型能够学习如何对输入图像的退化类型进行分类，以进行通用图像恢复预训练。与现有的自监督预训练方法不同，DCPT 将输入图像的退化类型用作极弱的监督，这种监督可以毫不费力地获得，甚至是所有图像恢复数据集中固有的。DCPT 包括两个主要阶段。首先，从编码器中提取图像特征。随后，利用轻量级解码器（例如 ResNet18）仅根据第一阶段提取的特征对输入图像的退化类型进行分类，而不使用输入图像。编码器使用简单但有效的 DCPT 进行预训练，用于解决通用图像恢复并实现出色的性能。遵循 DCPT，卷积神经网络 (CNN) 和 Transformer 都表现出性能改进，在 10D 一体化恢复任务中增益高达 2.55 dB，在混合退化场景中增益高达 6.53 dB。此外，以前的自监督预训练方法（例如蒙版图像建模）在预训练后会丢弃解码器，而我们的 DCPT 可以更有效地利用预训练的参数。这种优势源于在 DCPT 期间获得的退化分类器，它有助于在不同退化类型上训练的相同架构模型之间的迁移学习。源代码和模型可在此 https URL 上找到。

Title: Advancing Generative Artificial Intelligence and Large Language Models for Demand Side Management with Electric Vehicles

• Authors: Hanwen Zhang, Ruichen Zhang, Wei Zhang, Dusit Niyato, Yonggang Wen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.15544
• Pdf URL: https://arxiv.org/pdf/2501.15544
• Copy Paste: [[2501.15544]] Advancing Generative Artificial Intelligence and Large Language Models for Demand Side Management with Electric Vehicles(https://arxiv.org/abs/2501.15544)
• Keywords: generation, generative
• Abstract: Generative artificial intelligence, particularly through large language models (LLMs), is poised to transform energy optimization and demand side management (DSM) within microgrids. This paper explores the integration of LLMs into energy management, emphasizing their roles in automating the optimization of DSM strategies with electric vehicles. We investigate challenges and solutions associated with DSM and explore the new opportunities presented by leveraging LLMs. Then, We propose an innovative solution that enhances LLMs with retrieval-augmented generation for automatic problem formulation, code generation, and customizing optimization. We present a case study to demonstrate the effectiveness of our proposed solution in charging scheduling and optimization for electric vehicles, highlighting our solution's significant advancements in energy efficiency and user adaptability. This work underscores the potential of LLMs for energy optimization and fosters a new era of intelligent DSM solutions.
• 摘要：生成式人工智能，尤其是通过大型语言模型 (LLM)，有望改变微电网内的能源优化和需求侧管理 (DSM)。本文探讨了 LLM 与能源管理的集成，强调了它们在自动优化电动汽车 DSM 策略方面的作用。我们研究了与 DSM 相关的挑战和解决方案，并探索了利用 LLM 带来的新机遇。然后，我们提出了一种创新解决方案，通过检索增强生成来增强 LLM，以实现自动问题制定、代码生成和自定义优化。我们提供了一个案例研究来证明我们提出的解决方案在电动汽车充电调度和优化方面的有效性，并强调了我们的解决方案在能源效率和用户适应性方面的重大进步。这项工作强调了 LLM 在能源优化方面的潜力，并开启了智能 DSM 解决方案的新时代。

Title: CE-SDWV: Effective and Efficient Concept Erasure for Text-to-Image Diffusion Models via a Semantic-Driven Word Vocabulary

• Authors: Jiahang Tu, Qian Feng, Chufan Chen, Jiahua Dong, Hanbin Zhao, Chao Zhang, Hui Qian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.15562
• Pdf URL: https://arxiv.org/pdf/2501.15562
• Copy Paste: [[2501.15562]] CE-SDWV: Effective and Efficient Concept Erasure for Text-to-Image Diffusion Models via a Semantic-Driven Word Vocabulary(https://arxiv.org/abs/2501.15562)
• Keywords: generative
• Abstract: Large-scale text-to-image (T2I) diffusion models have achieved remarkable generative performance about various concepts. With the limitation of privacy and safety in practice, the generative capability concerning NSFW (Not Safe For Work) concepts is undesirable, e.g., producing sexually explicit photos, and licensed images. The concept erasure task for T2I diffusion models has attracted considerable attention and requires an effective and efficient method. To achieve this goal, we propose a CE-SDWV framework, which removes the target concepts (e.g., NSFW concepts) of T2I diffusion models in the text semantic space by only adjusting the text condition tokens and does not need to re-train the original T2I diffusion model's weights. Specifically, our framework first builds a target concept-related word vocabulary to enhance the representation of the target concepts within the text semantic space, and then utilizes an adaptive semantic component suppression strategy to ablate the target concept-related semantic information in the text condition tokens. To further adapt the above text condition tokens to the original image semantic space, we propose an end-to-end gradient-orthogonal token optimization strategy. Extensive experiments on I2P and UnlearnCanvas benchmarks demonstrate the effectiveness and efficiency of our method.
• 摘要：大规模文本到图像 (T2I) 扩散模型已针对各种概念实现了出色的生成性能。由于实践中隐私和安全的限制，有关 NSFW（不适合工作）概念的生成能力不尽如人意，例如生成色情照片和授权图片。T2I 扩散模型的概念擦除任务已引起广泛关注，需要一种有效且高效的方法。为了实现这一目标，我们提出了一个 CE-SDWV 框架，该框架仅通过调整文本条件标记即可在文本语义空间中删除 T2I 扩散模型的目标概念（例如，NSFW 概念），而无需重新训练原始 T2I 扩散模型的权重。具体而言，我们的框架首先构建目标概念相关词汇表以增强目标概念在文本语义空间中的表示，然后利用自适应语义成分抑制策略来消除文本条件标记中与目标概念相关的语义信息。为了进一步将上述文本条件标记适应原始图像语义空间，我们提出了一种端到端梯度正交标记优化策略。在 I2P 和 UnlearnCanvas 基准上进行的大量实验证明了我们方法的有效性和效率。

Title: Advancing TDFN: Precise Fixation Point Generation Using Reconstruction Differences

• Authors: Shuguang Wang, Yuanjing Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15603
• Pdf URL: https://arxiv.org/pdf/2501.15603
• Copy Paste: [[2501.15603]] Advancing TDFN: Precise Fixation Point Generation Using Reconstruction Differences(https://arxiv.org/abs/2501.15603)
• Keywords: generation
• Abstract: Wang and Wang (2025) proposed the Task-Driven Fixation Network (TDFN) based on the fixation mechanism, which leverages low-resolution information along with high-resolution details near fixation points to accomplish specific visual tasks. The model employs reinforcement learning to generate fixation points. However, training reinforcement learning models is challenging, particularly when aiming to generate pixel-level accurate fixation points on high-resolution images. This paper introduces an improved fixation point generation method by leveraging the difference between the reconstructed image and the input image to train the fixation point generator. This approach directs fixation points to areas with significant differences between the reconstructed and input images. Experimental results demonstrate that this method achieves highly accurate fixation points, significantly enhances the network's classification accuracy, and reduces the average number of required fixations to achieve a predefined accuracy level.
• 摘要：Wang 和 Wang (2025) 提出了基于注视机制的任务驱动注视网络 (TDFN)，它利用注视点附近的低分辨率信息和高分辨率细节来完成特定的视觉任务。该模型采用强化学习来生成注视点。然而，训练强化学习模型具有挑战性，特别是当旨在在高分辨率图像上生成像素级精确的注视点时。本文介绍了一种改进的注视点生成方法，利用重建图像和输入图像之间的差异来训练注视点生成器。该方法将注视点引导到重建图像和输入图像之间具有显著差异的区域。实验结果表明，该方法实现了高精度的注视点，显著提高了网络的分类准确率，并减少了达到预定义准确率所需的平均注视次数。

Title: GaussianToken: An Effective Image Tokenizer with 2D Gaussian Splatting

• Authors: Jiajun Dong, Chengkun Wang, Wenzhao Zheng, Lei Chen, Jiwen Lu, Yansong Tang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.15619
• Pdf URL: https://arxiv.org/pdf/2501.15619
• Copy Paste: [[2501.15619]] GaussianToken: An Effective Image Tokenizer with 2D Gaussian Splatting(https://arxiv.org/abs/2501.15619)
• Keywords: generation
• Abstract: Effective image tokenization is crucial for both multi-modal understanding and generation tasks due to the necessity of the alignment with discrete text data. To this end, existing approaches utilize vector quantization (VQ) to project pixels onto a discrete codebook and reconstruct images from the discrete representation. However, compared with the continuous latent space, the limited discrete codebook space significantly restrict the representational ability of these image tokenizers. In this paper, we propose GaussianToken: An Effective Image Tokenizer with 2D Gaussian Splatting as a solution. We first represent the encoded samples as multiple flexible featured 2D Gaussians characterized by positions, rotation angles, scaling factors, and feature coefficients. We adopt the standard quantization for the Gaussian features and then concatenate the quantization results with the other intrinsic Gaussian parameters before the corresponding splatting operation and the subsequent decoding module. In general, GaussianToken integrates the local influence of 2D Gaussian distribution into the discrete space and thus enhances the representation capability of the image tokenizer. Competitive reconstruction performances on CIFAR, Mini-ImageNet, and ImageNet-1K demonstrate the effectiveness of our framework. Our code is available at: this https URL.
• 摘要：由于需要与离散文本数据对齐，有效的图像标记对于多模态理解和生成任务都至关重要。为此，现有方法利用矢量量化 (VQ) 将像素投影到离散码本上，并从离散表示中重建图像。然而，与连续潜在空间相比，有限的离散码本空间显著限制了这些图像标记器的表示能力。在本文中，我们提出了 GaussianToken：一种具有 2D 高斯分层的有效图像标记器作为解决方案。我们首先将编码样本表示为多个灵活的特征 2D 高斯，这些特征由位置、旋转角度、缩放因子和特征系数表征。我们对高斯特征采用标准量化，然后在相应的分层操作和后续解码模块之前将量化结果与其他固有高斯参数连接起来。一般来说，GaussianToken 将 2D 高斯分布的局部影响整合到离散空间中，从而增强了图像标记器的表示能力。 CIFAR、Mini-ImageNet 和 ImageNet-1K 上的竞争性重建性能证明了我们框架的有效性。我们的代码可从此 https URL 获取。

Title: Bringing Characters to New Stories: Training-Free Theme-Specific Image Generation via Dynamic Visual Prompting

• Authors: Yuxin Zhang, Minyan Luo, Weiming Dong, Xiao Yang, Haibin Huang, Chongyang Ma, Oliver Deussen, Tong-Yee Lee, Changsheng Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15641
• Pdf URL: https://arxiv.org/pdf/2501.15641
• Copy Paste: [[2501.15641]] Bringing Characters to New Stories: Training-Free Theme-Specific Image Generation via Dynamic Visual Prompting(https://arxiv.org/abs/2501.15641)
• Keywords: generation, generative
• Abstract: The stories and characters that captivate us as we grow up shape unique fantasy worlds, with images serving as the primary medium for visually experiencing these realms. Personalizing generative models through fine-tuning with theme-specific data has become a prevalent approach in text-to-image generation. However, unlike object customization, which focuses on learning specific objects, theme-specific generation encompasses diverse elements such as characters, scenes, and objects. Such diversity also introduces a key challenge: how to adaptively generate multi-character, multi-concept, and continuous theme-specific images (TSI). Moreover, fine-tuning approaches often come with significant computational overhead, time costs, and risks of overfitting. This paper explores a fundamental question: Can image generation models directly leverage images as contextual input, similarly to how large language models use text as context? To address this, we present T-Prompter, a novel training-free TSI method for generation. T-Prompter introduces visual prompting, a mechanism that integrates reference images into generative models, allowing users to seamlessly specify the target theme without requiring additional training. To further enhance this process, we propose a Dynamic Visual Prompting (DVP) mechanism, which iteratively optimizes visual prompts to improve the accuracy and quality of generated images. Our approach enables diverse applications, including consistent story generation, character design, realistic character generation, and style-guided image generation. Comparative evaluations against state-of-the-art personalization methods demonstrate that T-Prompter achieves significantly better results and excels in maintaining character identity preserving, style consistency and text alignment, offering a robust and flexible solution for theme-specific image generation.
• 摘要：在我们成长的过程中，那些吸引我们的故事和角色塑造了独特的幻想世界，而图像是视觉体验这些世界的主要媒介。通过使用特定主题的数据进行微调来个性化生成模型已成为文本到图像生成的一种流行方法。然而，与专注于学习特定对象的对象定制不同，特定主题的生成包含角色、场景和对象等多种元素。这种多样性也带来了一个关键挑战：如何自适应地生成多角色、多概念和连续的特定主题图像 (TSI)。此外，微调方法通常会带来大量的计算开销、时间成本和过度拟合的风险。本文探讨了一个基本问题：图像生成模型能否直接利用图像作为上下文输入，就像大型语言模型使用文本作为上下文一样？为了解决这个问题，我们提出了 T-Prompter，一种新颖的无需训练的 TSI 生成方法。 T-Prompter 引入了视觉提示，这是一种将参考图像集成到生成模型中的机制，允许用户无缝指定目标主题而无需额外训练。为了进一步增强此过程，我们提出了一种动态视觉提示 (DVP) 机制，该机制迭代优化视觉提示以提高生成图像的准确性和质量。我们的方法支持多种应用，包括一致的故事生成、角色设计、逼真的角色生成和风格引导的图像生成。与最先进的个性化方法的比较评估表明，T-Prompter 取得了明显更好的结果，并且在保持角色身份保留、风格一致性和文本对齐方面表现出色，为特定主题的图像生成提供了强大而灵活的解决方案。

Title: StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel

• Authors: Dylan Cutler, Arun Kandoor, Nishanth Dikkala, Nikunj Saunshi, Xin Wang, Rina Panigrahy
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.15665
• Pdf URL: https://arxiv.org/pdf/2501.15665
• Copy Paste: [[2501.15665]] StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel(https://arxiv.org/abs/2501.15665)
• Keywords: generation
• Abstract: Standard decoding in a Transformer based language model is inherently sequential as we wait for a token's embedding to pass through all the layers in the network before starting the generation of the next token. In this work, we propose a new architecture StagFormer (Staggered Transformer), which staggered execution along the time axis and thereby enables parallelizing the decoding process along the depth of the model. We achieve this by breaking the dependency of the token representation at time step $i$ in layer $l$ upon the representations of tokens until time step $i$ from layer $l-1$. Instead, we stagger the execution and only allow a dependency on token representations until time step $i-1$. The later sections of the Transformer still get access to the ``rich" representations from the prior section but only from those token positions which are one time step behind. StagFormer allows for different sections of the model to be executed in parallel yielding at potential 33\% speedup in decoding while being quality neutral in our simulations. We also explore many natural variants of this idea. We present how weight-sharing across the different sections being staggered can be more practical in settings with limited memory. We show how one can approximate a recurrent model during inference using such weight-sharing. We explore the efficacy of using a bounded window attention to pass information from one section to another which helps drive further latency gains for some applications. We also explore demonstrate the scalability of the staggering idea over more than 2 sections of the Transformer.
• 摘要：基于 Transformer 的语言模型中的标准解码本质上是连续的，因为我们要等待一个 token 的嵌入通过网络中的所有层，然后才开始生成下一个 token。在这项工作中，我们提出了一种新的架构 StagFormer（交错 Transformer），它沿时间轴交错执行，从而能够沿模型深度并行化解码过程。我们通过打破层 $l$ 中时间步骤 $i$ 的 token 表示对层 $l-1$ 中时间步骤 $i$ 之前的 token 表示的依赖来实现这一点。相反，我们交错执行，并且只允许对时间步骤 $i-1$ 之前的 token 表示的依赖。 Transformer 的后面部分仍然可以访问前一部分的“丰富”表示，但只能访问落后一个时间步的标记位置。StagFormer 允许模型的不同部分并行执行，从而可能将解码速度提高 33% ，同时在我们的模拟中保持质量中性。我们还探索了这个想法的许多自然变体。我们介绍了在内存有限的环境中，如何在不同部分之间交错共享权重会更加实用。我们展示了如何使用这种权重共享在推理过程中近似递归模型。我们探讨了使用有界窗口注意力将信息从一个部分传递到另一个部分的有效性，这有助于进一步提高某些应用程序的延迟。我们还探索了在 Transformer 的 2 个以上部分上展示交错想法的可扩展性。

Title: Disentanglement Analysis in Deep Latent Variable Models Matching Aggregate Posterior Distributions

• Authors: Surojit Saha, Sarang Joshi, Ross Whitaker
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.15705
• Pdf URL: https://arxiv.org/pdf/2501.15705
• Copy Paste: [[2501.15705]] Disentanglement Analysis in Deep Latent Variable Models Matching Aggregate Posterior Distributions(https://arxiv.org/abs/2501.15705)
• Keywords: generative
• Abstract: Deep latent variable models (DLVMs) are designed to learn meaningful representations in an unsupervised manner, such that the hidden explanatory factors are interpretable by independent latent variables (aka disentanglement). The variational autoencoder (VAE) is a popular DLVM widely studied in disentanglement analysis due to the modeling of the posterior distribution using a factorized Gaussian distribution that encourages the alignment of the latent factors with the latent axes. Several metrics have been proposed recently, assuming that the latent variables explaining the variation in data are aligned with the latent axes (cardinal directions). However, there are other DLVMs, such as the AAE and WAE-MMD (matching the aggregate posterior to the prior), where the latent variables might not be aligned with the latent axes. In this work, we propose a statistical method to evaluate disentanglement for any DLVMs in general. The proposed technique discovers the latent vectors representing the generative factors of a dataset that can be different from the cardinal latent axes. We empirically demonstrate the advantage of the method on two datasets.
• 摘要：深度隐变量模型 (DLVM) 旨在以无监督的方式学习有意义的表示，以便隐藏的解释因素可以通过独立隐变量（又称解缠）进行解释。变分自动编码器 (VAE) 是一种流行的 DLVM，在解缠分析中得到广泛研究，因为它使用分解的高斯分布对后验分布进行建模，从而鼓励将隐变量与隐轴对齐。最近提出了几种指标，假设解释数据变化的隐变量与隐轴（基本方向）对齐。然而，还有其他 DLVM，例如 AAE 和 WAE-MMD（将聚合后验与先验相匹配），其中隐变量可能与隐轴不一致。在这项工作中，我们提出了一种统计方法来评估任何 DLVM 的解缠。所提出的技术发现了表示数据集生成因素的潜在向量，这些向量可能与基本隐轴不同。我们通过两个数据集实证证明了该方法的优势。

Title: INRet: A General Framework for Accurate Retrieval of INRs for Shapes

• Authors: Yushi Guan, Daniel Kwan, Ruofan Liang, Selvakumar Panneer, Nilesh Jain, Nilesh Ahuja, Nandita Vijaykumar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.15722
• Pdf URL: https://arxiv.org/pdf/2501.15722
• Copy Paste: [[2501.15722]] INRet: A General Framework for Accurate Retrieval of INRs for Shapes(https://arxiv.org/abs/2501.15722)
• Keywords: generation
• Abstract: Implicit neural representations (INRs) have become an important method for encoding various data types, such as 3D objects or scenes, images, and videos. They have proven to be particularly effective at representing 3D content, e.g., 3D scene reconstruction from 2D images, novel 3D content creation, as well as the representation, interpolation, and completion of 3D shapes. With the widespread generation of 3D data in an INR format, there is a need to support effective organization and retrieval of INRs saved in a data store. A key aspect of retrieval and clustering of INRs in a data store is the formulation of similarity between INRs that would, for example, enable retrieval of similar INRs using a query INR. In this work, we propose INRet, a method for determining similarity between INRs that represent shapes, thus enabling accurate retrieval of similar shape INRs from an INR data store. INRet flexibly supports different INR architectures such as INRs with octree grids, triplanes, and hash grids, as well as different implicit functions including signed/unsigned distance function and occupancy field. We demonstrate that our method is more general and accurate than the existing INR retrieval method, which only supports simple MLP INRs and requires the same architecture between the query and stored INRs. Furthermore, compared to converting INRs to other representations (e.g., point clouds or multi-view images) for 3D shape retrieval, INRet achieves higher accuracy while avoiding the conversion overhead.
• 摘要：隐式神经表征 (INR) 已成为编码各种数据类型（例如 3D 对象或场景、图像和视频）的重要方法。它们已被证明在表示 3D 内容方面特别有效，例如从 2D 图像重建 3D 场景、创建新颖的 3D 内容以及表示、插值和完成 3D 形状。随着 INR 格式的 3D 数据的广泛生成，需要支持有效组织和检索数据存储中保存的 INR。在数据存储中检索和聚类 INR 的一个关键方面是制定 INR 之间的相似性，例如，可以使用查询 INR 检索相似的 INR。在这项工作中，我们提出了 INRet，一种用于确定表示形状的 INR 之间的相似性的方法，从而能够从 INR 数据存储中准确检索相似形状的 INR。 INRet 灵活地支持不同的 INR 架构，例如具有八叉树网格、三平面和哈希网格的 INR，以及不同的隐式函数，包括有符号/无符号距离函数和占用字段。我们证明我们的方法比现有的 INR 检索方法更通用、更准确，后者仅支持简单的 MLP INR，并且要求查询和存储的 INR 之间具有相同的架构。此外，与将 INR 转换为其他表示形式（例如点云或多视图图像）以进行 3D 形状检索相比，INRet 实现了更高的准确性，同时避免了转换开销。

Title: Efficiency Bottlenecks of Convolutional Kolmogorov-Arnold Networks: A Comprehensive Scrutiny with ImageNet, AlexNet, LeNet and Tabular Classification

• Authors: Ashim Dahal, Saydul Akbar Murad, Nick Rahimi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.15757
• Pdf URL: https://arxiv.org/pdf/2501.15757
• Copy Paste: [[2501.15757]] Efficiency Bottlenecks of Convolutional Kolmogorov-Arnold Networks: A Comprehensive Scrutiny with ImageNet, AlexNet, LeNet and Tabular Classification(https://arxiv.org/abs/2501.15757)
• Keywords: generation
• Abstract: Algorithmic level developments like Convolutional Neural Networks, transformers, attention mechanism, Retrieval Augmented Generation and so on have changed Artificial Intelligence. Recent such development was observed by Kolmogorov-Arnold Networks that suggested to challenge the fundamental concept of a Neural Network, thus change Multilayer Perceptron, and Convolutional Neural Networks. They received a good reception in terms of scientific modeling, yet had some drawbacks in terms of efficiency. In this paper, we train Convolutional Kolmogorov Arnold Networks (CKANs) with the ImageNet-1k dataset with 1.3 million images, MNIST dataset with 60k images and a tabular biological science related MoA dataset and test the promise of CKANs in terms of FLOPS, Inference Time, number of trainable parameters and training time against the accuracy, precision, recall and f-1 score they produce against the standard industry practice on CNN models. We show that the CKANs perform fair yet slower than CNNs in small size dataset like MoA and MNIST but are not nearly comparable as the dataset gets larger and more complex like the ImageNet. The code implementation of this paper can be found on the link: \href{this https URL}{this https URL}
• 摘要：卷积神经网络、Transformer、注意力机制、检索增强生成等算法层面的发展改变了人工智能。柯尔莫哥洛夫-阿诺德网络观察到了最近的这种发展，它建议挑战神经网络的基本概念，从而改变多层感知器和卷积神经网络。它们在科学建模方面受到了好评，但在效率方面也存在一些缺点。在本文中，我们使用包含 130 万张图像的 ImageNet-1k 数据集、包含 6 万张图像的 MNIST 数据集和表格生物科学相关的 MoA 数据集训练卷积柯尔莫哥洛夫-阿诺德网络 (CKAN)，并根据它们产生的准确度、精确度、召回率和 f-1 分数与 CNN 模型的标准行业实践来测试 CKAN 在 FLOPS、推理时间、可训练参数数量和训练时间方面的前景。我们表明，CKAN 在 MoA 和 MNIST 等小型数据集上的表现与 CNN 相当但速度较慢，但​​随着数据集变得越来越大、越来越复杂（如 ImageNet），它们的表现就无法与 CNN 相比。本文的代码实现可在以下链接中找到：\href{此 https URL}{此 https URL}

Title: Risk-Aware Distributional Intervention Policies for Language Models

• Authors: Bao Nguyen, Binh Nguyen, Duy Nguyen, Viet Anh Nguyen
• Subjects: cs.LG, cs.CL, math.OC
• Abstract URL: https://arxiv.org/abs/2501.15758
• Pdf URL: https://arxiv.org/pdf/2501.15758
• Copy Paste: [[2501.15758]] Risk-Aware Distributional Intervention Policies for Language Models(https://arxiv.org/abs/2501.15758)
• Keywords: generation
• Abstract: Language models are prone to occasionally undesirable generations, such as harmful or toxic content, despite their impressive capability to produce texts that appear accurate and coherent. This paper presents a new two-stage approach to detect and mitigate undesirable content generations by rectifying activations. First, we train an ensemble of layerwise classifiers to detect undesirable content using activations by minimizing a smooth surrogate of the risk-aware score. Then, for contents that are detected as undesirable, we propose layerwise distributional intervention policies that perturb the attention heads minimally while guaranteeing probabilistically the effectiveness of the intervention. Benchmarks on several language models and datasets show that our method outperforms baselines in reducing the generation of undesirable output.
• 摘要：尽管语言模型能够生成准确连贯的文本，但它们偶尔也会出现不良内容，例如有害或有毒内容。本文提出了一种新的两阶段方法，通过纠正激活来检测和缓解不良内容的生成。首先，我们训练一组分层分类器，通过最小化风险意识分数的平滑替代值来使用激活来检测不良内容。然后，对于被检测为不良的内容，我们提出了分层分布式干预策略，这些策略对注意力头的干扰最小，同时在概率上保证干预的有效性。在多个语言模型和数据集上的基准测试表明，我们的方法在减少不良输出的生成方面优于基线。

Title: Efficient Attention-Sharing Information Distillation Transformer for Lightweight Single Image Super-Resolution

• Authors: Karam Park, Jae Woong Soh, Nam Ik Cho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15774
• Pdf URL: https://arxiv.org/pdf/2501.15774
• Copy Paste: [[2501.15774]] Efficient Attention-Sharing Information Distillation Transformer for Lightweight Single Image Super-Resolution(https://arxiv.org/abs/2501.15774)
• Keywords: super-resolution
• Abstract: Transformer-based Super-Resolution (SR) methods have demonstrated superior performance compared to convolutional neural network (CNN)-based SR approaches due to their capability to capture long-range dependencies. However, their high computational complexity necessitates the development of lightweight approaches for practical use. To address this challenge, we propose the Attention-Sharing Information Distillation (ASID) network, a lightweight SR network that integrates attention-sharing and an information distillation structure specifically designed for Transformer-based SR methods. We modify the information distillation scheme, originally designed for efficient CNN operations, to reduce the computational load of stacked self-attention layers, effectively addressing the efficiency bottleneck. Additionally, we introduce attention-sharing across blocks to further minimize the computational cost of self-attention operations. By combining these strategies, ASID achieves competitive performance with existing SR methods while requiring only around 300K parameters - significantly fewer than existing CNN-based and Transformer-based SR models. Furthermore, ASID outperforms state-of-the-art SR methods when the number of parameters is matched, demonstrating its efficiency and effectiveness. The code and supplementary material are available on the project page.
• 摘要：基于 Transformer 的超分辨率 (SR) 方法由于能够捕获长距离依赖关系而表现出比基于卷积神经网络 (CNN) 的 SR 方法更优异的性能。然而，它们的高计算复杂度使得开发轻量级方法以供实际使用成为必要。为了应对这一挑战，我们提出了注意力共享信息蒸馏 (ASID) 网络，这是一种轻量级 SR 网络，它集成了注意力共享和专为基于 Transformer 的 SR 方法设计的信息蒸馏结构。我们修改了最初为高效 CNN 操作设计的信息蒸馏方案，以减少堆叠自注意力层的计算负荷，从而有效解决效率瓶颈。此外，我们引入了跨块注意力共享，以进一步最大限度地降低自注意力操作的计算成本。通过结合这些策略，ASID 实现了与现有 SR 方法相媲美的性能，同时只需要大约 300K 个参数 - 比现有的基于 CNN 和基于 Transformer 的 SR 模型少得多。此外，当参数数量匹配时，ASID 的表现优于最先进的 SR 方法，证明了其效率和有效性。代码和补充材料可在项目页面上找到。

Title: Memorization and Regularization in Generative Diffusion Models

• Authors: Ricardo Baptista, Agnimitra Dasgupta, Nikola B. Kovachki, Assad Oberai, Andrew M. Stuart
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2501.15785
• Pdf URL: https://arxiv.org/pdf/2501.15785
• Copy Paste: [[2501.15785]] Memorization and Regularization in Generative Diffusion Models(https://arxiv.org/abs/2501.15785)
• Keywords: generative
• Abstract: Diffusion models have emerged as a powerful framework for generative modeling. At the heart of the methodology is score matching: learning gradients of families of log-densities for noisy versions of the data distribution at different scales. When the loss function adopted in score matching is evaluated using empirical data, rather than the population loss, the minimizer corresponds to the score of a time-dependent Gaussian mixture. However, use of this analytically tractable minimizer leads to data memorization: in both unconditioned and conditioned settings, the generative model returns the training samples. This paper contains an analysis of the dynamical mechanism underlying memorization. The analysis highlights the need for regularization to avoid reproducing the analytically tractable minimizer; and, in so doing, lays the foundations for a principled understanding of how to regularize. Numerical experiments investigate the properties of: (i) Tikhonov regularization; (ii) regularization designed to promote asymptotic consistency; and (iii) regularizations induced by under-parameterization of a neural network or by early stopping when training a neural network. These experiments are evaluated in the context of memorization, and directions for future development of regularization are highlighted.
• 摘要：扩散模型已成为生成建模的强大框架。该方法的核心是分数匹配：学习不同尺度上数据分布的噪声版本的对数密度系列的梯度。当使用经验数据而不是总体损失来评估分数匹配中采用的损失函数时，最小化器对应于时间相关的高斯混合的分数。然而，使用这种易于分析的最小化器会导致数据记忆：在无条件和有条件设置中，生成模型都会返回训练样本。本文包含对记忆背后动力学机制的分析。分析强调了正则化的必要性，以避免重现易于分析的最小化器；并且，这样做为从原则上理解如何正则化奠定了基础。数值实验研究了以下性质：(i) 吉洪诺夫正则化；(ii) 旨在促进渐近一致性的正则化；以及 (iii) 神经网络参数不足或训练神经网络时过早停止所导致的正则化。这些实验是在记忆的背景下进行评估的，并强调了正则化未来发展的方向。

Title: LemmaHead: RAG Assisted Proof Generation Using Large Language Models

• Authors: Tianbo Yang, Mingqi Yang, Hongyi Zhao, Tianshuo Yang
• Subjects: cs.LG, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.15797
• Pdf URL: https://arxiv.org/pdf/2501.15797
• Copy Paste: [[2501.15797]] LemmaHead: RAG Assisted Proof Generation Using Large Language Models(https://arxiv.org/abs/2501.15797)
• Keywords: generation
• Abstract: Developing the logic necessary to solve mathematical problems or write mathematical proofs is one of the more difficult objectives for large language models (LLMS). Currently, the most popular methods in literature consists of fine-tuning the model on written mathematical content such as academic publications and textbooks, so that the model can learn to emulate the style of mathematical writing. In this project, we explore the effectiveness of using retrieval augmented generation (RAG) to address gaps in the mathematical reasoning of LLMs. We develop LemmaHead, a RAG knowledge base that supplements queries to the model with relevant mathematical context, with particular focus on context from published textbooks. To measure our model's performance in mathematical reasoning, our testing paradigm focuses on the task of automated theorem proving via generating proofs to a given mathematical claim in the Lean formal language.
• 摘要：开发解决数学问题或编写数学证明所需的逻辑是大型语言模型 (LLMS) 最困难的目标之一。目前，文献中最流行的方法包括根据书面数学内容（例如学术出版物和教科书）对模型进行微调，以便模型可以学习模仿数学写作风格。在这个项目中，我们探索了使用检索增强生成 (RAG) 解决 LLM 数学推理缺陷的有效性。我们开发了 LemmaHead，这是一个 RAG 知识库，它使用相关的数学背景补充对模型的查询，特别关注已出版教科书中的背景。为了衡量我们模型在数学推理方面的表现，我们的测试范式侧重于通过用精益形式语言为给定的数学主张生成证明来完成自动定理证明的任务。

Title: MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretraining

• Authors: Ruiqi Wu, Na Su, Chenran Zhang, Tengfei Ma, Tao Zhou, Zhiting Cui, Nianfeng Tang, Tianyu Mao, Yi Zhou, Wen Fan, Tianxing Wu, Shenqi Jing, Huazhu Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15798
• Pdf URL: https://arxiv.org/pdf/2501.15798
• Copy Paste: [[2501.15798]] MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretraining(https://arxiv.org/abs/2501.15798)
• Keywords: generative
• Abstract: Vision-language pretraining (VLP) has been investigated to generalize across diverse downstream tasks for fundus image analysis. Although recent methods showcase promising achievements, they significantly rely on large-scale private image-text data but pay less attention to the pretraining manner, which limits their further advancements. In this work, we introduce MM-Retinal V2, a high-quality image-text paired dataset comprising CFP, FFA, and OCT image modalities. Then, we propose a novel fundus vision-language pretraining model, namely KeepFIT V2, which is pretrained by integrating knowledge from the elite data spark into categorical public datasets. Specifically, a preliminary textual pretraining is adopted to equip the text encoder with primarily ophthalmic textual knowledge. Moreover, a hybrid image-text knowledge injection module is designed for knowledge transfer, which is essentially based on a combination of global semantic concepts from contrastive learning and local appearance details from generative learning. Extensive experiments across zero-shot, few-shot, and linear probing settings highlight the generalization and transferability of KeepFIT V2, delivering performance competitive to state-of-the-art fundus VLP models trained on large-scale private image-text datasets. Our dataset and model are publicly available via this https URL.
• 摘要：视觉语言预训练 (VLP) 已被研究用于推广到眼底图像分析的各种下游任务。尽管最近的方法展示了有希望的成果，但它们严重依赖于大规模私有图像文本数据，而较少关注预训练方式，这限制了它们的进一步发展。在这项工作中，我们介绍了 MM-Retinal V2，这是一个包含 CFP、FFA 和 OCT 图像模态的高质量图像文本配对数据集。然后，我们提出了一种新颖的眼底视觉语言预训练模型，即 KeepFIT V2，该模型通过将精英数据火花中的知识整合到分类公共数据集中进行预训练。具体而言，采用初步的文本预训练，为文本编码器配备主要的眼科文本知识。此外，设计了一个混合图像文本知识注入模块用于知识转移，该模块本质上基于对比学习中的全局语义概念和生成学习中的局部外观细节的组合。在零样本、少样本和线性探测设置中进行的大量实验凸显了 KeepFIT V2 的泛化能力和可迁移性，其性能可与在大型私有图像文本数据集上训练的最先进的眼底 VLP 模型相媲美。我们的数据集和模型可通过此 https URL 公开获取。

Title: Controllable Hand Grasp Generation for HOI and Efficient Evaluation Methods

• Authors: Ishant, Rongliang Wu, Joo Hwee Lim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15839
• Pdf URL: https://arxiv.org/pdf/2501.15839
• Copy Paste: [[2501.15839]] Controllable Hand Grasp Generation for HOI and Efficient Evaluation Methods(https://arxiv.org/abs/2501.15839)
• Keywords: generation
• Abstract: Controllable affordance Hand-Object Interaction (HOI) generation has become an increasingly important area of research in computer vision. In HOI generation, the hand grasp generation is a crucial step for effectively controlling the geometry of the hand. Current hand grasp generation methods rely on 3D information for both the hand and the object. In addition, these methods lack controllability concerning the hand's location and orientation. We treat the hand pose as the discrete graph structure and exploit the geometric priors. It is well established that higher order contextual dependency among the points improves the quality of the results in general. We propose a framework of higher order geometric representations (HOR's) inspired by spectral graph theory and vector algebra to improve the quality of generated hand poses. We demonstrate the effectiveness of our proposed HOR's in devising a controllable novel diffusion method (based on 2D information) for hand grasp generation that outperforms the state of the art (SOTA). Overcoming the limitations of existing methods: like lacking of controllability and dependency on 3D information. Once we have the generated pose, it is very natural to evaluate them using a metric. Popular metrics like FID and MMD are biased and inefficient for evaluating the generated hand poses. Using our proposed HOR's, we introduce an efficient and stable framework of evaluation metrics for grasp generation methods, addressing inefficiencies and biases in FID and MMD.
• 摘要：可控可供性手-物体交互 (HOI) 生成已成为计算机视觉领域中越来越重要的研究领域。在 HOI 生成中，手握生成是有效控制手的几何形状的关键步骤。当前的手握生成方法依赖于手和物体的 3D 信息。此外，这些方法缺乏对手的位置和方向的可控性。我们将手势视为离散图结构并利用几何先验。众所周知，点之间的高阶上下文依赖性通常会提高结果的质量。我们提出了一个受谱图理论和向量代数启发的高阶几何表示 (HOR) 框架，以提高生成的手势的质量。我们证明了我们提出的 HOR 在设计一种可控的新型扩散方法（基于 2D 信息）方面是有效的，该方法用于手握生成，其性能优于最先进的技术 (SOTA)。克服现有方法的局限性：如缺乏可控性和对 3D 信息的依赖。一旦我们有了生成的姿势，使用指标来评估它们是很自然的。FID 和 MMD 等流行指标在评估生成的手势时存在偏差且效率低下。使用我们提出的 HOR，我们为抓握生成方法引入了一个高效且稳定的评估指标框架，解决了 FID 和 MMD 中的低效率和偏差问题。

Title: Can Location Embeddings Enhance Super-Resolution of Satellite Imagery?

• Authors: Daniel Panangian, Ksenia Bittner
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15847
• Pdf URL: https://arxiv.org/pdf/2501.15847
• Copy Paste: [[2501.15847]] Can Location Embeddings Enhance Super-Resolution of Satellite Imagery?(https://arxiv.org/abs/2501.15847)
• Keywords: super-resolution, generation, generative
• Abstract: Publicly available satellite imagery, such as Sentinel- 2, often lacks the spatial resolution required for accurate analysis of remote sensing tasks including urban planning and disaster response. Current super-resolution techniques are typically trained on limited datasets, leading to poor generalization across diverse geographic regions. In this work, we propose a novel super-resolution framework that enhances generalization by incorporating geographic context through location embeddings. Our framework employs Generative Adversarial Networks (GANs) and incorporates techniques from diffusion models to enhance image quality. Furthermore, we address tiling artifacts by integrating information from neighboring images, enabling the generation of seamless, high-resolution outputs. We demonstrate the effectiveness of our method on the building segmentation task, showing significant improvements over state-of-the-art methods and highlighting its potential for real-world applications.
• 摘要：公开的卫星图像（例如 Sentinel-2）通常缺乏精确分析遥感任务（包括城市规划和灾难响应）所需的空间分辨率。当前的超分辨率技术通常是在有限的数据集上进行训练的，导致在不同地理区域的泛化能力较差。在这项工作中，我们提出了一种新颖的超分辨率框架，该框架通过位置嵌入结合地理环境来增强泛化能力。我们的框架采用生成对抗网络 (GAN) 并结合扩散模型中的技术来提高图像质量。此外，我们通过整合来自邻近图像的信息来解决平铺伪影问题，从而生成无缝的高分辨率输出。我们证明了我们的方法在建筑物分割任务上的有效性，与最先进的方法相比有显著的改进，并突出了其在实际应用中的潜力。

Title: LLM-attacker: Enhancing Closed-loop Adversarial Scenario Generation for Autonomous Driving with Large Language Models

• Authors: Yuewen Mei, Tong Nie, Jian Sun, Ye Tian
• Subjects: cs.LG, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.15850
• Pdf URL: https://arxiv.org/pdf/2501.15850
• Copy Paste: [[2501.15850]] LLM-attacker: Enhancing Closed-loop Adversarial Scenario Generation for Autonomous Driving with Large Language Models(https://arxiv.org/abs/2501.15850)
• Keywords: generation
• Abstract: Ensuring and improving the safety of autonomous driving systems (ADS) is crucial for the deployment of highly automated vehicles, especially in safety-critical events. To address the rarity issue, adversarial scenario generation methods are developed, in which behaviors of traffic participants are manipulated to induce safety-critical events. However, existing methods still face two limitations. First, identification of the adversarial participant directly impacts the effectiveness of the generation. However, the complexity of real-world scenarios, with numerous participants and diverse behaviors, makes identification challenging. Second, the potential of generated safety-critical scenarios to continuously improve ADS performance remains underexplored. To address these issues, we propose LLM-attacker: a closed-loop adversarial scenario generation framework leveraging large language models (LLMs). Specifically, multiple LLM agents are designed and coordinated to identify optimal attackers. Then, the trajectories of the attackers are optimized to generate adversarial scenarios. These scenarios are iteratively refined based on the performance of ADS, forming a feedback loop to improve ADS. Experimental results show that LLM-attacker can create more dangerous scenarios than other methods, and the ADS trained with it achieves a collision rate half that of training with normal scenarios. This indicates the ability of LLM-attacker to test and enhance the safety and robustness of ADS. Video demonstrations are provided at: this https URL.
• 摘要：确保和提高自动驾驶系统 (ADS) 的安全性对于高度自动化车辆的部署至关重要，尤其是在安全关键事件中。为了解决稀缺性问题，开发了对抗场景生成方法，其中操纵交通参与者的行为以诱发安全关键事件。然而，现有方法仍然面临两个限制。首先，对抗参与者的识别直接影响生成的有效性。然而，现实世界场景的复杂性，参与者众多，行为多样，使得识别具有挑战性。其次，生成的安全关键场景持续提高 ADS 性能的潜力仍未得到充分挖掘。为了解决这些问题，我们提出了 LLM-attacker：一个利用大型语言模型 (LLM) 的闭环对抗场景生成框架。具体来说，设计和协调多个 LLM 代理以识别最佳攻击者。然后，优化攻击者的轨迹以生成对抗场景。这些场景根据 ADS 的性能进行迭代改进，形成一个反馈循环以改进 ADS。实验结果表明，LLM-attacker 可以创建比其他方法更危险的场景，而用它训练的 ADS 的碰撞率仅为用正常场景训练的一半。这表明 LLM-attacker 能够测试和增强 ADS 的安全性和鲁棒性。视频演示位于：此 https URL。

Title: CausalSR: Structural Causal Model-Driven Super-Resolution with Counterfactual Inference

• Authors: Zhengyang Lu, Bingjie Lu, Feng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15852
• Pdf URL: https://arxiv.org/pdf/2501.15852
• Copy Paste: [[2501.15852]] CausalSR: Structural Causal Model-Driven Super-Resolution with Counterfactual Inference(https://arxiv.org/abs/2501.15852)
• Keywords: restoration, super-resolution
• Abstract: Physical and optical factors interacting with sensor characteristics create complex image degradation patterns. Despite advances in deep learning-based super-resolution, existing methods overlook the causal nature of degradation by adopting simplistic black-box mappings. This paper formulates super-resolution using structural causal models to reason about image degradation processes. We establish a mathematical foundation that unifies principles from causal inference, deriving necessary conditions for identifying latent degradation mechanisms and corresponding propagation. We propose a novel counterfactual learning strategy that leverages semantic guidance to reason about hypothetical degradation scenarios, leading to theoretically-grounded representations that capture invariant features across different degradation conditions. The framework incorporates an adaptive intervention mechanism with provable bounds on treatment effects, allowing precise manipulation of degradation factors while maintaining semantic consistency. Through extensive empirical validation, we demonstrate that our approach achieves significant improvements over state-of-the-art methods, particularly in challenging scenarios with compound degradations. On standard benchmarks, our method consistently outperforms existing approaches by significant margins (0.86-1.21dB PSNR), while providing interpretable insights into the restoration process. The theoretical framework and empirical results demonstrate the fundamental importance of causal reasoning in understanding image restoration systems.
• 摘要：与传感器特性相互作用的物理和光学因素会产生复杂的图像退化模式。尽管基于深度学习的超分辨率技术取得了进展，但现有方法通过采用简单的黑盒映射而忽略了退化的因果性质。本文使用结构因果模型来推理图像退化过程，从而制定超分辨率。我们建立了一个数学基础，将因果推理的原理统一起来，推导出识别潜在退化机制和相应传播的必要条件。我们提出了一种新颖的反事实学习策略，利用语义指导来推理假设的退化场景，从而产生基于理论的表示，以捕捉不同退化条件下的不变特征。该框架结合了一种自适应干预机制，对治疗效果具有可证明的界限，允许精确操纵退化因素，同时保持语义一致性。通过广泛的实证验证，我们证明我们的方法比最先进的方法取得了显着的改进，特别是在具有复合退化的具有挑战性的场景中。在标准基准测试中，我们的方法始终以显著的优势 (0.86-1.21dB PSNR) 优于现有方法，同时为恢复过程提供了可解释的见解。理论框架和实证结果证明了因果推理在理解图像恢复系统中的根本重要性。

Title: Slot-Guided Adaptation of Pre-trained Diffusion Models for Object-Centric Learning and Compositional Generation

• Authors: Adil Kaan Akan, Yucel Yemez
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.15878
• Pdf URL: https://arxiv.org/pdf/2501.15878
• Copy Paste: [[2501.15878]] Slot-Guided Adaptation of Pre-trained Diffusion Models for Object-Centric Learning and Compositional Generation(https://arxiv.org/abs/2501.15878)
• Keywords: generation, generative
• Abstract: We present SlotAdapt, an object-centric learning method that combines slot attention with pretrained diffusion models by introducing adapters for slot-based conditioning. Our method preserves the generative power of pretrained diffusion models, while avoiding their text-centric conditioning bias. We also incorporate an additional guidance loss into our architecture to align cross-attention from adapter layers with slot attention. This enhances the alignment of our model with the objects in the input image without using external supervision. Experimental results show that our method outperforms state-of-the-art techniques in object discovery and image generation tasks across multiple datasets, including those with real images. Furthermore, we demonstrate through experiments that our method performs remarkably well on complex real-world images for compositional generation, in contrast to other slot-based generative methods in the literature. The project page can be found at $\href{this https URL}{\text{this https url}}$.
• 摘要：我们提出了 SlotAdapt，这是一种以对象为中心的学习方法，通过引入基于槽位的条件适配器，将槽位注意力与预训练扩散模型相结合。我们的方法保留了预训练扩散模型的生成能力，同时避免了它们以文本为中心的条件偏差。我们还将额外的指导损失纳入我们的架构中，以将适配器层的交叉注意力与槽位注意力对齐。这增强了我们的模型与输入图像中对象的对齐，而无需使用外部监督。实验结果表明，我们的方法在多个数据集（包括具有真实图像的数据集）中的对象发现和图像生成任务中优于最先进的技术。此外，我们通过实验证明，与文献中其他基于槽位的生成方法相比，我们的方法在复杂的现实世界图像的构图生成方面表现非常出色。项目页面可以在 $\href{this https URL}{\text{this https url}}$ 找到。

Title: Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Tasks

• Authors: Hailong Guo, Bohan Zeng, Yiren Song, Wentao Zhang, Chuang Zhang, Jiaming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.15891
• Pdf URL: https://arxiv.org/pdf/2501.15891
• Copy Paste: [[2501.15891]] Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Tasks(https://arxiv.org/abs/2501.15891)
• Keywords: generation
• Abstract: Image-based virtual try-on (VTON) aims to generate a virtual try-on result by transferring an input garment onto a target person's image. However, the scarcity of paired garment-model data makes it challenging for existing methods to achieve high generalization and quality in VTON. Also, it limits the ability to generate mask-free try-ons. To tackle the data scarcity problem, approaches such as Stable Garment and MMTryon use a synthetic data strategy, effectively increasing the amount of paired data on the model side. However, existing methods are typically limited to performing specific try-on tasks and lack user-friendliness. To enhance the generalization and controllability of VTON generation, we propose Any2AnyTryon, which can generate try-on results based on different textual instructions and model garment images to meet various needs, eliminating the reliance on masks, poses, or other conditions. Specifically, we first construct the virtual try-on dataset LAION-Garment, the largest known open-source garment try-on dataset. Then, we introduce adaptive position embedding, which enables the model to generate satisfactory outfitted model images or garment images based on input images of different sizes and categories, significantly enhancing the generalization and controllability of VTON generation. In our experiments, we demonstrate the effectiveness of our Any2AnyTryon and compare it with existing methods. The results show that Any2AnyTryon enables flexible, controllable, and high-quality image-based virtual try-on this http URL://lognthis http URL
• 摘要：基于图像的虚拟试穿 (VTON) 旨在通过将输入服装转移到目标人的图像上来生成虚拟试穿结果。然而，配对服装模型数据的稀缺性使得现有方法难以在 VTON 中实现高泛化和高质量。此外，它限制了生成无口罩试穿的能力。为了解决数据稀缺问题，Stable Garment 和 MMTryon 等方法使用合成数据策略，有效地增加了模型端的配对数据量。然而，现有方法通常仅限于执行特定的试穿任务，缺乏用户友好性。为了增强 VTON 生成的泛化和可控性，我们提出了 Any2AnyTryon，它可以根据不同的文本指令生成试穿结果并建模服装图像以满足各种需求，消除了对口罩、姿势或其他条件的依赖。具体来说，我们首先构建虚拟试穿数据集 LAION-Garment，这是已知最大的开源服装试穿数据集。然后，我们引入了自适应位置嵌入，使模型能够根据不同尺寸和类别的输入图像生成令人满意的着装模型图像或服装图像，从而显著增强了 VTON 生成的泛化和可控性。在实验中，我们证明了 Any2AnyTryon 的有效性，并将其与现有方法进行了比较。结果表明，Any2AnyTryon 可以实现灵活、可控、高质量的基于图像的虚拟试穿。这个 http URL://logn这个 http URL

Title: Freestyle Sketch-in-the-Loop Image Segmentation

• Authors: Subhadeep Koley, Viswanatha Reddy Gajjala, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, Ayan Kumar Bhunia, Yi-Zhe Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16022
• Pdf URL: https://arxiv.org/pdf/2501.16022
• Copy Paste: [[2501.16022]] Freestyle Sketch-in-the-Loop Image Segmentation(https://arxiv.org/abs/2501.16022)
• Keywords: generation
• Abstract: In this paper, we expand the domain of sketch research into the field of image segmentation, aiming to establish freehand sketches as a query modality for subjective image segmentation. Our innovative approach introduces a "sketch-in-the-loop" image segmentation framework, enabling the segmentation of visual concepts partially, completely, or in groupings - a truly "freestyle" approach - without the need for a purpose-made dataset (i.e., mask-free). This framework capitalises on the synergy between sketch-based image retrieval (SBIR) models and large-scale pre-trained models (CLIP or DINOv2). The former provides an effective training signal, while fine-tuned versions of the latter execute the subjective segmentation. Additionally, our purpose-made augmentation strategy enhances the versatility of our sketch-guided mask generation, allowing segmentation at multiple granularity levels. Extensive evaluations across diverse benchmark datasets underscore the superior performance of our method in comparison to existing approaches across various evaluation scenarios.
• 摘要：在本文中，我们将草图研究领域扩展到图像分割领域，旨在将手绘草图作为主观图像分割的查询模式。我们的创新方法引入了一个“草图在环”图像分割框架，可以部分、完全或分组地分割视觉概念——一种真正的“自由式”方法——而无需专门制作的数据集（即无掩模）。该框架利用了基于草图的图像检索 (SBIR) 模型和大规模预训练模型 (CLIP 或 DINOv2) 之间的协同作用。前者提供了有效的训练信号，而后者的微调版本执行主观分割。此外，我们专门设计的增强策略增强了我们草图引导掩模生成的多功能性，允许在多个粒度级别进行分割。对各种基准数据集的广泛评估强调了我们的方法与各种评估场景中的现有方法相比具有卓越的性能。

Title: Addressing Out-of-Label Hazard Detection in Dashcam Videos: Insights from the COOOL Challenge

• Authors: Anh-Kiet Duong, Petra Gomez-Krämer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16037
• Pdf URL: https://arxiv.org/pdf/2501.16037
• Copy Paste: [[2501.16037]] Addressing Out-of-Label Hazard Detection in Dashcam Videos: Insights from the COOOL Challenge(https://arxiv.org/abs/2501.16037)
• Keywords: generation
• Abstract: This paper presents a novel approach for hazard analysis in dashcam footage, addressing the detection of driver reactions to hazards, the identification of hazardous objects, and the generation of descriptive captions. We first introduce a method for detecting driver reactions through speed and sound anomaly detection, leveraging unsupervised learning techniques. For hazard detection, we employ a set of heuristic rules as weak classifiers, which are combined using an ensemble method. This ensemble approach is further refined with differential privacy to mitigate overconfidence, ensuring robustness despite the lack of labeled data. Lastly, we use state-of-the-art vision-language models for hazard captioning, generating descriptive labels for the detected hazards. Our method achieved the highest scores in the Challenge on Out-of-Label in Autonomous Driving, demonstrating its effectiveness across all three tasks. Source codes are publicly available at this https URL.
• 摘要：本文介绍了一种用于行车记录仪镜头中危险分析的新方法，该方法解决了驾驶员对危险的反应检测、危险物体的识别以及描述性字幕的生成问题。我们首先介绍一种通过速度和声音异常检测来检测驾驶员反应的方法，该方法利用了无监督学习技术。对于危险检测，我们使用一组启发式规则作为弱分类器，并使用集成方法将它们组合在一起。这种集成方法通过差分隐私进一步完善，以减轻过度自信，确保在缺乏标记数据的情况下也能保持稳健性。最后，我们使用最先进的视觉语言模型来制作危险字幕，为检测到的危险生成描述性标签。我们的方法在自动驾驶标签外挑战赛中获得了最高分，证明了其在所有三个任务中的有效性。源代码可在此 https URL 上公开获取。

Title: Generating Spatial Synthetic Populations Using Wasserstein Generative Adversarial Network: A Case Study with EU-SILC Data for Helsinki and Thessaloniki

• Authors: Vanja Falck
• Subjects: cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.16080
• Pdf URL: https://arxiv.org/pdf/2501.16080
• Copy Paste: [[2501.16080]] Generating Spatial Synthetic Populations Using Wasserstein Generative Adversarial Network: A Case Study with EU-SILC Data for Helsinki and Thessaloniki(https://arxiv.org/abs/2501.16080)
• Keywords: generation, generative
• Abstract: Using agent-based social simulations can enhance our understanding of urban planning, public health, and economic forecasting. Realistic synthetic populations with numerous attributes strengthen these simulations. The Wasserstein Generative Adversarial Network, trained on census data like EU-SILC, can create robust synthetic populations. These methods, aided by external statistics or EU-SILC weights, generate spatial synthetic populations for agent-based models. The increased access to high-quality micro-data has sparked interest in synthetic populations, which preserve demographic profiles and analytical strength while ensuring privacy and preventing discrimination. This study uses national data from Finland and Greece for Helsinki and Thessaloniki to explore balanced spatial synthetic population generation. Results show challenges related to balancing data with or without aggregated statistics for the target population and the general under-representation of fringe profiles by deep generative methods. The latter can lead to discrimination in agent-based simulations.
• 摘要：使用基于代理的社会模拟可以增强我们对城市规划、公共卫生和经济预测的理解。具有众多属性的现实合成人口可以加强这些模拟。在 EU-SILC 等人口普查数据上训练的 Wasserstein 生成对抗网络可以创建强大的合成人口。这些方法在外部统计数据或 EU-SILC 权重的帮助下，为基于代理的模型生成空间合成人口。高质量微观数据的获取途径增加，引发了人们对合成人口的兴趣，合成人口保留了人口统计资料和分析实力，同时确保了隐私并防止了歧视。本研究使用芬兰和希腊赫尔辛基和塞萨洛尼基的国家数据来探索平衡的空间合成人口生成。结果显示了平衡数据（无论是否有目标人群的汇总统计数据）的挑战，以及深度生成方法对边缘概况的普遍代表性不足。后者可能导致基于代理的模拟中的歧视。

Title: ARFlow: Autogressive Flow with Hybrid Linear Attention

• Authors: Mude Hui, Rui-Jie Zhu, Songlin Yang, Yu Zhang, Zirui Wang, Yuyin Zhou, Jason Eshraghian, Cihang Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16085
• Pdf URL: https://arxiv.org/pdf/2501.16085
• Copy Paste: [[2501.16085]] ARFlow: Autogressive Flow with Hybrid Linear Attention(https://arxiv.org/abs/2501.16085)
• Keywords: generation
• Abstract: Flow models are effective at progressively generating realistic images, but they generally struggle to capture long-range dependencies during the generation process as they compress all the information from previous time steps into a single corrupted image. To address this limitation, we propose integrating autoregressive modeling -- known for its excellence in modeling complex, high-dimensional joint probability distributions -- into flow models. During training, at each step, we construct causally-ordered sequences by sampling multiple images from the same semantic category and applying different levels of noise, where images with higher noise levels serve as causal predecessors to those with lower noise levels. This design enables the model to learn broader category-level variations while maintaining proper causal relationships in the flow process. During generation, the model autoregressively conditions the previously generated images from earlier denoising steps, forming a contextual and coherent generation trajectory. Additionally, we design a customized hybrid linear attention mechanism tailored to our modeling approach to enhance computational efficiency. Our approach, termed ARFlow, under 400k training steps, achieves 14.08 FID scores on ImageNet at 128 * 128 without classifier-free guidance, reaching 4.34 FID with classifier-free guidance 1.5, significantly outperforming the previous flow-based model SiT's 9.17 FID. Extensive ablation studies demonstrate the effectiveness of our modeling strategy and chunk-wise attention design.
• 摘要：流动模型可以有效地逐步生成逼真的图像，但它们通常难以在生成过程中捕捉到长距离依赖关系，因为它们将来自先前时间步骤的所有信息压缩成单个损坏的图像。为了解决这一限制，我们建议将自回归建模（以其在建模复杂、高维联合概率分布方面的出色表现而闻名）集成到流动模型中。在训练期间，在每个步骤中，我们通过从同一语义类别中采样多张图像并应用不同级别的噪声来构建因果排序序列，其中噪声水平较高的图像作为噪声水平较低的图像的因果前身。这种设计使模型能够学习更广泛的类别级变化，同时在流动过程中保持适当的因果关系。在生成过程中，该模型自回归地调节先前去噪步骤中生成的图像，形成上下文连贯的生成轨迹。此外，我们设计了一种定制的混合线性注意机制，以适应我们的建模方法，以提高计算效率。我们的方法称为 ARFlow，在 400k 个训练步骤下，在没有无分类器指导的情况下在 128 * 128 的 ImageNet 上实现了 14.08 FID 分数，在无分类器指导的情况下达到 4.34 FID 1.5，明显优于之前基于流的模型 SiT 的 9.17 FID。广泛的消融研究证明了我们的建模策略和块级注意力设计的有效性。

Title: Efficient Portrait Matte Creation With Layer Diffusion and Connectivity Priors

• Authors: Zhiyuan Lu, Hao Lu, Hua Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16147
• Pdf URL: https://arxiv.org/pdf/2501.16147
• Copy Paste: [[2501.16147]] Efficient Portrait Matte Creation With Layer Diffusion and Connectivity Priors(https://arxiv.org/abs/2501.16147)
• Keywords: generation
• Abstract: Learning effective deep portrait matting models requires training data of both high quality and large quantity. Neither quality nor quantity can be easily met for portrait matting, however. Since the most accurate ground-truth portrait mattes are acquired in front of the green screen, it is almost impossible to harvest a large-scale portrait matting dataset in reality. This work shows that one can leverage text prompts and the recent Layer Diffusion model to generate high-quality portrait foregrounds and extract latent portrait mattes. However, the portrait mattes cannot be readily in use due to significant generation artifacts. Inspired by the connectivity priors observed in portrait images, that is, the border of portrait foregrounds always appears connected, a connectivity-aware approach is introduced to refine portrait mattes. Building on this, a large-scale portrait matting dataset is created, termed LD-Portrait-20K, with $20,051$ portrait foregrounds and high-quality alpha mattes. Extensive experiments demonstrated the value of the LD-Portrait-20K dataset, with models trained on it significantly outperforming those trained on other datasets. In addition, comparisons with the chroma keying algorithm and an ablation study on dataset capacity further confirmed the effectiveness of the proposed matte creation approach. Further, the dataset also contributes to state-of-the-art video portrait matting, implemented by simple video segmentation and a trimap-based image matting model trained on this dataset.
• 摘要：学习有效的深度人像抠图模型需要高质量和大量的训练数据。然而，人像抠图的质量和数量都很难满足。由于最准确的真实人像抠图是在绿屏前获取的，因此在现实中几乎不可能收集大规模的人像抠图数据集。这项工作表明，可以利用文本提示和最近的层扩散模型来生成高质量的人像前景并提取潜在的人像抠图。然而，由于严重的生成伪影，人像抠图无法随时使用。受在人像图像中观察到的连通性先验的启发，即人像前景的边界总是看起来是连通的，我们引入了一种连通性感知方法来细化人像抠图。在此基础上，我们创建了一个大规模的人像抠图数据集，称为 LD-Portrait-20K，其中包含 $20,051$ 个人像前景和高质量的 alpha 抠图。大量实验证明了 LD-Portrait-20K 数据集的价值，在该数据集上训练的模型明显优于在其他数据集上训练的模型。此外，与色度键控算法的比较和对数据集容量的消融研究进一步证实了所提出的遮罩创建方法的有效性。此外，该数据集还为最先进的视频肖像抠像做出了贡献，通过简单的视频分割和在该数据集上训练的基于三分图的图像抠像模型来实现。

Title: BAG: Body-Aligned 3D Wearable Asset Generation

• Authors: Zhongjin Luo, Yang Li, Mingrui Zhang, Senbo Wang, Han Yan, Xibin Song, Taizhang Shang, Wei Mao, Hongdong Li, Xiaoguang Han, Pan Ji
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.16177
• Pdf URL: https://arxiv.org/pdf/2501.16177
• Copy Paste: [[2501.16177]] BAG: Body-Aligned 3D Wearable Asset Generation(https://arxiv.org/abs/2501.16177)
• Keywords: generation
• Abstract: While recent advancements have shown remarkable progress in general 3D shape generation models, the challenge of leveraging these approaches to automatically generate wearable 3D assets remains unexplored. To this end, we present BAG, a Body-aligned Asset Generation method to output 3D wearable asset that can be automatically dressed on given 3D human bodies. This is achived by controlling the 3D generation process using human body shape and pose information. Specifically, we first build a general single-image to consistent multiview image diffusion model, and train it on the large Objaverse dataset to achieve diversity and generalizability. Then we train a Controlnet to guide the multiview generator to produce body-aligned multiview images. The control signal utilizes the multiview 2D projections of the target human body, where pixel values represent the XYZ coordinates of the body surface in a canonical space. The body-conditioned multiview diffusion generates body-aligned multiview images, which are then fed into a native 3D diffusion model to produce the 3D shape of the asset. Finally, by recovering the similarity transformation using multiview silhouette supervision and addressing asset-body penetration with physics simulators, the 3D asset can be accurately fitted onto the target human body. Experimental results demonstrate significant advantages over existing methods in terms of image prompt-following capability, shape diversity, and shape quality. Our project page is available at this https URL.
• 摘要：虽然最近的进展表明通用 3D 形状生成模型取得了显著进展，但利用这些方法自动生成可穿戴 3D 资产的挑战仍未得到探索。为此，我们提出了 BAG，一种身体对齐资产生成方法，用于输出可自动穿戴在给定 3D 人体上的 3D 可穿戴资产。这是通过使用人体形状和姿势信息控制 3D 生成过程来实现的。具体来说，我们首先构建一个通用的单图像到一致的多视图图像扩散模型，并在大型 Objaverse 数据集上对其进行训练，以实现多样性和通用性。然后我们训练一个 Controlnet 来指导多视图生成器生成身体对齐的多视图图像。控制信号利用目标人体的多视图 2D 投影，其中像素值表示规范空间中身体表面的 XYZ 坐标。身体调节的多视图扩散生成身体对齐的多视图图像，然后将其输入到原生 3D 扩散模型中以生成资产的 3D 形状。最后，通过使用多视图轮廓监督恢复相似性变换并使用物理模拟器解决资产与人体的穿透问题，可以将 3D 资产准确地贴合到目标人体上。实验结果表明，在图像提示跟随能力、形状多样性和形状质量方面，该方法比现有方法具有显著优势。我们的项目页面位于此 https URL。

Title: SPECIAL: Zero-shot Hyperspectral Image Classification With CLIP

• Authors: Li Pang, Jing Yao, Kaiyu Li, Xiangyong Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16222
• Pdf URL: https://arxiv.org/pdf/2501.16222
• Copy Paste: [[2501.16222]] SPECIAL: Zero-shot Hyperspectral Image Classification With CLIP(https://arxiv.org/abs/2501.16222)
• Keywords: generation
• Abstract: Hyperspectral image (HSI) classification aims at categorizing each pixel in an HSI into a specific land cover class, which is crucial for applications like remote sensing, environmental monitoring, and agriculture. Although deep learning-based HSI classification methods have achieved significant advancements, existing methods still rely on manually labeled data for training, which is both time-consuming and this http URL address this limitation, we introduce a novel zero-shot hyperspectral image classification framework based on CLIP (SPECIAL), aiming to eliminate the need for manual annotations. The SPECIAL framework consists of two main stages: (1) CLIP-based pseudo-label generation, and (2) noisy label learning. In the first stage, HSI is spectrally interpolated to produce RGB bands. These bands are subsequently classified using CLIP, resulting in noisy pseudo-labels that are accompanied by confidence this http URL improve the quality of these labels, we propose a scaling strategy that fuses predictions from multiple spatial scales. In the second stage, spectral information and a label refinement technique are incorporated to mitigate label noise and further enhance classification accuracy. Experimental results on three benchmark datasets demonstrate that our SPECIAL outperforms existing methods in zero-shot HSI classification, showing its potential for more practical applications. The code is available at this https URL.
• 摘要：高光谱图像 (HSI) 分类旨在将 HSI 中的每个像素归类为特定的土地覆盖类别，这对于遥感、环境监测和农业等应用至关重要。尽管基于深度学习的 HSI 分类方法取得了重大进展，但现有方法仍然依赖手动标记的数据进行训练，这既耗时又费力。为了解决这一限制，我们引入了一种基于 CLIP (SPECIAL) 的新型零样本高光谱图像分类框架，旨在消除对手动注释的需求。SPECIAL 框架由两个主要阶段组成：(1) 基于 CLIP 的伪标签生成，以及 (2) 噪声标签学习。在第一阶段，对 HSI 进行光谱插值以产生 RGB 波段。随后使用 CLIP 对这些波段进行分类，从而产生带有置信度的噪声伪标签。为了提高这些标签的质量，我们提出了一种融合多个空间尺度预测的缩放策略。在第二阶段，结合光谱信息和标签细化技术来减轻标签噪声并进一步提高分类准确性。在三个基准数据集上的实验结果表明，我们的 SPECIAL 在零样本 HSI 分类中的表现优于现有方法，显示出其在更多实际应用中的潜力。代码可在此 https URL 上获取。

Title: Phase Transitions in Large Language Models and the $O(N)$ Model

• Authors: Youran Sun, Babak Haghighat
• Subjects: cs.LG, cs.CL, hep-th, physics.data-an
• Abstract URL: https://arxiv.org/abs/2501.16241
• Pdf URL: https://arxiv.org/pdf/2501.16241
• Copy Paste: [[2501.16241]] Phase Transitions in Large Language Models and the $O(N)$ Model(https://arxiv.org/abs/2501.16241)
• Keywords: generation
• Abstract: Large language models (LLMs) exhibit unprecedentedly rich scaling behaviors. In physics, scaling behavior is closely related to phase transitions, critical phenomena, and field theory. To investigate the phase transition phenomena in LLMs, we reformulated the Transformer architecture as an $O(N)$ model. Our study reveals two distinct phase transitions corresponding to the temperature used in text generation and the model's parameter size, respectively. The first phase transition enables us to estimate the internal dimension of the model, while the second phase transition is of \textit{higher-depth} and signals the emergence of new capabilities. As an application, the energy of the $O(N)$ model can be used to evaluate whether an LLM's parameters are sufficient to learn the training data.
• 摘要：大型语言模型 (LLM) 表现出前所未有的丰富缩放行为。在物理学中，缩放行为与相变、临界现象和场论密切相关。为了研究 LLM 中的相变现象，我们将 Transformer 架构重新表述为 $O(N)$ 模型。我们的研究揭示了两个不同的相变，分别对应于文本生成中使用的温度和模型的参数大小。第一个相变使我们能够估计模型的内部维度，而第二个相变具有 \textit{更高深度} 并标志着新功能的出现。作为一种应用，$O(N)$ 模型的能量可用于评估 LLM 的参数是否足以学习训练数据。

Title: RelightVid: Temporal-Consistent Diffusion Model for Video Relighting

• Authors: Ye Fang, Zeyi Sun, Shangzhan Zhang, Tong Wu, Yinghao Xu, Pan Zhang, Jiaqi Wang, Gordon Wetzstein, Dahua Lin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16330
• Pdf URL: https://arxiv.org/pdf/2501.16330
• Copy Paste: [[2501.16330]] RelightVid: Temporal-Consistent Diffusion Model for Video Relighting(https://arxiv.org/abs/2501.16330)
• Keywords: generation
• Abstract: Diffusion models have demonstrated remarkable success in image generation and editing, with recent advancements enabling albedo-preserving image relighting. However, applying these models to video relighting remains challenging due to the lack of paired video relighting datasets and the high demands for output fidelity and temporal consistency, further complicated by the inherent randomness of diffusion models. To address these challenges, we introduce RelightVid, a flexible framework for video relighting that can accept background video, text prompts, or environment maps as relighting conditions. Trained on in-the-wild videos with carefully designed illumination augmentations and rendered videos under extreme dynamic lighting, RelightVid achieves arbitrary video relighting with high temporal consistency without intrinsic decomposition while preserving the illumination priors of its image backbone.
• 摘要：扩散模型在图像生成和编辑方面取得了显著的成功，最近的进展使得保留反照率的图像重新照明成为可能。然而，由于缺乏成对的视频重新照明数据集以及对输出保真度和时间一致性的高要求，将这些模型应用于视频重新照明仍然具有挑战性，而扩散模型固有的随机性使情况进一步复杂化。为了应对这些挑战，我们推出了 RelightVid，这是一个灵活的视频重新照明框架，可以接受背景视频、文本提示或环境地图作为重新照明条件。RelightVid 使用精心设计的照明增强技术对野外视频进行训练，并在极端动态照明下渲染视频，实现了任意视频重新照明，具有高度的时间一致性，无需内在分解，同时保留了其图像主干的照明先验。