2025-01-29

Title: An Integrated Approach to AI-Generated Content in e-health

• Authors: Tasnim Ahmed, Salimur Choudhury
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16348
• Pdf URL: https://arxiv.org/pdf/2501.16348
• Copy Paste: [[2501.16348]] An Integrated Approach to AI-Generated Content in e-health(https://arxiv.org/abs/2501.16348)
• Keywords: generative
• Abstract: Artificial Intelligence-Generated Content, a subset of Generative Artificial Intelligence, holds significant potential for advancing the e-health sector by generating diverse forms of data. In this paper, we propose an end-to-end class-conditioned framework that addresses the challenge of data scarcity in health applications by generating synthetic medical images and text data, evaluating on practical applications such as retinopathy detection, skin infections and mental health assessments. Our framework integrates Diffusion and Large Language Models (LLMs) to generate data that closely match real-world patterns, which is essential for improving downstream task performance and model robustness in e-health applications. Experimental results demonstrate that the synthetic images produced by the proposed diffusion model outperform traditional GAN architectures. Similarly, in the text modality, data generated by uncensored LLM achieves significantly better alignment with real-world data than censored models in replicating the authentic tone.
• 摘要：人工智能生成内容是生成人工智能的一个子集，通过生成多种形式的数据，它对推动电子健康领域的发展具有巨大潜力。在本文中，我们提出了一个端到端的类条件框架，通过生成合成的医学图像和文本数据来解决健康应用中数据稀缺的挑战，并对视网膜病变检测、皮肤感染和心理健康评估等实际应用进行评估。我们的框架集成了扩散和大型语言模型 (LLM) 来生成与现实世界模式紧密匹配的数据，这对于提高电子健康应用中的下游任务性能和模型鲁棒性至关重要。实验结果表明，所提出的扩散模型生成的合成图像优于传统的 GAN 架构。同样，在文本模态中，未经审查的 LLM 生成的数据在复制真实语气方面与现实世界数据的一致性明显优于受审查的模型。

Title: The OpenLAM Challenges

• Authors: Anyang Peng, Xinzijian Liu, Ming-Yu Guo, Linfeng Zhang, Han Wang
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2501.16358
• Pdf URL: https://arxiv.org/pdf/2501.16358
• Copy Paste: [[2501.16358]] The OpenLAM Challenges(https://arxiv.org/abs/2501.16358)
• Keywords: generative
• Abstract: Inspired by the success of Large Language Models (LLMs), the development of Large Atom Models (LAMs) has gained significant momentum in scientific computation. Since 2022, the Deep Potential team has been actively pretraining LAMs and launched the OpenLAM Initiative to develop an open-source foundation model spanning the periodic table. A core objective is establishing comprehensive benchmarks for reliable LAM evaluation, addressing limitations in existing datasets. As a first step, the LAM Crystal Philately competition has collected over 19.8 million valid structures, including 1 million on the OpenLAM convex hull, driving advancements in generative modeling and materials science applications.
• 摘要：受大型语言模型 (LLM) 成功的启发，大型原子模型 (LAM) 的开发在科学计算领域获得了巨大的发展势头。自 2022 年以来，Deep Potential 团队一直在积极预训练 LAM，并启动了 OpenLAM 计划，以开发一个涵盖元素周期表的开源基础模型。核心目标是建立可靠的 LAM 评估的综合基准，解决现有数据集的局限性。作为第一步，LAM 水晶集邮比赛收集了超过 1980 万个有效结构，其中包括 OpenLAM 凸包上的 100 万个，推动了生成建模和材料科学应用的进步。

Title: Foundation Models for CPS-IoT: Opportunities and Challenges

• Authors: Ozan Baris, Yizhuo Chen, Gaofeng Dong, Liying Han, Tomoyoshi Kimura, Pengrui Quan, Ruijie Wang, Tianchen Wang, Tarek Abdelzaher, Mario Bergés, Paul Pu Liang, Mani Srivastava
• Subjects: cs.LG, cs.AI, eess.SY
• Abstract URL: https://arxiv.org/abs/2501.16368
• Pdf URL: https://arxiv.org/pdf/2501.16368
• Copy Paste: [[2501.16368]] Foundation Models for CPS-IoT: Opportunities and Challenges(https://arxiv.org/abs/2501.16368)
• Keywords: generation
• Abstract: Methods from machine learning (ML) have transformed the implementation of Perception-Cognition-Communication-Action loops in Cyber-Physical Systems (CPS) and the Internet of Things (IoT), replacing mechanistic and basic statistical models with those derived from data. However, the first generation of ML approaches, which depend on supervised learning with annotated data to create task-specific models, faces significant limitations in scaling to the diverse sensor modalities, deployment configurations, application tasks, and operating dynamics characterizing real-world CPS-IoT systems. The success of task-agnostic foundation models (FMs), including multimodal large language models (LLMs), in addressing similar challenges across natural language, computer vision, and human speech has generated considerable enthusiasm for and exploration of FMs and LLMs as flexible building blocks in CPS-IoT analytics pipelines, promising to reduce the need for costly task-specific engineering. Nonetheless, a significant gap persists between the current capabilities of FMs and LLMs in the CPS-IoT domain and the requirements they must meet to be viable for CPS-IoT applications. In this paper, we analyze and characterize this gap through a thorough examination of the state of the art and our research, which extends beyond it in various dimensions. Based on the results of our analysis and research, we identify essential desiderata that CPS-IoT domain-specific FMs and LLMs must satisfy to bridge this gap. We also propose actions by CPS-IoT researchers to collaborate in developing key community resources necessary for establishing FMs and LLMs as foundational tools for the next generation of CPS-IoT systems.
• 摘要：机器学习 (ML) 方法已经改变了信息物理系统 (CPS) 和物联网 (IoT) 中感知-认知-通信-行动循环的实现，用从数据中得出的模型取代了机械和基本统计​​模型。然而，第一代 ML 方法依赖于使用带注释数据的监督学习来创建特定于任务的模型，在扩展到表征现实世界 CPS-IoT 系统的各种传感器模式、部署配置、应用任务和操作动态方面面临着重大限制。任务无关基础模型 (FM)（包括多模态大型语言模型 (LLM)）在解决自然语言、计算机视觉和人类语音中的类似挑战方面取得了成功，这引起了人们对 FM 和 LLM 作为 CPS-IoT 分析管道中灵活构建块的极大热情和探索，有望减少对昂贵的特定于任务的工程的需求。尽管如此，CPS-IoT 领域中 FM 和 LLM 的当前能力与它们必须满足的 CPS-IoT 应用要求之间仍然存在巨大差距。在本文中，我们通过彻底检查最新技术水平和我们的研究来分析和描述这一差距，我们的研究在各个方面都超越了它。根据我们的分析和研究结果，我们确定了 CPS-IoT 领域特定的 FM 和 LLM 必须满足的基本要求才能弥补这一差距。我们还建议 CPS-IoT 研究人员采取行动，合作开发建立 FM 和 LLM 作为下一代 CPS-IoT 系统的基础工具所必需的关键社区资源。

Title: Optimal Signal Decomposition-based Multi-Stage Learning for Battery Health Estimation

• Authors: Vijay Babu Pamshetti, Wei Zhang, King Jet Tseng, Bor Kiat Ng, Qingyu Yan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16377
• Pdf URL: https://arxiv.org/pdf/2501.16377
• Copy Paste: [[2501.16377]] Optimal Signal Decomposition-based Multi-Stage Learning for Battery Health Estimation(https://arxiv.org/abs/2501.16377)
• Keywords: generation
• Abstract: Battery health estimation is fundamental to ensure battery safety and reduce cost. However, achieving accurate estimation has been challenging due to the batteries' complex nonlinear aging patterns and capacity regeneration phenomena. In this paper, we propose OSL, an optimal signal decomposition-based multi-stage machine learning for battery health estimation. OSL treats battery signals optimally. It uses optimized variational mode decomposition to extract decomposed signals capturing different frequency bands of the original battery signals. It also incorporates a multi-stage learning process to analyze both spatial and temporal battery features effectively. An experimental study is conducted with a public battery aging dataset. OSL demonstrates exceptional performance with a mean error of just 0.26%. It significantly outperforms comparison algorithms, both those without and those with suboptimal signal decomposition and analysis. OSL considers practical battery challenges and can be integrated into real-world battery management systems, offering a good impact on battery monitoring and optimization.
• 摘要：电池健康评估对于确保电池安全和降低成本至关重要。然而，由于电池复杂的非线性老化模式和容量再生现象，实现准确的评估一直具有挑战性。在本文中，我们提出了 OSL，一种基于最佳信号分解的多阶段机器学习，用于电池健康评估。OSL 对电池信号进行最佳处理。它使用优化的变分模态分解来提取捕获原始电池信号不同频带的分解信号。它还结合了多阶段学习过程，以有效地分析空间和时间电池特征。使用公共电池老化数据集进行了实验研究。OSL 表现出色，平均误差仅为 0.26%。它明显优于比较算法，无论是没有还是具有次优信号分解和分析的算法。OSL 考虑了实际的电池挑战，可以集成到现实世界的电池管理系统中，对电池监控和优化产生良好的影响。

Title: Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Update

• Authors: Qing Li, Jiahui Geng, Zongxiong Chen, Kun Song, Lei Ma, Fakhri Karray
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16378
• Pdf URL: https://arxiv.org/pdf/2501.16378
• Copy Paste: [[2501.16378]] Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Update(https://arxiv.org/abs/2501.16378)
• Keywords: generation
• Abstract: Vision-language models (VLMs) demonstrate strong multimodal capabilities but have been found to be more susceptible to generating harmful content compared to their backbone large language models (LLMs). Our investigation reveals that the integration of images significantly shifts the model's internal activations during the forward pass, diverging from those triggered by textual input. Moreover, the safety alignments of LLMs embedded within VLMs are not sufficiently robust to handle the activations discrepancies, making the models vulnerable to even the simplest jailbreaking attacks. To address this issue, we propose an \textbf{internal activation revision} approach that efficiently revises activations during generation, steering the model toward safer outputs. Our framework incorporates revisions at both the layer and head levels, offering control over the model's generation at varying levels of granularity. In addition, we explore three strategies for constructing positive and negative samples and two approaches for extracting revision vectors, resulting in different variants of our method. Comprehensive experiments demonstrate that the internal activation revision method significantly improves the safety of widely used VLMs, reducing attack success rates by an average of 48.94\%, 34.34\%, 43.92\%, and 52.98\% on SafeBench, Safe-Unsafe, Unsafe, and MM-SafetyBench, respectively, while minimally impacting model helpfulness.
• 摘要：视觉语言模型 (VLM) 展示了强大的多模态能力，但与它们的骨干大型语言模型 (LLM) 相比，它们更容易生成有害内容。我们的调查显示，图像的整合在前向传递过程中显著改变了模型的内部激活，与文本输入触发的激活不同。此外，嵌入在 VLM 中的 LLM 的安全对齐不够强大，无法处理激活差异，使得模型容易受到即使是最简单的越狱攻击的攻击。为了解决这个问题，我们提出了一种 \textbf{内部激活修订} 方法，可以在生成过程中有效地修改激活，引导模型实现更安全的输出。我们的框架在层和头两个层面都进行了修订，提供对不同粒度级别模型生成的控制。此外，我们探索了三种构建正样本和负样本的策略以及两种提取修订向量的方法，从而产生了我们方法的不同变体。综合实验表明，内部激活修订方法显著提高了广泛使用的 VLM 的安全性，在 SafeBench、Safe-Unsafe、Unsafe 和 MM-SafetyBench 上分别将攻击成功率平均降低了 48.94%、34.34%、43.92% 和 52.98%，同时对模型有用性的影响最小。

Title: UDiTQC: U-Net-Style Diffusion Transformer for Quantum Circuit Synthesis

• Authors: Zhiwei Chen, Hao Tang
• Subjects: cs.LG, cs.AI, quant-ph
• Abstract URL: https://arxiv.org/abs/2501.16380
• Pdf URL: https://arxiv.org/pdf/2501.16380
• Copy Paste: [[2501.16380]] UDiTQC: U-Net-Style Diffusion Transformer for Quantum Circuit Synthesis(https://arxiv.org/abs/2501.16380)
• Keywords: generation, generative
• Abstract: Quantum computing is a transformative technology with wide-ranging applications, and efficient quantum circuit generation is crucial for unlocking its full potential. Current diffusion model approaches based on U-Net architectures, while promising, encounter challenges related to computational efficiency and modeling global context. To address these issues, we propose UDiT,a novel U-Net-style Diffusion Transformer architecture, which combines U-Net's strengths in multi-scale feature extraction with the Transformer's ability to model global context. We demonstrate the framework's effectiveness on two tasks: entanglement generation and unitary compilation, where UDiTQC consistently outperforms existing methods. Additionally, our framework supports tasks such as masking and editing circuits to meet specific physical property requirements. This dual advancement, improving quantum circuit synthesis and refining generative model architectures, marks a significant milestone in the convergence of quantum computing and machine learning research.
• 摘要：量子计算是一种具有广泛应用的变革性技术，而高效的量子电路生成对于充分发挥其潜力至关重要。当前基于 U-Net 架构的扩散模型方法虽然前景光明，但也面临着与计算效率和建模全局上下文相关的挑战。为了解决这些问题，我们提出了 UDiT，这是一种新颖的 U-Net 式扩散 Transformer 架构，它将 U-Net 在多尺度特征提取方面的优势与 Transformer 建模全局上下文的能力相结合。我们在两个任务上证明了该框架的有效性：纠缠生成和单元编译，其中 UDiTQC 始终优于现有方法。此外，我们的框架还支持诸如掩蔽和编辑电路以满足特定物理属性要求等任务。这种双重进步，改进了量子电路合成并完善了生成模型架构，标志着量子计算和机器学习研究融合的重要里程碑。

Title: CoCoNUT: Structural Code Understanding does not fall out of a tree

• Authors: Claas Beger, Saikat Dutta
• Subjects: cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.16456
• Pdf URL: https://arxiv.org/pdf/2501.16456
• Copy Paste: [[2501.16456]] CoCoNUT: Structural Code Understanding does not fall out of a tree(https://arxiv.org/abs/2501.16456)
• Keywords: generation
• Abstract: Large Language Models (LLMs) have shown impressive performance across a wide array of tasks involving both structured and unstructured textual data. Recent results on various benchmarks for code generation, repair, or completion suggest that certain models have programming abilities comparable to or even surpass humans. In this work, we demonstrate that high performance on such benchmarks does not correlate to humans' innate ability to understand structural control flow in code. To this end, we extract solutions from the HumanEval benchmark, which the relevant models perform strongly on, and trace their execution path using function calls sampled from the respective test set. Using this dataset, we investigate the ability of seven state-of-the-art LLMs to match the execution trace and find that, despite their ability to generate semantically identical code, they possess limited ability to trace execution paths, especially for longer traces and specific control structures. We find that even the top-performing model, Gemini, can fully and correctly generate only 47% of HumanEval task traces. Additionally, we introduce a subset for three key structures not contained in HumanEval: Recursion, Parallel Processing, and Object-Oriented Programming, including concepts like Inheritance and Polymorphism. Besides OOP, we show that none of the investigated models achieve an accuracy over 5% on the relevant traces. Aggregating these specialized parts with HumanEval tasks, we present Benchmark CoCoNUT: Code Control Flow for Navigation Understanding and Testing, which measures a model's ability to trace execution of code upon relevant calls, including advanced structural components. We conclude that current LLMs need significant improvement to enhance code reasoning abilities. We hope our dataset helps researchers bridge this gap.
• 摘要：大型语言模型 (LLM) 在涉及结构化和非结构化文本数据的广泛任务中表现出色。最近在各种代码生成、修复或完成基准测试中的结果表明，某些模型的编程能力堪比甚至超过人类。在这项工作中，我们证明了此类基准测试中的高性能与人类理解代码中的结构控制流的先天能力无关。为此，我们从 HumanEval 基准测试中提取了相关模型表现强劲的解决方案，并使用从相应测试集中采样的函数调用来跟踪它们的执行路径。使用此数据集，我们研究了七个最先进的 LLM 匹配执行跟踪的能力，并发现尽管它们能够生成语义上相同的代码，但它们跟踪执行路径的能力有限，尤其是对于较长的跟踪和特定的控制结构。我们发现即使是表现最好的模型 Gemini，也只能完全正确地生成 47% 的 HumanEval 任务跟踪。此外，我们引入了 HumanEval 中未包含的三个关键结构的子集：递归、并行处理和面向对象编程，包括继承和多态性等概念。除了 OOP 之外，我们还表明，所研究的模型中没有一个在相关跟踪上达到超过 5% 的准确率。通过将这些专门的部分与 HumanEval 任务聚合在一起，我们提出了 Benchmark CoCoNUT：用于导航理解和测试的代码控制流，它衡量模型在相关调用时跟踪代码执行的能力，包括高级结构组件。我们得出结论，当前的 LLM 需要显著改进才能增强代码推理能力。我们希望我们的数据集能够帮助研究人员弥补这一差距。

Title: Cross-Domain Semantic Segmentation with Large Language Model-Assisted Descriptor Generation

• Authors: Philip Hughes, Larry Burns, Luke Adams
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16467
• Pdf URL: https://arxiv.org/pdf/2501.16467
• Copy Paste: [[2501.16467]] Cross-Domain Semantic Segmentation with Large Language Model-Assisted Descriptor Generation(https://arxiv.org/abs/2501.16467)
• Keywords: generation
• Abstract: Semantic segmentation plays a crucial role in enabling machines to understand and interpret visual scenes at a pixel level. While traditional segmentation methods have achieved remarkable success, their generalization to diverse scenes and unseen object categories remains limited. Recent advancements in large language models (LLMs) offer a promising avenue for bridging visual and textual modalities, providing a deeper understanding of semantic relationships. In this paper, we propose LangSeg, a novel LLM-guided semantic segmentation method that leverages context-sensitive, fine-grained subclass descriptors generated by LLMs. Our framework integrates these descriptors with a pre-trained Vision Transformer (ViT) to achieve superior segmentation performance without extensive model retraining. We evaluate LangSeg on two challenging datasets, ADE20K and COCO-Stuff, where it outperforms state-of-the-art models, achieving up to a 6.1% improvement in mean Intersection over Union (mIoU). Additionally, we conduct a comprehensive ablation study and human evaluation to validate the effectiveness of our method in real-world scenarios. The results demonstrate that LangSeg not only excels in semantic understanding and contextual alignment but also provides a flexible and efficient framework for language-guided segmentation tasks. This approach opens up new possibilities for interactive and domain-specific segmentation applications.
• 摘要：语义分割在使机器能够以像素级别理解和解释视觉场景方面起着至关重要的作用。虽然传统的分割方法取得了显著的成功，但它们对不同场景和看不见的对象类别的推广仍然有限。大型语言模型 (LLM) 的最新进展为连接视觉和文本模态提供了一条有希望的途径，从而更深入地理解语义关系。在本文中，我们提出了 LangSeg，这是一种新颖的 LLM 引导语义分割方法，它利用由 LLM 生成的上下文敏感的细粒度子类描述符。我们的框架将这些描述符与预训练的视觉变换器 (ViT) 相结合，无需大量模型再训练即可实现卓越的分割性能。我们在两个具有挑战性的数据集 ADE20K 和 COCO-Stuff 上评估了 LangSeg，它的表现优于最先进的模型，平均交并比 (mIoU) 提高了 6.1%。此外，我们还进行了全面的消融研究和人工评估，以验证我们的方法在现实场景中的有效性。结果表明，LangSeg 不仅在语义理解和上下文对齐方面表现出色，而且还为语言引导的分割任务提供了灵活而高效的框架。这种方法为交互式和特定领域的分割应用开辟了新的可能性。

Title: Smoothed Embeddings for Robust Language Models

• Authors: Ryo Hase, Md Rafi Ur Rashid, Ashley Lewis, Jing Liu, Toshiaki Koike-Akino, Kieran Parsons, Ye Wang
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.16497
• Pdf URL: https://arxiv.org/pdf/2501.16497
• Copy Paste: [[2501.16497]] Smoothed Embeddings for Robust Language Models(https://arxiv.org/abs/2501.16497)
• Keywords: generation
• Abstract: Improving the safety and reliability of large language models (LLMs) is a crucial aspect of realizing trustworthy AI systems. Although alignment methods aim to suppress harmful content generation, LLMs are often still vulnerable to jailbreaking attacks that employ adversarial inputs that subvert alignment and induce harmful outputs. We propose the Randomized Embedding Smoothing and Token Aggregation (RESTA) defense, which adds random noise to the embedding vectors and performs aggregation during the generation of each output token, with the aim of better preserving semantic information. Our experiments demonstrate that our approach achieves superior robustness versus utility tradeoffs compared to the baseline defenses.
• 摘要：提高大型语言模型 (LLM) 的安全性和可靠性是实现可信赖的 AI 系统的关键方面。尽管对齐方法旨在抑制有害内容的生成，但 LLM 通常仍然容易受到越狱攻击，这种攻击会使用对抗性输入来破坏对齐并诱导有害输出。我们提出了随机嵌入平滑和标记聚合 (RESTA) 防御，它在嵌入向量中添加随机噪声并在生成每个输出标记期间执行聚合，目的是更好地保留语义信息。我们的实验表明，与基线防御相比，我们的方法实现了卓越的稳健性与效用权衡。

Title: PackDiT: Joint Human Motion and Text Generation via Mutual Prompting

• Authors: Zhongyu Jiang, Wenhao Chai, Zhuoran Zhou, Cheng-Yen Yang, Hsiang-Wei Huang, Jenq-Neng Hwang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.16551
• Pdf URL: https://arxiv.org/pdf/2501.16551
• Copy Paste: [[2501.16551]] PackDiT: Joint Human Motion and Text Generation via Mutual Prompting(https://arxiv.org/abs/2501.16551)
• Keywords: generation, generative
• Abstract: Human motion generation has advanced markedly with the advent of diffusion models. Most recent studies have concentrated on generating motion sequences based on text prompts, commonly referred to as text-to-motion generation. However, the bidirectional generation of motion and text, enabling tasks such as motion-to-text alongside text-to-motion, has been largely unexplored. This capability is essential for aligning diverse modalities and supports unconditional generation. In this paper, we introduce PackDiT, the first diffusion-based generative model capable of performing various tasks simultaneously, including motion generation, motion prediction, text generation, text-to-motion, motion-to-text, and joint motion-text generation. Our core innovation leverages mutual blocks to integrate multiple diffusion transformers (DiTs) across different modalities seamlessly. We train PackDiT on the HumanML3D dataset, achieving state-of-the-art text-to-motion performance with an FID score of 0.106, along with superior results in motion prediction and in-between tasks. Our experiments further demonstrate that diffusion models are effective for motion-to-text generation, achieving performance comparable to that of autoregressive models.
• 摘要：随着扩散模型的出现，人体运动生成取得了显著进展。最近的研究主要集中在基于文本提示生成运动序列，通常称为文本到运动生成。然而，运动和文本的双向生成，使诸如运动到文本和文本到运动等任务成为可能，在很大程度上尚未得到探索。这种能力对于协调不同的模态至关重要，并支持无条件生成。在本文中，我们介绍了 PackDiT，这是第一个基于扩散的生成模型，能够同时执行各种任务，包括运动生成、运动预测、文本生成、文本到运动、运动到文本和联合运动文本生成。我们的核心创新利用相互块无缝集成不同模态的多个扩散变换器 (DiT)。我们在 HumanML3D 数据集上训练 PackDiT，实现了最先进的文本到运动性能，FID 得分为 0.106，并在运动预测和中间任务中取得了优异的成绩。我们的实验进一步证明，扩散模型对于运动到文本的生成是有效的，其性能可与自回归模型相媲美。

Title: LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation

• Authors: Farzad Farhadzadeh, Debasmit Das, Shubhankar Borse, Fatih Porikli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16559
• Pdf URL: https://arxiv.org/pdf/2501.16559
• Copy Paste: [[2501.16559]] LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation(https://arxiv.org/abs/2501.16559)
• Keywords: generation
• Abstract: The rising popularity of large foundation models has led to a heightened demand for parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), which offer performance comparable to full model fine-tuning while requiring only a few additional parameters tailored to the specific base model. When such base models are deprecated and replaced, all associated LoRA modules must be retrained, requiring access to either the original training data or a substantial amount of synthetic data that mirrors the original distribution. However, the original data is often inaccessible due to privacy or licensing issues, and generating synthetic data may be impractical and insufficiently representative. These factors complicate the fine-tuning process considerably. To address this challenge, we introduce a new adapter, Cross-Model Low-Rank Adaptation (LoRA-X), which enables the training-free transfer of LoRA parameters across source and target models, eliminating the need for original or synthetic training data. Our approach imposes the adapter to operate within the subspace of the source base model. This constraint is necessary because our prior knowledge of the target model is limited to its weights, and the criteria for ensuring the adapter's transferability are restricted to the target base model's weights and subspace. To facilitate the transfer of LoRA parameters of the source model to a target model, we employ the adapter only in the layers of the target model that exhibit an acceptable level of subspace similarity. Our extensive experiments demonstrate the effectiveness of LoRA-X for text-to-image generation, including Stable Diffusion v1.5 and Stable Diffusion XL.
• 摘要：大型基础模型的日益普及导致了对参数高效微调方法的需求增加，例如低秩自适应 (LoRA)，它提供与完整模型微调相当的性能，同时只需要针对特定​​基础模型定制一些额外的参数。当此类基础模型被弃用和替换时，所有相关的 LoRA 模块都必须重新训练，这需要访问原始训练数据或大量反映原始分布的合成数据。然而，由于隐私或许可问题，原始数据通常无法访问，而生成合成数据可能不切实际且代表性不足。这些因素使微调过程变得相当复杂。为了应对这一挑战，我们引入了一个新的适配器，即跨模型低秩自适应 (LoRA-X)，它能够在源模型和目标模型之间进行无需训练的 LoRA 参数传输，从而无需原始或合成训练数据。我们的方法要求适配器在源基础模型的子空间内运行。这种约束是必要的，因为我们对目标模型的先验知识仅限于其权重，而确保适配器可转移性的标准仅限于目标基础模型的权重和子空间。为了便于将源模型的 LoRA 参数转移到目标模型，我们仅在目标模型中表现出可接受的子空间相似度的层中使用适配器。我们进行了广泛的实验，证明了 LoRA-X 在文本到图像生成方面的有效性，包括 Stable Diffusion v1.5 和 Stable Diffusion XL。

Title: Directing Mamba to Complex Textures: An Efficient Texture-Aware State Space Model for Image Restoration

• Authors: Long Peng, Xin Di, Zhanfeng Feng, Wenbo Li, Renjing Pei, Yang Wang, Xueyang Fu, Yang Cao, Zheng-Jun Zha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16583
• Pdf URL: https://arxiv.org/pdf/2501.16583
• Copy Paste: [[2501.16583]] Directing Mamba to Complex Textures: An Efficient Texture-Aware State Space Model for Image Restoration(https://arxiv.org/abs/2501.16583)
• Keywords: restoration, super-resolution
• Abstract: Image restoration aims to recover details and enhance contrast in degraded images. With the growing demand for high-quality imaging (\textit{e.g.}, 4K and 8K), achieving a balance between restoration quality and computational efficiency has become increasingly critical. Existing methods, primarily based on CNNs, Transformers, or their hybrid approaches, apply uniform deep representation extraction across the image. However, these methods often struggle to effectively model long-range dependencies and largely overlook the spatial characteristics of image degradation (regions with richer textures tend to suffer more severe damage), making it hard to achieve the best trade-off between restoration quality and efficiency. To address these issues, we propose a novel texture-aware image restoration method, TAMambaIR, which simultaneously perceives image textures and achieves a trade-off between performance and efficiency. Specifically, we introduce a novel Texture-Aware State Space Model, which enhances texture awareness and improves efficiency by modulating the transition matrix of the state-space equation and focusing on regions with complex textures. Additionally, we design a {Multi-Directional Perception Block} to improve multi-directional receptive fields while maintaining low computational overhead. Extensive experiments on benchmarks for image super-resolution, deraining, and low-light image enhancement demonstrate that TAMambaIR achieves state-of-the-art performance with significantly improved efficiency, establishing it as a robust and efficient framework for image restoration.
• 摘要：图像修复旨在恢复细节并增强退化图像中的对比度。随着对高质量成像（\textit{例如}，4K 和 8K）的需求不断增长，实现修复质量和计算效率之间的平衡变得越来越重要。现有方法主要基于 CNN、Transformers 或它们的混合方法，在整个图像上应用统一的深度表示提取。然而，这些方法通常难以有效地模拟长距离依赖关系，并且在很大程度上忽略了图像退化的空间特征（纹理更丰富的区域往往遭受更严重的损坏），因此很难在修复质量和效率之间实现最佳平衡。为了解决这些问题，我们提出了一种新颖的纹理感知图像修复方法 TAMambaIR，它可以同时感知图像纹理并在性能和效率之间实现平衡。具体来说，我们引入了一种新颖的纹理感知状态空间模型，该模型通过调节状态空间方程的转移矩阵并关注具有复杂纹理的区域来增强纹理感知并提高效率。此外，我们设计了一个{多方向感知块}来改善多方向接受场，同时保持较低的计算开销。在图像超分辨率、去雨和低光图像增强的基准上进行的大量实验表明，TAMambaIR 实现了最先进的性能，效率显著提高，使其成为一个强大而高效的图像恢复框架。

Title: CascadeV: An Implementation of Wurstchen Architecture for Video Generation

• Authors: Wenfeng Lin, Jiangchuan Wei, Boyuan Liu, Yichen Zhang, Shiyue Yan, Mingyu Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16612
• Pdf URL: https://arxiv.org/pdf/2501.16612
• Copy Paste: [[2501.16612]] CascadeV: An Implementation of Wurstchen Architecture for Video Generation(https://arxiv.org/abs/2501.16612)
• Keywords: generation
• Abstract: Recently, with the tremendous success of diffusion models in the field of text-to-image (T2I) generation, increasing attention has been directed toward their potential in text-to-video (T2V) applications. However, the computational demands of diffusion models pose significant challenges, particularly in generating high-resolution videos with high frame rates. In this paper, we propose CascadeV, a cascaded latent diffusion model (LDM), that is capable of producing state-of-the-art 2K resolution videos. Experiments demonstrate that our cascaded model achieves a higher compression ratio, substantially reducing the computational challenges associated with high-quality video generation. We also implement a spatiotemporal alternating grid 3D attention mechanism, which effectively integrates spatial and temporal information, ensuring superior consistency across the generated video frames. Furthermore, our model can be cascaded with existing T2V models, theoretically enabling a 4$\times$ increase in resolution or frames per second without any fine-tuning. Our code is available at this https URL.
• 摘要：最近，随着扩散模型在文本转图像 (T2I) 生成领域取得巨大成功，人们越来越关注它们在文本转视频 (T2V) 应用中的潜力。然而，扩散模型的计算需求带来了重大挑战，特别是在生成高帧率的高分辨率视频时。在本文中，我们提出了 CascadeV，这是一种级联潜在扩散模型 (LDM)，能够生成最先进的 2K 分辨率视频。实验表明，我们的级联模型实现了更高的压缩比，大大降低了与高质量视频生成相关的计算挑战。我们还实现了时空交替网格 3D 注意机制，该机制有效地整合了空间和时间信息，确保生成的视频帧之间具有出色的一致性。此外，我们的模型可以与现有的 T2V 模型级联，理论上无需任何微调即可将分辨率或每秒帧数提高 4$\times$。我们的代码可在此 https URL 上找到。

Title: Polyp-Gen: Realistic and Diverse Polyp Image Generation for Endoscopic Dataset Expansion

• Authors: Shengyuan Liu, Zhen Chen, Qiushi Yang, Weihao Yu, Di Dong, Jiancong Hu, Yixuan Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16679
• Pdf URL: https://arxiv.org/pdf/2501.16679
• Copy Paste: [[2501.16679]] Polyp-Gen: Realistic and Diverse Polyp Image Generation for Endoscopic Dataset Expansion(https://arxiv.org/abs/2501.16679)
• Keywords: generation
• Abstract: Automated diagnostic systems (ADS) have shown significant potential in the early detection of polyps during endoscopic examinations, thereby reducing the incidence of colorectal cancer. However, due to high annotation costs and strict privacy concerns, acquiring high-quality endoscopic images poses a considerable challenge in the development of ADS. Despite recent advancements in generating synthetic images for dataset expansion, existing endoscopic image generation algorithms failed to accurately generate the details of polyp boundary regions and typically required medical priors to specify plausible locations and shapes of polyps, which limited the realism and diversity of the generated images. To address these limitations, we present Polyp-Gen, the first full-automatic diffusion-based endoscopic image generation framework. Specifically, we devise a spatial-aware diffusion training scheme with a lesion-guided loss to enhance the structural context of polyp boundary regions. Moreover, to capture medical priors for the localization of potential polyp areas, we introduce a hierarchical retrieval-based sampling strategy to match similar fine-grained spatial features. In this way, our Polyp-Gen can generate realistic and diverse endoscopic images for building reliable ADS. Extensive experiments demonstrate the state-of-the-art generation quality, and the synthetic images can improve the downstream polyp detection task. Additionally, our Polyp-Gen has shown remarkable zero-shot generalizability on other datasets. The source code is available at this https URL.
• 摘要：自动诊断系统 (ADS) 在内窥镜检查中早期发现息肉方面显示出巨大潜力，从而降低结直肠癌的发病率。然而，由于高昂的注释成本和严格的隐私问题，获取高质量的内窥镜图像对 ADS 的开发构成了相当大的挑战。尽管最近在生成合成图像以扩展数据集方面取得了进展，但现有的内窥镜图像生成算法无法准确生成息肉边界区域的细节，并且通常需要医学先验来指定息肉的合理位置和形状，这限制了生成图像的真实性和多样性。为了解决这些限制，我们提出了 Polyp-Gen，这是第一个基于扩散的全自动内窥镜图像生成框架。具体而言，我们设计了一种具有病变引导损失的空间感知扩散训练方案，以增强息肉边界区域的结构背景。此外，为了捕获用于定位潜在息肉区域的医学先验，我们引入了一种基于分层检索的采样策略来匹配类似的细粒度空间特征。通过这种方式，我们的 Polyp-Gen 可以生成逼真且多样化的内窥镜图像，从而构建可靠的 ADS。大量实验证明了最先进的生成质量，合成图像可以改善下游息肉检测任务。此外，我们的 Polyp-Gen 在其他数据集上表现出了显著的零样本通用性。源代码可在此 https URL 上找到。

Title: Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models

• Authors: Huijie Liu, Jingyun Wang, Shuai Ma, Jie Hu, Xiaoming Wei, Guoliang Kang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16714
• Pdf URL: https://arxiv.org/pdf/2501.16714
• Copy Paste: [[2501.16714]] Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models(https://arxiv.org/abs/2501.16714)
• Keywords: generation
• Abstract: Motion customization aims to adapt the diffusion model (DM) to generate videos with the motion specified by a set of video clips with the same motion concept. To realize this goal, the adaptation of DM should be possible to model the specified motion concept, without compromising the ability to generate diverse appearances. Thus, the key to solving this problem lies in how to separate the motion concept from the appearance in the adaptation process of DM. Typical previous works explore different ways to represent and insert a motion concept into large-scale pretrained text-to-video diffusion models, e.g., learning a motion LoRA, using latent noise residuals, etc. While those methods can encode the motion concept, they also inevitably encode the appearance in the reference videos, resulting in weakened appearance generation capability. In this paper, we follow the typical way to learn a motion LoRA to encode the motion concept, but propose two novel strategies to enhance motion-appearance separation, including temporal attention purification (TAP) and appearance highway (AH). Specifically, we assume that in the temporal attention module, the pretrained Value embeddings are sufficient to serve as basic components needed by producing a new motion. Thus, in TAP, we choose only to reshape the temporal attention with motion LoRAs so that Value embeddings can be reorganized to produce a new motion. Further, in AH, we alter the starting point of each skip connection in U-Net from the output of each temporal attention module to the output of each spatial attention module. Extensive experiments demonstrate that compared to previous works, our method can generate videos with appearance more aligned with the text descriptions and motion more consistent with the reference videos.
• 摘要：运动定制旨在使扩散模型 (DM) 适应一组具有相同运动概念的视频片段来生成具有指定运动的视频。为了实现这一目标，DM 的适应性应该能够对指定的运动概念进行建模，而不会损害生成多样化外观的能力。因此，解决这个问题的关键在于如何在 DM 的适应过程中将运动概念与外观分离。典型的先前工作探索了将运动概念表示和插入到大规模预训练的文本到视频扩散模型中的不同方法，例如学习运动 LoRA、使用潜在噪声残差等。虽然这些方法可以对运动概念进行编码，但它们并不可避免地对参考视频中的外观进行编码，从而导致外观生成能力减弱。在本文中，我们遵循学习运动 LoRA 的典型方法来编码运动概念，但提出了两种新策略来增强运动-外观分离，包括时间注意力净化 (TAP) 和外观高速公路 (AH)。具体来说，我们假设在时间注意模块中，预训练的值嵌入足以作为产生新运动所需的基本组件。因此，在 TAP 中，我们选择仅使用运动 LoRA 重塑时间注意，以便可以重新组织值嵌入以产生新运动。此外，在 AH 中，我们将 U-Net 中每个跳跃连接的起点从每个时间注意模块的输出更改为每个空间注意模块的输出。大量实验表明，与之前的研究相比，我们的方法可以生成外观与文本描述更一致、运动与参考视频更一致的视频。

Title: Outlier Synthesis via Hamiltonian Monte Carlo for Out-of-Distribution Detection

• Authors: Hengzhuang Li, Teng Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.16718
• Pdf URL: https://arxiv.org/pdf/2501.16718
• Copy Paste: [[2501.16718]] Outlier Synthesis via Hamiltonian Monte Carlo for Out-of-Distribution Detection(https://arxiv.org/abs/2501.16718)
• Keywords: generative
• Abstract: Out-of-distribution (OOD) detection is crucial for developing trustworthy and reliable machine learning systems. Recent advances in training with auxiliary OOD data demonstrate efficacy in enhancing detection capabilities. Nonetheless, these methods heavily rely on acquiring a large pool of high-quality natural outliers. Some prior methods try to alleviate this problem by synthesizing virtual outliers but suffer from either poor quality or high cost due to the monotonous sampling strategy and the heavy-parameterized generative models. In this paper, we overcome all these problems by proposing the Hamiltonian Monte Carlo Outlier Synthesis (HamOS) framework, which views the synthesis process as sampling from Markov chains. Based solely on the in-distribution data, the Markov chains can extensively traverse the feature space and generate diverse and representative outliers, hence exposing the model to miscellaneous potential OOD scenarios. The Hamiltonian Monte Carlo with sampling acceptance rate almost close to 1 also makes our framework enjoy great efficiency. By empirically competing with SOTA baselines on both standard and large-scale benchmarks, we verify the efficacy and efficiency of our proposed HamOS.
• 摘要：分布外 (OOD) 检测对于开发值得信赖且可靠的机器学习系统至关重要。使用辅助 OOD 数据进行训练的最新进展证明了其在增强检测能力方面的有效性。尽管如此，这些方法严重依赖于获取大量高质量的自然异常值。一些先前的方法试图通过合成虚拟异常值来缓解这个问题，但由于单调的采样策略和重参数化的生成模型，导致质量差或成本高。在本文中，我们通过提出汉密尔顿蒙特卡罗异常值合成 (HamOS) 框架克服了所有这些问题，该框架将合成过程视为从马尔可夫链中采样。仅基于分布内数据，马尔可夫链就可以广泛地遍历特征空间并生成多样且具有代表性的异常值，从而将模型暴露于各种潜在的 OOD 场景。采样接受率几乎接近 1 的汉密尔顿蒙特卡罗也使我们的框架具有很高的效率。通过在标准和大规模基准上与 SOTA 基线进行实证竞争，我们验证了我们提出的 HamOS 的有效性和效率。

Title: B-RIGHT: Benchmark Re-evaluation for Integrity in Generalized Human-Object Interaction Testing

• Authors: Yoojin Jang, Junsu Kim, Hayeon Kim, Eun-ki Lee, Eun-sol Kim, Seungryul Baek, Jaejun Yoo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16724
• Pdf URL: https://arxiv.org/pdf/2501.16724
• Copy Paste: [[2501.16724]] B-RIGHT: Benchmark Re-evaluation for Integrity in Generalized Human-Object Interaction Testing(https://arxiv.org/abs/2501.16724)
• Keywords: generation
• Abstract: Human-object interaction (HOI) is an essential problem in artificial intelligence (AI) which aims to understand the visual world that involves complex relationships between humans and objects. However, current benchmarks such as HICO-DET face the following limitations: (1) severe class imbalance and (2) varying number of train and test sets for certain classes. These issues can potentially lead to either inflation or deflation of model performance during evaluation, ultimately undermining the reliability of evaluation scores. In this paper, we propose a systematic approach to develop a new class-balanced dataset, Benchmark Re-evaluation for Integrity in Generalized Human-object Interaction Testing (B-RIGHT), that addresses these imbalanced problems. B-RIGHT achieves class balance by leveraging balancing algorithm and automated generation-and-filtering processes, ensuring an equal number of instances for each HOI class. Furthermore, we design a balanced zero-shot test set to systematically evaluate models on unseen scenario. Re-evaluating existing models using B-RIGHT reveals substantial the reduction of score variance and changes in performance rankings compared to conventional HICO-DET. Our experiments demonstrate that evaluation under balanced conditions ensure more reliable and fair model comparisons.
• 摘要：人与物交互 (HOI) 是人工智能 (AI) 中的一个重要问题，旨在理解涉及人与物之间复杂关系的视觉世界。然而，当前的基准测试（如 HICO-DET）面临以下限制：(1) 严重的类别不平衡和 (2) 某些类别的训练集和测试集数量各不相同。这些问题可能会导致评估过程中模型性能的膨胀或缩水，最终破坏评估分数的可靠性。在本文中，我们提出了一种系统的方法来开发一个新的类别平衡数据集，即广义人与物交互测试中完整性的基准重新评估 (B-RIGHT)，以解决这些不平衡问题。B-RIGHT 通过利用平衡算法和自动生成和过滤过程实现类别平衡，确保每个 HOI 类别的实例数量相等。此外，我们设计了一个平衡的零样本测试集来系统地评估看不见的场景中的模型。使用 B-RIGHT 重新评估现有模型表明，与传统的 HICO-DET 相比，分数差异和性能排名的变化显著减少。我们的实验表明，在平衡条件下进行评估可确保更可靠、更公平的模型比较。

Title: DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation

• Authors: Chenguo Lin, Panwang Pan, Bangbang Yang, Zeming Li, Yadong Mu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16764
• Pdf URL: https://arxiv.org/pdf/2501.16764
• Copy Paste: [[2501.16764]] DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation(https://arxiv.org/abs/2501.16764)
• Keywords: generation, generative
• Abstract: Recent advancements in 3D content generation from text or a single image struggle with limited high-quality 3D datasets and inconsistency from 2D multi-view generation. We introduce DiffSplat, a novel 3D generative framework that natively generates 3D Gaussian splats by taming large-scale text-to-image diffusion models. It differs from previous 3D generative models by effectively utilizing web-scale 2D priors while maintaining 3D consistency in a unified model. To bootstrap the training, a lightweight reconstruction model is proposed to instantly produce multi-view Gaussian splat grids for scalable dataset curation. In conjunction with the regular diffusion loss on these grids, a 3D rendering loss is introduced to facilitate 3D coherence across arbitrary views. The compatibility with image diffusion models enables seamless adaptions of numerous techniques for image generation to the 3D realm. Extensive experiments reveal the superiority of DiffSplat in text- and image-conditioned generation tasks and downstream applications. Thorough ablation studies validate the efficacy of each critical design choice and provide insights into the underlying mechanism.
• 摘要：从文本或单个图像生成 3D 内容的最新进展面临着有限的高质量 3D 数据集和与 2D 多视图生成的不一致性的问题。我们介绍了 DiffSplat，这是一种新颖的 3D 生成框架，它通过驯服大规模文本到图像扩散模型原生生成 3D 高斯图。它与以前的 3D 生成模型的不同之处在于有效利用了网络规模的 2D 先验，同时在统一模型中保持 3D 一致性。为了引导训练，提出了一个轻量级重建模型，以立即生成多视图高斯图网格，用于可扩展的数据集管理。结合这些网格上的常规扩散损失，引入了 3D 渲染损失以促进任意视图之间的 3D 一致性。与图像扩散模型的兼容性使众多图像生成技术能够无缝适应 3D 领域。大量实验揭示了 DiffSplat 在文本和图像条件生成任务以及下游应用中的优势。彻底的消融研究验证了每个关键设计选择的有效性，并提供了对潜在机制的见解。

Title: FlexMotion: Lightweight, Physics-Aware, and Controllable Human Motion Generation

• Authors: Arvin Tashakori, Arash Tashakori, Gongbo Yang, Z. Jane Wang, Peyman Servati
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.16778
• Pdf URL: https://arxiv.org/pdf/2501.16778
• Copy Paste: [[2501.16778]] FlexMotion: Lightweight, Physics-Aware, and Controllable Human Motion Generation(https://arxiv.org/abs/2501.16778)
• Keywords: generation
• Abstract: Lightweight, controllable, and physically plausible human motion synthesis is crucial for animation, virtual reality, robotics, and human-computer interaction applications. Existing methods often compromise between computational efficiency, physical realism, or spatial controllability. We propose FlexMotion, a novel framework that leverages a computationally lightweight diffusion model operating in the latent space, eliminating the need for physics simulators and enabling fast and efficient training. FlexMotion employs a multimodal pre-trained Transformer encoder-decoder, integrating joint locations, contact forces, joint actuations and muscle activations to ensure the physical plausibility of the generated motions. FlexMotion also introduces a plug-and-play module, which adds spatial controllability over a range of motion parameters (e.g., joint locations, joint actuations, contact forces, and muscle activations). Our framework achieves realistic motion generation with improved efficiency and control, setting a new benchmark for human motion synthesis. We evaluate FlexMotion on extended datasets and demonstrate its superior performance in terms of realism, physical plausibility, and controllability.
• 摘要：轻量级、可控且物理上合理的人体运动合成对于动画、虚拟现实、机器人和人机交互应用至关重要。现有方法通常在计算效率、物理真实感或空间可控性之间做出妥协。我们提出了 FlexMotion，这是一个新颖的框架，它利用在潜在空间中运行的计算轻量级扩散模型，消除了对物理模拟器的需求并实现了快速高效的训练。FlexMotion 采用多模态预训练的 Transformer 编码器-解码器，集成关节位置、接触力、关节驱动和肌肉激活，以确保生成运动的物理合理性。FlexMotion 还引入了一个即插即用模块，它增加了一系列运动参数（例如关节位置、关节驱动、接触力和肌肉激活）的空间可控性。我们的框架实现了逼真的运动生成，提高了效率和控制力，为人体运动合成树立了新的标杆。我们在扩展数据集上评估 FlexMotion，并在真实性、物理合理性和可控性方面展示其卓越的性能。

Title: Flow Matching: Markov Kernels, Stochastic Processes and Transport Plans

• Authors: Christian Wald, Gabriele Steidl
• Subjects: cs.LG, math.PR
• Abstract URL: https://arxiv.org/abs/2501.16839
• Pdf URL: https://arxiv.org/pdf/2501.16839
• Copy Paste: [[2501.16839]] Flow Matching: Markov Kernels, Stochastic Processes and Transport Plans(https://arxiv.org/abs/2501.16839)
• Keywords: generative
• Abstract: Among generative neural models, flow matching techniques stand out for their simple applicability and good scaling properties. Here, velocity fields of curves connecting a simple latent and a target distribution are learned. Then the corresponding ordinary differential equation can be used to sample from a target distribution, starting in samples from the latent one. This paper reviews from a mathematical point of view different techniques to learn the velocity fields of absolutely continuous curves in the Wasserstein geometry. We show how the velocity fields can be characterized and learned via i) transport plans (couplings) between latent and target distributions, ii) Markov kernels and iii) stochastic processes, where the latter two include the coupling approach, but are in general broader. Besides this main goal, we show how flow matching can be used for solving Bayesian inverse problems, where the definition of conditional Wasserstein distances plays a central role. Finally, we briefly address continuous normalizing flows and score matching techniques, which approach the learning of velocity fields of curves from other directions.
• 摘要：在生成神经模型中，流匹配技术因其简单的适用性和良好的扩展性而脱颖而出。在这里，学习连接简单潜在分布和目标分布的曲线的速度场。然后，可以使用相应的常微分方程从目标分布中采样，从潜在分布的样本开始。本文从数学的角度回顾了在 Wasserstein 几何中学习绝对连续曲线速度场的不同技术。我们展示了如何通过 i) 潜在分布和目标分布之间的传输计划（耦合）、ii) 马尔可夫核和 iii) 随机过程来表征和学习速度场，其中后两者包括耦合方法，但通常更广泛。除了这个主要目标之外，我们还展示了如何使用流匹配来解决贝叶斯逆问题，其中条件 Wasserstein 距离的定义起着核心作用。最后，我们简要介绍了连续正则化流和分数匹配技术，它们从其他方向接近曲线的速度场学习。

Title: Extending Information Bottleneck Attribution to Video Sequences

• Authors: Veronika Solopova, Lucas Schmidt, Dorothea Kolossa
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16889
• Pdf URL: https://arxiv.org/pdf/2501.16889
• Copy Paste: [[2501.16889]] Extending Information Bottleneck Attribution to Video Sequences(https://arxiv.org/abs/2501.16889)
• Keywords: generation
• Abstract: We introduce VIBA, a novel approach for explainable video classification by adapting Information Bottlenecks for Attribution (IBA) to video sequences. While most traditional explainability methods are designed for image models, our IBA framework addresses the need for explainability in temporal models used for video analysis. To demonstrate its effectiveness, we apply VIBA to video deepfake detection, testing it on two architectures: the Xception model for spatial features and a VGG11-based model for capturing motion dynamics through optical flow. Using a custom dataset that reflects recent deepfake generation techniques, we adapt IBA to create relevance and optical flow maps, visually highlighting manipulated regions and motion inconsistencies. Our results show that VIBA generates temporally and spatially consistent explanations, which align closely with human annotations, thus providing interpretability for video classification and particularly for deepfake detection.
• 摘要：我们引入了 VIBA，这是一种通过将归因信息瓶颈 (IBA) 应用于视频序列来实现可解释视频分类的新方法。虽然大多数传统的可解释性方法都是为图像模型设计的，但我们的 IBA 框架解决了用于视频分析的时间模型中可解释性的需求。为了证明其有效性，我们将 VIBA 应用于视频深度伪造检测，并在两种架构上对其进行了测试：用于空间特征的 Xception 模型和用于通过光流捕捉运动动态的基于 VGG11 的模型。使用反映最近深度伪造生成技术的自定义数据集，我们调整 IBA 以创建相关性和光流图，在视觉上突出显示被操纵的区域和运动不一致。我们的结果表明，VIBA 生成时间和空间一致的解释，与人类注释紧密一致，从而为视频分类和特别是深度伪造检测提供了可解释性。

Title: ToolFactory: Automating Tool Generation by Leveraging LLM to Understand REST API Documentations

• Authors: Xinyi Ni (1), Qiuyang Wang (1), Yukun Zhang (1), Pengyu Hong (1) ((1) Brandeis University)
• Subjects: cs.LG, cs.AI, cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.16945
• Pdf URL: https://arxiv.org/pdf/2501.16945
• Copy Paste: [[2501.16945]] ToolFactory: Automating Tool Generation by Leveraging LLM to Understand REST API Documentations(https://arxiv.org/abs/2501.16945)
• Keywords: generation
• Abstract: LLM-based tool agents offer natural language interfaces, enabling users to seamlessly interact with computing services. While REST APIs are valuable resources for building such agents, they must first be transformed into AI-compatible tools. Automatically generating AI-compatible tools from REST API documents can greatly streamline tool agent development and minimize user learning curves. However, API documentation often suffers from a lack of standardization, inconsistent schemas, and incomplete information. To address these issues, we developed \textbf{ToolFactory}, an open-source pipeline for automating tool generation from unstructured API documents. To enhance the reliability of the developed tools, we implemented an evaluation method to diagnose errors. Furthermore, we built a knowledge base of verified tools, which we leveraged to infer missing information from poorly documented APIs. We developed the API Extraction Benchmark, comprising 167 API documents and 744 endpoints in various formats, and designed a JSON schema to annotate them. This annotated dataset was utilized to train and validate ToolFactory. The experimental results highlight the effectiveness of ToolFactory. We also demonstrated ToolFactory by creating a domain-specific AI agent for glycomaterials research. ToolFactory exhibits significant potential for facilitating the seamless integration of scientific REST APIs into AI workflows.
• 摘要：基于 LLM 的工具代理提供自然语言界面，使用户能够无缝地与计算服务交互。虽然 REST API 是构建此类代理的宝贵资源，但它们必须首先转换为与 AI 兼容的工具。从 REST API 文档自动生成与 AI 兼容的工具可以大大简化工具代理开发并最大限度地缩短用户学习曲线。但是，API 文档通常缺乏标准化、模式不一致和信息不完整。为了解决这些问题，我们开发了 \textbf{ToolFactory}，这是一个开源管道，用于自动从非结构化 API 文档生成工具。为了提高所开发工具的可靠性，我们实施了一种评估方法来诊断错误。此外，我们建立了一个经过验证的工具知识库，我们利用该知识库从记录不全的 API 中推断出缺失的信息。我们开发了 API Extraction Benchmark，包含 167 个 API 文档和 744 个各种格式的端点，并设计了一个 JSON 模式来注释它们。这个带注释的数据集用于训练和验证 ToolFactory。实验结果突出了 ToolFactory 的有效性。我们还通过创建用于糖基材料研究的特定领域 AI 代理来演示 ToolFactory。ToolFactory 在促进科学 REST API 与 AI 工作流程无缝集成方面展现出巨大潜力。

Title: Image-based Geo-localization for Robotics: Are Black-box Vision-Language Models there yet?

• Authors: Sania Waheed, Bruno Ferrarini, Michael Milford, Sarvapali D. Ramchurn, Shoaib Ehsan
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.16947
• Pdf URL: https://arxiv.org/pdf/2501.16947
• Copy Paste: [[2501.16947]] Image-based Geo-localization for Robotics: Are Black-box Vision-Language Models there yet?(https://arxiv.org/abs/2501.16947)
• Keywords: generation
• Abstract: The advances in Vision-Language models (VLMs) offer exciting opportunities for robotic applications involving image geo-localization, the problem of identifying the geo-coordinates of a place based on visual data only. Recent research works have focused on using a VLM as embeddings extractor for geo-localization, however, the most sophisticated VLMs may only be available as black boxes that are accessible through an API, and come with a number of limitations: there is no access to training data, model features and gradients; retraining is not possible; the number of predictions may be limited by the API; training on model outputs is often prohibited; and queries are open-ended. The utilization of a VLM as a stand-alone, zero-shot geo-localization system using a single text-based prompt is largely unexplored. To bridge this gap, this paper undertakes the first systematic study, to the best of our knowledge, to investigate the potential of some of the state-of-the-art VLMs as stand-alone, zero-shot geo-localization systems in a black-box setting with realistic constraints. We consider three main scenarios for this thorough investigation: a) fixed text-based prompt; b) semantically-equivalent text-based prompts; and c) semantically-equivalent query images. We also take into account the auto-regressive and probabilistic generation process of the VLMs when investigating their utility for geo-localization task by using model consistency as a metric in addition to traditional accuracy. Our work provides new insights in the capabilities of different VLMs for the above-mentioned scenarios.
• 摘要：视觉语言模型 (VLM) 的进步为涉及图像地理定位的机器人应用提供了激动人心的机会，图像地理定位是仅基于视觉数据识别某个地点的地理坐标的问题。最近的研究工作集中于使用 VLM 作为地理定位的嵌入提取器，但是，最复杂的 VLM 可能仅作为可通过 API 访问的黑盒提供，并且具有许多限制：无法访问训练数据、模型特征和梯度；无法重新训练；预测数量可能受 API 限制；通常禁止对模型输出进行训练；查询是开放式的。使用单个基于文本的提示将 VLM 用作独立的零样本地理定位系统在很大程度上尚未得到探索。为了弥补这一空白，本文进行了首次系统研究，据我们所知，研究一些最先进的 VLM 作为具有现实约束的黑盒设置中的独立零样本地理定位系统的潜力。我们考虑了三种主要场景来进行这项彻底的研究：a) 固定的基于文本的提示；b) 语义等效的基于文本的提示；c) 语义等效的查询图像。在研究 VLM 对地理定位任务的实用性时，我们还考虑了 VLM 的自回归和概率生成过程，除了传统的准确性之外，我们还使用模型一致性作为衡量标准。我们的工作为不同 VLM 在上述场景中的功能提供了新的见解。

Title: What Really Matters for Learning-based LiDAR-Camera Calibration

• Authors: Shujuan Huang, Chunyu Lin, Yao Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16969
• Pdf URL: https://arxiv.org/pdf/2501.16969
• Copy Paste: [[2501.16969]] What Really Matters for Learning-based LiDAR-Camera Calibration(https://arxiv.org/abs/2501.16969)
• Keywords: generation
• Abstract: Calibration is an essential prerequisite for the accurate data fusion of LiDAR and camera sensors. Traditional calibration techniques often require specific targets or suitable scenes to obtain reliable 2D-3D correspondences. To tackle the challenge of target-less and online calibration, deep neural networks have been introduced to solve the problem in a data-driven manner. While previous learning-based methods have achieved impressive performance on specific datasets, they still struggle in complex real-world scenarios. Most existing works focus on improving calibration accuracy but overlook the underlying mechanisms. In this paper, we revisit the development of learning-based LiDAR-Camera calibration and encourage the community to pay more attention to the underlying principles to advance practical applications. We systematically analyze the paradigm of mainstream learning-based methods, and identify the critical limitations of regression-based methods with the widely used data generation pipeline. Our findings reveal that most learning-based methods inadvertently operate as retrieval networks, focusing more on single-modality distributions rather than cross-modality correspondences. We also investigate how the input data format and preprocessing operations impact network performance and summarize the regression clues to inform further improvements.
• 摘要：校准是实现 LiDAR 和相机传感器精确数据融合的重要前提。传统的校准技术通常需要特定的目标或合适的场景才能获得可靠的 2D-3D 对应关系。为了应对无目标和在线校准的挑战，人们引入了深度神经网络以数据驱动的方式解决问题。虽然以前的基于学习的方法在特定数据集上取得了令人印象深刻的性能，但它们在复杂的现实场景中仍然举步维艰。大多数现有工作都侧重于提高校准精度，但忽略了底层机制。在本文中，我们重新审视了基于学习的 LiDAR-Camera 校准的发展，并鼓励社区更多地关注底层原理以推进实际应用。我们系统地分析了主流基于学习的方法的范式，并确定了基于回归的方法在广泛使用的数据生成管道中的关键局限性。我们的研究结果表明，大多数基于学习的方法无意中充当了检索网络，更多地关注单模态分布而不是跨模态对应关系。我们还研究输入数据格式和预处理操作如何影响网络性能，并总结回归线索以指导进一步改进。

Title: Modulating CNN Features with Pre-Trained ViT Representations for Open-Vocabulary Object Detection

• Authors: Xiangyu Gao, Yu Dai, Benliu Qiu, Hongliang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16981
• Pdf URL: https://arxiv.org/pdf/2501.16981
• Copy Paste: [[2501.16981]] Modulating CNN Features with Pre-Trained ViT Representations for Open-Vocabulary Object Detection(https://arxiv.org/abs/2501.16981)
• Keywords: generative
• Abstract: Owing to large-scale image-text contrastive training, pre-trained vision language model (VLM) like CLIP shows superior open-vocabulary recognition ability. Most existing open-vocabulary object detectors attempt to utilize the pre-trained VLM to attain generative representation. F-ViT uses the pre-trained visual encoder as the backbone network and freezes it during training. However, the frozen backbone doesn't benefit from the labeled data to strengthen the representation. Therefore, we propose a novel two-branch backbone network design, named as ViT-Feature-Modulated Multi-Scale Convolutional network (VMCNet). VMCNet consists of a trainable convolutional branch, a frozen pre-trained ViT branch and a feature modulation module. The trainable CNN branch could be optimized with labeled data while the frozen pre-trained ViT branch could keep the representation ability derived from large-scale pre-training. Then, the proposed feature modulation module could modulate the multi-scale CNN features with the representations from ViT branch. With the proposed mixed structure, detector is more likely to discover novel categories. Evaluated on two popular benchmarks, our method boosts the detection performance on novel category and outperforms the baseline. On OV-COCO, the proposed method achieves 44.3 AP$_{50}^{\mathrm{novel}}$ with ViT-B/16 and 48.5 AP$_{50}^{\mathrm{novel}}$ with ViT-L/14. On OV-LVIS, VMCNet with ViT-B/16 and ViT-L/14 reaches 27.8 and 38.4 mAP$_{r}$.
• 摘要：由于大规模图像-文本对比训练，像 CLIP 这样的预训练视觉语言模型 (VLM) 表现出卓越的开放词汇识别能力。大多数现有的开放词汇物体检测器都试图利用预训练的 VLM 来获得生成表示。F-ViT 使用预训练的视觉编码器作为主干网络并在训练期间将其冻结。然而，冻结的主干不会从标记数据中受益以加强表示。因此，我们提出了一种新颖的双分支主干网络设计，称为 ViT-Feature-Modulated 多尺度卷积网络 (VMCNet)。VMCNet 由可训练的卷积分支、冻结的预训练 ViT 分支和特征调制模块组成。可训练的 CNN 分支可以使用标记数据进行优化，而冻结的预训练 ViT 分支可以保持从大规模预训练中获得的表示能力。然后，提出的特征调制模块可以使用来自 ViT 分支的表示来调制多尺度 CNN 特征。借助所提出的混合结构，检测器更有可能发现新类别。在两个流行的基准上进行评估，我们的方法提高了新类别的检测性能并优于基线。在 OV-COCO 上，所提出的方法使用 ViT-B/16 实现 44.3 AP$_{50}^{\mathrm{novel}}$，使用 ViT-L/14 实现 48.5 AP$_{50}^{\mathrm{novel}}$。在 OV-LVIS 上，使用 ViT-B/16 和 ViT-L/14 的 VMCNet 达到 27.8 和 38.4 mAP$_{r}$。

Title: MAUCell: An Adaptive Multi-Attention Framework for Video Frame Prediction

• Authors: Shreyam Gupta (1), P. Agrawal (2), Priyam Gupta (3) ((1) Indian Institute of Technology (BHU), Varanasi, India, (2) University of Colorado, Boulder, USA, (3) Intelligent Field Robotic Systems (IFROS), University of Girona, Spain)
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.16997
• Pdf URL: https://arxiv.org/pdf/2501.16997
• Copy Paste: [[2501.16997]] MAUCell: An Adaptive Multi-Attention Framework for Video Frame Prediction(https://arxiv.org/abs/2501.16997)
• Keywords: generative
• Abstract: Temporal sequence modeling stands as the fundamental foundation for video prediction systems and real-time forecasting operations as well as anomaly detection applications. The achievement of accurate predictions through efficient resource consumption remains an ongoing issue in contemporary temporal sequence modeling. We introduce the Multi-Attention Unit (MAUCell) which combines Generative Adversarial Networks (GANs) and spatio-temporal attention mechanisms to improve video frame prediction capabilities. Our approach implements three types of attention models to capture intricate motion sequences. A dynamic combination of these attention outputs allows the model to reach both advanced decision accuracy along with superior quality while remaining computationally efficient. The integration of GAN elements makes generated frames appear more true to life therefore the framework creates output sequences which mimic real-world footage. The new design system maintains equilibrium between temporal continuity and spatial accuracy to deliver reliable video prediction. Through a comprehensive evaluation methodology which merged the perceptual LPIPS measurement together with classic tests MSE, MAE, SSIM and PSNR exhibited enhancing capabilities than contemporary approaches based on direct benchmark tests of Moving MNIST, KTH Action, and CASIA-B (Preprocessed) datasets. Our examination indicates that MAUCell shows promise for operational time requirements. The research findings demonstrate how GANs work best with attention mechanisms to create better applications for predicting video sequences.
• 摘要：时间序列建模是视频预测系统和实时预测操作以及异常检测应用的基础。通过高效的资源消耗实现准确预测仍然是当代时间序列建模中持续存在的问题。我们引入了多注意单元 (MAUCell)，它结合了生成对抗网络 (GAN) 和时空注意机制来提高视频帧预测能力。我们的方法实现了三种类型的注意模型来捕捉复杂的运动序列。这些注意力输出的动态组合使模型能够达到高级决策精度和卓越质量，同时保持计算效率。GAN 元素的集成使生成的帧看起来更逼真，因此该框架创建了模仿真实世界镜头的输出序列。新的设计系统在时间连续性和空间准确性之间保持平衡，以提供可靠的视频预测。通过一种将感知 LPIPS 测量与经典测试 MSE、MAE、SSIM 和 PSNR 相结合的综合评估方法，与基于移动 MNIST、KTH Action 和 CASIA-B（预处理）数据集的直接基准测试的当代方法相比，MSE、MAE、SSIM 和 PSNR 表现出了增强的功能。我们的研究表明，MAUCell 有望满足操作时间要求。研究结果表明，GAN 如何与注意力机制配合使用，以创建更好的应用程序来预测视频序列。

Title: Token-by-Token Regeneration and Domain Biases: A Benchmark of LLMs on Advanced Mathematical Problem-Solving

• Authors: Evgenii Evstafev
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.17084
• Pdf URL: https://arxiv.org/pdf/2501.17084
• Copy Paste: [[2501.17084]] Token-by-Token Regeneration and Domain Biases: A Benchmark of LLMs on Advanced Mathematical Problem-Solving(https://arxiv.org/abs/2501.17084)
• Keywords: generation
• Abstract: Large language models (LLMs) excel in many natural language tasks, yet they struggle with complex mathemat-ical problem-solving, particularly in symbolic reasoning and maintaining consistent output. This study evalu-ates 10 LLMs with 7 to 8 billion parameters using 945 competition-level problems from the MATH dataset. The focus is on their ability to generate executable Python code as a step in their reasoning process, involving over 9,450 code executions. The research introduces an evaluation framework using mistral-large-2411 to rate answers on a 5-point scale, which helps address inconsistencies in mathematical notation. It also examines the impact of regenerating output token-by-token on refining results. The findings reveal a significant 34.5% per-formance gap between the top commercial model (gpt-4o-mini, scoring 83.7%) and the least effective open-source model (open-codestral-mamba:v0.1, scoring 49.2%). This disparity is especially noticeable in complex areas like Number Theory. While token-by-token regeneration slightly improved accuracy (+0.8%) for the model llama3.1:8b, it also reduced code execution time by 36.7%, highlighting a trade-off between efficiency and precision. The study also noted a consistent trend where harder problems correlated with lower accuracy across all models. Despite using controlled execution environments, less than 1% of the generated code was unsafe, and 3.17% of problems remained unsolved after 10 attempts, suggesting that hybrid reasoning methods may be beneficial.
• 摘要：大型语言模型 (LLM) 在许多自然语言任务中表现出色，但它们在解决复杂的数学问题方面却举步维艰，尤其是在符号推理和保持一致的输出方面。本研究使用来自 MATH 数据集的 945 个竞赛级问题评估了 10 个具有 70 亿到 80 亿个参数的 LLM。重点是它们在推理过程中生成可执行 Python 代码的能力，涉及超过 9,450 次代码执行。该研究引入了一个评估框架，使用 mistral-large-2411 以 5 分制对答案进行评分，这有助于解决数学符号不一致的问题。它还研究了逐个标记重新生成输出对细化结果的影响。研究结果显示，顶级商业模型 (gpt-4o-mini，得分 83.7%) 和最不有效的开源模型 (open-codestral-mamba:v0.1，得分 49.2%) 之间的性能差距显著为 34.5%。这种差异在数论等复杂领域尤其明显。虽然逐个标记的再生略微提高了模型 llama3.1:8b 的准确率 (+0.8%)，但也缩短了 36.7% 的代码执行时间，凸显了效率和精度之间的权衡。该研究还指出了一个一致的趋势，即所有模型中，较难的问题与较低的准确率相关。尽管使用了受控执行环境，但生成的代码中只有不到 1% 是不安全的，3.17% 的问题在 10 次尝试后仍未解决，这表明混合推理方法可能有益。

Title: Unlocking Transparent Alignment Through Enhanced Inverse Constitutional AI for Principle Extraction

• Authors: Carl-Leander Henneking, Claas Beger
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.17112
• Pdf URL: https://arxiv.org/pdf/2501.17112
• Copy Paste: [[2501.17112]] Unlocking Transparent Alignment Through Enhanced Inverse Constitutional AI for Principle Extraction(https://arxiv.org/abs/2501.17112)
• Keywords: generation
• Abstract: Traditional methods for aligning Large Language Models (LLMs), such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on implicit principles, limiting interpretability. Constitutional AI (CAI) offers an explicit, rule-based framework for guiding model outputs. Building on this, we refine the Inverse Constitutional AI (ICAI) algorithm, which extracts constitutions from preference datasets. By improving principle generation, clustering, and embedding processes, our approach enhances the accuracy and generalizability of extracted principles across synthetic and real-world datasets. While in-context alignment yields modest improvements, our results highlight the potential of these principles to foster more transparent and adaptable alignment methods, offering a promising direction for future advancements beyond traditional fine-tuning.
• 摘要：传统的大型语言模型 (LLM) 对齐方法，例如从人类反馈中强化学习 (RLHF) 和直​​接偏好优化 (DPO)，依赖于隐式原则，限制了可解释性。宪法人工智能 (CAI) 提供了一个明确的、基于规则的框架来指导模型输出。在此基础上，我们改进了逆宪法人工智能 (ICAI) 算法，该算法从偏好数据集中提取宪法。通过改进原则生成、聚类和嵌入过程，我们的方法提高了在合成和真实世界数据集中提取的原则的准确性和通用性。虽然上下文对齐产生了适度的改进，但我们的结果突出了这些原则促进更透明和适应性更强的对齐方法的潜力，为超越传统微调的未来进步提供了一个有希望的方向。

Title: Optimizing Large Language Model Training Using FP4 Quantization

• Authors: Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang, Baining Guo, Zhengjun Zha, Peng Cheng
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.17116
• Pdf URL: https://arxiv.org/pdf/2501.17116
• Copy Paste: [[2501.17116]] Optimizing Large Language Model Training Using FP4 Quantization(https://arxiv.org/abs/2501.17116)
• Keywords: generation
• Abstract: The growing computational demands of training large language models (LLMs) necessitate more efficient methods. Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce these costs. While FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge due to significant quantization errors and limited representational capacity. This work introduces the first FP4 training framework for LLMs, addressing these challenges with two key innovations: a differentiable quantization estimator for precise weight updates and an outlier clamping and compensation strategy to prevent activation collapse. To ensure stability, the framework integrates a mixed-precision training scheme and vector-wise quantization. Experimental results demonstrate that our FP4 framework achieves accuracy comparable to BF16 and FP8, with minimal degradation, scaling effectively to 13B-parameter LLMs trained on up to 100B tokens. With the emergence of next-generation hardware supporting FP4, our framework sets a foundation for efficient ultra-low precision training.
• 摘要：训练大型语言模型 (LLM) 的计算需求不断增长，因此需要更高效的方法。量化训练通过启用低位算术运算来降低这些成本，提供了一种有前途的解决方案。虽然 FP8 精度已证明是可行的，但由于量化误差大且表示容量有限，利用 FP4 仍然是一个挑战。这项工作引入了第一个用于 LLM 的 FP4 训练框架，通过两项关键创新解决了这些挑战：用于精确权重更新的可微量化估计器以及用于防止激活崩溃的异常值钳制和补偿策略。为了确保稳定性，该框架集成了混合精度训练方案和矢量量化。实验结果表明，我们的 FP4 框架实现了与 BF16 和 FP8 相当的精度，且性能下降最小，可以有效扩展到在多达 100B 个 token 上训练的 13B 参数 LLM。随着支持 FP4 的下一代硬件的出现，我们的框架为高效的超低精度训练奠定了基础。

Title: CoRe-Net: Co-Operational Regressor Network with Progressive Transfer Learning for Blind Radar Signal Restoration

• Authors: Muhammad Uzair Zahid, Serkan Kiranyaz, Alper Yildirim, Moncef Gabbouj
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.17125
• Pdf URL: https://arxiv.org/pdf/2501.17125
• Copy Paste: [[2501.17125]] CoRe-Net: Co-Operational Regressor Network with Progressive Transfer Learning for Blind Radar Signal Restoration(https://arxiv.org/abs/2501.17125)
• Keywords: restoration
• Abstract: Real-world radar signals are frequently corrupted by various artifacts, including sensor noise, echoes, interference, and intentional jamming, differing in type, severity, and duration. This pilot study introduces a novel model, called Co-Operational Regressor Network (CoRe-Net) for blind radar signal restoration, designed to address such limitations and drawbacks. CoRe-Net replaces adversarial training with a novel cooperative learning strategy, leveraging the complementary roles of its Apprentice Regressor (AR) and Master Regressor (MR). The AR restores radar signals corrupted by various artifacts, while the MR evaluates the quality of the restoration and provides immediate and task-specific feedback, ensuring stable and efficient learning. The AR, therefore, has the advantage of both self-learning and assistive learning by the MR. The proposed model has been extensively evaluated over the benchmark Blind Radar Signal Restoration (BRSR) dataset, which simulates diverse real-world artifact scenarios. Under the fair experimental setup, this study shows that the CoRe-Net surpasses the Op-GANs over a 1 dB mean SNR improvement. To further boost the performance gain, this study proposes multi-pass restoration by cascaded CoRe-Nets trained with a novel paradigm called Progressive Transfer Learning (PTL), which enables iterative refinement, thus achieving an additional 2 dB mean SNR enhancement. Multi-pass CoRe-Net training by PTL consistently yields incremental performance improvements through successive restoration passes whilst highlighting CoRe-Net ability to handle such a complex and varying blend of artifacts.
• 摘要：现实世界中的雷达信号经常受到各种干扰的破坏，包括传感器噪声、回声、干扰和故意干扰，这些干扰的类型、严重程度和持续时间各不相同。这项初步研究引入了一种用于盲雷达信号恢复的新型模型，称为协同操作回归网络 (CoRe-Net)，旨在解决此类限制和缺点。CoRe-Net 用一种新颖的合作学习策略取代了对抗性训练，利用其学徒回归器 (AR) 和主回归器 (MR) 的互补作用。AR 恢复被各种干扰破坏的雷达信号，而 MR 评估恢复的质量并提供即时和特定于任务的反馈，确保稳定高效的学习。因此，AR 兼具自学和 MR 辅助学习的优势。所提出的模型已在基准盲雷达信号恢复 (BRSR) 数据集上进行了广泛评估，该数据集模拟了各种现实世界的干扰场景。在公平的实验设置下，本研究表明，CoRe-Net 的平均 SNR 改进超过 Op-GAN 1 dB。为了进一步提高性能增益，本研究提出了通过级联 CoRe-Nets 进行多遍恢复的方法，这些 CoRe-Nets 经过一种名为渐进式迁移学习 (PTL) 的新范式进行训练，从而实现迭代细化，从而实现额外的 2 dB 平均 SNR 增强。通过 PTL 进行多遍 CoRe-Net 训练，通过连续的恢复过程持续产生增量性能改进，同时突出了 CoRe-Net 处理如此复杂和多变的工件混合的能力。

Title: IC-Portrait: In-Context Matching for View-Consistent Personalized Portrait

• Authors: Han Yang, Enis Simsar, Sotiris Anagnostidi, Yanlong Zang, Thomas Hofmann, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.17159
• Pdf URL: https://arxiv.org/pdf/2501.17159
• Copy Paste: [[2501.17159]] IC-Portrait: In-Context Matching for View-Consistent Personalized Portrait(https://arxiv.org/abs/2501.17159)
• Keywords: generation
• Abstract: Existing diffusion models show great potential for identity-preserving generation. However, personalized portrait generation remains challenging due to the diversity in user profiles, including variations in appearance and lighting conditions. To address these challenges, we propose IC-Portrait, a novel framework designed to accurately encode individual identities for personalized portrait generation. Our key insight is that pre-trained diffusion models are fast learners (e.g.,100 ~ 200 steps) for in-context dense correspondence matching, which motivates the two major designs of our IC-Portrait framework. Specifically, we reformulate portrait generation into two sub-tasks: 1) Lighting-Aware Stitching: we find that masking a high proportion of the input image, e.g., 80%, yields a highly effective self-supervisory representation learning of reference image lighting. 2) View-Consistent Adaptation: we leverage a synthetic view-consistent profile dataset to learn the in-context correspondence. The reference profile can then be warped into arbitrary poses for strong spatial-aligned view conditioning. Coupling these two designs by simply concatenating latents to form ControlNet-like supervision and modeling, enables us to significantly enhance the identity preservation fidelity and stability. Extensive evaluations demonstrate that IC-Portrait consistently outperforms existing state-of-the-art methods both quantitatively and qualitatively, with particularly notable improvements in visual qualities. Furthermore, IC-Portrait even demonstrates 3D-aware relighting capabilities.
• 摘要：现有的扩散模型在身份保留生成方面表现出巨大潜力。然而，由于用户资料的多样性，包括外观和光照条件的变化，个性化肖像生成仍然具有挑战性。为了应对这些挑战，我们提出了 IC-Portrait，这是一个新颖的框架，旨在准确编码个人身份以生成个性化肖像。我们的主要见解是，预训练的扩散模型是上下文密集对应匹配的快速学习者（例如 100 至 200 步），这激发了我们 IC-Portrait 框架的两大设计。具体来说，我们将肖像生成重新表述为两个子任务：1）照明感知拼接：我们发现，掩盖输入图像的大部分（例如 80%），可以产生对参考图像照明的高效自监督表示学习。2）视图一致自适应：我们利用合成的视图一致配置文件数据集来学习上下文对应关系。然后可以将参考配置文件扭曲成任意姿势，以进行强大的空间对齐视图调节。通过简单地连接潜在函数来形成类似 ControlNet 的监督和建模，将这两种设计结合起来，使我们能够显著提高身份保存的保真度和稳定性。大量评估表明，IC-Portrait 在数量和质量上始终优于现有的最先进方法，尤其是在视觉质量方面有显著的改善。此外，IC-Portrait 甚至展示了 3D 感知的重新照明功能。

Title: CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation

• Authors: Nikolai Kalischek, Michael Oechsle, Fabian Manhardt, Philipp Henzler, Konrad Schindler, Federico Tombari
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.17162
• Pdf URL: https://arxiv.org/pdf/2501.17162
• Copy Paste: [[2501.17162]] CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation(https://arxiv.org/abs/2501.17162)
• Keywords: generation
• Abstract: We introduce a novel method for generating 360° panoramas from text prompts or images. Our approach leverages recent advances in 3D generation by employing multi-view diffusion models to jointly synthesize the six faces of a cubemap. Unlike previous methods that rely on processing equirectangular projections or autoregressive generation, our method treats each face as a standard perspective image, simplifying the generation process and enabling the use of existing multi-view diffusion models. We demonstrate that these models can be adapted to produce high-quality cubemaps without requiring correspondence-aware attention layers. Our model allows for fine-grained text control, generates high resolution panorama images and generalizes well beyond its training set, whilst achieving state-of-the-art results, both qualitatively and quantitatively. Project page: this https URL
• 摘要：我们介绍了一种从文本提示或图像生成 360° 全景图的新方法。我们的方法利用 3D 生成方面的最新进展，采用多视图扩散模型来联合合成立方体贴图的六个面。与以前依赖于处理等距矩形投影或自回归生成的方法不同，我们的方法将每个面视为标准透视图像，简化了生成过程并允许使用现有的多视图扩散模型。我们证明这些模型可以适应生成高质量的立方体贴图，而无需对应感知注意层。我们的模型允许细粒度的文本控制，生成高分辨率全景图并远远超出其训练集，同时在定性和定量方面都实现了最先进的结果。项目页面：此 https URL